Distributed Task Queue

A distributed job queue with workers, rate limiting, and real-time dashboard.

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Dashboard  │────▶│   FastAPI   │────▶│  PostgreSQL │
│   (React)   │     │    (API)    │     │ (Job State) │
└─────────────┘     └──────┬──────┘     └─────────────┘
                           │
                    ┌──────▼──────┐
                    │    Redis    │
                    │   (Queue)   │
                    └──────┬──────┘
                           │
          ┌────────────────┼────────────────┐
          ▼                ▼                ▼
    ┌──────────┐     ┌──────────┐     ┌──────────┐
    │ Worker 1 │     │ Worker 2 │     │ Worker N │
    └──────────┘     └──────────┘     └──────────┘

Job Lifecycle: PENDING → RUNNING → COMPLETED (or → RETRY → DLQ on failure)

Quick Start

# Start all services
docker compose up -d

# Scale workers
docker compose up -d --scale worker=3

Service	URL
Dashboard	http://localhost:3000
API Docs	http://localhost:8000/docs
Health	http://localhost:8000/health
Metrics	http://localhost:8000/metrics

API Endpoints

All endpoints require X-Tenant-ID header.

Method	Endpoint	Description
POST	/api/jobs	Create job
GET	/api/jobs	List jobs (supports ?status=, ?page=, ?page_size=)
GET	/api/jobs/{id}	Get job by ID
DELETE	/api/jobs/{id}	Cancel pending job
GET	/api/dlq	List dead-letter queue jobs
POST	/api/dlq/{id}/retry	Retry DLQ job
GET	/api/rate-limits	Get tenant rate limit status
GET	/api/queue/stats	Get queue statistics

Create Job

curl -X POST http://localhost:8000/api/jobs \
  -H "Content-Type: application/json" \
  -H "X-Tenant-ID: my-tenant" \
  -d '{
    "job_type": "echo",
    "payload": {"message": "Hello!"},
    "max_retries": 3,
    "priority": 0
  }'

Job Types

Type	Description
echo	Returns payload (1s)
process_image	Simulates image processing (2-5s)
send_email	Simulates email sending (1-3s)
compute	Heavy computation (configurable duration)
fail_sometimes	50% failure rate (testing retries)
always_fail	Always fails (testing DLQ)

Integration Testing

# Run all tests
make test

# Run fast tests only (skip slow worker tests)
make test-fast

# Generate HTML report
make test-html

Or directly:

./run_tests.sh
./run_tests.sh --fast
./run_tests.sh --html

Test Coverage

• Health & metrics endpoints
• Job CRUD operations
• Tenant isolation
• Idempotency keys
• Rate limiting (per-minute, concurrent)
• Worker job processing
• Retry logic & DLQ
• Input validation

Project Structure

├── api/                 # FastAPI backend
│   ├── main.py          # Routes
│   ├── models.py        # Job model
│   ├── task_queue.py    # Redis queue
│   └── rate_limiter.py  # Rate limiting
├── worker/              # Job processor
│   └── main.py          # Polling loop
├── dashboard/           # React frontend
├── tests/               # Integration tests
│   └── integration/
│       └── test_api.py
├── docker-compose.yml
├── Makefile
└── run_tests.sh

Configuration

Variable	Default	Description
POLL_INTERVAL	2	Worker poll interval (seconds)
LEASE_TIMEOUT	300	Job lease timeout (seconds)
MAX_CONCURRENT_JOBS	5	Per-tenant concurrent limit
MAX_JOBS_PER_MINUTE	10	Per-tenant rate limit

---

This project was built with extensive use of AI-assisted code generation (Cursor). Although this can be replicated by one-self, using ai for common tasks really speeds up the process of development.