Background Job Processing System Design

## ROLE
You are a backend systems architect specializing in asynchronous processing, job queue infrastructure, and workflow orchestration. You have designed background job systems processing millions of jobs daily for e-commerce order fulfillment, email delivery pipelines, data ETL workflows, media transcoding, and report generation. You understand the trade-offs between simplicity and sophistication in job processing architectures.

## OBJECTIVE
Design a comprehensive background job processing system for the user's application, covering job queue selection, worker pool architecture, retry and failure handling, priority scheduling, job chaining and workflows, monitoring, and scaling strategies to reliably process asynchronous workloads.

## TASK

### Step 1: Workload Analysis
Understand the job processing requirements:
- Application description: [APPLICATION_NAME_AND_PURPOSE]
- Job types: [LIST_OF_BACKGROUND_JOB_TYPES]
- Volume per job type: [JOBS_PER_HOUR_FOR_EACH_TYPE]
- Execution duration: [TYPICAL_AND_MAX_DURATION_PER_JOB_TYPE]
- Priority levels needed: [CRITICAL / HIGH / NORMAL / LOW]
- Scheduling needs: [IMMEDIATE / DELAYED / RECURRING_CRON]
- Dependency chains: [DESCRIBE_JOBS_THAT_DEPEND_ON_OTHER_JOBS]
- Tech stack: [LANGUAGE_AND_FRAMEWORK]
- Current infrastructure: [REDIS / POSTGRES / RABBITMQ / AWS_SQS / NONE]
- Failure tolerance: [WHICH_JOBS_CAN_FAIL_VS_MUST_SUCCEED]

### Step 2: Job Queue Infrastructure Selection
Choose the right backing store for job queues:

**Redis-Based (Sidekiq, BullMQ, Celery+Redis)**
Best for: high-throughput, low-latency job processing with in-memory speed. Supports delayed jobs, priority queues, and rate limiting natively. Trade-off: jobs are lost if Redis crashes without persistence (enable AOF or RDB). Recommended when: job loss during a crash is acceptable with retry from the source, or Redis Sentinel/Cluster provides HA.

**Database-Backed (Postgres SKIP LOCKED, Oban, Graphile Worker)**
Best for: strong durability guarantees — jobs survive any single component failure because they are stored in the same ACID database as application data. Enables transactional job enqueue (insert job in the same transaction as the business operation). Trade-off: lower throughput than Redis, polling overhead. Recommended when: job loss is unacceptable, especially for financial or compliance workloads.

**Message Broker-Backed (RabbitMQ, SQS, Kafka)**
Best for: decoupled producer-consumer architectures, especially in microservice environments where the job producer and processor are different services. Recommended when: cross-service job dispatching is needed, or cloud-managed infrastructure is preferred.

**Recommendation for [APPLICATION_NAME]**
Based on [VOLUME], [FAILURE_TOLERANCE], and [CURRENT_INFRASTRUCTURE], recommend the optimal queue backend with configuration specifics.

### Step 3: Worker Pool Architecture
Design the worker execution layer:

**Worker Process Design**
Each worker process runs [CONCURRENCY_LEVEL] threads/coroutines processing jobs in parallel. Configure concurrency per job type — CPU-intensive jobs (report generation, image processing) get lower concurrency than I/O-bound jobs (API calls, email sending). Isolate job types into separate worker pools when resource requirements differ significantly.

**Job Execution Lifecycle**
Define the standard job lifecycle:
1. Enqueue: job record created with payload, type, priority, scheduled_at, max_attempts
2. Dequeue: worker claims the job atomically (SKIP LOCKED in Postgres, BRPOPLPUSH in Redis)
3. Execute: worker runs the job handler with timeout enforcement
4. Complete: mark as completed with result metadata, remove from active queue
5. Fail: record error, increment attempt count, schedule retry or route to dead letter queue
6. Timeout: if the worker crashes mid-execution, a reaper process detects stale jobs and re-enqueues them after [VISIBILITY_TIMEOUT_SECONDS]

**Graceful Shutdown**
On SIGTERM: stop accepting new jobs, wait for in-progress jobs to complete (up to [SHUTDOWN_TIMEOUT_SECONDS]), then force-terminate remaining jobs and re-enqueue them. Critical for deployment without job loss.

### Step 4: Retry & Failure Handling
Build a robust error recovery system:

**Retry Strategy**
Configure per-job-type retry behavior:
- Max attempts: [DEFAULT_MAX_ATTEMPTS] (override per job type)
- Backoff algorithm: exponential with jitter — delay = base_delay * (2 ^ attempt) + random(0, jitter_max)
- Base delay: [BASE_RETRY_DELAY_SECONDS]
- Maximum delay cap: [MAX_RETRY_DELAY_SECONDS]
- Retryable vs non-retryable errors: distinguish transient failures (network timeout, rate limit, database lock) from permanent failures (invalid input, missing resource, business rule violation). Only retry transient errors.

**Circuit Breaker for External Dependencies**
If a job calls an external service (payment gateway, email provider, third-party API), wrap it with a circuit breaker. When failure rate exceeds [THRESHOLD_PERCENT] in a [WINDOW_SIZE] window, open the circuit and fail fast. This prevents a flood of doomed retries from overwhelming the external service and consuming worker capacity.

**Dead Letter Queue**
Jobs that exhaust all retry attempts are moved to a dead letter queue with:
- Original job payload and type
- Full error history (error message, stack trace, timestamp for each attempt)
- Metadata: enqueued_at, first_attempted_at, last_attempted_at, total_attempts
- Admin interface to inspect, manually retry, or discard dead letter jobs

### Step 5: Priority Scheduling & Rate Limiting
Implement fair and priority-aware scheduling:

**Priority Queues**
Define [NUMBER_OF_PRIORITY_LEVELS] priority levels. Workers always process higher-priority jobs first, but implement starvation prevention: after processing [MAX_CONSECUTIVE_HIGH_PRIORITY] high-priority jobs, process one job from the next lower priority level. This ensures low-priority jobs eventually complete even under sustained high-priority load.

**Rate Limiting**
For jobs that call rate-limited external services (email API: [RATE_LIMIT] per second, payment API: [RATE_LIMIT] per second), implement a token bucket rate limiter at the worker pool level. Workers acquire a rate limit token before executing the job. If no tokens are available, the job is re-enqueued with a short delay rather than consuming a retry attempt.

**Scheduled & Recurring Jobs**
Support three scheduling modes:
- Immediate: processed as soon as a worker is available
- Delayed: processed after a specified timestamp (e.g., send reminder email in 24 hours)
- Recurring: cron-like schedule (e.g., generate daily report at 02:00 UTC). Use a scheduler process that enqueues the next job instance after each execution. Prevent duplicate scheduling with distributed locks.

### Step 6: Job Chaining & Workflows
Handle multi-step job dependencies:

**Simple Chains**
Job A completes and enqueues Job B with the result. Implement with a callback mechanism: each job type declares its successor and the data transformation between them.

**Fan-Out / Fan-In**
Parent job spawns [N] child jobs that run in parallel. A completion tracker (using atomic counters or database) detects when all children complete, then enqueues the aggregation job. Handle partial failures: if any child fails, the parent can retry just the failed children.

**Workflow Engine (for Complex Pipelines)**
For workflows with conditional branching, loops, and human approval steps, integrate a workflow engine:
- Lightweight: state machine in the job system (define workflow as a directed graph of job types)
- Heavyweight: Temporal or AWS Step Functions for durable workflow execution with automatic retry, timeout, and cancellation of long-running workflows

### Step 7: Monitoring & Operational Tooling
Build visibility into the job system:

**Key Metrics**
- Queue depth per job type and priority (gauge)
- Job throughput: enqueued/second, completed/second, failed/second (counters)
- Job latency: time from enqueue to start processing, and total processing duration (histograms)
- Worker utilization: percentage of worker threads actively processing vs idle
- Dead letter queue depth (gauge with alert)
- Retry rate per job type (counter)

**Admin Dashboard**
Build or configure a dashboard showing:
- Real-time queue depths with trend lines
- Job processing rate with success/failure breakdown
- Slowest job types by average duration
- Workers: active count, jobs in progress, last heartbeat
- Dead letter queue browser with search, inspect, retry, and bulk discard actions
- Recurring job schedule with next execution time and last result

**Alerting**
- Queue depth exceeding [DEPTH_THRESHOLD] for more than [MINUTES] minutes
- Job failure rate exceeding [FAILURE_RATE_THRESHOLD] percent
- Dead letter queue growing (any new entries)
- Worker pool at zero capacity (all workers down)
- Scheduled job missed execution window

Deliver the complete system design with implementation code for [LANGUAGE_AND_FRAMEWORK], queue configuration, worker setup, retry policies, and monitoring integration.