Background Job Processing Pipeline

You are a backend infrastructure engineer specializing in asynchronous job processing systems for high-throughput applications. Design a complete background job processing pipeline for the following system.

System Context:
Job Types: [EMAIL SENDING/REPORT GENERATION/DATA PROCESSING/MEDIA ENCODING/WEBHOOKS/IMPORTS]
Job Volume: [JOBS PER HOUR ESTIMATE]
Latency Sensitivity: [NEAR-REAL-TIME/MINUTES ACCEPTABLE/HOURS ACCEPTABLE]
Job Duration: [SECONDS/MINUTES/HOURS]
Infrastructure: [AWS/GCP/AZURE/SELF-HOSTED]
Queue Technology: [REDIS-BASED/SQS/RABBITMQ/DATABASE-BACKED/UNDECIDED]

Section 1 - Queue Architecture and Job Design:
Define the queue topology including separate queues for different job types, priority levels, and processing requirements. Design the job payload structure including job type identifier, arguments, metadata, scheduling information, and correlation IDs for tracing. Specify the serialization format for job payloads and the maximum payload size with strategies for jobs that reference large data. Create the job lifecycle state machine showing transitions from enqueued through processing, completed, failed, and dead states. Define the queue selection strategy that routes jobs to the appropriate queue based on type, priority, and resource requirements. Design the job deduplication mechanism that prevents the same job from being enqueued multiple times using unique job keys or content-based fingerprints.

Section 2 - Worker Architecture and Concurrency:
Design the worker process architecture including the number of worker processes, threads or coroutines per process, and memory allocation per worker. Specify the job claiming mechanism including how workers pull jobs from queues, visibility timeout configuration, and heartbeat reporting for long-running jobs. Define the concurrency control strategy including maximum concurrent jobs per worker, per-queue concurrency limits, and global rate limiting for resource-sensitive operations. Create the worker pool management including how workers are started, stopped, and scaled based on queue depth. Design the graceful shutdown procedure that allows in-progress jobs to complete within a timeout before the worker terminates. Specify the worker specialization strategy if certain job types require dedicated workers with specific resources or dependencies.

Section 3 - Scheduling and Priority Management:
Define the job scheduling system for recurring jobs including cron expression support, timezone handling, and overlap prevention for jobs that run longer than their interval. Design the delayed job mechanism that enqueues jobs for future execution with configurable precision. Create the priority queue implementation with distinct priority levels and the scheduling algorithm that balances priority with fairness to prevent starvation of lower-priority jobs. Specify the batch job coordination for workflows where multiple jobs must complete before a downstream job begins. Design the rate-limited job execution that respects external API rate limits by throttling job processing for specific integrations. Define the job dependency chain where one job triggers subsequent jobs upon completion with error propagation between dependent jobs.

Section 4 - Retry Policy and Error Handling:
Define the retry strategy for each job type including maximum retry count, backoff algorithm with jitter, and retry delay configuration. Design the error classification system that determines whether a failure is retryable such as network timeouts and rate limits, or permanent such as invalid input and missing resources. Specify the dead letter queue handling for jobs that exhaust all retries including metadata preservation, inspection tooling, and manual replay capabilities. Create the circuit breaker integration that pauses job processing for specific external dependencies when failure rates exceed thresholds. Design the partial failure handling for batch jobs that process multiple items specifying whether to fail the entire batch or track individual item results. Define the alerting thresholds for job failure rates, retry rates, and dead letter queue growth that trigger engineering notification.

Section 5 - Job Lifecycle and Data Management:
Design the job result storage strategy including where results are stored, how long they are retained, and how callers retrieve completed job results. Specify the job progress reporting mechanism for long-running jobs that updates a progress percentage, current step description, and estimated completion time. Create the job cancellation system that allows in-progress jobs to be cancelled with cleanup of partial work. Define the job logging strategy including structured logs for each lifecycle event, log correlation with the originating request, and log retention policies. Design the cleanup procedures for completed job data including archival schedules and storage optimization. Specify the job analytics pipeline that tracks processing times, success rates, and throughput by job type for capacity planning.

Section 6 - Scaling, Monitoring, and Operations:
Define the auto-scaling rules for worker fleets based on queue depth, processing latency, and time-of-day patterns. Design the monitoring dashboard showing real-time queue depths, processing rates, error rates, and worker utilization across all queues. Create the capacity planning model for projecting worker requirements based on job volume forecasts and processing time distributions. Specify the deployment strategy for worker code updates including rolling deploys that avoid processing the same job on different code versions. Design the operational runbook for common incidents including queue backlogs, stuck jobs, worker crashes, and queue infrastructure failures. Document the disaster recovery plan for job queue data loss including message persistence guarantees and replay from source systems.