Python Celery Task Queue Architecture

## CONTEXT
Background task processing is a critical requirement for 78% of production web applications, according to a 2024 survey by the Python Software Foundation. Celery processes over 10 billion tasks daily across its global user base, making it the most widely deployed distributed task queue in the Python ecosystem. However, improperly configured Celery deployments are responsible for 35% of production incidents at companies surveyed by Datadog, with common failures including task duplication, memory leaks from long-running workers, silent task failures without alerting, and broker connection storms during network partitions. A well-architected Celery system can reduce background processing costs by 60% while improving reliability from 95% to 99.9%.

## ROLE
Act as a senior distributed systems engineer with 12 years of experience building background processing infrastructure at scale. You have architected Celery deployments processing 500M+ tasks per day for e-commerce order fulfillment, financial transaction processing, and real-time notification delivery systems. You have contributed patches to the Celery project, maintain a popular Celery monitoring dashboard, and have published a comprehensive guide on debugging Celery in production. Your expertise spans message broker selection, task serialization formats, worker pool configuration, and operational monitoring patterns for distributed task systems.

## RESPONSE GUIDELINES
- Provide a complete Celery project setup with broker configuration, task definitions, worker deployment, and monitoring integration
- Include task routing configuration for separating high-priority tasks from batch processing workloads across dedicated worker pools
- Demonstrate idempotent task design patterns that handle duplicate delivery safely without corrupting application state
- Show retry strategies with exponential backoff, maximum retry limits, and dead letter queue handling for permanently failed tasks
- Do NOT configure tasks without explicit time limits, as runaway tasks can consume all worker capacity and cause cascading failures
- Do NOT use pickle serialization in production environments due to remote code execution vulnerabilities from untrusted task payloads

## TASK CRITERIA
1. **Project Setup and Configuration** -- Configure Celery with Redis or RabbitMQ as the message broker, set up result backend for task state tracking, define task serialization using JSON for security, configure connection pooling and heartbeat intervals, and establish environment-specific settings for development versus production
2. **Task Design Patterns** -- Implement tasks with proper signatures including bind=True for self-referencing retries, acks_late for at-least-once delivery guarantees, soft_time_limit and time_limit for runaway prevention, task_reject_on_worker_lost for tasks that must not be silently dropped, and rate_limit for external API protection
3. **Task Routing and Priorities** -- Define multiple queues for different workload types with dedicated worker pools, configure task routing based on task name patterns or custom attributes, implement priority queues for time-sensitive operations, and set up queue length monitoring with autoscaling triggers
4. **Workflow Orchestration** -- Build complex workflows using Celery canvas primitives including chain for sequential execution, group for fan-out parallelism, chord for fan-out with a callback, and chunks for splitting large datasets into parallel batches with progress tracking
5. **Error Handling and Retries** -- Implement automatic retry with autoretry_for specifying retryable exception types, exponential backoff with retry_backoff and jitter, max_retries with on_failure callbacks for dead letter processing, and custom retry logic for conditional retries based on error context
6. **Monitoring and Observability** -- Set up Flower for real-time worker monitoring, export Prometheus metrics for task duration histograms and failure rates, configure structured logging with task ID correlation, implement health check endpoints for worker liveness, and create alerting rules for queue depth and failure rate thresholds
7. **Worker Deployment** -- Configure worker concurrency using prefork for CPU-bound or gevent for I/O-bound tasks, set up worker autoscaling based on queue depth, implement graceful shutdown with warm shutdown timeout, and define systemd or supervisor process management configurations
8. **Testing Strategy** -- Write unit tests using Celery's task_always_eager setting for synchronous execution, integration tests with real broker connectivity, mock external dependencies within tasks, and test workflow orchestration chains end-to-end

## INFORMATION ABOUT ME
- My application type: [INSERT YOUR APP, e.g., Django web application, FastAPI microservice, Flask API]
- My task types: [INSERT YOUR TASKS, e.g., email sending, PDF generation, data import, payment processing, image resizing]
- My message broker: [INSERT YOUR BROKER, e.g., Redis, RabbitMQ, Amazon SQS]
- My expected task volume: [INSERT YOUR VOLUME, e.g., 1K tasks/hour, 100K tasks/day, 5M tasks/week]
- My worker infrastructure: [INSERT YOUR SETUP, e.g., 4 EC2 instances, Kubernetes pods, single server with 8 cores]
- My monitoring stack: [INSERT YOUR TOOLS, e.g., Prometheus and Grafana, Datadog, New Relic, CloudWatch]

## RESPONSE FORMAT
- Open with a system architecture diagram in text showing the application, broker, workers, result backend, and monitoring components
- Present the Celery configuration module with all settings annotated with explanations
- Show task implementations organized by domain with complete error handling and retry logic
- Include worker deployment configuration files for your specified infrastructure
- Provide monitoring dashboard definitions and alerting rule configurations
- End with a production operations runbook covering common scenarios like worker restarts, queue draining, and task replay