Message Queue & Pub/Sub Design

You are a messaging infrastructure architect with experience designing reliable message-based systems for enterprise and cloud-native applications. Design a comprehensive messaging architecture for the following system.

System Context:
Messaging Patterns Needed: [POINT-TO-POINT/PUB-SUB/REQUEST-REPLY/MIXED]
Message Volume: [MESSAGES PER SECOND]
Message Size: [SMALL KB/MEDIUM MB/LARGE BINARY]
Ordering Requirements: [STRICT FIFO/PARTITION-ORDERED/UNORDERED]
Technology Options: [RABBITMQ/AWS SQS-SNS/GOOGLE PUB-SUB/AZURE SERVICE BUS/NATS]
Integration Points: [NUMBER OF PRODUCING AND CONSUMING SERVICES]

Section 1 - Topology and Exchange Design:
Define the messaging topology including exchanges, topics, queues, and subscriptions organized by business domain. Design the exchange types choosing between direct, topic, fanout, and headers routing with specific use cases for each. Specify the topic naming convention using a hierarchical dot or slash-separated namespace that supports wildcard subscriptions. Create the queue naming and binding strategy including durable versus transient queues, exclusive queues for specific consumers, and auto-delete queues for temporary workloads. Define the virtual host or namespace isolation strategy for separating environments and business domains on shared infrastructure. Design the federation or shovel configuration for connecting messaging infrastructure across data centers or cloud regions.

Section 2 - Message Design and Routing:
Define the message envelope format including headers for message ID, correlation ID, reply-to address, content type, timestamp, and custom routing properties. Design the message routing rules that direct messages to the correct consumers based on message type, content properties, or routing keys. Specify the message serialization format choosing between JSON, Protocol Buffers, Avro, or MessagePack with schema evolution considerations. Create the message versioning strategy for handling format changes without breaking existing consumers. Define the message size limits and the chunking strategy for messages that exceed the broker maximum. Design the request-reply pattern implementation including reply queue management, correlation matching, and timeout handling.

Section 3 - Delivery Guarantees and Reliability:
Specify the delivery guarantee level for each message type choosing between at-most-once, at-least-once, and exactly-once with the implementation mechanism for each. Design the publisher confirmation mechanism that ensures messages are durably stored before the producer proceeds. Define the consumer acknowledgment strategy including manual versus automatic acknowledgment, batch acknowledgment, and negative acknowledgment for redelivery. Create the message persistence configuration specifying which messages are written to disk versus memory-only for different durability requirements. Specify the message TTL and expiration policies for time-sensitive messages that should not be processed after their relevance window. Design the transaction support for operations that need to publish multiple messages atomically or combine message publishing with database operations.

Section 4 - Consumer Patterns and Scaling:
Define the competing consumers pattern for distributing work across multiple consumer instances with fair dispatch and prefetch count tuning. Design the consumer group management including how consumers join and leave groups, partition assignment, and rebalancing behavior. Create the message filtering strategy that allows consumers to receive only relevant messages without wasting network and processing resources on discarded messages. Specify the consumer concurrency model including the number of concurrent message handlers, thread pool configuration, and back-pressure signaling when consumers are overwhelmed. Design the consumer priority configuration for scenarios where certain consumers should receive messages preferentially. Define the exclusive consumer pattern for ensuring exactly one consumer processes messages from critical queues.

Section 5 - Error Handling and Dead Letter Management:
Design the dead letter exchange and queue configuration including the routing rules for dead lettered messages and the metadata preserved with each dead letter. Define the retry strategy using delayed redelivery with configurable retry counts, exponential backoff, and per-message retry tracking. Create the poison message detection that identifies messages causing repeated consumer failures and routes them to quarantine. Specify the dead letter queue monitoring and alerting with thresholds that trigger engineering notification. Design the DLQ processing tools including message inspection, selective replay, bulk replay, and permanent discard with audit trail. Define the error callback mechanism that notifies the original publisher when a message cannot be processed.

Section 6 - Monitoring, Capacity, and Operations:
Define the monitoring metrics including queue depth, publish rate, consume rate, acknowledgment rate, message age, and consumer lag for each queue. Create the alerting rules for queue depth growth exceeding consumption rate, consumer disconnections, and broker resource utilization thresholds. Design the capacity planning model for projecting broker resources based on message volume growth, retention requirements, and consumer processing time. Specify the broker cluster management including node addition, removal, and rolling upgrade procedures without message loss. Create the disaster recovery plan including cross-region replication, backup procedures, and failover automation. Document the operational runbook for common messaging incidents including stuck queues, slow consumers, broker memory pressure, and split-brain cluster recovery.