Event-Driven Architecture Design

You are a distributed systems engineer specializing in event-driven architectures and asynchronous messaging patterns. Design a comprehensive event-driven system for the following platform.

Platform Context:
Business Domain: [E-COMMERCE/FINTECH/IOT/SOCIAL/LOGISTICS/OTHER]
Event Volume: [EVENTS PER SECOND ESTIMATE]
Processing Requirements: [REAL-TIME/NEAR-REAL-TIME/BATCH]
Ordering Requirements: [STRICT ORDERING/PARTIAL ORDERING/NO ORDERING]
Message Broker: [KAFKA/RABBITMQ/AWS SQS-SNS/GOOGLE PUB-SUB/NATS]
Consumer Count: [NUMBER OF CONSUMING SERVICES]

Section 1 - Event Design and Schema Strategy:
Define the event taxonomy categorizing events as domain events, integration events, and notification events with clear distinctions between each type. Design the event schema using a standard envelope format that includes event ID, type, source, timestamp, correlation ID, causation ID, and schema version. Create the event naming convention following past-tense verb-noun patterns like OrderPlaced, PaymentProcessed, and InventoryReserved. Specify the event payload design principles including what data to include in the event versus what to look up from the source service. Define the schema registry strategy for managing event schemas including compatibility rules for schema evolution such as backward compatible, forward compatible, and full compatible modes. Design the event catalog that documents all events in the system with their producers, consumers, payload schemas, and SLA requirements.

Section 2 - Message Broker Topology and Configuration:
Design the topic and queue structure including naming conventions, partitioning strategy, and replication configuration. Specify the partition key selection for each event type ensuring related events are processed in order while maximizing parallelism. Define the retention policy for each topic balancing storage costs with consumer replay requirements. Configure the broker cluster for high availability including the minimum in-sync replicas, acknowledgment settings, and failover behavior. Design the multi-region replication strategy if the system spans data centers including active-active versus active-passive configurations. Specify the capacity planning model including throughput per partition, consumer lag thresholds, and scaling triggers for adding partitions or broker nodes.

Section 3 - Producer Patterns and Reliability:
Design the transactional outbox pattern that ensures events are published atomically with database changes preventing lost events. Specify the event publishing retry strategy including which failures are retryable, maximum retry attempts, and backoff configuration. Create the idempotent producer configuration that prevents duplicate events from being written to the broker. Define the event enrichment strategy specifying whether events are enriched at production time or consumption time with tradeoffs. Design the change data capture pipeline as an alternative event source that publishes events based on database changes. Specify the producer monitoring metrics including publish success rate, latency percentiles, and retry frequency.

Section 4 - Consumer Patterns and Processing Guarantees:
Define the consumer group strategy including group naming conventions, partition assignment, and rebalancing behavior. Design the idempotent consumer implementation using deduplication keys, idempotency tables, or natural idempotency to achieve effectively once processing. Specify the consumer offset management strategy including when offsets are committed, how to handle processing failures, and how to replay events from a specific point. Create the consumer scaling rules including the relationship between partitions and consumer instances. Design the competing consumers pattern for work distribution and the event sourcing consumer pattern for building read models. Define the consumer lag monitoring and alerting thresholds that indicate processing is falling behind.

Section 5 - Error Handling and Dead Letter Queues:
Design the dead letter queue strategy including when messages are routed to the DLQ, the maximum retry count before dead lettering, and the metadata preserved with dead lettered messages. Create the DLQ processing workflow including manual inspection tools, bulk replay capabilities, and automated retry scheduling. Define the error classification system that distinguishes between transient errors worth retrying, permanent errors requiring code fixes, and poison messages that should be discarded. Specify the circuit breaker integration that stops consuming events when a downstream dependency is unavailable rather than filling the DLQ. Design the compensating event strategy for handling failures in saga workflows where earlier steps must be undone. Create the alerting and escalation procedures for DLQ growth including on-call notification, severity classification, and resolution SLA.

Section 6 - Testing, Monitoring, and Evolution:
Define the event-driven testing strategy including unit tests for event handlers, integration tests with embedded brokers, and end-to-end tests that verify complete event chains. Design the event flow visualization dashboard that shows real-time event throughput, consumer lag, and processing success rates across the entire system. Specify the event versioning migration strategy for upgrading consumers to new schema versions without downtime. Create the chaos engineering experiments for testing event system resilience including broker node failures, consumer crashes, and network partitions. Define the event replay runbook for rebuilding read models or reprocessing historical events after bug fixes. Document the capacity planning process for forecasting event volume growth and proactively scaling the messaging infrastructure.