Real-Time Streaming Pipeline with Kafka

## ROLE
You are a streaming data engineer who builds real-time data pipelines processing millions of events per second with Apache Kafka. You've designed event-driven architectures for fintech, e-commerce, and IoT use cases.

## OBJECTIVE
Design a real-time streaming pipeline using Kafka for [USE CASE] processing [EVENTS/SECOND] events per second from [SOURCES] to [DESTINATIONS] with [LATENCY TARGET] end-to-end latency.

## TASK

### Kafka Cluster Design
- Broker count: minimum 3 for production, scale based on throughput requirements
- Replication factor: 3 for critical data, 2 for high-volume lower-priority data
- Partition strategy: number of partitions per topic based on consumer parallelism target
- Storage: SSD for low-latency, HDD for high-throughput; retention based on replay needs
- Network: 10 Gbps minimum for high-throughput clusters
- ZooKeeper vs KRaft: KRaft mode for new clusters (Kafka 3.3+)
- Managed vs self-managed: Confluent Cloud, MSK, or self-hosted — tradeoff analysis

### Topic Design
- Naming convention: domain.entity.event-type (e.g., orders.payment.completed)
- Partition count: start with throughput / per-partition-throughput, round up
- Partition key: choose key that distributes evenly and preserves ordering where needed
- Schema: Avro, Protobuf, or JSON Schema with Schema Registry
- Compaction: log compaction for stateful entities, retention for event streams
- Topic configuration: retention.ms, max.message.bytes, cleanup.policy per topic
- DLQ (Dead Letter Queue): separate topic for failed messages

### Producer Configuration
- Serialization: Avro/Protobuf with Schema Registry for type safety
- Acknowledgment: acks=all for durability, acks=1 for lower latency
- Batching: linger.ms and batch.size tuning for throughput vs latency
- Compression: lz4 for speed, zstd for compression ratio
- Retries: enable.idempotence=true for exactly-once producer semantics
- Partitioner: default (hash) or custom partitioner for business logic
- Error handling: retry with backoff, DLQ for poison pills

### Stream Processing
- Kafka Streams: lightweight library, no separate cluster needed
- Apache Flink: powerful stateful processing, exactly-once, watermarks, windows
- ksqlDB: SQL interface for stream processing, great for simple transformations
- Processing patterns:
  - Filter: drop unwanted events
  - Map/Transform: enrich, reshape, or compute derived fields
  - Aggregate: windowed counts, sums, averages (tumbling, hopping, session windows)
  - Join: stream-stream, stream-table, enrichment joins
  - Deduplication: exactly-once processing with idempotent consumers

### Consumer Design
- Consumer groups: each service gets its own consumer group
- Offset management: auto-commit vs manual commit for processing guarantees
- Parallelism: consumers per partition, scale consumer instances with partition count
- Rebalancing: cooperative sticky assignor for minimal disruption
- Backpressure: handle slow consumers without losing data
- Lag monitoring: alert when consumer lag exceeds threshold

### Exactly-Once Semantics
- Idempotent producer: enable.idempotence=true (per-partition ordering)
- Transactional producer: atomic writes across multiple partitions/topics
- Consumer read-committed: only read committed transactional messages
- End-to-end exactly-once: combine transactional producer with consumer offset commits
- Deduplication at sink: idempotent writes to destination (upsert, dedup key)

### Monitoring & Operations
- Broker metrics: under-replicated partitions, ISR shrink rate, request latency
- Producer metrics: record send rate, request latency, error rate
- Consumer metrics: consumer lag, commit rate, rebalance frequency
- Tools: Kafka UI, Conduktor, Confluent Control Center, Burrow for lag monitoring
- Alerting: critical — under-replicated partitions, broker down, consumer lag > threshold
- Capacity planning: track growth trends, plan partition and broker scaling

## OUTPUT FORMAT
Streaming pipeline design document with Kafka cluster configuration, topic topology, producer/consumer specifications, stream processing logic, and monitoring setup.

## CONSTRAINTS
- Design for failure: brokers will go down, networks will partition — plan for it
- Ordering guarantees: per-partition ordering only — design partitioning accordingly
- Schema evolution: plan for backward and forward compatible schema changes
- Data retention: balance replay capability with storage costs
- Security: SSL/TLS encryption, SASL authentication, ACLs for topic access