Build a composable streaming data pipeline in Go using channels, stages, and fan-out fan-in with cancellation and error handling.
## CONTEXT I need to process a stream of data in Go through multiple transformation stages, with concurrency where it helps, bounded memory, cancellation, and clean error propagation. I want a composable pipeline built from channel-based stages that I can extend without rewriting. Assume idiomatic Go pipeline patterns in 2026. ## ROLE Act as a Go concurrency engineer who builds streaming pipelines that stay correct under cancellation and failure. You compose stages cleanly, fan out only where it helps, and ensure every goroutine terminates and every error surfaces. ## RESPONSE GUIDELINES - Build the pipeline from small, composable channel-based stages. - Propagate cancellation through context to every stage. - Bound memory by limiting buffering and parallelism. - Surface errors without leaking goroutines. ## TASK CRITERIA ### Design The Stages - Define each stage as a function from an input to an output channel. - Keep each stage single-purpose and independently testable. - Pass context so every stage observes cancellation. - Make the pipeline easy to extend with new stages. ### Add Concurrency Where It Helps - Fan out CPU or I/O bound stages to multiple workers. - Fan in results into a single downstream channel. - Choose parallelism based on the stage workload. - Avoid parallelizing stages that must preserve order. ### Handle Cancellation - Stop all stages promptly when context is canceled. - Ensure no stage blocks forever on a full or empty channel. - Drain or abandon in-flight items per a clear policy. - Verify every goroutine exits on cancellation. ### Propagate Errors - Carry errors alongside data or through a dedicated channel. - Cancel upstream stages when a fatal error occurs. - Distinguish recoverable from fatal stage errors. - Return a final aggregated error to the caller. ### Bound Resources And Test - Limit channel buffering to control memory use. - Apply backpressure so fast producers do not overwhelm consumers. - Test the pipeline for correctness, cancellation, and leaks. - Benchmark throughput and tune parallelism with data. ## ASK THE USER FOR - The data source and the transformations each stage performs. - Whether ordering must be preserved through the pipeline. - Which stages are CPU bound versus I/O bound. - Your memory limits and expected throughput.
Or press ⌘C to copy