Go architectures · 3 MIN READ
Pipes and filters: data through independent stages
Compose transformations while making backpressure and errors explicit.
Compose transformations while making backpressure and errors explicit. A pipes-and-filters design passes data through small stages such as decode, validate and transform. Trace the worked Go example, then compare the benefit with the cost before applying this design to your own project.
Architecture · A MAP OF THE IDEA
- Decode or normalize
- Validate and transform
- Encode or emit
Useful when
- A data flow has independently testable stages
- Streaming can reduce whole-input memory use
Watch out
- Stage order and error propagation need a contract
- Concurrency adds cancellation and backpressure work
Implementation & operation cost: Medium
Fit depends on the task
Editorial suitability, not benchmarks or a universal ranking. Styles describe different axes and can be combined.
The design decision
A pipes-and-filters design passes data through small stages such as decode, validate and transform. Stages can be ordinary functions before concurrency is needed. Streaming changes the memory budget; concurrent stages also need cancellation and a bounded handoff.
A complete Go example
Read this program from the entry point and follow the calls shown in the diagram. The output is included at the end so you can check the behavior without a compiler.
package main
import (
"fmt"
"strings"
)
type Stage func(string) string
func Pipeline(input string, stages ...Stage) string {
for _, stage := range stages {
input = stage(input)
}
return input
}
func main() {
result := Pipeline(" go ", strings.TrimSpace, strings.ToUpper)
fmt.Println(result)
}
// Output:
// GO
The pipeline first trims and then capitalizes, so order matters. Function values make each stage independently reusable. This synchronous version avoids goroutine management; later streaming versions can preserve the same stage boundaries with explicit error propagation.
Tradeoffs and the simpler alternative
Use stages when transformations compose naturally and need separate tests. Shared state between stages undermines isolation. A single expression is simpler for one trivial transformation, and unbounded channel buffers do not solve a slow downstream consumer.
Define stage contracts and resource bounds
A stage consumes one representation and produces another: parse, normalize, validate and summarize. Start with ordinary function composition for finite in-memory data. Introduce channels only when streaming or independent work justifies coordination. Every stage needs a policy for malformed input, cancellation and output ownership. An unbounded buffer can turn temporary downstream slowness into a memory failure.
Propagate cancellation through the whole pipeline
If a downstream consumer exits early, upstream producers must be able to stop rather than block forever on send. Pass context, select on cancellation where sends can block, and let the owner of a channel close it after all producers finish. Multiple concurrent stages may reorder results; preserve an input index if order is required. A goroutine per item is not a bounded worker pool.
Test pressure as well as transformation
Verify a stage with empty input, invalid input, a slow consumer and cancellation halfway through. Check that completion waits for owned goroutines and that no sender is left writing to a closed channel. Benchmark representative data before adding concurrency: parsing small records in separate goroutines can cost more than a loop. Pipes and filters suit stream transformations; they are less helpful when each stage requires shared mutable domain state and distributed transactions.
TAKE THIS WITH YOU
Composes ordered transformations with explicit handoffs. Use stages when transformations compose naturally and need separate tests. Shared state between stages undermines isolation. A single expression is simpler for one trivial transformation, and unbounded channel buffers do not solve a slow downstream consumer.
Check your understanding
Which statement best describes Pipes and filters: data through independent stages?
Go to the source
Progress stays in this browser. No account needed.