For over a decade, the idiomatic way to implement lazy generators or data pipelines in Go was the "concurrency pattern": spin up a goroutine, push data into a channel, and close the channel when done. While elegant, this pattern abuses Go's concurrency primitives for sequential logic.
Using channels for simple iteration incurs significant performance penalties: heavy Garbage Collector (GC) pressure from short-lived goroutine stacks, scheduler overhead (context switching), and the risk of goroutine leaks if the consumer exits early.
With Go 1.23, the standard library introduced the iter package. This allows us to refactor push-based channel generators into pull-based iterators, eliminating the concurrency overhead entirely while maintaining the ergonomics of for-range loops.
The Root Cause: Why Channels Are Expensive for Iteration
When you use a channel merely to stream data from point A to point B without parallel processing, you incur costs at three layers:
- Memory Allocation (GC Pressure): Every generator requires a new goroutine. A goroutine starts with a 2KB stack. While small, creating and destroying millions of these (e.g., in a high-throughput request handler) creates "garbage" that the runtime must track and collect. Additionally, the channel itself is a heap-allocated struct with internal locks.
- Scheduler Latency: Passing a value through a channel (
ch <- valtoval := <-ch) often involves park/unpark operations in the runtime scheduler. This forces context switches, trashing CPU caches compared to a simple function call. - Goroutine Leaks: This is the most dangerous operational risk. If the consumer stops iterating (e.g., via
breakor an error), the producer goroutine blocks forever onch <- val, waiting for a read that never happens. To fix this, you must introduce adonechannel orcontext, complicating the code significantly.
The Legacy Pattern: Channel-Based Generators
Consider a typical scenario: reading a stream of large log lines, parsing them, and filtering them. Here is the legacy channel implementation.
package main
import (
"fmt"
"math/rand"
"time"
)
// LogEntry represents a parsed data point.
type LogEntry struct {
ID int
Timestamp int64
}
// LegacyGenerator: Uses a channel and a goroutine.
// PROBLEM: If the caller breaks early, the goroutine leaks.
func StreamLogsLegacy(count int) <-chan LogEntry {
out := make(chan LogEntry)
go func() {
defer close(out)
for i := 0; i < count; i++ {
// Simulate work
entry := LogEntry{
ID: i,
Timestamp: time.Now().UnixNano(),
}
// BLOCKING: If caller stops reading, we hang here forever.
out <- entry
}
}()
return out
}
func main() {
// Consuming the channel
for entry := range StreamLogsLegacy(5) {
fmt.Printf("Legacy: %d\n", entry.ID)
}
}
The Fix: Refactoring to iter.Seq
Go 1.23 standardizes iteration via function signatures. Instead of returning a channel, we return an iter.Seq[T]. This is a function type defined as func(yield func(T) bool).
This approach keeps the logic on the caller's goroutine. There is no new stack allocation, no channel lock, and no scheduler involvement.
The Refactored Code
package main
import (
"fmt"
"iter"
"time"
)
type LogEntry struct {
ID int
Timestamp int64
}
// StreamLogsModern returns an iterator sequence.
// No goroutines are spawned. No channels are allocated.
func StreamLogsModern(count int) iter.Seq[LogEntry] {
return func(yield func(LogEntry) bool) {
for i := 0; i < count; i++ {
entry := LogEntry{
ID: i,
Timestamp: time.Now().UnixNano(),
}
// YIELD: We pass data to the caller's loop body.
// The return value 'continue' tells us if the caller
// wants more data.
if !yield(entry) {
// The caller used 'break' or returned.
// We return cleanly, performing any cleanup needed here.
return
}
}
}
}
// Example of chaining iterators (middleware pattern)
func FilterEvenIDs(seq iter.Seq[LogEntry]) iter.Seq[LogEntry] {
return func(yield func(LogEntry) bool) {
// We loop over the upstream sequence
for entry := range seq {
if entry.ID%2 == 0 {
if !yield(entry) {
return
}
}
}
}
}
func main() {
// 1. Basic Consumption
// The syntax looks identical to channel ranging.
fmt.Println("--- Basic Iteration ---")
for entry := range StreamLogsModern(3) {
fmt.Printf("Modern: %d\n", entry.ID)
}
// 2. Composed Pipeline
// This compiles down to nested function calls, extremely fast.
fmt.Println("\n--- Pipeline Iteration ---")
pipeline := FilterEvenIDs(StreamLogsModern(10))
for entry := range pipeline {
if entry.ID > 4 {
fmt.Println("Breaking early...")
break // This causes 'yield' to return false inside the generator
}
fmt.Printf("Filtered: %d\n", entry.ID)
}
}
Why This Works: Technical Breakdown
1. Inversion of Control
In the channel model, the producer runs independently and "pushes" data. In the iter model, the compiler rewrites the for range loop into a closure passed to your generator.
When you write:
for v := range StreamLogsModern(10) {
fmt.Println(v)
}
The Go compiler effectively transforms it into:
StreamLogsModern(10)(func(v LogEntry) bool {
fmt.Println(v)
return true // Continue loop
})
2. Stack Locality & Zero Allocation
Because StreamLogsModern runs on the caller's stack, arguments and variables are often kept in CPU registers. The iter.Seq is just a function pointer. There is no heap escape for the synchronization structure because there is no synchronization structure.
3. Automatic Cleanup
Notice the if !yield(entry) { return } check. If the consumer calls break in their loop, the yield function returns false. This allows your generator to execute defer statements or close resources immediately on the same stack frame. The "goroutine leak" class of bugs is mathematically impossible here because there is only one goroutine.
Conclusion
If your data pipeline does not require asynchronous processing (i.e., you don't need to process item N+1 while item N is being consumed), stop using channels.
Refactoring to iter reduces GC pressure, eliminates deadlock risks, and drastically improves CPU cache locality. Use channels for orchestration and signaling; use iter for data streams.