Skip to main content

Posts

Showing posts with the label OTP

Elixir Production Pitfalls: Diagnosing GenServer Mailbox Overflows

  The Silent Memory Killer The most insidious failure mode in the BEAM (Erlang VM) is not the loud exception that crashes a process and restarts it. It is the silent accumulation of messages in a single GenServer's mailbox. You have likely seen the symptoms: Latency Spikes:  Requests involving a specific subsystem start timing out. Memory Bloat:  Node memory usage climbs steadily (sawtooth pattern or linear) despite no apparent load increase. The Cliff:  The node suddenly becomes unresponsive to heartbeats and gets killed by the orchestrator or OOM killer. No Stacktrace:  Because the process didn't crash logic-wise—it just drowned—there is often no error log pointing to the culprit. This occurs because standard  GenServer.cast/2  is a "fire-and-forget" mechanism. If the producer sends messages faster than the consumer can process them, the mailbox (a linked list in memory) grows infinitely. The Root Cause: Unbounded Mailboxes & GC Debt In Erlang/El...