Skip to main content

Posts

Showing posts with the label OTP

Debugging Erlang Mailbox Overflows with OTP 27 Monitors

  The most insidious failure mode in the BEAM virtual machine is the silent mailbox overflow. Unlike a stack overflow or a logic error, a message queue buildup doesn't immediately crash the process. Instead, it slowly consumes memory, degrades garbage collection performance, and eventually triggers the system-wide Out-Of-Memory (OOM) killer, taking down the entire node. Until recently, detecting this required reactive polling loops or expensive introspection tools like  recon  that could exacerbate the load. With the release of OTP 27, we now have a native, event-driven mechanism to handle this at the VM level: the  message_queue_len  monitor. The Root Cause: Asynchronous Coupling and GC Erlang processes communicate asynchronously. When Process A sends a message to Process B, the message is copied to B's heap (or the shared heap for large binaries) and appended to its mailbox linked list. Process B consumes these messages via  receive  or  gen_ser...

Elixir OTP Strategy: Handling 'GenServer timeout' in Heavy Processing

  The Hook: The 5000ms Wall Every senior Elixir developer has seen this stack trace. It usually appears during a traffic spike, cascading through your logs and triggering pager alerts: ** (exit) exited in: GenServer.call(MySystem.HeavyWorker, :process_data, 5000) ** (EXIT) time out The default 5000ms timeout in  GenServer.call/3  is not arbitrary; it is a fail-safe. However, in high-load systems, hitting this timeout usually isn't a symptom of network latency—it is a symptom of  mailbox congestion . When a GenServer performs heavy processing directly in its main loop, it violates the cardinal rule of the Actor Model:  Keep the mailbox flowing. The Root Cause: Serial Execution in the Actor Model Under the hood, a GenServer is a single Erlang process with a mailbox (a FIFO queue) and a recursive loop. When you execute a blocking function inside  handle_call/3 : The process halts message consumption. It executes your logic (e.g., XML parsing, image resizin...