Every Elixir developer eventually encounters the dreaded 5000ms timeout error. It usually appears in your logs like this:
** (exit) {:timeout, {GenServer, :call, [MyServer, :process_data, 5000]}}
This error is deceptive. It rarely means the network is slow or the database is down. In the context of the BEAM (Erlang VM), this error indicates an architectural bottleneck: your process mailbox is overflowing.
When a GenServer crashes due to a timeout, it means the process could not process the message and send a reply within the default window. Increasing the timeout is rarely the correct solution. To fix this permanently, we must understand the mechanics of the process mailbox and apply concurrency patterns like Task.Supervisor or PartitionSupervisor.
The Root Cause: Serial Execution in a Concurrent World
To fix the bottleneck, you must understand the actor model’s limitation. An Elixir process (including a GenServer) is a strictly serial entity. It handles one message at a time.
Under the hood, every process on the BEAM has a mailbox. When you issue a GenServer.call/3, a message is appended to the tail of that process's mailbox. The process pulls messages from the head of the queue, processes them, and moves to the next.
The Mathematics of the Crash
If your GenServer takes 50ms to process a single message, it has a maximum throughput of 20 messages per second.
If your system receives 30 messages per second, the queue grows by 10 messages every second. By the time the GenServer picks up the 100th message, it has been sitting in the mailbox for several seconds. Eventually, the latency exceeds 5000ms, and the caller crashes with a timeout.
Diagnosing the Mailbox
Before applying a fix, confirm the mailbox is the culprit. We can inspect a running process using Process.info/2.
If you have a live remote console (IEx) attached to your production node, run the following to find processes with large queues:
# Find the top 5 processes by message queue length
Process.list()
|> Enum.map(fn pid ->
info = Process.info(pid, [:registered_name, :message_queue_len])
{pid, info[:registered_name], info[:message_queue_len]}
end)
|> Enum.sort_by(fn {_, _, len} -> len end, :desc)
|> Enum.take(5)
If you see a message_queue_len in the hundreds or thousands, you have found your bottleneck.
Solution 1: Offloading Blocking Work with Task.Supervisor
The most common cause of timeouts is performing blocking operations (HTTP requests, heavy JSON parsing, database transactions) directly inside the handle_call/3 callback.
While the GenServer waits for the database, it cannot process other messages. The fix is to move the work out of the GenServer's critical path using Task.Supervisor.
The Vulnerable Code
This GenServer blocks the entire process loop while generating a report:
def handle_call({:generate_report, data}, _from, state) do
# BLOCKING: This takes 2 seconds
report = HeavyWork.process(data)
{:reply, report, state}
end
The Fix
We use a Task to perform the work. However, since handle_call expects a reply, we must be careful. If the client needs the answer immediately, we cannot make it async without changing the contract.
If the client needs the result, the bottleneck is unavoidable unless we parallelize. If the client doesn't need the result immediately (fire and forget), we should change the architecture to return :ok immediately and process async.
Here is a robust pattern where the GenServer delegates work but keeps the mailbox free:
# lib/my_app/report_processor.ex
defmodule MyApp.ReportProcessor do
use GenServer
# 1. Start a TaskSupervisor in your application.ex first!
# {Task.Supervisor, name: MyApp.TaskSupervisor}
def start_link(_opts) do
GenServer.start_link(__MODULE__, %{}, name: __MODULE__)
end
def init(state) do
{:ok, state}
end
# We change call to cast if we don't need a synchronous reply.
# If we MUST reply, see Solution 2 (Partitioning).
def handle_cast({:process_job, data}, state) do
Task.Supervisor.start_child(MyApp.TaskSupervisor, fn ->
perform_work(data)
end)
# Returns immediately. The GenServer is free to process the next msg.
{:noreply, state}
end
defp perform_work(data) do
# Simulate heavy work
Process.sleep(1000)
IO.puts("Processed #{inspect(data)}")
end
end
By spawning a Task, the GenServer's only job is to instantiate the task. The reduction count stays low, and the mailbox stays empty.
Solution 2: PartitionSupervisor (For High-Volume Synchronous Calls)
Sometimes you cannot use Task. You might need to maintain state, or the caller strictly requires a synchronous reply (GenServer.call).
If a single GenServer cannot keep up with the volume of calls, you must shard (partition) the process. Since Elixir 1.14, PartitionSupervisor is the standard, production-ready way to do this without external libraries.
This creates multiple instances of your GenServer (e.g., one per CPU core) and routes messages based on a key (like a User ID or Transaction ID).
Step 1: Define the Worker
This is a standard GenServer. No special logic is required inside the module.
defmodule MyApp.Worker do
use GenServer
def start_link(opts) do
GenServer.start_link(__MODULE__, opts)
end
def init(_opts) do
{:ok, %{}}
end
def handle_call({:process, item}, _from, state) do
# Simulate work
Process.sleep(50)
{:reply, :done, state}
end
end
Step 2: Configure the PartitionSupervisor
In your application.ex, instead of starting a single worker, you start the partition supervisor.
# lib/my_app/application.ex
def start(_type, _args) do
children = [
{PartitionSupervisor,
child_spec: MyApp.Worker.child_spec([]),
name: MyApp.WorkerPartition}
]
Supervisor.start_link(children, strategy: :one_for_one)
end
Step 3: Routing Requests
When calling the GenServer, you use a via tuple. The PartitionSupervisor automatically routes the message to a specific dynamic child based on the key provided.
defmodule MyApp.Router do
def process_item(item_id) do
# The {:via, ...} tuple handles the routing logic
GenServer.call(
{:via, PartitionSupervisor, {MyApp.WorkerPartition, item_id}},
{:process, item_id}
)
end
end
Why This Fixes Timeouts
If you have 8 cores, PartitionSupervisor starts 8 workers by default. Your throughput capacity effectively multiplies by 8. If one "shard" is blocked processing a heavy user's request, requests for other users (routed to different shards) continue to process instantly.
The Trap: Why Increasing Timeout is Fatal
A common "hotfix" is increasing the timeout:
GenServer.call(pid, :msg, 30_000) # Don't do this
This exacerbates the problem.
- Backpressure Failure: The timeout acts as a mechanism to signal that the system is overloaded.
- Memory Leaks: If you allow the queue to grow for 30 seconds instead of 5, the process consumes more RAM to store those messages.
- Cascading Failure: If the caller waits 30 seconds, the HTTP request that triggered the caller likely timed out at the load balancer level long ago. You are processing work for a client that has already disconnected.
Common Edge Case: The "Cast" Avalanche
Be wary of switching from call to cast blindly.
GenServer.cast returns :ok immediately, regardless of the mailbox size. This removes the timeout error but introduces a memory overflow risk. If you cast messages faster than the worker can process them, the VM will eventually run out of memory and crash the entire node.
If you need high-volume asynchronous processing, use Broadway or GenStage to implement backpressure, ensuring you only pull as many jobs as you can handle.
Summary
GenServer timeouts are a symptom of serial execution bottlenecks. To resolve them in a production environment:
- Measure: Check
process_info(pid, :message_queue_len). - Offload: Use
Task.Supervisorfor heavy, stateless work. - Scale: Use
PartitionSupervisorto shard stateful, synchronous work across multiple processes. - Avoid: Do not simply increase the timeout value.
By respecting the constraints of the actor model and leveraging OTP's supervision trees, you ensure your Elixir applications remain resilient under heavy load.