Erlang OTP 27 Upgrade: Replacing External Parsers with Native JSON

For over a decade, the Erlang and Elixir ecosystem has fragmented over JSON handling. We’ve relied on NIF-based libraries like jiffy for raw speed (at the cost of compilation friction) or pure-Erlang/Elixir libraries like Poison and Jason for stability. These dependencies bloat the release process, complicate cross-platform builds, and introduce version resolution conflicts.

With the release of OTP 27, these external dependencies are effectively obsolete. The OTP team has introduced the json module—a native, high-performance, validating parser and generator built directly into the runtime.

The Root Cause: The Terminology Mismatch

The historical reliance on external libraries stems from a fundamental mismatch between JSON types and Erlang terms. JSON is a strict subset of object notation; Erlang is a system of tuples, lists, and binaries.

Atom Exhaustion: Early parsers naively converted JSON keys to Erlang atoms. Since the atom table is not garbage collected, parsing arbitrary user input could crash the VM (DoS).
String Representation: Before UTF-8 binaries became the standard, Erlang handled strings as lists of integers. This made JSON string parsing prohibitively expensive.
NIF Overhead: To solve performance issues, libraries like jiffy used C implementations (NIFs). However, a crashing NIF takes down the entire VM, violating the Erlang supervision principle.

The OTP 27 json module solves this by strictly using binaries for strings (avoiding atom exhaustion), generating iolists for efficient I/O, and operating entirely within the managed memory of the BEAM.

The Fix: Migrating to Native JSON

The following solution demonstrates how to replace Jason with OTP 27’s native :json. We will implement a robust wrapper that handles Elixir structs and atom keys safely, mimicking the developer experience of Jason without the dependency.

Prerequisites: Erlang/OTP 27+ and Elixir 1.17+.

1. Basic Decoding and Encoding

The native module uses :json.encode/1 and :json.decode/1. Note that :json.encode/1 returns an iolist (a deep list of binaries), not a single binary string. This is significantly more performant for network sockets (Phoenix/Cowboy) as it avoids memory copying.

defmodule NativeDemo do
  @doc """
  Simple roundtrip demonstration.
  """
  def run do
    # 1. Encoding a standard map
    payload = %{"id" => 123, "active" => true, "tags" => ["otp", "27"]}
    
    # Returns iolist: ["{", [["\"id\"", ":", "123"], ",", ...], "}"]
    encoded_iolist = :json.encode(payload)

    # 2. Decoding back to a map
    # Input must be a binary.
    json_binary = IO.iodata_to_binary(encoded_iolist)
    
    # Returns: {:ok, %{"id" => 123, "active" => true, "tags" => ["otp", "27"]}}
    {:ok, decoded} = :json.decode(json_binary)
    
    decoded
  end
end

2. Production-Grade Implementation (Handling Structs & Atoms)

The raw :json module does not know about Elixir Protocols or Structs. If you pass a generic struct to :json.encode/1, it will treat it as a map and include the __struct__ key, exposing implementation details.

We need a serializer that:

Implements a Protocol for custom encoding (like Jason.Encoder).
Safely decodes keys to atoms (optional, but common in Elixir internal services).

Step A: Define the Protocol

defprotocol App.JSON.Encoder do
  @fallback_to_any true
  def encode(data)
end

defimpl App.JSON.Encoder, for: Any do
  def encode(data) do
    # Fallback: remove struct metadata or pass through
    Map.from_struct(data)
  end
end

# Example: Custom encoding for a User struct
defmodule App.User do
  defstruct [:id, :name, :password_hash]
end

defimpl App.JSON.Encoder, for: App.User do
  def encode(user) do
    # Explicitly select fields to expose, effectively sanitizing output
    %{
      id: user.id,
      name: user.name
    }
  end
end

Step B: The Serializer Module

This module wraps OTP 27 functions to provide a Jason-compatible API.

defmodule App.JSON do
  @moduledoc """
  High-performance JSON wrapper using OTP 27 native :json module.
  """

  @doc """
  Encodes an Elixir term to a JSON iolist.
  Uses the App.JSON.Encoder protocol for structs.
  """
  def encode!(term) do
    # We pass a custom encoder function to :json.encode/2
    :json.encode(term, &custom_encode/2)
  end

  @doc """
  Decodes JSON binary to an Elixir term.
  default: keys are binaries.
  """
  def decode!(binary) do
    case :json.decode(binary) do
      {:ok, result} -> result
      {:error, reason} -> raise "JSON Decode Error: #{inspect(reason)}"
    end
  end

  @doc """
  Decodes JSON and safely converts known keys to atoms.
  WARNING: Only use atoms: true if you are certain the keys exist
  in the atom table to prevent DoS.
  """
  def decode!(binary, [keys: :atoms]) do
    :json.decode(binary, [], &decode_callbacks/2)
  end

  # -- Internals --

  # Callback for :json.encode/2
  # This recursively handles structs via our Protocol
  defp custom_encode(struct = %_{}, _encode_continue) do
    # Resolve the protocol and recursively encode the result
    map_representation = App.JSON.Encoder.encode(struct)
    :json.encode(map_representation, &custom_encode/2)
  end

  defp custom_encode(other, encode_continue) do
    # Let the native encoder handle standard types (maps, lists, numbers)
    encode_continue.(other)
  end

  # Callback configuration for decoding to atoms
  defp decode_callbacks(:object_push, _state) do
    # Start of an object
    fn key, value, acc -> 
      atom_key = String.to_existing_atom(key)
      Map.put(acc, atom_key, value)
    rescue
      _ -> Map.put(acc, key, value) # Fallback to string if atom doesn't exist
    end
  end
  
  defp decode_callbacks(:object_finish, _state) do
    # When object finishes, start with empty map accumulator
    %{}
  end

  # Pass through other types (arrays, strings, numbers)
  defp decode_callbacks(_other, state), do: state
end

3. Integration

You can now use this module in your Phoenix Views or API controllers.

# In a Phoenix Controller
def index(conn, _params) do
  users = [%App.User{id: 1, name: "Alice", password_hash: "secret"}]
  
  # Encodes only id and name, returns iolist
  json_data = App.JSON.encode!(users)
  
  conn
  |> put_resp_content_type("application/json")
  |> send_resp(200, json_data)
end

The Explanation: How It Works

The Push-Parser Architecture

Unlike older libraries that parse the entire JSON string into an Abstract Syntax Tree (AST) in one pass, OTP 27's :json utilizes a callback-driven push parser.

When you call :json.decode/3, the runtime traverses the binary. Every time it encounters a JSON token (start of object, key, value), it triggers a specific callback function.

In our decode_callbacks/2 implementation above:

:object_push defines how to aggregate key-value pairs.
We intercept the key (which is a binary slice) and attempt String.to_existing_atom/1.
We construct the map incrementally.

This approach is revolutionary for Erlang because it allows selective decoding. If you have a 10MB JSON payload but only need the "id" field, you can write a callback that ignores all other keys, drastically reducing memory allocation compared to Jason which constructs the entire map in memory before you can access the field.

IO Lists vs. Binaries

The :json.encode/1 function returns an iolist. In the BEAM, an iolist is a nested list of binaries and integers (e.g., ["{", ["\"key\"", ":", "1"], "}"]).

When Phoenix sends a response to a socket:

Jason approach: Encode to a massive binary string $\to$ Copy to socket buffer.
Native approach: Generate iolist (referencing existing data on heap) $\to$ The OS writev system call gathers the data directly to the socket.

This reduces garbage collection pressure significantly on high-throughput systems.

Conclusion

OTP 27's native JSON support is not just a standard library addition; it is a fundamental infrastructure improvement. By replacing Jason or Poison with native :json, you align your application with the VM's internal memory model, remove a C-compilation step from your build pipeline, and gain access to powerful streaming decode capabilities.

Refactor your serialization layer today. The best dependency is the one you don't have to install.

Programming Tutorials

Search This Blog