Skip to main content

Event Sourcing Pitfalls: Managing Schema Evolution and Event Versioning

 

The "Grey Screen" of Replays

You have a mature Event Sourced system. The UserRegistered event has been production-stable for two years. Today, you decided to refactor. The fullName string field in the payload is technically debt; you need structured data. You split it into firstName and lastName, update your domain models, run the tests, and deploy.

Ten minutes later, your projection replay service crashes.

Error: Validation Failed. 
Path: ['firstName'] - Required
Path: ['lastName'] - Required
Source: { "fullName": "John Doe", ... }

You just broke the cardinal rule of Event Sourcing: The Event Store is an immutable ledger. You cannot simply run an UPDATE SQL statement to migrate historical JSON blobs to the new schema because that corrupts the cryptographic or logical integrity of the log. Yet, your new code cannot understand the old language.

The Root Cause: Immutable Facts vs. Mutable Code

The core conflict lies in the temporal decoupling of storage and interpretation.

  1. Serialization (Write time): Occurred in the past (e.g., 2022) using Schema V1.
  2. Storage: The blob is frozen in time.
  3. Deserialization (Read time): Occurs now (2024) using Schema V2.

When you refactor a standard CRUD application, you write a database migration script to align the data with the new code. in Event Sourcing, you do not migrate the data. If you modify the event store, you destroy the audit trail.

Instead of migrating data at rest, we must migrate data in flight. We need a translation layer that lazily upgrades historical events to match the current domain understanding before they reach your domain logic.

The Fix: The Upcasting Pattern

We will implement an Upcaster. This is a middleware component that sits between the raw event stream and your domain event handler. Its job is to detect old versions of an event and transform the JSON payload into the current version's shape.

We will handle a scenario where we evolve a User event through three versions:

  1. V1: fullName (string)
  2. V2: firstName, lastName (split)
  3. V3: name object containing first and last (structural nesting)

1. Define the Versioned Schemas

We use Zod for runtime validation to ensure our upcasting logic results in a valid type.

import { z } from 'zod';

// --- Version 1 (Legacy) ---
// Stored as: { "type": "UserRegistered", "version": 1, "payload": { "fullName": "Jane Doe" } }
const UserRegisteredV1Schema = z.object({
  fullName: z.string(),
});

// --- Version 2 (Transitional) ---
const UserRegisteredV2Schema = z.object({
  firstName: z.string(),
  lastName: z.string(),
});

// --- Version 3 (Current) ---
const UserRegisteredV3Schema = z.object({
  name: z.object({
    first: z.string(),
    last: z.string(),
  }),
  email: z.string().email(),
});

// The goal is to always consume V3 in our domain logic
type UserRegisteredCurrent = z.infer<typeof UserRegisteredV3Schema>;

2. Implement the Upcasters

An upcaster is a pure function: (oldPayload) => newPayload. We chain them to allow an event to travel from V1 -> V2 -> V3 automatically.

type UnknownPayload = Record<string, unknown>;

const upcastV1toV2 = (payload: UnknownPayload): UnknownPayload => {
  // Logic: Split string into first/last
  const fullName = payload.fullName as string;
  const [firstName, ...rest] = fullName.split(' ');
  const lastName = rest.join(' ') || ''; // Handle mononyms gracefully

  return {
    firstName,
    lastName,
    email: payload.email, // Pass through existing fields
  };
};

const upcastV2toV3 = (payload: UnknownPayload): UnknownPayload => {
  // Logic: Nest fields into a 'name' object
  return {
    name: {
      first: payload.firstName,
      last: payload.lastName,
    },
    email: payload.email,
  };
};

3. The Upcasting Registry

We need a registry that knows how to route a specific event type and version through the correct transformation pipeline.

type UpcasterFn = (payload: any) => any;

class EventUpcasterRegistry {
  // Map<EventType, Map<Version, Upcaster>>
  private registry = new Map<string, Map<number, UpcasterFn>>();

  register(eventType: string, fromVersion: number, upcaster: UpcasterFn) {
    if (!this.registry.has(eventType)) {
      this.registry.set(eventType, new Map());
    }
    this.registry.get(eventType)!.set(fromVersion, upcaster);
  }

  // Recursively apply upcasters until no more upcasters exist for the version
  upcast(eventType: string, version: number, payload: any): { version: number; payload: any } {
    const versionMap = this.registry.get(eventType);
    
    // Base case: No upcasters for this event type
    if (!versionMap) return { version, payload };

    const upcaster = versionMap.get(version);

    // Base case: We reached the latest version (no upcaster for this version)
    if (!upcaster) return { version, payload };

    // Recursive step: Apply transform and try to upcast the next version
    const nextPayload = upcaster(payload);
    const nextVersion = version + 1;

    console.log(`[Upcasting] ${eventType}: V${version} -> V${nextVersion}`);
    
    return this.upcast(eventType, nextVersion, nextPayload);
  }
}

4. Integration (The Deserializer)

Here is how you wire this into your event processing loop. This simulates reading from an Event Store (like EventStoreDB or Kafka).

// --- Setup Registry ---
const registry = new EventUpcasterRegistry();

// Register the chain: V1 -> V2, and V2 -> V3
registry.register('UserRegistered', 1, upcastV1toV2);
registry.register('UserRegistered', 2, upcastV2toV3);

// --- Simulation Data ---
// Imagine these are pulled from your database
const historicalEvents = [
  { 
    id: 'evt_1', 
    type: 'UserRegistered', 
    version: 1, 
    payload: { fullName: "Grace Hopper", email: "grace@navy.mil" } 
  },
  { 
    id: 'evt_2', 
    type: 'UserRegistered', 
    version: 2, 
    payload: { firstName: "Alan", lastName: "Turing", email: "alan@bletchley.uk" } 
  }
];

// --- Processing Loop ---
async function replayEvents() {
  for (const rawEvent of historicalEvents) {
    
    // 1. Upcast raw JSON to latest structure
    const { payload: finalPayload } = registry.upcast(
      rawEvent.type, 
      rawEvent.version, 
      rawEvent.payload
    );

    // 2. Validate against the CURRENT domain schema (V3)
    // If upcasting failed logic, this Zod parse will throw, protecting domain integrity.
    try {
      const domainEvent = UserRegisteredV3Schema.parse(finalPayload);
      
      console.log(`Processed ${rawEvent.id} as V3:`, JSON.stringify(domainEvent, null, 2));
      
      // await projectToReadModel(domainEvent);
    } catch (error) {
      console.error(`Failed to process event ${rawEvent.id}`, error);
      // In prod: Dead Letter Queue logic here
    }
  }
}

// Execute
replayEvents();

Output

When you run the code above, the output demonstrates how V1 and V2 events are normalized into the V3 structure before the domain touches them.

[Upcasting] UserRegistered: V1 -> V2
[Upcasting] UserRegistered: V2 -> V3
Processed evt_1 as V3: {
  "name": {
    "first": "Grace",
    "last": "Hopper"
  },
  "email": "grace@navy.mil"
}

[Upcasting] UserRegistered: V2 -> V3
Processed evt_2 as V3: {
  "name": {
    "first": "Alan",
    "last": "Turing"
  },
  "email": "alan@bletchley.uk"
}

Why This Works

1. Lazy Migration

We avoid "Stop the World" database migrations. If you have 100 million events, you don't need to update 100 million rows during deployment. The transformation happens in memory, on-demand, only for the events actually being accessed.

2. Encapsulated Complexity

The domain layer (UserAggregate or UserProjector) remains clean. It only knows about UserRegisteredV3. It does not contain messy if (event.version === 1) switches. The complexity of backwards compatibility is isolated entirely within the anti-corruption layer (the Serializer/Upcaster).

3. Non-Destructive

Because we never alter the source event in the persistence store, we retain the original context. If our V1->V2 upcaster had a bug (e.g., it mishandled names with hyphens), we can fix the upcaster code and replay again. If we had migrated the database rows, the original data would have been destroyed.

Conclusion

In distributed systems, the definition of "current state" is fluid. Event Sourcing forces us to acknowledge that while history is immutable, our interpretation of history evolves.

Do not pollute your domain logic with version checks, and do not mutate your event log. Use Upcasters to create a strictly typed bridge between your legacy data and your modern code. This keeps your write path fast, your storage immutable, and your domain logic pure.

Labels: