Skip to main content

Posts

Showing posts with the label Kafka

Solving the Dual-Write Problem: Implementing the Transactional Outbox Pattern

  It’s the distributed system nightmare every backend engineer encounters eventually. Your application successfully commits a record to the database (e.g., a new user sign-up). Immediately after, it attempts to publish an event to Apache Kafka to trigger downstream services (e.g., send a welcome email). But the network blips. The message broker is unreachable. The application throws an error, but the database transaction is already committed. You now have a  dual-write inconsistency . The user exists in your primary database, but the rest of your microservices architecture has no idea. The system is out of sync, and manual reconciliation is painful. This guide details how to solve this strictly using the  Transactional Outbox Pattern  with PostgreSQL, Kafka, and Debezium. The Anatomy of the Dual-Write Problem Before fixing it, we must understand why the naive approach fails. In a monolithic architecture, you might rely on a single ACID transaction. In microservices, ...

Handling Event Schema Changes: Implementing Upcasters to Fix Deserialization Errors

  You have successfully implemented Event Sourcing using the Axon Framework. Your event store is immutable, capturing every state change your application has ever seen. It works perfectly until your domain logic evolves. You add a new mandatory field to an existing event class. You deploy the new version. Suddenly, your replay fails, or your projection groups stall. The logs are flooded with serialization exceptions: com.fasterxml.jackson.databind.exc.MismatchedInputException: Missing required creator property 'currency' This is the "Schema Evolution" problem. Since you cannot modify the historical events in your append-only store, you must bridge the gap between the old data structure and your new Java class definition. The solution is not to relax your serialization rules, but to implement  Upcasters . The Root Cause: The Immutability Paradox In a CRUD-based system, schema changes are handled via database migration scripts (e.g., Flyway or Liquibase). You simply run...