Saga Patterns: Distributed Transactions Without Distributed Pain

What you need to know

• Sagas replace distributed transactions with a sequence of local transactions plus compensating transactions for rollback. No distributed locks required.
• Choreography sagas use events to coordinate steps with no central coordinator. Good for simple linear flows.
• Orchestration sagas use a central coordinator that commands each step. Better for complex flows where observability of overall transaction state matters.
• Compensating transactions must be designed upfront for every saga step. An uncommitted step with no compensating transaction is a correctness hole.
• Saga state must be persisted to durable storage before each step executes. This is what makes sagas recoverable across service restarts.

The core argument

The distributed transaction problem surfaces the moment a business operation touches more than one service. Order creation touches inventory, payment, fulfillment, and notification services. A failure halfway through leaves the system in an inconsistent state unless each step can be rolled back. Two-phase commit solves this with distributed locks, but distributed locks in production create availability problems, deadlock risks, and dependency chains between services that should be independent.

Sagas solve the same problem without locking. Each step is a local transaction. If a step fails, the saga runs compensating transactions for previously completed steps. The system reaches a consistent state through a series of single-service operations rather than a cross-service atomic transaction. The trade-off is eventual consistency: there is a window where the system is in an intermediate state, which is acceptable for most business operations and unacceptable for a small number (financial settlements, for example, where intermediate states have regulatory implications).

The choice between choreography and orchestration comes down to observability and complexity. For Nexli, we implemented a checkout saga using choreography initially because the flow was linear and the services were few. When the checkout flow added conditional branches (subscription products requiring different fulfillment than one-time purchases), the choreography model became hard to reason about because the overall transaction flow was implicit across multiple services. We migrated to orchestration, which made the transaction state visible in one place and made debugging saga failures straightforward.

Common mistakes

1. Not persisting saga state before executing each step. If the saga coordinator crashes after step 2 completes but before step 3 is recorded as pending, restart recovery cannot determine what has been done and what needs compensation. Persist the intended next step before executing the current step. Write-ahead style: record what you are about to do before doing it.

1. Designing compensating transactions as exact reversals when side effects have occurred. A compensating transaction for a step that sent an email cannot unsend the email. The compensating transaction must perform the appropriate real-world response (send a cancellation notification, record a credit) rather than reversing an irreversible action. Design compensating transactions for what can actually be undone.

1. Not making compensating transactions idempotent. Compensating transactions are retried on transient failure. If a compensating transaction is not idempotent, retrying it causes double-compensation (crediting an account twice, releasing inventory twice). Treat compensating transactions the same as the original steps: idempotent, with deduplication on the operation ID.

1. Building sagas for operations that fit within a single database. Adding saga infrastructure for operations that could be a single database transaction adds complexity without benefit. Sagas are justified when the operation genuinely crosses service or database boundaries. For a user account update that touches five tables in one database, a local transaction is always preferable.

1. Not monitoring saga failure rates as an operational metric. Sagas that fail and require compensation are business events that should be visible to the operations team. A spike in saga failures indicates infrastructure problems, data integrity issues, or downstream service degradation. Instrument saga failure rates as a key metric with alerting thresholds.

Where to start

1. Identify the one business operation in the product that most needs distributed consistency. In most SaaS products this is the checkout or subscription upgrade flow. Map the steps, the services involved, and what happens to the system if each step fails. This mapping reveals which steps need compensating transactions and what those compensations should do.

1. Implement an orchestration saga with persisted state for that operation. Create a saga_executions table with columns for the operation type, current step, step history (JSON), status, and created/updated timestamps. Write a coordinator that progresses through steps by updating this table. This gives immediate visibility into in-flight and failed sagas.

1. Write integration tests that simulate failure at each step. The most common saga bugs are in compensating transaction logic and recovery paths that are never exercised in normal operation. Write tests that force failure at each step and verify that compensating transactions restore the system to the correct initial state. These tests are the primary correctness check for saga implementations.

Related reading

• Resilience Patterns: Circuit Breakers, Retries, Bulkheads
• SaaS Webhook Reliability: From At Most Once to At Least Once to Exactly Once
• Schema Design Decisions That Haunt You at Million User Scale
• Service Decomposition: Drawing the Right Lines