Sagas and distributed transactions
- Lesson 30 of 40
- 13 min
- Advanced
- 11 steps
Booking a whole trip touches three services, each with its own database. We will see why no transaction covers all three, why two-phase commit gets stuck and how a saga commits step by step and undoes its work when something fails.
- Trip all-or-nothing
- eventually
- Locks left waiting
- none
- Coordination
- orchestrator
- request
- response
- failed request
- event
- slow request
Step 1 of 11
Problem
One trip, three databases
Someone wants to book a trip: a flight, a hotel and a rental car. Each part is handled by a different service with its own database, as we saw with microservices.
The booking service asks for the flight and gets it. It asks for the hotel and gets it too. It asks for the car and hears that no cars are left. The result: a flight and a hotel that are useless on their own, and two charges on the card.
Step 2 of 11
Technique
Two-phase commit
A classic answer is two-phase commit (2PC). In phase 1, a coordinator asks every participant to prepare: do the work, lock the rows and promise it will be able to commit. Each one replies with its vote.
In phase 2, if everyone voted yes, the coordinator tells them to commit; if anyone voted no, it tells all of them to undo. Here Cars votes no, so nothing is booked anywhere: all or nothing.
What changed
- Bookings becomes a 2PC coordinator
- Prepare → vote → commit or undo
Step 3 of 11
Problem
If the coordinator goes down
This time all three participants vote yes and stay prepared. Right then the coordinator goes down, before sending its decision. A prepared participant cannot decide alone: if it commits while the others undo, the trip ends up half done.
So it waits, with the seat, the room and the car locked. Anyone else trying to book those same places waits too, until the coordinator comes back.
What changed
- Coordinator goes down after prepare
- Flight, hotel and car stay locked
Step 4 of 11
Solution
A saga: commit step by step, compensate on failure
A saga splits the trip into local transactions that commit right away, one after another: reserve flight, reserve hotel, rent car. Nobody keeps locks open waiting for the others.
Every step has a compensating action that cancels its business effect. When the car fails, the saga runs the compensations in reverse order: first it cancels the hotel, then the flight.
What changed
- Each step commits in its own DB
- Compensations: cancel hotel, then flight
- Locks across services
Step 5 of 11
Technique
Orchestration: one service runs the saga
In an orchestrated saga, one component, the orchestrator, works as a small state machine: it tells each service what to do, waits for its answer and decides whether to move on or compensate.
After every step it saves the saga state in its own database. If it restarts, it reads that state and carries on where it left off, without starting over.
What changed
- Bookings becomes the orchestrator
- Saga state saved after every step
Step 6 of 11
Technique
Choreography: each service reacts to events
In a choreographed saga nobody is in charge. Flights reserves and publishes FlightReserved; Hotels hears it, reserves and publishes HotelReserved; Cars hears that and reserves, or publishes CarFailed, and then Hotels and Flights cancel their part.
There is no central component to build and run, but the trip logic ends up spread across every service: it is harder to follow, test and change.
What changed
- Orchestrator
- Event bus
- Each service reacts to the previous step’s event
Step 7 of 11
Trade-off
The price: no isolation
While the saga runs, other people see its half-done state: that seat is no longer free even though the trip is not confirmed yet. And some steps cannot really be undone: a «trip confirmed» email that already went out does not come back.
You design for it: pending states («on hold», not «confirmed»), the step that fails most often first, and the hardest to undo, like charging the card or sending emails, last.
What changed
- Bookings start «on hold»
- Car availability checked first
- Card charged at the end
Step 8 of 11
Problem
Case: crash and resume
The orchestrator crashes right after Hotels reserved the room, but before it saved that fact. When it restarts it reads «hotel: pending» and sends the reservation again.
If Hotels reserved another room, the traveller would pay for two. So every step carries an idempotency key (saga id + step, here 57-hotel) and Hotels returns the reservation that already exists instead of creating a new one.
What changed
- Orchestrator restarts mid-saga
- «Reserve hotel» arrives twice
- Same key → same reservation
Step 9 of 11
Problem
Case: the compensation fails too
No cars are left, so the saga tries to cancel the hotel, but the Hotels API is down. Giving up is not an option: the traveller would pay for a room they will not use, and the flight is still waiting its turn to be cancelled.
Compensations are retried with growing waits (backoff) until they succeed, and must be safe to repeat. If one is still stuck after a while, for example 4 hours, it goes to a queue for a person to resolve.
What changed
- Hotels API down while cancelling
- Retries with backoff
- To support if still stuck (example: 4 h)
Step 10 of 11
Technique
Case: a timeout means «don’t know»
The call to rent the car runs past its timeout. Did Cars reserve it before the connection dropped? A timeout is not a failure: the outcome is unknown.
Compensating blindly could cancel a reservation that never existed or leave alive one that did. Better to ask: query Cars for the status of 57-car, or retry with the same idempotency key and get the real answer.
What changed
- «Rent car» timed out
- Status unknown → query 57-car
- Answer: reserved → carry on
Step 11 of 11
Summary
Recap
A database transaction cannot span several services.
2PC gets all-or-nothing, but blocks everyone if the coordinator goes down.
A saga commits step by step and runs compensations when something fails.
Orchestrate or choreograph; either way, make every step idempotent.
Compensations must succeed sooner or later; a timeout means «don’t know», not «failed».
← → step · R replay · Space pause
You will learn
- Why a transaction cannot cross services (covered)
- Two-phase commit and why it blocks (covered)
- Sagas and compensating actions (covered)
- Orchestration versus choreography (covered)
- Failing compensations and unknown outcomes (covered)