When a later step fails: compensation and sagas

Lesson 6 made sure o7 is charged at most once. But what if Shipping then says "out of stock, never"? The charge already happened at PayCo, outside your database. You can't roll it back. You can only write a new fact that cancels it.

By the end you'll be able to undo an earlier step with a compensation (a refund) that is itself safe to repeat, and confirm it from PayCo's webhooks, which arrive late, twice, out of order, or forged.

See it fail first

Run Part A of the new demo. It has 200 paid orders, and Shipping rejects the out-of-stock ones (10%) permanently. It tries three naive shops:

$ npm run l7
Part A · 200 paid orders, Shipping rejects the out-of-stock ones for good (seed 1)

  version             rejected   charged, not shipped, not refunded   shipped AND refunded   refunded 2+   "already refunded" errors
  ignore                    21                                   21                      0             0                           0
  refund-inline             21                                    3                      0             0                           4
  refund-on-silence         21                                    0                     15             0                           0

• ignore is your Lesson 6 flow. Nobody handles the rejection, so 21 customers paid for nothing.
• refund-inline calls PayCo's refund right away, with no key and no outbox. 3 orders were lost to a crash between "rejected" and the call: nothing remembered them. In 4 cases the refund response was lost, and the retry got 400 charge_already_refunded. PayCo did refund, but the shop read the 400 as "the refund failed".
• refund-on-silence refunds whenever Shipping doesn't confirm within 30 ms. Shipping was only slow, so 15 customers got their parcel and their money back.

Each row is a lesson you already know, showing up somewhere new. Silence isn't "no" (L1). A step that can be repeated needs a key (L4). Something that must happen after a commit belongs in the outbox (L5).

Why not one big transaction?

The textbook answer to "all or nothing across systems" is a distributed transaction (two-phase commit, 2PC). Every participant locks its data and waits for a coordinator to say commit. PayCo will never hold a lock on its ledger while your warehouse checks a shelf. Neither will Stripe, and neither will a shipping carrier. Even inside one company, locks held for a long time hurt. That was the motivation for sagas in 1987: long-lived transactions "hold on to database resources for relatively long periods of time, significantly delaying the termination of shorter and more common transactions"^[1].

A saga gives up atomicity on purpose. Garcia-Molina and Salem: "either all the transactions in a saga are successfully completed or compensating transactions are run to amend a partial execution"^[1]. Chris Richardson's modern version: "A saga is a reliably coordinated sequence of local transactions"^[2].

Mental model: a saga keeps a receipt book, not an eraser. You can't delete the charge. PayCo's ledger will show ch_7 forever. A compensation adds a new line that cancels the old one by meaning. The paper is precise about this: the compensating transaction "undoes, from a semantic point of view, any of the actions performed by T_i, but does not necessarily return the database to the state that existed when the execution of T_i began"^[1]. That's why orders.status stays paid (it's still true that PayCo charged), and the saga's progress lives in its own fulfilment row: shipped, compensating or refunded.

The model breaks down in two places. First, some steps have no compensation: you can't un-ship a parcel or un-send an email. Order the saga so these come last (the pivot). Second, the in-between states are visible. For a while the customer sees a charge that will be refunded. Richardson lists this as the saga's main drawback: "Lack of isolation"^[2]. Design your UI and emails for it ("refund on its way"), because no lock is going to hide it.

Three rules, each from a row of the table

1 · Compensate only on a definite "no"

refund-on-silence broke Lesson 1's rule. A timeout tells you nothing about what happened. If Shipping's event is late or lost, ask Shipping: that's sweepStuck(), the saga's version of L6's reconcile(). If Shipping says "rejected", compensate. If it says "never heard of it", or the question itself times out, do nothing and ask again later.

2 · The compensation travels through the outbox, with its own key

On a rejection, onShippingEvent writes the compensating row and a refund.requested outbox message in one transaction, and returns. It never calls PayCo itself. Your L5 relay delivers the message to issueRefund at least once. That covers the 3 orders refund-inline forgot. Because the message can arrive twice, the refund carries a key derived from the order, refund:o7, just like the charge's charge:o7. Without the key, PayCo's own rule protects the customer from a double refund: "you can't refund a total greater than the original charge amount"^[3]. But your shop would then see a 400 for a refund that actually worked. With the key, every retry gets back the same re_7.

3 · Webhooks: verify, ask, then dedupe

A refund starts pending. The bank decides later, and PayCo tells you with a webhook. Stripe documents three facts that shape your handler:

• Forgeable. Anyone can POST to your URL, so check the signature first: "Always verify that webhook events originate from Stripe before acting on them"^[4].
• Out of order. "Stripe doesn't guarantee the delivery of events in the order that they're generated"^[4]. An old refund.created (pending) can arrive after the refund succeeded. So don't trust the payload's status. Ask PayCo for the charge now (getCharge), which Stripe suggests too: "You can also use the API to retrieve any missing objects"^[4].
• Duplicated and retried. "Webhook endpoints might occasionally receive the same event more than once. You can guard against duplicated event receipts by logging the event IDs you've processed"^[4]. That's your L5 inbox, now facing outward. Stripe retries delivery "for up to three days", so if your lookup fails, throw and record nothing. The redelivery will find you again.

Here is the reference handler, annotated:

export async function onWebhook(deps, ev) {
  if (!verifyWebhook(ev)) throw new ValidationError('not from PayCo');         1
  const charge = await remote(deps, 'get-charge', () => deps.provider.getCharge(ev.chargeId)); 2
  const done = charge.refunds.find((r) => r.id === ev.refundId && r.status === 'succeeded');
  return tx(deps.db, () => {                                                    3
    if (insertInbox(ev.id).changes === 0) return 'duplicate';
    if (!done) return 'ignored';
    const moved = deps.db.prepare(`UPDATE fulfilment SET state = 'refunded', refund_id = ?
      WHERE order_id = ? AND state = 'compensating'`).run(done.id, ev.orderId).changes === 1;
    if (!moved) return 'ignored';                                                4
    insertLedger(ev.orderId, done.amountCents, done.id);
    return 'applied';
  });
}

1. Signature first. A forged event costs you nothing, not even a lookup.
2. Ask, don't believe. The event is a hint that something changed. PayCo's current state is the fact. If this call fails, the function throws before the inbox insert, so PayCo's retry gets a second chance.
3. One transaction for "seen it" plus the effect, exactly as in L5's consumer.
4. A second guard. Two different events (refund.created arriving late, refund.updated) can both see the refund succeeded. The inbox can't tell they mean the same fact, but the conditional UPDATE can. Only the first one writes the ledger entry.

Now watch one rejected order go through all three rules. Press ▶, use the chapters, or turn on 🔇 narration.

Your turn: the rejection

Fill in the heart of onShippingEvent, the branch for shipment.rejected. Use the exact table, state and topic names from lib/db.ts.

const started = db.prepare(
  `INSERT INTO fulfilment (order_id, state)
   VALUES (?, '')
   ON CONFLICT(order_id) DO NOTHING`)
  .run(ev.orderId).changes === 1;
if (!started) return 'ignored';
db.prepare(`INSERT INTO 
  (id, topic, payload) VALUES (?, ?, ?)`)
  .run(randomUUID(),
       '',
       JSON.stringify(msg));
return 'applied';  // inside tx(): both rows commit, or neither

The lab

Implement the four functions in src/compensation.ts: onShippingEvent, issueRefund, onWebhook and sweepStuck. The contract is in the stub's comments. Every remote call uses your L2 withTimeout inside your L3 retry, and the relay is your L5 relayOnce.

cd labs/payflow
npm run test:07                 # 14 tests: duplicates, crashes, forged / late / failed webhooks, silence
npm run l7                      # Part A again, then Part B: YOUR saga in 50 chaotic worlds

Part B (lib/saga.ts) runs every world with 10% of orders rejected, 5% of each network fault, relay and handler crashes, 20% of Shipping's events lost, 30% of deliveries duplicated, webhooks shuffled and retried until you answer, PayCo's getCharge down for about 30% of passes, and one forged webhook. Then it checks ground truth for every order: shipped once or refunded once; never both; never neither. It also checks that the shop's fulfilment agrees with PayCo and Shipping, and that the ledger has one entry per refund. One example run of the reference solution (counts vary run to run):

Part B · your saga · 50 world(s), reject rate 0.1, crash rate 0.1, …

  orders   rejected   shipped   refunded   refunds at PayCo   crashes   events lost   duplicate deliveries   sweep resolved
     600         55       545         55                 55       162           143                    244              106

  webhooks delivered 210 · forged webhooks rejected 50/50 · network calls 3309

  ✓ every charge was shipped once or refunded once (never both, never neither), …

Read it as a ledger: 55 rejections, 55 refunds, each confirmed once, and 106 orders rescued by asking Shipping after their event was lost. As in Lesson 6, only a full run of src/ (no SEED, SEEDS ≥ 50, default REJECT) writes your evidence file results/07-compensation.json. Every other run writes a labelled file.

What the tests can and can't see. We tried deliberate bugs against this lab: a refund without its key, a refund called directly instead of through the outbox, compensating on silence, marking an event processed before the lookup succeeded, and skipping the signature check are each caught by chaos and by test:07. Two are caught only by test:07, because a second layer hides them: removing the webhook inbox (the conditional UPDATE still stops a second ledger entry), and trusting the payload's status instead of asking PayCo (in the lab, a correctly signed event never lies about the refund it names). As in Lesson 6, layers protect customers and per-layer tests keep each layer honest.

Practice: from memory

No scrolling up.

Put the compensation in order

Which rule was broken?

cd labs/payflow
npm test                        # every test green, including test:07
SEEDS=200 npm run l7            # ~2 min → results/07-compensation.json  (your evidence)

Read or watch next

Primary source: Applying the Saga Pattern, Caitie McCaffrey (GOTO 2015). She carries the 1987 paper from one database to many services that can't share a transaction, the situation your shop is in now.

• Applying the Saga Pattern▶ Video · 34 min · Caitie McCaffrey, GOTO 2015Watch how she adapts the 1987 paper's compensations to services that only talk over a network, then compare it with your fulfilment states.
• Pattern: Saga📄 Article · 10 min · Chris RichardsonChoreography (events trigger the next step, as here) vs orchestration (one coordinator), and the drawbacks: no isolation, no automatic rollback.
• Sagas📄 Paper · §1–2, 15 min · Garcia-Molina & Salem, 1987The original. Read the airline-seat example of a compensation that undoes "from a semantic point of view".
• Receive Stripe events in your webhook endpoint📄 Docs · 15 min · StripeThe real contract behind rule 3: signatures, ordering, duplicates, three days of retries.
• Refund and cancel payments📄 Docs · 10 min · StripeRefunds that stay pending, that fail (refund.failed), and that become reversals. Your lab skips the failed path; production can't.

💬 Questions are part of the lesson. Is it unclear why the shop needs both the inbox and the conditional UPDATE? Do you want to design the saga for your own project? Ask in the terminal. I'm your teacher for this course.