Final: a checkout that never charges twice

Five lessons, four files in src/. Now you compose them into one checkout, and prove it correct by attacking it with every fault the course has shown you.

By the end you'll be able to build a checkout flow that charges and ships each order at most once under every fault the course has shown you, and show it with a chaos test that checks the real ledger rather than your code's opinion of itself. That's the mission's last milestone.

The whole flow on one page

Every box below is a lesson you've already done. The only new code is the glue in src/checkout.ts: checkout() and reconcile().

Mental model: every step is a door that's safe to walk through twice. You can't stop failures, and you can't know where the last attempt stopped. So you make each step re-runnable: the claim is atomic, the charge carries a key PayCo dedupes on (sent with withTimeout inside retry, defaults { maxAttempts: 4, baseMs: 5, capMs: 40 }, which the tests assume. The lab bounds each call and the number of attempts, not the whole request: it doesn't thread Lesson 2's Deadline through checkout, and chaos runs without a retry budget. In production you'd pass both in.), markPaid only acts if the order isn't paid yet, and the consumer remembers messages it has applied. Then "start again from the top" is always a correct recovery, whoever does it: a client retry, a takeover after a crash, or reconciliation. The model breaks down at the edges of your system. A step that's re-runnable only within 24 hours (Stripe may prune keys "after they're at least 24 hours old"^[1]) stops being safe after that.

Two keys, two jobs

The flow uses two idempotency keys, and confusing them is the classic bug:

• The client key (client-key-o7) makes the request to the shop safe to repeat: a retry gets the saved response instead of running checkout again (Lesson 4).
• The provider key (charge:o7) makes the charge safe to repeat. It's derived from the order, not from the client key and not per attempt. Whatever path reaches PayCo (a retry inside retry(), a takeover after a crash, a second browser tab with a different client key) the same order sends the same key, so PayCo makes at most one charge.

The second key is what makes the first one's "release on error, run again" policy safe. Brandur Leach calls charges like this foreign state mutations: "We've pushed data into a system beyond our own boundaries and we shouldn't lose track of it"^[2]. You can't roll a charge back with your database transaction, so you make it repeatable instead.

Crash points: what makes each one safe

Unknown is not failed

When every attempt times out, checkout sets the order to 'unknown' and rethrows. Marking it 'failed' would often be a lie: a dropped or slow response means the card was charged (Lesson 1). Only a 400 (PayCo rejected the request as invalid, before doing anything) means "not charged", so only that sets 'failed'. A 503 also means "not charged", but only for that one attempt: once retries run out, the attempts may have mixed 503s with timeouts, so the order becomes 'unknown' and reconcile decides. A 422 from PayCo means charge:o7 was already used with other parameters, so a charge may exist: that's a bug to alert on, never 'failed'.

Both error writes carry a guard: WHERE status = 'pending'. Two requests can work on o7 at once (a second tab, or a takeover while the first worker still waits on PayCo). If one has already paid it and the other then writes 'unknown' over 'paid', reconcile later calls markPaid again, which writes a second outbox row, and o7 ships twice. So checkout's error writes only move an order out of pending. Don't confuse this with markPaid's own guard from Lesson 5, status != 'paid': that one must stay, because reconcile needs it to move unknown to paid.

Then who resolves 'unknown'? Kleppmann's answer, from Lesson 1: "the shop can wait until the connection is restored, and then query the payments service to find out the status of any transactions whose outcome was unknown"^[3]. That's reconcile(). For every pending or unknown order it asks PayCo with findByOrder, sent through deps.net with timeout and retry like the charge (it's a read, so retrying is safe):

reconcile() returns the three counts, { paid, failed, unresolved }.

The timing trap. Reconcile only once in-flight requests have settled. If a slow charge lands after reconcile said "failed", you have a charged customer and a failed order. The chaos harness waits for net.idle() first. Real systems can't wait forever, so they lean on the other half of Kleppmann's note: "a payment is something that can be undone". When reconciliation finds a charge for an order you've already given up on, you refund it, or void it (cancel before it settles).

Follow one order through every fault the harness can throw at it. Press ▶, use the chapters, or turn on 🔇 narration.

Testing: ask the ledger, not your code

A unit test asks "did checkout() return paid?". That's your code grading itself. The chaos test in lib/chaos.ts builds a fresh world per seed: 12 orders, clients that retry with the same key, 30% double clicks, some clients that walk away after an error, some orders reopened in a second session with a new client key, 8% of each network fault, and our process crashing at random crash points 10% of the time. Then it runs recovery: wait for the network, reconcile once while PayCo's lookups are down (nothing may become failed), reconcile again, and drain the outbox through a relay and a consumer that both crash. Then it checks ground truth. For every order:

This is property-based testing with faults: instead of a few hand-picked examples, you state what must always hold and let many random worlds hunt for a counterexample. Every world is seeded. A seed fixes the fault schedule, but the network uses real timers, so the interleaving, the counts and the charge ids shift a little between runs of the same seed. Use one seed to look at a world; confirm a bug or a fix across many seeds (SEEDS=50 or more). On the reference solution:

One example run. Counts, fault counts and charge ids vary run to run.

$ IMPL=solutions npm run chaos
Chaos · 50 world(s), crash rate 0.1, network faults {"drop-request":0.08,…}

  orders   paid   failed   charges   shipments   crashes   client errors   network calls
     600    594        6       594         594       376             536            1275

  faults injected: ok 839 · drop-response 109 · slow 111 · reject-503 98 · drop-request 118
  recovery: 65 clients walked away · reconcile resolved 2 → paid, 6 → failed · 8 lookups failed

  ✓ every invariant held: ≤1 charge and ≤1 shipment per order, paid ⇔ charged ⇔ shipped,
    no order left unknown, and no order marked 'failed' while PayCo's lookups were down.

Read it as a ledger: 594 paid orders, 594 charges, 594 shipments, and 6 orders that were never charged and are honestly failed. The recovery line shows reconcile doing real work. The counts move between runs; the invariants don't.

A test is only as good as the bugs it catches

A green chaos run means nothing until you've seen it go red. So break the code on purpose (this is called mutation testing). IMPL=<folder> makes the tests and demos load any folder directly inside labs/payflow (a nested folder breaks the ../lib imports):

cd labs/payflow
cp -r src mutant                # your own code (never edit src/ itself)
# edit mutant/checkout.ts: delete the provider's idempotencyKey
IMPL=mutant SEEDS=50 npm run chaos   # → results/06-chaos.mutant.json, not your evidence file
rm -rf mutant

Use at least 50 seeds (200 is better): some bugs only show when several faults line up on one order. One example run (counts and charge ids vary run to run):

  orders   paid   failed   charges   shipments   ...
     600    594        6      1001         594

  ✗ 50 world(s) broke an invariant. First ones:
    seed 1: o1-4: charged 4 times (ch_10, ch_14, ch_20, ch_23)
    seed 3: o3-12: client was told paid/ch_13, truth is paid/ch_10

  Look at one world: IMPL=mutant SEED=1 npm run chaos
  A seed fixes the fault SCHEDULE, not the timer interleaving …

Only a full run of src/ (no SEED, SEEDS ≥ 50) writes your evidence file results/06-chaos.json. Mutant and single-seed runs write results/06-chaos.<label>.json and say so, so experimenting can't overwrite your evidence.

We tried many deliberate bugs against this lab. Chaos catches the big ones: no provider key (charged 2–4×), a provider key copied from the client key (charged twice when a second session reopens the order), an unconditional markPaid and a consumer without dedupe (shipped 2×), "timeout ⇒ failed" and a reconcile that marks orders failed when the lookup fails (charged but order is 'failed'; these need several faults on one order, so they show at 50+ seeds). Others pass chaos and are caught only by a unit test. A key claim that isn't exclusive (an upsert, an INSERT that overwrites an existing row instead of failing) passes chaos because the provider key absorbs the duplicate work. That's defence in depth hiding a bug. Lesson 4's test "concurrent duplicate → ConflictError" still catches it, and the double-click and 422 tests in test:06 catch a checkout that skips the store altogether. Layers protect your customers; per-layer tests keep each layer honest, so a hole in one doesn't stay hidden behind another.

Reading a test before you write one

Your challenge includes writing one test. Here is an existing one from test/06-checkout.test.ts, annotated:

test('lost response, then retries: still ONE charge', async () => {
  const d = world(['drop-response', 'drop-response']);   1
  const res = await checkout(d, req);                     2
  assert.equal(res.status, 'paid');
  assert.equal(d.provider.charges.length, 1);             3
});

1. A scripted world. world(script) builds a fresh DB, PayCo and network with order A (500¢). The script is the fault for each network call, in order; after it runs out, every call is ok. So: attempts 1 and 2 lose their responses, attempt 3 succeeds.
2. Act like a client. req is { key: 'client-1', orderId: 'A', amountCents: 500 }. Call checkout as many times, and as concurrently, as your scenario needs.
3. Assert on ground truth. Not just the return value: PayCo's ledger (d.provider.charges) and the order row (getOrder(d.db, 'A')).

Now the same shape with blanks: a double click while the first request is still in flight. (It's in the suite as test 6; write it from memory before you look.)

test('double click while the first is slow → one charge', async () => {
  const d = world([/* fault for the first charge call: ___ */]);
  const first = checkout(d, req);                    // don't await yet
  await assert.rejects(checkout(d, req), /* which error? ___ */);
  assert.equal((await first).status, 'paid');
  assert.equal(d.provider.charges.length, /* ___ */);
});

Which mutants turn this test red? A checkout that skips the client-key store, or a claim that isn't exclusive: both let the second click through instead of a 409. Removing the "already paid → return it" check does not: the 409 fires before that check is reached. That check has its own test, "crash right after commit", which asserts that PayCo isn't called again (d.net.log stays the same length). Pick your assertions to match the bug you want to catch: ground truth alone can't see a wasted call that PayCo deduplicated.

Practice: from memory

These mix all six lessons on purpose. No scrolling up.

Put the flow in order

Which invariant catches it?

cd labs/payflow
npm test                        # all 57 tests green, including test:06
SEEDS=200 npm run chaos         # → results/06-chaos.json  (your evidence)
SEED=17 npm run chaos           # look at one world → results/06-chaos.seed17.json

Read or watch next

Primary source: Implementing Stripe-like Idempotency Keys in Postgres, Brandur Leach. It's the production version of what you just built: the same idempotency_keys table and locked_at, plus recovery points, so a takeover resumes from the last finished phase instead of starting over.

• Implementing Stripe-like Idempotency Keys in Postgres📄 Article · ~30 min · Brandur LeachRead "foreign state mutations" and "recovery points". Compare its atomic phases with your three crash points.
• Stripe API: Idempotent requests📄 Docs · 3 min · StripeA real contract to hold yours against: what's saved, what's compared, when keys are pruned, and why concurrent requests aren't saved.
• Distributed Systems lecture notes📄 Notes · re-read §2.1 Slide 25 · Martin KleppmannThe shop-and-payments problem from Lesson 1. Now you've built its solution: retry safely, then query to resolve unknowns.
• Distributed Systems lecture series▶ Playlist · 8 lectures · Martin KleppmannWhere to go next: replication, consensus and consistency models, the parts this course left out of scope.

💬 Stuck on a chaos failure you can't explain? Paste the violation lines and the seed into the terminal. We'll replay it and walk the trace together, the same way you'd debug a real incident.