Reference · Lesson 6

The whole flow

A checkout that charges and ships each order at most once, and the test that proves it. One page.

The rule

Make every step safe to run twice. Then "start again from the top" is always a correct recovery, whoever does it: a client retry, a takeover after a crash, or reconciliation.

The flow: decisions, not code

step	decision	from
1. wrap	everything runs inside the client-key store	L4
2. check	missing order / wrong amount → `ValidationError` · already paid → return it, no call · `failed` → `ValidationError` · `pending` or `unknown` → continue
3. charge	per-call timeout inside retry (defaults `{ maxAttempts: 4, baseMs: 5, capMs: 40 }`), through `net`, with a provider key derived from the order	L2 · L3 · L4
4. on error	400 → `failed` · anything else (timeout, 503, 422) → `unknown`, never `failed` (alert on 422) · these writes only leave `pending` · rethrow	L1
5. commit	`markPaid`: order + outbox row in one transaction, guard `!= 'paid'`	L5
crash points	`before-charge` · `after-charge` · `after-commit`

Reconcile

for each `pending` / `unknown` order, the lookup (through `net`, timeout + retry)…	then
finds a charge	`markPaid` with PayCo's charge id
finds none	`failed`
itself fails	leave it; count unresolved; next pass

Run it only after in-flight requests settle. Returns { paid, failed, unresolved }. Then relayOnce → ShippingConsumer.handle: at-least-once + inbox = exactly-once effect.

Two keys

key	example	protects
client key	`client-key-o7`	the request to the shop: retries get the saved response
provider key	`charge:o7`	the charge: any path to PayCo for this order makes ≤ 1 charge

Crash points

crash at	world after	retry is safe because
`before-charge`	nothing charged	it charges for the first time
`after-charge`	charged, order pending	retry re-runs with the same provider key → same charge. Real death: key stays in_flight until stale → takeover. Lab's `SimulatedCrash` unwinds through `store.run`, releasing the claim, so the retry just re-claims.
`after-commit`	paid + outbox row, response unsaved	order already paid → returned; `markPaid` is conditional

Invariants (`lib/chaos.ts`, checked per order against ground truth)

invariant	protected by
≤ 1 charge (PayCo ledger)	provider key
≤ 1 shipment	conditional `markPaid` + consumer inbox
no order left pending / unknown	`reconcile()`
paid ⇔ charged; `charge_id` matches PayCo	unknown ≠ failed + reconcile
paid ⇔ shipped	outbox in the same transaction
every "paid" told to a client is true	saved response under the client key
nothing marked `failed` while lookups were down	a failed lookup leaves the order unresolved

Chaos worlds also include clients that walk away, second sessions with a new client key, a reconcile pass during a lookup outage, and crashes inside the relay and the consumer.

Commands

npm run test:06                 # 12 tests, the last is a 25-seed chaos run
SEEDS=200 npm run chaos         # → results/06-chaos.json (the evidence: src/, ≥ 50 seeds)
SEED=17 npm run chaos           # look at one world → results/06-chaos.seed17.json
IMPL=solutions npm run chaos    # what passing looks like → 06-chaos.solutions.json
CRASH=0.3 npm run chaos         # more crashes → 06-chaos.crash0.3.json

A seed fixes the fault schedule, not the timer interleaving: the same seed can break on a different order, or pass. Confirm bugs and fixes with SEEDS=50+.

Mutation check: cp -r src mutant (directly inside labs/payflow; nested folders break the ../lib imports), break one thing, IMPL=mutant SEEDS=50 npm run chaos (200 is better), then rm -rf mutant. Mutant runs write 06-chaos.mutant.json, never your evidence file.

bug	caught by
no provider key · provider key = client key	chaos: charged 2–4×
unconditional `markPaid` · consumer without dedupe	chaos: shipped 2×
timeout ⇒ failed · reconcile marks failed when the lookup fails	chaos (50+ seeds) + test:06: charged but order is 'failed'
checkout skips the client-key store	test:06 double click (409) and 422
"already paid → return it" removed	test:06 after-commit (asserts no new PayCo call)
non-exclusive key claim (upsert)	test:04 only; chaos passes (masked by the provider key)
SELECT-then-INSERT claim · takeover without compare-and-set	nothing in this lab (one synchronous process can't race); real in production

Still not handled

A charge landing after reconcile said "failed" → refund it, or void it: cancel before it settles (payments can be undone).
Keys pruned (Stripe: after ≥ 24 h) → a very late retry is a new request. Stripe: retry with the same key only in the first 24 h, then reconcile.
Real lookups lag: Stripe Search is not for read-after-write ("searchable in under 1 minute"). Reconcile by a stored provider id, or from webhooks.
A declined card: one key per order replays the decline (Stripe caches errors too). Key each payment attempt instead.
5xx from a real provider may be indeterminate (Stripe), not "not done".
Orders stuck unknown because PayCo is down for long → alert, keep reconciling.

Sources

Brandur Leach: Implementing Stripe-like Idempotency Keys in Postgres📄 Foreign state mutations, locked_at, recovery points
Stripe API: Idempotent requests📄 Saved responses, parameter checks, 24 h pruning
Stripe: Advanced error handling📄 5xx indeterminate · 24 h retry window, then reconcile · webhooks
Stripe: Search📄 Not for read-after-write flows
Kleppmann, Distributed Systems notes§2.1 Slide 25: query the payment service to resolve unknown outcomes; payments can be undone