Reference · Lesson 4

Idempotency keys

Making a non-idempotent operation safe to retry. One page, for quick lookup.

The rule

The key names the operation, not the attempt. Generate it once when the user starts the action, persist it, and send it on every retry. The server claims it atomically and replays the first answer.

Idempotent or not

idempotent: f(x) = f(f(x))	not idempotent
add to a set · set a field to a value · `PUT` · `DELETE` · `GET`	increment a counter · append · charge · send an email · `POST`

RFC 9110 §9.2.2: a client "SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent". The key is that means.

The table (`lib/db.ts`)

CREATE TABLE idempotency_keys (
  key          TEXT PRIMARY KEY,   -- the atomic claim
  fingerprint  TEXT NOT NULL,      -- fingerprint(params), as is or hashed
  status       TEXT NOT NULL,      -- 'in_flight' | 'done'
  response     TEXT,               -- first answer, replayed
  locked_at    INTEGER NOT NULL    -- old lock = worker probably died; also YOUR fencing token
);

The decision, in order

state when the request arrives	`run()` does	status
no row → `INSERT … ON CONFLICT(key) DO NOTHING` wins (`.changes === 1`)	run work · ok → save `done` + response (fenced) · error → delete our claim (fenced), rethrow	result
row exists, fingerprint differs	reject; never run, never replay	422 `MismatchError`
`done`, same params	replay saved response; work doesn't run	original result
`in_flight`, lock younger than `lockMs`	reject; client retries later; nothing saved	409 `ConflictError`
`in_flight`, lock stale	take over with compare-and-set on `locked_at` (one winner), run work	result

Fencing: writes after the claim

write	only if	0 rows changed means
save the response	`status = 'in_flight' AND locked_at = <mine>`	someone took over: throw `ConflictError`
release on error	`status = 'in_flight' AND locked_at = <mine>`	the claim isn't mine any more: leave it

lockMs must outlast any honest run of the work (≥ the request deadline + the downstream timeout), or you take over live workers by design. Kleppmann: "include a fencing token with every write request".

Classic bugs

bug	effect	fix
new key per attempt	each retry is a new operation (lab: 24/100 double-charged, same as no key)	one key per operation, persisted before the first send
SELECT, then INSERT	two concurrent requests both see "none", both act (can't happen in the lab's single synchronous process, so no test catches it; it does in production)	claim with a single atomic INSERT on a unique key
unfenced save or delete	a slow worker overwrites the new owner's answer, or deletes its claim	condition every write on your `locked_at`
key only at the top hop	crash after the charge, before saving → re-run charges again	pass your own key downstream (`charge:<orderId>`)
replay on different params	hides a client bug, returns the wrong answer	compare fingerprints, 422
retention shorter than retry window	pruned key → retry becomes a new charge	keep keys longer than any client retries (Stripe v1: prunable after ≥24 h)

Lab numbers (`npm run l4`, seed 1)

  version       charges   double-charged orders
  no-key            132                      24
  key-per-try       132                      24
  same-key          100                       0

The two error codes

error	means	client should
`ConflictError` · 409	key still in flight	retry later, same key (retryable)
`MismatchError` · 422	key reused with different params	never retry: a client bug

Design note: save errors, or release the claim?

Stripe's API v1 saves the status code and body of the first request "including 500 errors", so a retry gets the same error back. This lab's store does the opposite: if the work throws, it deletes its claim so a retry can run the work again. That is safe only because the work is idempotent downstream (the PayCo key charge:<orderId>). If re-running the work could double-charge, releasing the claim would bring the Lesson 1 bug back. Pick one deliberately.

Limits in a real system

Stripe allows keys "up to 255 characters", suggests random IDs (V4 UUIDs, e.g. crypto.randomUUID()), and may prune keys once they're "at least 24 hours old" (API v1). After pruning, a reused key is a new request: retry window < retention window. Stripe: "Limit retries of an unresolved operation to the first 24 hours", then reconcile. AWS: keep tokens for "the lifetime of the resource, plus an interval".
Client rules (Stripe): on a 409, wait and retry with the same key and params, since "A new key can start a second operation". To change params after an unknown outcome, "first establish that the original operation didn't succeed and isn't still executing", then use a new key.
Scope keys per account or tenant, so two customers' keys can't collide.
An IETF draft tried to standardise an Idempotency-Key HTTP header. It expired (draft-07) without becoming an RFC: treat it as a description of common practice, not a standard.

Vocabulary

term	meaning
idempotent	Doing it twice has the same effect as once.
idempotency key	Client-chosen unique ID for one logical operation, sent on every retry.
fingerprint	A string computed from the request params (stable JSON, or a hash of it), stored with the key to catch reuse with different data.
fencing token	A value proving you still hold a lock; every later write checks it.
atomic	Happens completely or not at all; no other request can see it half-done.
compare-and-set	`UPDATE … WHERE col = <old value>`: succeeds only if nobody changed the row since you read it.
claim	Atomically recording "I'm working on this key"; only one request wins.
stale lock	An `in_flight` claim older than `lockMs`: its worker probably died.
foreign state mutation	A change in another system (a charge) that your database can't roll back.

Sources

AWS: Making retries safe with idempotent APIsclient request tokens · semantically equivalent responses · parameter mismatch · token + change recorded atomically (same-database case) · retention = resource lifetime + interval
Kleppmann: How to do distributed lockingfencing tokens: why a lock that "probably" expired isn't enough
Stripe: Advanced error handling409 → same key · changing params after an unknown outcome · 24 h retry window
Stripe: Idempotent requestsrandom (V4 UUID) keys · ≤255 chars · saves status + body incl. 500s · prune ≥24 h (v1)
Brandur Leach: Stripe-like idempotency keys in Postgreslocked_at · 409 · recovery points · foreign state mutations
IETF draft: The Idempotency-Key HTTP Header Fieldexpired draft-07 · common practice, not a standard
RFC 9110 §9.2.2 Idempotent Methodswhich HTTP methods are idempotent, and when clients may retry
Kleppmann, Distributed Systems notesSlide 90: f(x) = f(f(x)), counter vs set, exactly-once = retry + idempotence or dedup