Retries, backoff and jitter

What to retry, how long to wait, and how many retries to allow. One page, for quick lookup.

The rule

Retry only what an identical later request could fix, only if the operation is idempotent (a timeout may already have done the work), at one layer, with capped exponential backoff + full jitter, under a retry budget, within the deadline.

What to retry

Error classes from labs/payflow/lib/errors.ts.

General HTTP: "request is wrong" (400, 401, 403, 404, 422) → don't retry. "Try later" (408, 429 with Retry-After, 503, and 409 only when it means "same idempotency key still in flight") → retry after waiting. A general 409 Conflict (the resource's state conflicts with the request) usually fails again. 500/502/504 and timeouts: unknown outcome → retry only if the call is idempotent. Check each API's contract: Stripe treats every 5xx as indeterminate, and its Stripe-Should-Retry header can say "retry" or "don't" outright.

The formula (full jitter)

// attempt = 0 before the first retry
ceiling = min(capMs, baseMs × 2^attempt)
wait    = random() × ceiling          // uniform in [0, ceiling]

Ceilings with base 10 ms, cap 2000 ms, for attempt 0, 1, 2…: 10 → 20 → 40 → 80 → 160 → 320 → 640 → 1280 → 2000 → 2000 ms.

The retry loop: stop checks, in order

Any "stop" → rethrow the last error object itself, not wrapped, so the caller can still tell a timeout from a 503. Every success, including a first try, refills the budget.

Retry budget (token bucket)

A retry needs a whole token (≥ 1). Float drift: 0.1 added ten times is 0.9999999999999999, so round when adding (or count whole tenths).

Steady state ≈ 1 retry per 10 successes. Google SRE example of a server-wide budget: 60 retries per minute per process.

Amplification

k attempts per layer, n layers → kⁿ at the bottom. Lesson: 3 layers × 3 attempts → 27. Brooker: five layers, "the load on the database will increase 243x" (he says "three retries" per layer but counts 3 tries each: 3⁵ = 243). Google SRE: three layers that "all issue 3 retries (4 attempts)" → "64 attempts (4^3)". Fix: retry at a single point in the stack.

Lab numbers (npm run l3, simulated, seed 1)

500 clients, capacity 50 per 10 ms tick, each 503 costs 0.1 of a slot, max 10 attempts.

Immediate and no-jitter exponential tie exactly because the simulation has no natural spread: identical waves meet an identical server. It's a worst case. The collapse also depends on rejections costing work: with REJECT_COST=0 or CAPACITY=100, immediate retry finishes faster than jitter (jitter still makes fewer calls). Among jittered variants, Brooker calls equal jitter "the loser"; full vs decorrelated "is less clear".

Vocabulary

Sources

• Brooker, Exponential Backoff And Jitter📄 "The no-jitter exponential backoff approach is the clear loser."
• Brooker, Timeouts, retries, and backoff with jitter📄 Selfish retries · 243× · token bucket · retry at a single point
• Google SRE, Addressing Cascading Failures📄 Randomized exponential backoff · per-request limits · retry budget · 64×