Reference · Lesson 2
Timeouts and deadlines
How long to wait, and how to share one time budget across hops.
The rules
- Every remote call gets a timeout. A lost message is silence, and many clients (gRPC included) set no deadline by default.
- A timeout ends the waiting, not the work. The outcome after a timeout is unknown.
- Choose the error rate; the data chooses the number. Acceptable false-timeout rate → matching percentile (0.1% → p99.9).
- Mind where percentiles fail (Brooker): clients over the internet need room for worst-case network latency; when p99.9 ≈ p50, add padding.
- One deadline per operation, set at the top. Each hop waits
min(its cap, time left)and passes time left down, not a clock time, minus a margin for the reply's trip back (SRE: "a few hundred milliseconds"). - No time left → don't send. Fail fast; nobody is waiting for the answer.
Picking a timeout
| false-timeout budget | percentile | lab PayCo (synthetic, seed 1) |
|---|---|---|
| 50% | p50 | 40 ms (never do this) |
| 10% | p90 | 64 ms |
| 1% | p99 | 206 ms (smallest within 1%: 178 ms; 150 ms is 1.004%, just over) |
| 0.1% | p99.9 | 1816 ms |
| too short | too long |
|---|---|
| false timeouts → unknown outcomes, errors, retries, extra load on a slow server | stuck users, threads and connections held while something is truly broken |
Contracts
| function | must |
|---|---|
withTimeout(ms, work) | reject with TimeoutError after ms even if work ignores its signal (race them) · abort the AbortSignal handed to work · return the work's result unchanged · clear the timer on every path |
Deadline.remaining() | ms left until the deadline, never negative |
withDeadline(d, cap, work) | wait min(cap, remaining) · nothing left → TimeoutError without calling work |
| tool | what it gives you |
|---|---|
AbortController | .signal to hand out; .abort(reason) flips signal.aborted and fires 'abort' |
signal.reason | whatever you passed to abort() (here a TimeoutError) |
try … finally | the place to clear the timer |
Across machines: send the remaining duration, not an absolute time (clock skew). Servers should check whether the caller already gave up before doing expensive work.
Lab
npm run test:02 | 8 tests for src/timeout.ts |
TIMEOUT_MS=250 npm run l2 | Part A: percentiles, false-timeout rate (3 decimals) and a within/OVER verdict for 1% and 0.1% budgets. Part B: your withTimeout against lost messages on the lab network, always at a fixed 30 ms. → results/02-timeouts.json |