Timeouts and deadlines

Lesson 1 showed that a lost message is silence. A timeout is how you decide to stop listening to it. Choosing when is a trade-off you can measure.

By the end you'll be able to choose a timeout from a latency distribution instead of guessing, and bound a whole multi-hop operation with one deadline. In the lab you'll write labs/payflow/src/timeout.ts, which every later lesson builds on.

Why every remote call needs one

Look at drop-request in lib/net.ts: the promise it returns never settles. No error, no reply, nothing. Real networks behave the same way, and many libraries don't protect you by default. gRPC (a popular framework for one service to call another; an RPC, remote procedure call, is the remote call from Lesson 1) says it plainly in its docs: "By default, gRPC does not set a deadline", so a client can wait without limit, and "you should always explicitly set a realistic deadline in your clients"^[1]. A call with no timeout doesn't fail. It just holds a connection, some memory and a customer forever.

Mental model: a timeout is a bet, not a measurement. When the timer fires you haven't learned anything new about PayCo. You've only decided you've waited long enough. Kleppmann's notes describe the typical failure detector as "label node as crashed if no reply within some timeout", and its problem: it "cannot tell the difference between crashed node, temporarily unresponsive node, lost message, and delayed message"^[2]. So the outcome after a timeout is still unknown, exactly as in Lesson 1. Where the model breaks down: a bet usually has a winner revealed later. A timed-out call may never reveal anything unless you go and ask (Lesson 6).

One more rule from the lab, and it surprises most people: giving up doesn't cancel the work. When your timer fires, the request may already be on the wire or inside PayCo. PayCo doesn't know you stopped listening, so it charges the card anyway. Once you've built withTimeout (below), npm run l2 Part B shows it. Here is the reference run:

  ok             ok ch_1 after a few ms · PayCo charges: 1
  drop-request   TimeoutError after 30 ms · PayCo charges: 0
  drop-response  TimeoutError after 30 ms · PayCo charges: 1

Watch the whole story. Press ▶, use the chapters, or turn on 🔇 narration.

How long to wait: read it off the distribution

Latency isn't one number. Most calls are fast and a few are very slow (a garbage-collection pause, a cold cache, a retransmitted packet). That long tail is why you describe latency with percentiles, and why the timeout is a trade between two kinds of harm:

• Too short: calls that would have succeeded time out. Each one is an unknown outcome, an error for the user, and usually a retry: extra load on a server that was only a bit slow.
• Too long: when something really is broken, every caller sits waiting, holding threads and connections, and the user stares at a spinner.

Marc Brooker describes how Amazon resolves this. "We choose an acceptable rate of false timeouts (such as 0.1%)", then "we look at the corresponding latency percentile on the downstream service (p99.9 in this example)"^[3]. In other words, you choose the error rate, and the data chooses the number. Here is the lab's PayCo, from npm run l2. The distribution is synthetic but seeded: most calls take about 40 ms, and 1% are slow outliers (0.3–2 s).

  percentile   ms
  p50            40
  p90            64
  p99           206
  p99.9        1816
  max          1997

  TIMEOUT_MS=60   → 13.356% false timeouts   OVER a 1% budget
  TIMEOUT_MS=100  →  1.416%                  OVER a 1% budget
  TIMEOUT_MS=150  →  1.004%                  OVER a 1% budget (by a hair)
  TIMEOUT_MS=178  →  1.000%                  within a 1% budget
  TIMEOUT_MS=1900 →  0.048%                  within a 0.1% budget
  TIMEOUT_MS=2500 →  0.000%

Notice the cliff. The smallest timeout within a 1% budget is 178 ms; within 0.1% it's about 1.8 s (p99.9 is 1816 ms), because of the 1% of slow outliers. Read the third decimal: 150 ms looks like "1%" but is 1.004%, just over budget. The demo prints a within/OVER verdict for you. The percentile tells you what a budget costs. Whether that's worth it is a product decision: a 1.8 s wait may be fine for a payment and unacceptable for autocomplete.

Where this method breaks. Brooker names its limits in the same article. It "doesn't work in cases where clients have substantial network latency, such as over the internet"; there you add a reasonable worst-case network latency on top. And it "doesn't work with services that have tight latency bounds, where p99.9 is close to p50"; there "adding some padding helps"^[3]. Our phone → shop hop crosses the internet, so its timeout needs that extra room; the shop → PayCo percentiles above are for a server-to-server call.

Try it: find the timeout

Your false-timeout budget is 1%. Drag the line to the smallest timeout that stays within it, then lock it in. Bars in red are calls that would time out.

Deadlines: one budget for the whole trip

A checkout isn't one call. The phone calls the shop, the shop calls PayCo, PayCo calls the card network. If every hop picks its own timeout, the times add up: the phone may give up after 3 s while the shop is still patiently waiting 5 s for PayCo, doing work for a customer who has already left. Google's SRE book says: "Rather than inventing a deadline when sending RPCs to backends, servers should employ deadline propagation"^[4]. The deadline is set once, at the top, and each hop passes the remaining time down. The same chapter adds a refinement: "You may want to reduce the outgoing deadline a bit (e.g., a few hundred milliseconds) to account for network transit times and post-processing in the client"^[4]. The reply still has to travel back, so don't hand the next hop every last millisecond.

Two practical details. Across machines, pass the time left ("2700 ms"), not a timestamp ("finish by 12:00:03.4"): clocks on different machines disagree. gRPC does exactly this: it converts the deadline into a timeout on the wire, which avoids clock-skew problems^[1]. And the receiving side should check whether its caller has already given up before doing expensive work. gRPC's advice is to "periodically check if the RPC that initiated it has been cancelled"^[1].

Try it once by hand. The phone sets a 3000 ms deadline. The shop spends 300 ms before it calls PayCo, and its per-call cap is 2000 ms.

Build it: src/timeout.ts

JavaScript's tool for "stop waiting" is the AbortController. Think of it as a remote control. You keep the controller and hand its signal to the work. Calling controller.abort(reason) flips signal.aborted to true and fires an 'abort' event that the work can listen to. fetch accepts a signal, and so does the lab's net.call(op, handler, { signal }). In both, aborting stops your waiting; it can't undo or reliably stop work the server already received.

Your withTimeout(ms, work) must do four things, and the tests in test/02-timeout.test.ts check each one:

1. reject with TimeoutError once ms have passed, even if work ignores the signal (so race the work against the timer, don't just trust it to stop);
2. abort the signal it gave to work, so work that listens can stop early;
3. return the result unchanged when the work wins;
4. always clear the timer, on success and on failure. A forgotten timer fires later and keeps the process alive.

Fill in the blanks from the four rules above. This is the shape, not every line:

async function withTimeout<T>(ms: number,
    work: (signal: AbortSignal) => Promise<T>) {
  const ctl = new ();
  const timer = setTimeout(
    () => ctl.(new TimeoutError()), ms);
  try {
    return await new Promise<T>((resolve, reject) => {
      // the race (rule 1): whichever settles first wins
      ctl.signal.addEventListener('abort',
        () => reject(ctl.signal.reason), { once: true });
      work(ctl.).then(resolve, reject);
    });
  }  {
    (timer);  // rule 4
  }
}

Then the deadline. Deadline.remaining() returns the milliseconds left, never negative. withDeadline(deadline, perCallMs, work) waits min(perCallMs, deadline.remaining()), and if that's zero it throws TimeoutError without calling work. The stub's comments spell out the contract. Run npm run test:02 until all 8 tests are green, then npm run l2.

Two different simulations live in npm run l2. Part A is the synthetic PayCo distribution above (median 40 ms). Part B runs your withTimeout on the lab network used by every other lesson, where each leg takes 1–4 ms and slow adds 60 ms. Part B always uses a fixed 30 ms and ignores TIMEOUT_MS. That fast network is why Lesson 6 can use 20–25 ms timeouts without breaking this lesson's rule.

Practice: from memory

No scrolling up.

cd labs/payflow
npm run test:02                      # all green with YOUR src/timeout.ts
TIMEOUT_MS=60  npm run l2            # try at least three values;
TIMEOUT_MS=250 npm run l2            # read the within/OVER verdict line
TIMEOUT_MS=2500 npm run l2
TIMEOUT_MS=<your choice> npm run l2  # last run = the results file you send

Read or watch next

Primary source: Timeouts, retries, and backoff with jitter, Marc Brooker, Amazon Builders' Library. Read the "Timeouts" section now; the rest is Lesson 3.

• Timeouts, retries, and backoff with jitter📄 Article · ~15 min · Marc Brooker, Amazon Builders' LibraryHow Amazon picks timeouts from a false-timeout rate and a percentile, and what goes wrong when timeouts are set too low.
• gRPC: Deadlines📄 Docs · ~8 min · gRPCWhy there's no default deadline, how deadlines propagate between services, and why servers should check for cancellation.
• Addressing Cascading Failures📄 Book chapter · long; read "Latency and Deadlines" · Google SRE bookWhat happens to a fleet when deadlines are missing or too long, and deadline propagation in practice.
• Distributed Systems lecture series▶ Playlist · Martin Kleppmann · lecture 2 covers system models and failure detectorsThe theory behind "a timeout can't tell crashed from slow". No dedicated high-quality video on picking timeouts exists, so this is the best watchable source.

💬 Stuck on a failing test, or unsure your deadline budget adds up? Ask me in the terminal. Paste the test output or your numbers and we'll go through them.