Partial failure: the third outcome
Inside one program, a function call either returns or throws. Across a network there is a third outcome, and every technique in this course exists because of it.
By the end you'll be able to look at any failed remote call and say what the caller can know: it was done, it wasn't done, or it's unknown. You'll also see, with real numbers from the lab, why the obvious fix ("no answer? send it again") charges customers twice.
The problem: you can't see the other machine
Our running example is a shop. When a customer checks out, the shop calls PayCo, a payment provider like Stripe, to charge their card. In code it looks like a function call: await payco.charge(order). But it isn't one. It's two messages between two computers that share nothing except a network. The request travels to PayCo, PayCo does the work, and a response travels back. Either message can be lost or delayed, and either computer can crash, independently of the other.
Martin Kleppmann's Cambridge course puts the consequence plainly: "there is no way for one node to have certainty about the state of another node. The only way how a node can know something is by having that knowledge communicated in a message"[1]. If the message doesn't come, the shop knows nothing.
Mental model: a remote call is a letter, not a function call. You post a letter asking your bank to transfer money. If no reply comes, you can't tell whether your letter was lost, the bank did the transfer and its reply was lost, or the reply is just slow. Silence carries no information. The model breaks down in one way: real networks also duplicate and reorder messages, and the server can crash halfway through the work. Stripe's engineers list exactly these three ways a call can fail: the connection fails before reaching the server, the call fails "midway while the server is fulfilling the operation", or "the call could succeed, but the connection break before the server can tell its client about it"[2].
Watch it happen. Press ▶, use the chapters, or turn on 🔇 narration. The terminal shows the real output of npm run l1.
Three outcomes, not two
After a remote call, the caller is in exactly one of three states. What decides it is what came back, not what the caller hoped for:
| What came back | The caller knows | Example |
|---|---|---|
| A success response | Done. | 200 · charge ch_41 |
| A refusal the server promises it made before doing any work | Not done. | 503 from an overloaded PayCo · 400 "amount must be positive" |
| Nothing (yet), or the connection broke after the request was sent | Unknown. | a timeout · a connection reset |
Two details matter. First, "not done" depends on a contract: a 503 only means "not charged" if PayCo promises to refuse before doing work. The lab's PayCo makes that promise; check the docs before you assume it of a real API. Stripe, the model for PayCo, does not: "Treat the outcome of a request that returns 500 as indeterminate: the operation might have produced side effects even though you received an error"[3]. For Stripe, a 5xx is unknown. Second, a reply that arrives late is still a reply: if it says "charged", the charge happened, even though you already gave up waiting. (The lab's network quietly discards replies that arrive after you've timed out, so you'll only see their effect in the ledger.)
Real HTTP clients give you one useful refinement. A connect timeout means the connection was never set up, so the request was never sent: not done. A timeout after the request was sent, while waiting for the reply, is unknown.
The two generals, and why a shop isn't doomed
The classic version of this problem is the two generals problem: two armies must attack together, and can only communicate by messengers who might be captured. Kleppmann shows that no protocol, however many confirmations it sends, gives both generals certainty[1]. His notes then apply it to an online shop and a payment service, where we want "online shop dispatches if and only if payment made".
Then comes the hopeful part. The shop's version is solvable, because "a payment is something that can be undone (unlike an army being defeated)". And: "If the communication between shop and payment service is interrupted, the shop can wait until the connection is restored, and then query the payments service to find out the status of any transactions whose outcome was unknown"[1]. Keep that sentence in mind. Lesson 6 builds exactly that query, called reconciliation.
What the lab shows
The lab, labs/payflow/, simulates the network inside one Node process, so you can make each fault happen on purpose. lib/net.ts has five faults. Read its header comment: it's the table above, in code. Part A of npm run l1 sends one charge per fault, with a 30 ms timeout:
fault caller saw charges at PayCo caller can know
ok ok ch_1 1 charged
drop-request timeout 0 ??? unknown
drop-response timeout 1 ??? unknown
slow timeout 1 ??? unknown
reject-503 error 503 0 not chargedPart B runs 200 checkouts with each fault at 5% and the naive policy: "on a timeout or a 503, send the charge again, up to 3 tries". The result, from the PayCo ledger:
charged once, shop says paid 178
not charged, shop says failed 0
charged 2+ times ← double charge 22
charged, but shop says failed 022 of 200 customers paid twice (seed 1). Watch the last row on other seeds too: with SEED=42 one customer was charged while the shop told them the payment failed. That's the other half of the harm. The resend was the right move for a lost request and for a 503, but the caller couldn't tell those apart from a lost or slow response. Not retrying isn't safe either: then every lost request becomes a lost sale, and an order may sit there charged while the shop says "payment failed". Kleppmann names the two simple policies: at-most-once ("send request, don't retry, update may not happen") and at-least-once ("retry request until acknowledged, may repeat update")[1]. Neither is what a shop wants. The rest of the course gets you to the third: retries that are safe.
labs/payflow/src/, and the final lesson composes them.Practice: from memory
No scrolling up.
Your charge call to PayCo times out after 30 ms. Was the customer charged?
PayCo replies 503, and its docs promise it refuses before doing any work. The charge…
Why can't the caller tell a lost request from a lost response?
With the naive "no answer? send it again" policy, which fault turns into a double charge?
The two generals can never be certain. Why is the shop-and-payments version still solvable?
Sort the observations
Which leg broke?
🏆 Milestone challenge: Partial failure
Two parts. The first is in the lab, the second is about a system you actually use.
cd labs/payflow
npm run l1 # seed 1
cp results/01-partial-failure.json results/01-seed1.json # the next run overwrites it
SEED=42 npm run l1 # pick any other seed too- Read your results. In
results/01-partial-failure.json, find Part B's double charges andpartB.faultsBehindDoubles(the faults hit by the calls of double-charged orders; an order may also have hit a harmless fault on another attempt). In 2–3 sentences: which fault types caused them, and why would not retrying be wrong too? If your second seed has a "charged, but shop says failed" order, explain it as well. - Transfer it. Pick one remote call from something you've built or use: sending an email through an API, creating a GitHub issue, an LLM tool call that writes to a database, a webhook. Write a 4-row table: where it breaks (request lost · server crashes midway · response lost · server refuses; the lab has no "crash midway" fault, but its
drop-responselooks the same to the caller) → did the side effect happen? → what the caller sees → done / not done / unknown. Then say what goes wrong for this call if you retry blindly, and if you never retry. - Look ahead. For your unknown rows, name one piece of information the caller could get afterwards to resolve the outcome. Does the real API offer a way to get it?
Send me back: both result files (01-seed1.json and your seed's 01-partial-failure.json), your answer to 1, and the table plus answer for 2 and 3.
Pass criteria (check yourself before sending):
- The result files come from real runs (two different seeds), and your explanation names
drop-responseandslowas the causes of the double charges (checkfaultsBehindDoubles), with the reason. - Your table classifies all four rows correctly, and at least one row is unknown with a reason that isn't "the network is down".
- You say what goes wrong with each simple policy for your call: retrying blindly (a duplicate side effect) and never retrying (lost or orphaned work).
- For the unknown rows, you name a concrete way to find out afterwards (for example, a read endpoint, listing by your own ID, a webhook), or state that the API has none.
When I've reviewed it, I'll write a learning record and unlock Partial failure.
Read or watch next
Primary source: Distributed Systems 2.1: The two generals problem, Martin Kleppmann. It's about 12 minutes, and ends with the shop-and-payment version of the problem this lesson used.
- Distributed Systems 2.1: The two generals problemWhy no number of confirmations gives certainty, and why a shop can still get it right. Watch for the table of four outcomes.
- Distributed Systems lecture notesThe written version of the video, plus Slide 90 on at-most-once, at-least-once and the exactly-once effect. The quotes in this lesson come from here.
- Designing robust and predictable APIs with idempotencyRead the opening section on the three ways a call can fail. The rest previews Lessons 3 and 4.
- Distributed Systems lecture seriesThe whole course, if you want the theory behind everything here. Lectures 1–2 match this lesson.
💬 Not sure whether a row in your table is "unknown" or "not done"? Ask me in the terminal. Paste the API's docs or error list and we'll work through it together.