Consistency without a shared transaction

Lesson 4 made one call safe to repeat. But a checkout doesn't end at PayCo: the shop must also record the payment and tell Shipping. Two systems, two writes, and a crash can land between them.

By the end you'll be able to replace a "write here, then tell them" dual write with a transactional outbox and a consumer inbox, and say exactly what consistency the customer sees. The code goes in labs/payflow/src/outbox.ts.

The problem: two writes, nothing holding them together

PayCo said yes. Now the shop must do two things: set order o7 to paid in its database, and send an order.paid message to Shipping, a separate service with its own database. Both must happen, or the customer gets a mess.

Inside one database, this is a solved problem. A transaction wraps several writes: you say BEGIN, make the writes, then COMMIT. The database guarantees the writes become durable together, or, if anything fails before COMMIT (a crash, an error, a ROLLBACK), none of them happened. That all-or-nothing property is called atomicity. In the lab, lib/db.ts gives you tx(db, fn), which does exactly this around a synchronous function.

But a transaction lives inside one database. Shipping's database is somewhere else. The textbook way to commit across two systems is two-phase commit (2PC): a coordinator asks both to "prepare", then tells both to "commit". The microservices.io pattern page is blunt about it for this case: "2PC is not an option"^[1]. The page's reason: "The database and/or the message broker might not support 2PC" (a broker is a service that stores messages and forwards them to receivers, like a queue). Third-party APIs like Shipping's certainly don't. So the shop does two separate writes, and Lesson 1's lesson applies: the process can die between any two steps.

The lab's npm run l5 tries both orders on 200 paid orders, with the shop crashing 10% of the time between the two writes (seed 1):

  version           consistent   paid, not shipped   shipped, not paid   shipped 2+   messages delivered
  db-then-publish          179                  21                   0            0                  179
  publish-then-db          179                   0                  21            0                  200

Write the database first and a crash leaves orders paid but never shipped: only 179 messages ever left, and the customer paid and gets nothing. Publish first and a crash leaves orders shipped but never paid in our records. Swapping the order only swaps the damage. No ordering of two independent writes fixes it.

Watch both fail, then watch the fix. Press ▶, use the chapters, or turn on 🔇 narration.

The fix: make it one write, then deliver it until it lands

Mental model: the outbox is a to-do note written on the same page as the change. Instead of doing the second write now, the shop writes "tell Shipping that o7 is paid" as a row in its own database, inside the same transaction as the order update. Either both the change and the note exist, or neither does. A separate relay later reads unsent notes and delivers them. microservices.io describes it as: "the service that sends the message to first store the message in the database" as part of the business transaction, with a message relay forwarding stored messages to the message broker (here, straight to Shipping)^[1].

Where the model breaks: a to-do note can be acted on twice. The relay sends a message and then marks it sent. If it crashes between those two steps, it sends the message again on restart. The pattern page says so directly: "The Message relay might publish a message more than once", and therefore "a message consumer must be idempotent"^[1]. The note fixes losing messages. The receiver has to fix repeating them.

The shape, annotated

This is what markPaid and relayOnce do. It's the transaction shape, not the TypeScript; writing that is your exercise.

-- markPaid(orderId, chargeId), inside tx(db, …)
BEGIN
UPDATE orders SET status = 'paid', charge_id = ?
 WHERE id = ? AND status != 'paid'                        1
-- only if that UPDATE changed a row:
INSERT INTO outbox (id, topic, payload)
VALUES (<new uuid>, 'order.paid', '{"orderId":…,"chargeId":…}')   2
COMMIT                                                    3

-- relayOnce(db, publish)
SELECT seq, id, topic, payload FROM outbox
 WHERE published_at IS NULL ORDER BY seq                  4
for each row:  await publish(msg)
               UPDATE outbox SET published_at = now WHERE seq = ?   5

1. Guarded update. AND status != 'paid' makes the step safe to run twice. Lesson 6 will re-run it after crashes and retries, and a second run must not create a second message. It returns false when nothing changed. The guard is deliberately not = 'pending': Lesson 6's reconciliation must be able to move an unknown order to paid.
2. The message gets its own id. That id, not the order id, is what the consumer uses to recognise a repeat delivery.
3. One commit point. A crash before it: neither row exists, and the order is still pending, so a retry can redo the whole step. A crash after it: both exist, and the relay will send the message eventually.
4. Order by seq. The pattern requires that "messages must be sent to the message broker in the order they were sent by the service"^[1]. The relay keeps that promise by publishing in seq order and stopping at the first failure, so it never skips ahead. Two caveats for production. Stopping at the first failure is head-of-line blocking: one message that always fails stops every message behind it, so real relays move it aside (a "dead-letter" table) after a few tries. And reading by seq is only safe here because SQLite lets one writer commit at a time. In Postgres or MySQL a row with a lower id can commit after a higher one, and a relay that remembers "last seq sent" skips it forever. That's one reason the pattern page also lists transaction log tailing as a relay.
5. Mark after sending. Mark first and a crash loses the message: back to the dual-write bug. Mark after and a crash means a resend. You choose duplicates over losses, because duplicates are fixable.

Your turn: the consumer

Fill in Shipping's handler. It must record the message id and create the shipment in one transaction, so a crash can't record "seen" without the shipment, or the shipment without "seen". Use the exact SQL keywords and values.

handle(m: Message): boolean {
  return tx(this.db, () => {
    const fresh = this.db
      .prepare('INSERT INTO  ' +
               '(id) VALUES (?) ON CONFLICT(id) DO ')
      .run(m.id).changes === ;
    if (!fresh) return ; // seen: skip
    this.db
      .prepare('INSERT INTO shipments (order_id, message_id) ' +
               'VALUES (?, ?)')
      .run(m.payload.orderId, m.id);
    return true;
  });
}

Then implement all three functions in src/outbox.ts and run npm run test:05 until all 10 tests are green. Some of them break a write on purpose (the order update, the shipment, the inbox insert) and check that nothing half-applied is left behind: they only pass if each side really uses one transaction. Run npm run l5 again and your version appears as a third row:

  outbox (yours)           200                   0                   0            0                  225

All 200 orders consistent, with 225 messages delivered: the relay crashed and resent 25 times, and the inbox absorbed every duplicate.

So what is "consistent", exactly?

Notice what the outbox does not give you: there is a moment where o7 is paid in the shop's database but Shipping hasn't heard yet. Real shops add another lag: they read from replicas, copies of the database that trail the primary by a little. Whether these windows are acceptable depends on which consistency model you promised. Three are worth knowing now:

Why not demand linearizability everywhere? Across services it costs a lot of waiting and coordination (Kleppmann's §7.2 has the details; the reference sheet summarises them). For "ship the order", a delay of a second is fine, as long as it is never lost. That's the trade the outbox makes: eventual consistency between services, and no message lost.

The customer is a different story. A customer who pays and refreshes may hit a replica that hasn't caught up and see "pending". Kleppmann's social-network version: you post, refresh, and your post is gone, which "is confusing for users"^[2]. Try it: drag the refresh time inside and outside the window, then switch on read-your-writes.

Illustrative numbers, not measurements. The point is the shape: the window is as long as the slowest copy, it always closes because nothing is lost, and read-your-writes is a promise to one client, usually kept by sending that client's reads to the primary for a while.

Practice: from memory

No scrolling up.

Sort the crashes

Rebuild the flow

cd labs/payflow
npm run test:05            # all green
npm run l5                 # your outbox appears as the third row
# the next run overwrites the results file, so keep a copy:
cp results/05-dual-write.json results/05-seed1.json
SEED=9 npm run l5          # and once more with another seed

Read or watch next

Primary source: Pattern: Transactional outbox, Chris Richardson, microservices.io. Short: read the whole page. It's this lesson's argument in five minutes.

• Pattern: Transactional outbox📄 Article · ~5 min · Chris Richardson, microservices.ioThe problem, why 2PC is ruled out, the relay, and the consequences: duplicates, idempotent consumers (an inbox), ordering.
• Distributed Systems 7.3: Eventual consistency▶ Video · Martin Kleppmann, University of Cambridge · lecture 7.3Why we'd give up linearizability, and what eventual and strong eventual consistency promise instead. Watch for the calendar-app example.
• Distributed Systems 7.2: Linearizability▶ Video · Martin KleppmannThe precise meaning of "strong consistency", if the table above left you wanting more. Optional at this stage.
• Consistency Models📄 Reference · skim · JepsenThe map of consistency models and what each forbids. Look at where linearizable and read-your-writes sit, then stop: the rest is beyond this course.

💬 Unsure whether your dual write can use an outbox, or what the consumer's dedupe key should be? Ask me in the terminal. Describe the two systems and I'll work through the redesign with you.