Consistency and the outbox
Keeping two services consistent without a shared transaction. One page, for quick lookup.
The rule
Never update two systems one after the other and hope. Write the change and the message in one local transaction, deliver at least once, and make the receiver skip repeats.
Dual write: what a crash between the writes does
| order of writes | crash between them leaves | npm run l5, seed 1, 200 orders |
|---|---|---|
| DB, then publish | paid, never shipped (lost message) | 21 broken · 179 messages delivered |
| publish, then DB | shipped, not paid (ghost message) | 21 broken · 200 messages delivered |
| outbox + relay + inbox | nothing broken; duplicates absorbed | 0 broken · 225 messages delivered for 200 |
The three pieces
| piece | where | does | guarantee |
|---|---|---|---|
| outbox write | shop DB, in the same transaction as the state change | guarded update (status != 'paid', so unknown → paid still works); only if it changed a row, insert the outbox row | the message exists iff the change does |
| relay | a loop or job next to the shop DB | unsent rows in seq order → publish → then mark published_at; stop at the first failure | at-least-once, in order |
| consumer inbox | receiver's DB, one transaction | record the message id (atomic insert into processed_messages); already there → skip; else apply the effect | effect applied once per message id |
Lab tests break each write on purpose (order update, shipment, inbox insert): they pass only if nothing half-applied is left, i.e. each side really uses one transaction.
Mark after sending: a crash then causes a duplicate (fixable), never a loss (not fixable). If the consumer's effect is an external call that can't join its transaction, pass the message id downstream as an idempotency key (Lesson 4).
Production caveats
- Ordering by
seqis SQLite-safe only. SQLite commits one writer at a time. In Postgres/MySQL a lower id can commit after a higher one; a relay that remembers "last seq sent" skips it forever. Alternatives: re-scan all unsent rows, or transaction log tailing. - Head-of-line blocking. Stopping at the first failure means one poison message stops everything behind it. Move it aside (dead-letter table) after a few tries.
- Both tables grow. Prune published outbox rows, and keep inbox ids longer than the longest redelivery horizon.
Consistency models (just enough)
| model | promise |
|---|---|
| linearizable (strong) | behaves as if there were a single copy, with operations taking effect in real-time order. A single database node gives it for reads and writes of one row. Across replicas it costs messages and waiting: Kleppmann notes "every operation requires communication with a quorum of replicas" (a majority), so a node cut off from enough of them can't serve requests, and a leader can become a bottleneck. |
| eventual | if updates stop, all reads eventually return the last value. Shop ↔ Shipping with an outbox. |
| read-your-writes | a client always sees its own writes. Usually: route that client's reads to the primary for a while after it writes. |
Inconsistency window = how long the slowest copy lags (relay interval, queue lag, replica lag). With an outbox it always closes, because nothing is lost.
Vocabulary
| term | meaning |
|---|---|
| transaction | Writes that become durable together at COMMIT, or not at all. |
| dual write | Updating two systems in sequence with nothing making them atomic. |
| outbox | A table of messages to send, written in the same transaction as the change. |
| relay | The process that sends unsent outbox rows and marks them sent. |
| inbox | The consumer's table of applied message ids (processed_messages), written in the same transaction as the effect, so repeats are skipped. It makes the consumer idempotent. |
| exactly-once effect | At-least-once delivery plus an inbox: the message may arrive twice, the effect happens once. (Not exactly-once delivery.) |
| message broker | A service that stores messages and forwards them to receivers, like a queue. |
| replica | A copy of a database that follows the primary, usually a little behind. |
Lab
labs/payflow/src/outbox.ts · npm run test:05 · npm run l5 → results/05-dual-write.json. Tables in lib/db.ts: outbox, processed_messages, shipments.