Distributed correctness

Your architect wants exactly-once delivery between two services. What do you tell them?

Exactly-once delivery is not achievable across a network. The proof is short enough to say out loud: the sender cannot distinguish "the message was lost" from "the message arrived and the acknowledgement was lost". It must either resend (risking a duplicate — at-least-once) or not resend (risking loss — at-most-once). There is no third option, and no amount of protocol removes the ambiguity. This is the Two Generals problem.

What is achievable is exactly-once processing: at-least-once delivery plus idempotent consumers, so duplicates have no observable effect.

def handle(msg):
    # The deduplication point must be the same transaction as the effect.
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO processed (message_id) VALUES (%s) ON CONFLICT DO NOTHING",
            [msg.id],
        ).rowcount
        if not inserted:
            return                              # duplicate; already applied
        apply_effect(msg)                       # same transaction — atomic with the marker

The critical detail is that the dedup marker and the effect commit together. If you record the message id in Redis and apply the effect in Postgres, a crash between them gives you either a lost update or a duplicate — you have moved the problem, not solved it.

When the effect is not in a database you control (sending an email, calling a third-party API), you cannot get atomicity, and you must choose which failure you prefer: mark-then-act risks losing the effect, act-then-mark risks duplicating it. For an email, duplicate is usually better than lost. For a charge, use the provider's idempotency key so they deduplicate.

Two services both think they are the leader. How did that happen, and what breaks?

Split brain happens because you cannot distinguish a crashed node from a slow or partitioned one. Node A stops responding to heartbeats; the cluster elects B; A was actually alive the whole time — garbage collecting, or on the wrong side of a network partition — and has no idea it was replaced. Now both act as leader.

What breaks depends on what the leader does: two writers to the same shard produce conflicting writes; two schedulers each fire the nightly job; two consumers process the same partition and duplicate every effect; two nodes both grant a lock.

The mitigations, in increasing order of strength:

1. Leases with expiry — the leader must renew every N seconds or lose leadership. Reduces the window but does not close it, because the old leader may be paused past its own expiry check and still act on resumption.
2. Fencing tokens — leadership carries a monotonically increasing number, and the resource (the database, the storage layer) rejects any write with a token lower than the highest it has seen. This actually solves it, but it requires the downstream resource to participate.
3. Consensus (Raft/Paxos) — a quorum decides, so a minority partition cannot elect a leader and cannot commit. This is why etcd, ZooKeeper and Consul exist and why you should use one rather than writing leader election yourself.
4. Design it away — make the operation idempotent and safe under concurrent execution, so leadership becomes an efficiency concern rather than a correctness one. This is usually the cheapest real answer.

Messages must be processed in order. How do you guarantee it, and what does it cost?

First, scope it. Global ordering across all messages is almost never the requirement and is enormously expensive — it means one partition, one consumer, no parallelism, and throughput capped by your slowest message. What is usually needed is per-entity ordering: events for one user, one order, one document must be applied in order; events for different entities are independent.

That is what partition keys are for. Kafka guarantees order within a partition, so partitioning by user_id gives per-user ordering with as much parallelism as you have partitions. RabbitMQ gives per-queue order with a single consumer, which is why consistent-hash exchanges exist.

The costs and the sharp edges:

• Head-of-line blocking. One poison message in a partition blocks every subsequent message for that key. You must decide up front: skip it (breaking ordering), stop the partition (blocking a subset of users), or dead-letter it (accepting that a later event may be applied without its predecessor). All three are wrong in some sense; pick deliberately.
• Partition count is hard to change. Adding partitions rehashes keys, so a key that lived in partition 3 now lives in partition 7 and its old messages may still be unconsumed — ordering breaks precisely during the resharding. Over-provision partitions up front.
• Hot keys. One entity producing 60% of events makes one partition the bottleneck, and you cannot split it without breaking ordering.
• Retries reorder. A message that fails and is retried with backoff will land after messages that came behind it. If ordering matters, a retry must block the partition, not go to a delay queue.

The alternative that avoids the whole problem: make the consumer order-insensitive. Include a version or timestamp in each event and apply it as a conditional update — UPDATE ... WHERE version < $new_version. Then out-of-order and duplicate delivery are both harmless, and you can process the partition in parallel. This is usually the better design, and offering it unprompted is a strong signal.

Two servers disagree about what time it is. Where does that hurt you?

Clocks drift, NTP corrections can step time backwards, virtual machines suffer clock jumps after live migration, and leap seconds have historically caused real outages. Concrete damage:

• Last-write-wins conflict resolution silently discards the correct value when the loser's clock was ahead. This is a genuine, well-documented source of data loss in eventually-consistent stores using LWW.
• Token and certificate expiry at boundaries: a JWT issued by a server whose clock is two minutes fast is rejected as "not yet valid" (nbf) by a correct verifier. Hence a small allowed skew in every JWT library.
• Lock leases computed as now + ttl on one machine and checked on another.
• Rate limiters using wall-clock windows can be gamed or can misfire across nodes.
• Ordering events by created_at from multiple producers is simply not sound — you will interleave incorrectly, and the errors will be invisible in testing where everything runs on one machine.
• Cache expiry and Expires headers across a fleet.

The fixes, in the order I would apply them:

1. Use monotonic clocks for durations. time.monotonic() / performance.now() never go backwards, and are the correct tool for measuring elapsed time, timeouts and leases. Using wall time for a duration is a bug even on a single machine.
2. Use logical clocks for ordering. A per-entity sequence number, a Lamport timestamp, or a vector clock captures causality without depending on physical time.
3. Let one authority assign order. A database sequence, a single partition, or a leader assigning monotonic ids. Centralised ordering is often the cheapest correct answer.
4. Allow explicit skew tolerance where physical time is unavoidable (token validation), and monitor clock offset across the fleet as a first-class metric.
5. If you genuinely need globally-ordered timestamps, that is what Spanner's TrueTime (with GPS and atomic clocks, and an explicit uncertainty interval it waits out) exists for — and knowing that is the answer to "can this be solved with better clocks?" is: yes, at enormous cost.

A saga's third step fails after the first two succeeded. What now?

Sagas replace atomicity with compensating transactions: for each completed step, a semantically inverse action executed in reverse order — refund the card, release the flight reservation.

The parts that make this hard, and that a good answer raises unprompted:

• Compensation is not rollback. A refund is not the inverse of a charge; it is a second transaction. The customer saw a charge and then a refund on their statement, the money was gone for three days, and if they were near their limit, something else declined. The intermediate state was visible, which means it must be acceptable to the business.
• Some actions cannot be compensated. An email has been sent. A webhook has fired. A physical package has shipped. The design response is to order the steps so that irreversible actions come last — send the confirmation email only after every reservation succeeded.
• Compensations can fail too. The refund API is down. So compensations must be retried with backoff, be idempotent themselves, and have a terminal path to a human — a queue an operations person works, with enough context to resolve it manually.
• Isolation is gone. Between step 2 and its compensation, another transaction can observe the intermediate state. If that matters, you need semantic locks (mark the record "pending" so other operations skip it) or you must accept the anomaly explicitly.
• You need to know what happened. Every step and compensation must be persisted as it occurs, or a crash mid-saga leaves you unable to determine what to undo. That state machine is the actual implementation, and it belongs in a durable store, not in memory.

Your service reads from a replica and a user reports their own update disappeared. Explain, and fix it.

The write went to the primary; the subsequent read went to a replica that had not yet applied it. Replication is asynchronous, so a lag of even 50 ms is enough: the user saves their profile, the page reloads, and the old name comes back. Then it fixes itself, which makes it maddening to debug and easy to dismiss as a caching bug.

The fixes, from cheapest to strongest:

1. Read from the primary for a short window after a write by that user — a "sticky" flag in the session or a cookie with a TTL slightly above your p99 replication lag. Simple, effective, and the standard answer.
2. Read your own writes via the client: return the updated resource in the write response so the UI does not need to re-read at all. This eliminates the round trip as well as the bug.
3. Monotonic reads by pinning a session to one replica, so the user never observes time going backwards even if they see slightly stale data.
4. Wait for the write's LSN. Capture the primary's log position on write, pass it along, and have the replica wait until it has applied at least that position (pg_wal_lsn_diff, MySQL's MASTER_POS_WAIT). Precise, and more machinery than most systems need.
5. Synchronous replication for the critical path — correct, and it makes every write pay the slowest replica's latency, which is usually the wrong trade.

Whatever you choose, monitor replication lag and alert on it, because every one of these degrades when lag grows from 50 ms to 30 seconds — and lag spikes are caused by exactly the things that also cause incidents: a long transaction, a bulk update, a vacuum, a network problem.