Replication that checks its own homework

Raft-style replication gives you agreement on what the log should contain. It does not, by itself, tell you that a follower’s disk actually contains it. Bits rot, filesystems lie under crash, a restored-from-backup node can be internally valid and quietly stale. TulaDB closes that gap by making log equality something the leader checks on every acknowledgement — before the acknowledgement counts toward durability.

Developer-preview note: this describes the shipped behaviour of wal_chain_mode >= 1, the default. A stricter full-payload mode exists for audit-grade deployments. See durability and recovery for the operator’s view.

A 32-byte answer to “do we have the same log?”

Every node maintains, per WAL lane, a running SHA3-256 chain over each frame’s replication-invariant content:

head' = SHA3-256(head ‖ canonical(frame))

The canonical form excludes node-local sealing, so the chain has a useful property: equal logs imply equal heads. The 32-byte head is a log-equality oracle — comparing two heads compares two entire log histories.

Verification is negotiated per connection. A verifying follower attaches its post-append chain head to every successful AppendEntries acknowledgement, and the leader compares it against its own — before the response is recorded. A follower whose log has silently diverged cannot help commit anything.

On mismatch: locate first, then decide

A mismatched head doesn’t say where the logs diverged — the chain is cumulative, so corruption at sequence D presents as a mismatch at every later head. So the leader doesn’t guess: it binary-searches, using internal lookup RPCs on the replication fabric, asking for the follower’s head at specific sequence numbers until it finds the first disagreeing one.

What happens next depends on one question: is the divergence point above or below the follower’s committed watermark?

Above the committed watermark — the divergent suffix is uncommitted, so it’s safe to discard. The leader sends a repair RPC, authenticated by the current term and recorded leader identity, that truncates only the follower’s uncommitted suffix. That round’s ack isn’t counted; the next AppendEntries rewrites the repaired suffix. The event is visible (chain_repair_total), but the cluster heals itself.

At or below the committed watermark — two nodes disagree about committed history, and there is no safe automatic answer. The leader fails closed: the mismatch is logged critical, a divergence counter increments, and the acknowledgement is never counted toward quorum. A diverged replica pages a human instead of participating in durability.

There’s a guard on the boundary between these cases, too: a follower that keeps needing repairs at the same position — or accumulates repairs without a sustained clean run — escalates from “repair” to “hard divergence.” A persistently corrupt disk should page someone, not be silently rewritten forever.

The chain outlives the cluster

The same head is auditable offline. A verification tool walks a cold WAL directory — read-only, plaintext or encrypted — recomputes each lane’s head with the identical canonicalization, and attests it against an expected lane:seq:head. So “this archived WAL is exactly the log the cluster committed” is checkable without booting the engine: a useful property when the archive is the thing your auditor is holding.

Recovery re-derives the chain from replay, seeded from a persisted checkpoint sidecar; snapshots embed the writer’s per-lane heads, so a freshly snapshot-installed follower resumes leader-side verification immediately — and a wrongly seeded head would be caught as divergence on its very first acknowledgement.

Why we built it this way

The theme is one we keep returning to: checks belong at the moment of decision, not in a nightly job. Quorum is a durability decision, so log equality is verified at quorum time, per acknowledgement, with the safe subset of failures healed automatically and the unsafe subset stopped loudly. For a ledger, a replica that’s confidently wrong is worse than a replica that’s down.

In the docs: Replication and consistency covers the consensus layer this sits on; durability and recovery covers the chain from the operator’s side.