Durability and recovery

Durability in TulaDB is a contract, not a configuration accident: a write is acknowledged only after the write-ahead log confirms it, and everything else — state, indexes, snapshots — is derivable from that log.

The write-ahead log

Every committed batch is appended to the WAL and made durable (fdatasync, or the io_uring backend on Linux) before the client sees an acknowledgement. Group commit amortises one flush across many batches without weakening the ordering: nothing is acknowledged ahead of its durability. By default a flush group closes after 2 ms (wal_group_flush_ns) or 1,024 frames (wal_group_max_batches), whichever comes first.

The WAL is segmented (256 MiB per segment by default, with prepared segments kept warm), and the durability discipline extends to metadata — files are flushed, then their directories are flushed, so a crash can’t leave acknowledged bytes in an unlinked limbo.

For deployments that consciously trade the guarantee for throughput, async_flush acknowledges ahead of the flush — measure before reaching for it: on a single node, group commit already amortises the flush, so async has little to remove there, and the engine requires an explicit acknowledgement of the loss window (allow_single_node_async_ack_loss = 1) to run async on a single node. That loss window is real and measured: a modelled power-loss test (make test-async-ack-power-loss, 2026-10-26) that cuts the machine at a random sync lost none of 37,392 acknowledged transfers under the durable default, and 0.8–1.4% of acknowledgements over 40 cuts with the async opt-in — everything acknowledged since the last background sync. On a single node, treat the async posture as benchmark-only.

In a cluster, async_flush moves the fsync off the leader’s critical path but does not move the durability point: an acknowledgement still waits for a majority of voters, a follower fsyncs before it acknowledges, and the leader counts its own copy only once that copy is fsynced. An acknowledged write therefore always has fsynced copies on a majority, and the reference AWS cluster configuration (tuladb-run-tier-aws-cluster.toml) runs this way. The loss window exists only when the live voter majority is one — a single node, or one voter plus learners — which is why the engine refuses that topology with async unless you set allow_single_node_async_ack_loss = 1.

Hash-chained log

Each WAL frame extends a hash chain over its replication-invariant content, so the log is self-verifying: silent corruption or an altered suffix breaks the chain arithmetic. In a cluster, the leader verifies each follower’s chain head before granting quorum credit — a follower whose log has silently diverged cannot count toward durability. Divergence involving committed entries fails closed rather than self-healing, because a committed record that two nodes disagree about is an incident, not a repair. An offline verifier (tuladb-chain-verify, source build only — not in the release binaries or the image) recomputes a stopped WAL’s chain heads, comparable to a live node’s wal_chain_head metric.

The chain has three modes (wal_chain_mode): 1, the default, chains frame headers and CRC32s — an oracle against accidental divergence; 2 chains a full-payload SHA3 digest and is the tamper-evident choice for audit and compliance deployments; 0 turns the chain off (and with it history retention, which requires the chain). The mode is fixed per data directory: a node refuses to start in a mode other than the one its directory records, and a snapshot install from a node in another mode is refused. Choose a cluster’s mode when you create it; moving one node to another mode means wiping and re-seeding it.

Snapshots bound recovery time

On restart, the engine loads the latest state snapshot and replays only the WAL tail since it — recovery cost is O(WAL tail), not O(history). Snapshots are taken on a configurable cadence (snapshot_interval_batches) and on demand (SIGUSR1 forces a checkpoint). On a graceful stop, after handing off leadership and closing every listener, tuladbd takes one final checkpoint within shutdown_checkpoint_budget_ms (default 5 s; 0 turns it off), so a planned restart replays almost nothing; past the budget it publishes nothing and the stop proceeds as before. Keep your service manager’s stop timeout above that budget. The snapshot is a recovery boundary, never a substitute for the log: correctness never depends on snapshot timing.

Recovery replays the same committed order the cluster agreed to, and duplicate protection is part of the replayed state — a restart cannot double-apply, and it cannot forget an idempotency decision.

Recovery fails closed when the log it would replay is incomplete:

• A hole in the WAL is found at boot. If a segment is missing between two retained ones, the node’s retained floor starts above the hole, so it never claims frames it can’t serve. A missing segment above the snapshot is refused outright on a single node — it names the missing positions; restore the segment from the WAL archive, restore a backup, or re-seed from a peer — rather than replaying later frames over a gap and reporting success. A clustered node instead starts latched (it serves no writes and never campaigns) until a snapshot install that covers the hole clears it.
• A committed frame that replay can’t apply refuses a solo start, naming the position and status; a clustered node latches the same way and is repaired by a snapshot install from a healthy peer.
• On-disk formats are checked before anything is written. A file in a format this build doesn’t write — or from an unreleased development build — refuses the boot, naming the file, the version found and the version supported. Derived artifacts (indexes, checkpoints of derived state) are rebuilt instead. There is no in-place downgrade: to return to an older binary, restore a backup taken on it.

Disk-full behaviour

Every filesystem the node writes to keeps a free-space reserve: the largest of disk_min_free_pct (default 1%) of its size, disk_min_free_bytes (default 2 GiB), and twice the largest recent checkpoint burst plus room for two segment activations, capped at half the filesystem. Below the reserve, new transfers are refused with the retryable ERR_CAPACITY and history retention is asked for a pass; below half of it, the node holds back its candidacy and a leader without a quorum of voters with room refuses new transfers too; below a quarter, a follower refuses replicated transfers rather than fill its disk. The levels, metrics and events are in Self-protection. Setting both to 0 turns the guard off, and a node that then fills its disk dies hard. Leave it on.

History retention

By default nothing is ever deleted: history_retention_hours = 0 keeps every transaction record — and with it, every idempotency decision — forever. Setting a window turns on a replicated purge: the leader decides a cutoff (older than the window by replicated time, below every open hold, and — by default, history_purge_requires_export = true — at or below what every export pipeline has durably acknowledged), and commits it to the log, so every replica advances the same floor at the same position. A read below the floor answers ERR_HISTORY_PURGED (43) on every node, identically, and a retry of a transfer below it is admitted as new — duplicate protection lasts exactly as long as history does. With no exporter acknowledging the leader, nothing is purged. Purge requires the single-lane WAL and the hash chain on. See Analytics export and Idempotency.

WAL archiving

For long retention and point-in-time restore, archive WAL segments as they seal — to a directory or through a custom archive command (an S3 upload, say). Archiving runs behind the write path with pacing and backlog bounds, and is fail-closed by default (wal_archive_fail_closed = true): an archive failure keeps the source segment and latches the WAL unhealthy, so writes stop rather than silently widen the recovery gap. Set it to false only on a non-DR deployment that prefers reclaim progress over archive retention. Archive health is fully observable (backlog, failures, retention) — see Observability.

Backup and restore

A dedicated tool produces a single-file backup (.tulab) of a stopped node — or a strict online backup from a live one (online-full) — with verification built in at every step:

tuladb_backup backup       --data-dir /var/lib/tuladb --output nightly.tulab --verify
tuladb_backup verify       --input nightly.tulab
tuladb_backup inspect      --input nightly.tulab
tuladb_backup restore      --input nightly.tulab --data-dir /var/lib/tuladb-restore
tuladb_backup verify-restore --input nightly.tulab     # rehearse the restore in a temp dir

Backups stream to and from stdout/stdin, so --output - | aws s3 cp - s3://… works without staging disks. verify-restore ships with tuladb_backup, so a single restore rehearsal is always available. A catalog companion (tuladb_backup_catalog.py, source build only — not in the release binaries) adds retention and — the part most backup regimes skip — recurring restore drills, so “the backup works” is a tested property, not an assumption.

tuladb_verify_store ships alongside it and checks a store rather than a backup file: it opens a data directory through the normal recovery path, rebuilds the volatile indexes, runs the engine’s read-only consistency checks, and exits non-zero on any failure. Run it against a stopped node or a filesystem snapshot — never a live data directory — and pass the node’s config so it opens the store with the same sizing:

tuladb_verify_store --data-dir /var/lib/tuladb-restore --config tuladb.toml [--quiet]

Point-in-time restore

With a backup plus archived WAL, restore to an exact position — by batch sequence (--target-batch-seq), wall-clock time (--target-unix-ms), or HLC timestamp (--target-hlc):

tuladb_backup pitr-restore --input nightly.tulab --data-dir /var/lib/tuladb-restore \
  --wal-archive-dir /backup/wal-archive --target-batch-seq 12500000 --verify

Because the target is a coordinate in the committed order, the restored state is exactly the ledger as of that point — balanced, with its idempotency index consistent with its content.

Fail-stop, not fail-strange

When the engine detects a condition it cannot reason about — an invariant violation, an arithmetic overflow on a replica, corruption in a critical structure — it halts rather than continuing on a maybe-wrong ledger: the fault is recorded, the tuladb_engine_halted metric trips, and the daemon exits with a distinct code. In a cluster the halted node simply drops out and quorum continues; a restarted node re-derives correct state from the log. For a system of record, a loud stop is a feature — silent drift is the failure mode that actually costs money.