Replication and consistency

A TulaDB cluster replicates one totally-ordered log of committed batches across nodes using Raft-family consensus. Writes are linearizable; reads are linearizable by default; and every weaker read mode is opt-in and explicitly bounded.

Voters, learners, quorum

A cluster is a set of voters (typically 3, or 5) plus optional learners. Voters form quorum: a write is committed when a majority of voters hold it durably in their WALs. Learners receive the full log but never vote and never count toward quorum — they’re for read fan-out, analytics isolation, warm standbys, and the safe way to grow a cluster (add as learner, catch up, promote).

Followers proxy writes to the current leader, so a client pointed at any node still lands its write correctly; leader-aware SDK clients skip the hop by routing to the leader directly. A proxied write that reaches a node which is no longer the leader is never executed there: it answers the retryable ERR_QUORUM_TIMEOUT, and the client’s retry under the same identity finds the new leader.

A new node joins as a learner and becomes a voter only once it holds an exact full copy of the cluster’s data — including retained history, which in this release it syncs from the voters and verifies. See Joining a node.

Elections, without the folklore failure modes

Leader election includes the hardening that separates a production consensus implementation from a textbook one:

• Pre-vote — a partitioned node returning to the cluster can’t bump the term and depose a healthy leader.
• Check-quorum step-down — a leader that can’t reach a majority stops accepting writes and steps down, closing the stale-leader window.
• Candidacy suppression — a node that is fail-stopped, badly lagged, or absent from the persisted voter set doesn’t stand for election.
• Planned leadership transfer — operators move leadership deliberately (tulactl transfer-leadership) before maintenance instead of inducing a failover.

Consensus control traffic (heartbeats, votes) runs on its own port, isolated from bulk replication — snapshot installs can’t delay heartbeats into a spurious election.

Timing defaults. The heartbeat is 50 ms and the election timeout is randomized in 250–500 ms, so a crashed leader is detected within half a second of its last heartbeat; check-quorum step-down defaults to 1 s. A voter also persists the vote fence of the latest leader it acknowledged, so after a restart it refuses (pre-)votes for long enough to honour any lease that leader may still hold — which is what makes a rolling change of heartbeat timing lease-safe once every node runs the same build. Do a binary upgrade and a timing change in separate rolls.

Version fencing. Nodes speak the client wire at exactly WIRE_VERSION 4 and the control protocol at exactly one version; a node on another version can neither vote nor lead, and its frames are dropped and logged rather than half-understood.

Transport security is not optional off loopback. A node whose fabric or heartbeat transport is reachable beyond loopback — a single node included — refuses to start unless mutual TLS is required (network_mtls_required = 1, with TCP heartbeat mTLS), because peer authentication binds each node id to its certificate only under mTLS. Only loopback-bound test clusters run without certificates.

Membership changes are log entries

Cluster configuration is itself replicated through the log and persisted crash-safely. Single-node changes use single-server reconfiguration; anything larger goes through joint consensus, where quorum temporarily requires a majority of both the old and new voter sets — for replication, elections, and read barriers alike. Operators drive changes through tulactl reconfig; the engine derives the safe intermediate configurations itself and refuses client-submitted joint configs.

The operational consequences are simple: grow by adding learners and promoting them one at a time; never shrink by deleting nodes from a config file; after the first committed configuration, the replicated config — not the TOML — is authoritative.

Every promotion path — an operator’s reconfig --promote, promote-standby, and policy-driven auto-promotion — requires the learner to report that it holds the full idempotency set, has no migration in flight, and holds its retained history (HISTORY_OK); there is no override. A change whose proposed voters don’t yet have a live quorum is refused with the retryable ERR_RECONFIG_NO_QUORUM (48), and the same change succeeds once they are up. A reconfig that removes or demotes the current leader reports success when it commits.

Read guarantees, precisely

Default reads are linearizable. The engine implements Raft’s ReadIndex protocol: before answering, the leader captures the commit index and confirms it still has quorum — via a leadership-lease fast path when timing bounds allow, or a heartbeat round otherwise — then waits for local apply to catch up. A deposed leader that hasn’t noticed yet cannot serve you a stale balance, and a newly elected leader serves no linearizable read until an entry of its own term has committed, so it can’t answer from a log it hasn’t yet confirmed. The lease fast path is guarded at startup: the lease is the voters’ recent-leader fence (2 × raft_heartbeat_ms) minus the clock-skew allowance, so cluster_clock_skew_max_ms must be less than 2 × raft_heartbeat_ms (defaults: 25 ms against a 100 ms fence). An unsafe combination refuses to boot rather than serving subtly wrong reads.

Follower reads are opt-in and bounded. A client may explicitly allow a specific read to be served by a follower. Two protections remain: the follower must have recent proof of leader contact, and it will never answer below that client’s own last write — read-your-writes holds. That wait is bounded: a read-your-writes position beyond every batch the node knows of is refused at once, and a node caps how many reads may wait at a time; both answer the retryable ERR_QUORUM_TIMEOUT. Staleness is bounded by the follower’s apply lag. It is not linearizable; use it for dashboards, not admission decisions.

Derived query indexes are asynchronous. History, listings, and search trail applied state slightly by default — see Queries for the gating options.

What failure looks like

Event	Behavior
Leader crash	Followers elect a new leader; committed writes are preserved; in-flight indeterminate writes resolve on idempotent resubmit
Network partition	The majority side continues; the minority side refuses writes and stale-risky reads — no split brain
Follower down	Quorum unaffected (3-node cluster tolerates one); on return it catches up from the log, or from a snapshot install followed by a verified history sync
Quorum lost	The cluster stops accepting writes — loudly — rather than diverging; reads that require quorum confirmation refuse. A node whose applied-but-uncommitted writes must be truncated fails its WAL closed (quorum_failstop_halt_on_stranded, on by default) and re-derives state from the committed log on restart — run tuladbd under a supervisor that restarts it
Conflicting entry at the same index and term	Impossible under correct Raft, so it is treated as corruption: the replica latches ERR_CORRUPTION instead of truncating, and names both entries
Slow follower	Lag is visible (tuladb_follower_lag_entries); an optional admission guard can shed writes rather than let a replica fall unrecoverably behind

The theme is consistent with the rest of the system: when safety and availability conflict, TulaDB chooses safety, visibly. For how these behaviors are actually exercised — partitions, crashes, clock skew, storage faults — see How we verify correctness.