Joining a node

A node that may become leader must be able to do everything the current leader does: accept writes, detect a retried duplicate, resolve an open hold, answer history and search, and serve the analytics exporter. So TulaDB holds every voter to one invariant: a node becomes a voter only once it holds an exact full copy of the cluster’s data. A learner may lag and catch up; a voter may not be missing anything.

New in 0.0.1-preview, this is enforced through the full-copy join, and it is on by default.

What a joining node receives

A node added as a learner catches up from the leader in one of two ways:

• The log tail. If the leader still holds the log from where the node left off, it simply ships the missing entries.
• A snapshot install, then the tail. If the leader has already reclaimed that part of its log, it ships a snapshot and resumes the log after it. The snapshot carries balances, registries, cluster config, feature state, open two-phase holds, the WAL chain head, and the complete idempotency set — the resident part and everything evicted to disk. Money and duplicate detection are therefore whole the moment the install finishes.

What a snapshot does not carry is the transaction history itself — the records behind history, search and listing reads — or each account’s metadata-history log. Before this release, a node joined by install answered those reads over the missing range with ERR_QUERY_UNAVAILABLE, permanently with the default unlimited retention. The full-copy join fills that range.

The history sync

After an install at log position L, the joiner syncs the retained history it lacks: everything from the replicated purge floor F up to L. That range is immutable — committed, applied, never rewritten — and new writes above L keep arriving through ordinary replication the whole time.

• Partitioned. The range is split into partitions, each the unit of transfer, verification, resume and progress.
• Pulled from the voters, not the leader. Sources are the other voters, then learners that have finished their own join, with the leader asked last. Pulls run over the voters’ fabric ports — node-authenticated under mTLS, like replication — and each source rate-limits what it serves across all joiners (join_sync_source_mib_per_s, default 64 MiB/s), so a join never stalls a source’s own work.
• Trusted only when verified. A partition is pulled only once join_sync_verify_voters sources (default 2) agree on its digest; a disagreeing source is marked suspect and a third breaks the tie. Every pulled record must then fold to the agreed digest, and every record whose idempotency identity is live must agree with the installed idempotency set — same position, phase and result. The idempotency set came from the snapshot the cluster already trusts, so a forged or forked record is refused even if a source is wrong. A source never passes on a range it is itself missing.
• Adopted, then the gap closed last. Verified records are written into new local history segments, the node rebuilds its query indexes from them and checkpoints, and only then is the gap marked filled. A crash at any earlier point leaves reads over the gap refusing and the node unpromotable — never serving partial history.
• Resumable. Progress lives in a durable journal; a restarted node resumes the sync where it stopped.

Account metadata history is synced the same way, verified against the voters’ digests, for the part of each gap above the position where the full-copy join was activated (see upgrades below).

HISTORY_OK gates promotion

A node reports HISTORY_OK when no unfilled install gap meets its retained range and no sync is in force. Every promotion path requires it, alongside the existing checks that the node holds the full idempotency set and has no migration in flight:

• tulactl reconfig --promote,
• tulactl reconfig promote-standby,
• policy-driven automatic promotion.

There is no override: a node that can’t prove a full copy can’t become a voter.

A voter can also become gapped — by falling far enough behind the leader’s log to need a snapshot install itself. It then clears its HISTORY_OK and runs the same sync. If that node is the leader, it hands leadership to a caught-up voter that has HISTORY_OK (leader_require_history, default on) — but it never refuses writes for it. A history gap affects only history reads over the gap, which already refuse; it can’t change what a write commits, so turning it into a write outage would be the wrong trade. With no eligible voter to hand to, it keeps leading.

Watching a join

tulactl join-status --data-dir /var/lib/tuladb [--json]

join-status reports the range being synced, partitions adopted (records and bytes), the driver’s state, HISTORY_OK, and the account-metadata-history step. It reads the node’s journal and metrics file, so it needs no running daemon connection. It exits 1 while a sync journal is in force, the metadata step is still running, or the node reports HISTORY_OK unset; 0 when none of those holds; and 2 on a usage error or an unreadable journal — usable as a wait-until-done probe in automation. The same state is exported as metrics: fullsync_state, fullsync_partitions_adopted, fullsync_last_agreement (sources agreeing per partition, normally 2), fullsync_digest_mismatch_total, fullsync_meta_state, fullsync_meta_gaps, join_history_ok, and leader_history_handoffs.

Configuration

Key	Default	Meaning
join_full_sync	"on"	"off" leaves the install gap in place (history reads over it refuse) and the node never reports HISTORY_OK — so it can’t be promoted while the feature is active. Leave it on.
join_sync_verify_voters	2	Sources whose digest of a partition must agree before it’s pulled. 1 trusts a single source.
join_sync_source_mib_per_s	64	Source side: history bytes this node serves to all joiners together. 0 = unlimited.
join_sync_max_bytes	0 (unlimited)	A sync whose history would exceed this stops; the node stays an unpromotable learner (fullsync_state = over_budget).
leader_require_history	true	A leader without HISTORY_OK hands off to a voter that has it. false keeps leadership wherever elections put it.

Size the sync against your retention. With bounded retention (history_retention_hours > 0) a join copies at most that window. With the default — keep everything — a join copies all history since the cluster was created; set join_sync_max_bytes on such clusters if you’d rather seed large nodes another way than copy it all.

Clusters formed fresh, and upgraded clusters

The full-copy join is an upgrade gate. A cluster formed from empty data directories on this release activates it at genesis, so it’s active before any account exists and nothing needs finalizing. Because 0.0.1-preview is the first release, every 0.0.1-preview cluster starts this way.

A cluster upgraded from a build that predates the feature activates it with tulactl upgrade-finalize, once every member — learners included — is up and supports it. On such a cluster, account metadata history from before the finalize is not positioned the same way on every voter, so no node can vouch for it: on a node that installed a snapshot covering that time, accounts created before the finalize answer metadata-history reads with ERR_QUERY_UNAVAILABLE for good. Route those reads to a node that never installed — any original voter answers them. Accounts created after the finalize answer everywhere once the sync has run. Balances, holds, idempotency and transaction history are unaffected.

Limits of this release

• No read forwarding yet. While a partition is still syncing, a history or search read over it refuses with ERR_QUERY_UNAVAILABLE rather than being forwarded to a voter; retry it against another node.
• One partition at a time, and no warm start: a node seeded from a backup still pulls its history rather than verifying what it already holds.
• Source identity is network- and certificate-trusted. A source checks that a request comes from a cluster member by the node id the request states. Under mTLS, any holder of a cluster certificate can make that claim; on a plaintext fabric, any host that can reach the port can. Keep the fabric private and mTLS on.
• A learner’s digest counts toward agreement. It is a copy the learner verified against the voters — weaker evidence than an independent voter, and present so that a voter re-syncing in a three-voter cluster still has a tie-breaker.
• One metadata merge pause per sync. Merging the synced account-metadata history re-indexes that node’s log once, about 1.2 s per million events, during which account creates and updates on that node wait.
• Several joiners at once can time out on a source. Each source serves joins on one worker per fabric listener; with three or more simultaneous joiners a queued request can outlast the joiner’s timeout, and the joiner tries another source after 30 s.