Docs/Deploy & operatev0.0.1-preview
Configuration
The config file, discovery order, the keys that matter, and the traps the engine refuses to let you fall into.
tuladbd is configured by a single TOML-style file of key = value lines. The design bias is fail-closed: an unknown key or unparseable value is a startup failure, not a silent default — a typo’d durability setting will never quietly run with the wrong semantics. Keys this release removed are refused the same way, by name, with a message saying what replaced them.
Discovery
Pass a path (tuladbd /etc/tuladb/tuladb.toml, or tuladbd --config FILE) or let the daemon search, in order: $TULADB_CONFIG → ./tuladb.toml → $XDG_CONFIG_HOME/tuladb/tuladb.toml → ~/.config/tuladb/tuladb.toml → /etc/tuladb/tuladb.toml → built-in defaults. tuladbd --check-config validates without starting; it opens no listeners and touches no data directory.
A minimal single node
node_id = 1
node_addr = "0.0.0.0:7700"
query_port = 7800
fabric_port = 8700
heartbeat_port = 9700
data_dir = "/var/lib/tuladb"
replica_count = 0 # single node: no peers, no quorum waits
max_accounts = 100000
max_transfers = 10000000
partition_count = 64
async_flush = false # ack after fdatasync — the default posture
log_level = "info"
log_format = "plain" # or "json"
# Binding beyond loopback is an explicit opt-in...
security_allow_non_loopback_bind = 1
# ...and then requires mTLS, single nodes included (see below)
network_tls_enabled = 1
network_mtls_required = 1
network_tls_cert_file = "/run/secrets/tuladb/node.crt"
network_tls_key_file = "/run/secrets/tuladb/node.key"
network_tls_ca_file = "/run/secrets/tuladb/ca.crt"
unauthenticated_role = "denied"
role_grant_0 = "payments-api-* -> ledger-operator"
Two startup refusals shape this example:
- A non-loopback bind is opt-in.
node_addron a wildcard or non-loopback host refuses to start withoutsecurity_allow_non_loopback_bind = 1. The built-in default is127.0.0.1:7700. - A reachable fabric port requires mTLS — single nodes included. The fabric listener opens on
node_addr’s host whether or not the node has peers, and it grants every connection cluster-admin; peers authenticate only under mTLS. So a node whose fabric or heartbeat transport is reachable beyond loopback refuses to start unlessnetwork_mtls_required = 1(which needsnetwork_tls_enabled = 1and a CA file) and the heartbeat runs over TCP with mTLS (it inherits the setting; aheartbeat_mtls_required = 0override or UDP heartbeats on an exposed node are refused). A node bound entirely to127.0.0.1needs no certificates.
Two more network rules apply to every node:
- Listeners are IPv4 only. Every listener binds
node_addr’s host, which must be an IPv4 literal (localhostis read as127.0.0.1); an IPv6 literal such as[::1]or a hostname is refused at startup. - Client connections from public addresses are refused by default. With
security_private_ingress_only = 1(the default), the client and query listeners close any connection whose source is not loopback, private (10/8, 172.16/12, 192.168/16), link-local or carrier-grade NAT (100.64/10) — before a byte is read. Put the engine behind a private network or load balancer; set it to0only if clients must genuinely connect from public addresses, and then only with mTLS.
Adding a cluster
Each node names its peers and its own role; replica_count counts voter peers only. Peer addresses name each peer’s client port — replication itself runs on each node’s separate fabric and heartbeat listeners, derived from the per-node fabric_port/heartbeat_port settings (or overridden per peer with peer_N_fabric / peer_N_heartbeat):
node_role = "voter"
peer_2 = "node2:7700"
peer_3 = "node3:7700"
replica_count = 2
# These are the built-in defaults, shown for reference
raft_election_min_ms = 250
raft_election_max_ms = 500
raft_heartbeat_ms = 50
cluster_clock_skew_max_ms = 25
leader_step_down_ms = 1000
quorum_ack_timeout_ms = 5000
proxy_forward_timeout_ms = 5000
client_request_timeout_ms = 15000
A clustered node with any non-loopback peer address is held to the same mTLS rule as above, and each node needs its own certificate: a node accepts a peer’s fabric connection or heartbeat only when the certificate matches that peer.
Rules the engine enforces at startup rather than letting you discover in production:
- Timeout ordering:
client_request_timeout_ms ≥ proxy_forward_timeout_ms ≥ quorum_ack_timeout_ms. A client must outwait the proxy, which must outwait quorum — an inverted ordering turns every slow write into a cascade of spurious failures. - Election timing:
raft_heartbeat_ms < raft_election_min_ms < raft_election_max_ms. - Lease-read safety:
cluster_clock_skew_max_msmust be less than2 × raft_heartbeat_ms. A voter refuses votes for2 × raft_heartbeat_msafter it last heard the leader, and the leader’s read lease is that fence minus the skew bound — so the skew budget must leave a positive lease. An unsafe combination refuses to boot. (A warning, not a refusal, fires when the vote fence reachesraft_election_min_ms, and whenleader_step_down_ms ≤ raft_election_max_ms.)
Bootstrap config seeds the cluster once; after the first committed configuration, membership lives in the replicated cluster config and changes go through tulactl reconfig — editing peers in the TOML does not promote or demote anyone. Grow a cluster by adding a learner and promoting it once it is caught up; see Joining a node for what a new node must hold before any promotion path accepts it.
Topology and standby promotion
A node can declare its placement — topology_region, topology_zone, topology_domain (failure domain), topology_group and topology_priority (higher is promoted first). These seed the node’s entry in the cluster config on first boot; after that the replicated config is the source of truth. They are what tulactl reconfig promote-standby --group G uses: the leader picks the highest-priority caught-up learner in group G and promotes it to voter. A learner with no topology_group is never picked, by either path — learners are often deliberate read replicas, so promotion needs a declared group.
topology_auto_promote(off by default) lets the leader do that on its own once a voter in the group has been unreachable fortopology_auto_promote_after_ms(default 30,000). It is the only setting that changes outage behaviour automatically, so it is opt-in.topology_awareness+topology_policy(single_region_ha,regional_voters,primary_with_dr,custom; off by default) validate the layout at startup and make reconfig refuse a layout the policy rejects — a voter with notopology_regionis always rejected. Check a layout offline withtulactl topology check. On a cluster bootstrapped from staticpeer_Nentries, peers’ placement isn’t known to the config, so an active policy can refuse every later reconfig; enable a policy on a cluster formed by reconfig.
Run tuladbd under a supervisor (systemd, Kubernetes). quorum_failstop_halt_on_stranded is on by default: a node whose applied-but-uncommitted writes are truncated by the cluster fails its WAL closed rather than keep balances no log backs, and a restart re-derives state from the committed log.
The keys that matter most
| Area | Keys | Notes |
|---|---|---|
| Identity & network | node_id, node_addr, query_port, fabric_port, heartbeat_port, data_dir, peer_N, peer_N_role, replica_count, security_allow_non_loopback_bind, security_private_ingress_only |
One process, four listeners, IPv4 only |
| Capacity | max_accounts, max_transfers, partition_count (≤ 4096), hot_ring_slots, max_inflight_batches |
Preallocated up front — see Limits |
| Batching | batch_max_count, batch_window_ns |
Throughput/latency trade |
| Durability | wal_segment_bytes, wal_segment_count, wal_group_max_batches, wal_group_flush_ns, async_flush, snapshot_interval_batches, wal_backend, wal_direct_io, shutdown_checkpoint_budget_ms |
wal_backend = "auto" picks io_uring on Linux; wal_direct_io (off by default) bypasses the page cache for WAL frame writes; a graceful stop takes a final checkpoint within shutdown_checkpoint_budget_ms (default 5000) so a planned restart replays almost nothing |
| WAL hash chain | wal_chain_mode |
0 off, 1 header + CRC (default), 2 full-payload SHA3 (tamper-evident). Fixed per data directory: a node refuses to start in a mode other than the one its data directory records |
| WAL archiving | wal_archive_enabled, wal_archive_dir, wal_archive_command, wal_archive_fail_closed, wal_archive_retention_hours |
Feeds PITR |
| Topology | topology_region, topology_zone, topology_domain, topology_group, topology_priority, topology_awareness, topology_policy, topology_auto_promote, topology_auto_promote_after_ms |
Placement, standby promotion — see topology |
| Raft timing | raft_election_min_ms/max_ms, raft_heartbeat_ms, heartbeat_io_timeout_ms, cluster_clock_skew_max_ms, quorum_ack_timeout_ms, leader_step_down_ms |
Defaults 250/500/50/25/25/5000/1000 ms; subject to the enforced inequalities |
| Read consistency | cluster_linearizable_reads, cluster_linearizable_query_reads, cluster_read_index_timeout_ms, read_catchup_timeout_ms, index_read_wait_ms |
See Queries |
| Write SLO guard | write_latency_slo_ms, write_slo_mode, write_slo_max_inflight, write_admit_max_peer_lag |
Shed early instead of queueing into collapse — see below |
| Memory and disk | memory_limit_bytes, memory_budget_bytes, memory_budget_pct, memory_budget_resume_pct, memory_fit_check, disk_min_free_pct, disk_min_free_bytes |
See resource limits |
| History retention | history_retention_hours, history_purge_requires_export, history_purge_interval_ms, history_export_*, export_lease_ttl_ms, export_lease_max_hold_batches |
Off by default — see history retention |
| Dedup backend | dedup_backend, dedup_run_shards, dedup_run_filter, dedup_hot_target_entries, dedup_hot_hard_limit_entries, dedup_index_queue_slots, idem_slab_max_slots |
See the dedup backend |
| Node join | join_full_sync, join_sync_verify_voters, join_sync_source_mib_per_s, join_sync_max_bytes, leader_require_history |
See Joining a node |
| Autotune | autotune, autotune_record, and "auto" for wal_group_flush_ns, write_latency_slo_ms, wal_segment_bytes |
See autotune |
| Security | network_tls_*, network_mtls_required, network_require_authenticated_clients, unauthenticated_role, role_grant_N, api_key_file, oidc_*, auth_max_failures_per_conn, auth_failure_delay_ms, storage_encryption_*, audit_log_* |
See Security |
| Observability | log_level, log_format, log_sink, otlp_endpoint |
See Observability |
History retention and duplicate protection
Duplicate protection lasts exactly as long as history. There is no separate, clock-based idempotency window in this release: idem_index_ttl_ns and dedup_retention_epoch_ms were removed, and a config that still sets either refuses to start. A transfer’s identity retires only when the replicated history-purge floor passes it; a retry of a purged identity is then admitted as new — identically on every node.
history_retention_hours = 0(the default) keeps everything forever — every record and every identity. The cost is disk, which grows with every transfer.- With
history_retention_hours > 0, the leader decides a purge cutoff and commits it to the log, so every replica moves the same floor at the same log position. A record is purged only when it is older than the window, at or below what every export pipeline has durably acknowledged to the leader, below the lowest open 2PC hold, and applied. Reads below the floor answerERR_HISTORY_PURGED. history_purge_requires_export = true(default) means purge needs an exporter. The exporter records its position through an export lease, so this also needsexport_lease_ttl_ms > 0(default0, leases off) — with leases off the engine warns at boot that no history will be purged. With no export pipeline acknowledging the leader, nothing is purged and history plus the idempotency index keep growing. Setting it tofalsepurges by age alone — history that was never exported is deleted.- Retention needs a single-lane WAL and the WAL hash chain on:
history_retention_hours > 0withwal_chain_mode = 0refuses to start.
A short duplicate window is therefore a short history window — size it for your longest possible retry path (stalled queues, disaster recovery), not for p99 latency.
The dedup backend
Where the durable idempotency index lives is a first-class choice (dedup_backend):
auto— the default. On a single-lane WAL (the only supported layout) it resolves torun, and if neither eviction target is set it uses 200,000 resident identities with a 600,000 hard limit. Two exceptions keep existing data safe: a data directory that already holds a store created undermemorystays onmemory(logged once at startup — setrunexplicitly to move it), and a config that names a backend keeps it.run— a bounded resident set in RAM over sharded, immutable sorted runs on disk, each with a Binary Fuse absence filter. Publish cost per entry doesn’t decay as history grows, and the resident identity set stays neardedup_hot_target_entriesregardless of total distinct transfers.dedup_run_filterpicks the filter for new runs:fuse16(default) orfuse32(fewer false-positive disk reads at about twice the filter RAM).memory— the entire set resident in RAM: simplest, and bounded only by host memory and checkpoint time, both of which grow with every resident identity.
The flash backend is retired: dedup_backend = "flash" refuses to start, and so does a data directory that still holds a flash tier.
Three rules the fault-injection campaigns taught, now encoded in the profiles:
dedup_run_shardsis routing, not tuning. It’s recorded in the tier on first open, and a mismatch fails the open closed — re-routing would make existing entries unreachable, which reads as a false absence, which is a double-apply.- Size the publish queue for your rate.
dedup_index_queue_slotsdefaults to 1M; sustained overflow diverts publishing to a slower path and can latch backpressure that halts commits. The engine warns at startup whenrunis set explicitly with the default queue; high-rate profiles set 8M. - Backend changes are one-way safe. A store created under
memoryopens cleanly underrun; arunstore must never be reopened asmemory, and the engine fails that open closed rather than risk double-applies.
The write SLO guard
write_latency_slo_ms > 0 with write_slo_mode = "reject" turns on an adaptive admission limit on the leader: it caps admitted, unfinished transfer and reserve writes, and cuts the limit when their p99 is over the target. Excess writes are refused before they touch the WAL with ERR_DEADLINE_EXCEEDED — retryable, and safe to retry under the same identity. Commits and rollbacks are never shed. write_slo_max_inflight is the limit’s ceiling (0 = auto). On a follower the same value bounds how long a proxied write may be held. It is off (0) by default; it must be less than client_request_timeout_ms. It is process-local load shedding, not per-tenant fairness or an SLA. How the limit moves, its metrics, and the other gates that shed load are in Self-protection.
Memory and disk limits
- Free-space reserve. On every filesystem the node writes, the reserve is the largest of
disk_min_free_pctof its size,disk_min_free_bytes, and twice the largest recent checkpoint burst (plus room for two segment activations), capped at half its size — by default at least 1% or 2 GiB. Below it, new transfers are refused withERR_CAPACITY(retryable) and history retention is asked for a pass; below half of it the node holds back its candidacy, and below a quarter a follower refuses replicated transfers instead of filling the disk. Setting both to0turns the reserve off, and a full disk then kills the process. - Memory budget.
memory_budget_bytes, ormemory_budget_pctof the machine’s limit (cgroup,memory_limit_bytes, or physical RAM), closes write admission when anonymous resident memory plus the next checkpoint’s projected growth reaches it: new transfers getERR_CAPACITY, while retries, 2PC commits and rollbacks, reads and replication continue. Admission reopens belowmemory_budget_resume_pct(default 95%). Off by default; settingmemory_limit_bytesalone implies a budget of 90% of it. It is admission control, not a hard cap — use a cgroup for that, and keep headroom for transients the projection doesn’t cover (a snapshot install, allocator retention). - Memory fit check. At startup the engine computes a memory plan from your capacity keys and compares it with the machine’s limit:
memory_fit_check = 1(default) warns,2refuses to start,0is off.tuladbd --check-config --assume-memory 16Gruns the check for another machine size, andtuladbd --memory-plan-jsonprints the plan.
What each gate looks like to clients and in metrics and logs: Self-protection.
Autotune
autotune is off by default; the production profiles set apply (the single-node AWS reference config doesn’t).
observe— on the first start of a fresh data directory,tuladbdprobes the WAL and data disks once (bounded to 2 s and 64 MiB, under the data-dir lock, before recovery) and keeps the result in a hardware-fingerprinted record (autotune.jsonin the data directory, orautotune_record). Later starts read the record instead of measuring again. It reports what it would derive — logs, metrics, andtuladbd --autotune-explain— and changes nothing. If the first record write fails, anautotune.pendingmarker makes the next start probe again.apply— everythingobservedoes, then the knobs set to"auto"are resolved from the record before the engine starts:wal_group_flush_ns(from a device-class table),write_latency_slo_ms(the node’s rated SLO; the guard turns on only on a rated node with 2+ CPUs), andwal_segment_bytes(sized from the node’s rating). Only knobs set to"auto"change; an unresolvable knob keeps its default with a warning;"auto"withoutautotune = "apply"is a config error.- Rating. A node is rated from 3 to 16 sustained-rate calibration runs against it (
tuladbd --autotune-rate). A cluster’s rating is its slowest voter’s. Without a rating, the rate-based behaviour — including the write-admission resource ceiling underapply— stays off.tulactl autotune-planproposes cluster-uniform sizing (max_accounts, the dedup hard limit) from every voter’s memory and the cluster rating; it is read-only.
The probe, the record, the rating and the resource ceiling in full: Autotune.
Shipped profiles
The distribution includes reviewed starting points rather than one config to mutate: a safe profile (durability-first defaults), a secure profile (mTLS required, anonymous access denied, tamper-evident chain mode 2, storage encryption, audit log on), run-tier and single-node production profiles, a cross-region profile that keeps WAN-sized Raft timing, and performance/bench profiles for load testing. The production profiles set the run-tier dedup backend and autotune = "apply"; the bench profiles set memory explicitly and are not a durability baseline. safe and secure leave dedup_index_queue_slots at its 1M default, so they log the dedup_posture warning at startup — size the queue for your rate (the dedup backend). Of the reviewed profiles only secure (8 GiB) and the Linux variant of safe (48 GiB) set memory_limit_bytes, which implies a 90% memory budget — tune it to the box; the others leave the budget off. Every profile binds 127.0.0.1 until you change it. Start from safe or secure and diff your changes — the profiles are small enough to review line by line.
Two settings deserve a permanent warning label:
unauthenticated_roledefaults toread-only. Roles aredenied,read-only,exporter,ledger-operatorandcluster-admin. On an exposed bind without required client authentication, an anonymous role ofexporteror higher refuses to start; demo stacks widen it only behind loopback binds. Production is"denied"plus explicit grants.wal_lane_countis1, and1is the only supported value. A larger value refuses to start without an unsafe-acknowledgement flag, and changing lane count on a populated data directory orphans committed frames.