Docs/Deploy & operatev0.0.1-preview

Configuration

The config file, discovery order, the keys that matter, and the traps the engine refuses to let you fall into.

On this page
  1. Discovery
  2. A minimal single node
  3. Adding a cluster
  4. The keys that matter most
  5. History retention and duplicate protection
  6. The dedup backend
  7. The write SLO guard
  8. Memory and disk limits
  9. Autotune
  10. Shipped profiles

tuladbd is configured by a single TOML-style file of key = value lines. The design bias is fail-closed: an unknown key or unparseable value is a startup failure, not a silent default — a typo’d durability setting will never quietly run with the wrong semantics. Keys this release removed are refused the same way, by name, with a message saying what replaced them.

Discovery

Pass a path (tuladbd /etc/tuladb/tuladb.toml, or tuladbd --config FILE) or let the daemon search, in order: $TULADB_CONFIG → ./tuladb.toml → $XDG_CONFIG_HOME/tuladb/tuladb.toml → ~/.config/tuladb/tuladb.toml → /etc/tuladb/tuladb.toml → built-in defaults. tuladbd --check-config validates without starting; it opens no listeners and touches no data directory.

A minimal single node

node_id        = 1
node_addr      = "0.0.0.0:7700"
query_port     = 7800
fabric_port    = 8700
heartbeat_port = 9700
data_dir       = "/var/lib/tuladb"
replica_count  = 0                 # single node: no peers, no quorum waits

max_accounts    = 100000
max_transfers   = 10000000
partition_count = 64

async_flush = false                # ack after fdatasync — the default posture
log_level   = "info"
log_format  = "plain"              # or "json"

# Binding beyond loopback is an explicit opt-in...
security_allow_non_loopback_bind = 1
# ...and then requires mTLS, single nodes included (see below)
network_tls_enabled   = 1
network_mtls_required = 1
network_tls_cert_file = "/run/secrets/tuladb/node.crt"
network_tls_key_file  = "/run/secrets/tuladb/node.key"
network_tls_ca_file   = "/run/secrets/tuladb/ca.crt"
unauthenticated_role  = "denied"
role_grant_0 = "payments-api-* -> ledger-operator"

Two startup refusals shape this example:

  • A non-loopback bind is opt-in. node_addr on a wildcard or non-loopback host refuses to start without security_allow_non_loopback_bind = 1. The built-in default is 127.0.0.1:7700.
  • A reachable fabric port requires mTLS — single nodes included. The fabric listener opens on node_addr’s host whether or not the node has peers, and it grants every connection cluster-admin; peers authenticate only under mTLS. So a node whose fabric or heartbeat transport is reachable beyond loopback refuses to start unless network_mtls_required = 1 (which needs network_tls_enabled = 1 and a CA file) and the heartbeat runs over TCP with mTLS (it inherits the setting; a heartbeat_mtls_required = 0 override or UDP heartbeats on an exposed node are refused). A node bound entirely to 127.0.0.1 needs no certificates.

Two more network rules apply to every node:

  • Listeners are IPv4 only. Every listener binds node_addr’s host, which must be an IPv4 literal (localhost is read as 127.0.0.1); an IPv6 literal such as [::1] or a hostname is refused at startup.
  • Client connections from public addresses are refused by default. With security_private_ingress_only = 1 (the default), the client and query listeners close any connection whose source is not loopback, private (10/8, 172.16/12, 192.168/16), link-local or carrier-grade NAT (100.64/10) — before a byte is read. Put the engine behind a private network or load balancer; set it to 0 only if clients must genuinely connect from public addresses, and then only with mTLS.

Adding a cluster

Each node names its peers and its own role; replica_count counts voter peers only. Peer addresses name each peer’s client port — replication itself runs on each node’s separate fabric and heartbeat listeners, derived from the per-node fabric_port/heartbeat_port settings (or overridden per peer with peer_N_fabric / peer_N_heartbeat):

node_role     = "voter"
peer_2        = "node2:7700"
peer_3        = "node3:7700"
replica_count = 2

# These are the built-in defaults, shown for reference
raft_election_min_ms      = 250
raft_election_max_ms      = 500
raft_heartbeat_ms         = 50
cluster_clock_skew_max_ms = 25
leader_step_down_ms       = 1000
quorum_ack_timeout_ms     = 5000
proxy_forward_timeout_ms  = 5000
client_request_timeout_ms = 15000

A clustered node with any non-loopback peer address is held to the same mTLS rule as above, and each node needs its own certificate: a node accepts a peer’s fabric connection or heartbeat only when the certificate matches that peer.

Rules the engine enforces at startup rather than letting you discover in production:

  • Timeout ordering: client_request_timeout_ms ≥ proxy_forward_timeout_ms ≥ quorum_ack_timeout_ms. A client must outwait the proxy, which must outwait quorum — an inverted ordering turns every slow write into a cascade of spurious failures.
  • Election timing: raft_heartbeat_ms < raft_election_min_ms < raft_election_max_ms.
  • Lease-read safety: cluster_clock_skew_max_ms must be less than 2 × raft_heartbeat_ms. A voter refuses votes for 2 × raft_heartbeat_ms after it last heard the leader, and the leader’s read lease is that fence minus the skew bound — so the skew budget must leave a positive lease. An unsafe combination refuses to boot. (A warning, not a refusal, fires when the vote fence reaches raft_election_min_ms, and when leader_step_down_ms ≤ raft_election_max_ms.)

Bootstrap config seeds the cluster once; after the first committed configuration, membership lives in the replicated cluster config and changes go through tulactl reconfig — editing peers in the TOML does not promote or demote anyone. Grow a cluster by adding a learner and promoting it once it is caught up; see Joining a node for what a new node must hold before any promotion path accepts it.

Topology and standby promotion

A node can declare its placement — topology_region, topology_zone, topology_domain (failure domain), topology_group and topology_priority (higher is promoted first). These seed the node’s entry in the cluster config on first boot; after that the replicated config is the source of truth. They are what tulactl reconfig promote-standby --group G uses: the leader picks the highest-priority caught-up learner in group G and promotes it to voter. A learner with no topology_group is never picked, by either path — learners are often deliberate read replicas, so promotion needs a declared group.

  • topology_auto_promote (off by default) lets the leader do that on its own once a voter in the group has been unreachable for topology_auto_promote_after_ms (default 30,000). It is the only setting that changes outage behaviour automatically, so it is opt-in.
  • topology_awareness + topology_policy (single_region_ha, regional_voters, primary_with_dr, custom; off by default) validate the layout at startup and make reconfig refuse a layout the policy rejects — a voter with no topology_region is always rejected. Check a layout offline with tulactl topology check. On a cluster bootstrapped from static peer_N entries, peers’ placement isn’t known to the config, so an active policy can refuse every later reconfig; enable a policy on a cluster formed by reconfig.

Run tuladbd under a supervisor (systemd, Kubernetes). quorum_failstop_halt_on_stranded is on by default: a node whose applied-but-uncommitted writes are truncated by the cluster fails its WAL closed rather than keep balances no log backs, and a restart re-derives state from the committed log.

The keys that matter most

Area Keys Notes
Identity & network node_id, node_addr, query_port, fabric_port, heartbeat_port, data_dir, peer_N, peer_N_role, replica_count, security_allow_non_loopback_bind, security_private_ingress_only One process, four listeners, IPv4 only
Capacity max_accounts, max_transfers, partition_count (≤ 4096), hot_ring_slots, max_inflight_batches Preallocated up front — see Limits
Batching batch_max_count, batch_window_ns Throughput/latency trade
Durability wal_segment_bytes, wal_segment_count, wal_group_max_batches, wal_group_flush_ns, async_flush, snapshot_interval_batches, wal_backend, wal_direct_io, shutdown_checkpoint_budget_ms wal_backend = "auto" picks io_uring on Linux; wal_direct_io (off by default) bypasses the page cache for WAL frame writes; a graceful stop takes a final checkpoint within shutdown_checkpoint_budget_ms (default 5000) so a planned restart replays almost nothing
WAL hash chain wal_chain_mode 0 off, 1 header + CRC (default), 2 full-payload SHA3 (tamper-evident). Fixed per data directory: a node refuses to start in a mode other than the one its data directory records
WAL archiving wal_archive_enabled, wal_archive_dir, wal_archive_command, wal_archive_fail_closed, wal_archive_retention_hours Feeds PITR
Topology topology_region, topology_zone, topology_domain, topology_group, topology_priority, topology_awareness, topology_policy, topology_auto_promote, topology_auto_promote_after_ms Placement, standby promotion — see topology
Raft timing raft_election_min_ms/max_ms, raft_heartbeat_ms, heartbeat_io_timeout_ms, cluster_clock_skew_max_ms, quorum_ack_timeout_ms, leader_step_down_ms Defaults 250/500/50/25/25/5000/1000 ms; subject to the enforced inequalities
Read consistency cluster_linearizable_reads, cluster_linearizable_query_reads, cluster_read_index_timeout_ms, read_catchup_timeout_ms, index_read_wait_ms See Queries
Write SLO guard write_latency_slo_ms, write_slo_mode, write_slo_max_inflight, write_admit_max_peer_lag Shed early instead of queueing into collapse — see below
Memory and disk memory_limit_bytes, memory_budget_bytes, memory_budget_pct, memory_budget_resume_pct, memory_fit_check, disk_min_free_pct, disk_min_free_bytes See resource limits
History retention history_retention_hours, history_purge_requires_export, history_purge_interval_ms, history_export_*, export_lease_ttl_ms, export_lease_max_hold_batches Off by default — see history retention
Dedup backend dedup_backend, dedup_run_shards, dedup_run_filter, dedup_hot_target_entries, dedup_hot_hard_limit_entries, dedup_index_queue_slots, idem_slab_max_slots See the dedup backend
Node join join_full_sync, join_sync_verify_voters, join_sync_source_mib_per_s, join_sync_max_bytes, leader_require_history See Joining a node
Autotune autotune, autotune_record, and "auto" for wal_group_flush_ns, write_latency_slo_ms, wal_segment_bytes See autotune
Security network_tls_*, network_mtls_required, network_require_authenticated_clients, unauthenticated_role, role_grant_N, api_key_file, oidc_*, auth_max_failures_per_conn, auth_failure_delay_ms, storage_encryption_*, audit_log_* See Security
Observability log_level, log_format, log_sink, otlp_endpoint See Observability

History retention and duplicate protection

Duplicate protection lasts exactly as long as history. There is no separate, clock-based idempotency window in this release: idem_index_ttl_ns and dedup_retention_epoch_ms were removed, and a config that still sets either refuses to start. A transfer’s identity retires only when the replicated history-purge floor passes it; a retry of a purged identity is then admitted as new — identically on every node.

  • history_retention_hours = 0 (the default) keeps everything forever — every record and every identity. The cost is disk, which grows with every transfer.
  • With history_retention_hours > 0, the leader decides a purge cutoff and commits it to the log, so every replica moves the same floor at the same log position. A record is purged only when it is older than the window, at or below what every export pipeline has durably acknowledged to the leader, below the lowest open 2PC hold, and applied. Reads below the floor answer ERR_HISTORY_PURGED.
  • history_purge_requires_export = true (default) means purge needs an exporter. The exporter records its position through an export lease, so this also needs export_lease_ttl_ms > 0 (default 0, leases off) — with leases off the engine warns at boot that no history will be purged. With no export pipeline acknowledging the leader, nothing is purged and history plus the idempotency index keep growing. Setting it to false purges by age alone — history that was never exported is deleted.
  • Retention needs a single-lane WAL and the WAL hash chain on: history_retention_hours > 0 with wal_chain_mode = 0 refuses to start.

A short duplicate window is therefore a short history window — size it for your longest possible retry path (stalled queues, disaster recovery), not for p99 latency.

The dedup backend

Where the durable idempotency index lives is a first-class choice (dedup_backend):

  • auto — the default. On a single-lane WAL (the only supported layout) it resolves to run, and if neither eviction target is set it uses 200,000 resident identities with a 600,000 hard limit. Two exceptions keep existing data safe: a data directory that already holds a store created under memory stays on memory (logged once at startup — set run explicitly to move it), and a config that names a backend keeps it.
  • run — a bounded resident set in RAM over sharded, immutable sorted runs on disk, each with a Binary Fuse absence filter. Publish cost per entry doesn’t decay as history grows, and the resident identity set stays near dedup_hot_target_entries regardless of total distinct transfers. dedup_run_filter picks the filter for new runs: fuse16 (default) or fuse32 (fewer false-positive disk reads at about twice the filter RAM).
  • memory — the entire set resident in RAM: simplest, and bounded only by host memory and checkpoint time, both of which grow with every resident identity.

The flash backend is retired: dedup_backend = "flash" refuses to start, and so does a data directory that still holds a flash tier.

Three rules the fault-injection campaigns taught, now encoded in the profiles:

  1. dedup_run_shards is routing, not tuning. It’s recorded in the tier on first open, and a mismatch fails the open closed — re-routing would make existing entries unreachable, which reads as a false absence, which is a double-apply.
  2. Size the publish queue for your rate. dedup_index_queue_slots defaults to 1M; sustained overflow diverts publishing to a slower path and can latch backpressure that halts commits. The engine warns at startup when run is set explicitly with the default queue; high-rate profiles set 8M.
  3. Backend changes are one-way safe. A store created under memory opens cleanly under run; a run store must never be reopened as memory, and the engine fails that open closed rather than risk double-applies.

The write SLO guard

write_latency_slo_ms > 0 with write_slo_mode = "reject" turns on an adaptive admission limit on the leader: it caps admitted, unfinished transfer and reserve writes, and cuts the limit when their p99 is over the target. Excess writes are refused before they touch the WAL with ERR_DEADLINE_EXCEEDED — retryable, and safe to retry under the same identity. Commits and rollbacks are never shed. write_slo_max_inflight is the limit’s ceiling (0 = auto). On a follower the same value bounds how long a proxied write may be held. It is off (0) by default; it must be less than client_request_timeout_ms. It is process-local load shedding, not per-tenant fairness or an SLA. How the limit moves, its metrics, and the other gates that shed load are in Self-protection.

Memory and disk limits

  • Free-space reserve. On every filesystem the node writes, the reserve is the largest of disk_min_free_pct of its size, disk_min_free_bytes, and twice the largest recent checkpoint burst (plus room for two segment activations), capped at half its size — by default at least 1% or 2 GiB. Below it, new transfers are refused with ERR_CAPACITY (retryable) and history retention is asked for a pass; below half of it the node holds back its candidacy, and below a quarter a follower refuses replicated transfers instead of filling the disk. Setting both to 0 turns the reserve off, and a full disk then kills the process.
  • Memory budget. memory_budget_bytes, or memory_budget_pct of the machine’s limit (cgroup, memory_limit_bytes, or physical RAM), closes write admission when anonymous resident memory plus the next checkpoint’s projected growth reaches it: new transfers get ERR_CAPACITY, while retries, 2PC commits and rollbacks, reads and replication continue. Admission reopens below memory_budget_resume_pct (default 95%). Off by default; setting memory_limit_bytes alone implies a budget of 90% of it. It is admission control, not a hard cap — use a cgroup for that, and keep headroom for transients the projection doesn’t cover (a snapshot install, allocator retention).
  • Memory fit check. At startup the engine computes a memory plan from your capacity keys and compares it with the machine’s limit: memory_fit_check = 1 (default) warns, 2 refuses to start, 0 is off. tuladbd --check-config --assume-memory 16G runs the check for another machine size, and tuladbd --memory-plan-json prints the plan.

What each gate looks like to clients and in metrics and logs: Self-protection.

Autotune

autotune is off by default; the production profiles set apply (the single-node AWS reference config doesn’t).

  • observe — on the first start of a fresh data directory, tuladbd probes the WAL and data disks once (bounded to 2 s and 64 MiB, under the data-dir lock, before recovery) and keeps the result in a hardware-fingerprinted record (autotune.json in the data directory, or autotune_record). Later starts read the record instead of measuring again. It reports what it would derive — logs, metrics, and tuladbd --autotune-explain — and changes nothing. If the first record write fails, an autotune.pending marker makes the next start probe again.
  • apply — everything observe does, then the knobs set to "auto" are resolved from the record before the engine starts: wal_group_flush_ns (from a device-class table), write_latency_slo_ms (the node’s rated SLO; the guard turns on only on a rated node with 2+ CPUs), and wal_segment_bytes (sized from the node’s rating). Only knobs set to "auto" change; an unresolvable knob keeps its default with a warning; "auto" without autotune = "apply" is a config error.
  • Rating. A node is rated from 3 to 16 sustained-rate calibration runs against it (tuladbd --autotune-rate). A cluster’s rating is its slowest voter’s. Without a rating, the rate-based behaviour — including the write-admission resource ceiling under apply — stays off. tulactl autotune-plan proposes cluster-uniform sizing (max_accounts, the dedup hard limit) from every voter’s memory and the cluster rating; it is read-only.

The probe, the record, the rating and the resource ceiling in full: Autotune.

Shipped profiles

The distribution includes reviewed starting points rather than one config to mutate: a safe profile (durability-first defaults), a secure profile (mTLS required, anonymous access denied, tamper-evident chain mode 2, storage encryption, audit log on), run-tier and single-node production profiles, a cross-region profile that keeps WAN-sized Raft timing, and performance/bench profiles for load testing. The production profiles set the run-tier dedup backend and autotune = "apply"; the bench profiles set memory explicitly and are not a durability baseline. safe and secure leave dedup_index_queue_slots at its 1M default, so they log the dedup_posture warning at startup — size the queue for your rate (the dedup backend). Of the reviewed profiles only secure (8 GiB) and the Linux variant of safe (48 GiB) set memory_limit_bytes, which implies a 90% memory budget — tune it to the box; the others leave the budget off. Every profile binds 127.0.0.1 until you change it. Start from safe or secure and diff your changes — the profiles are small enough to review line by line.

Two settings deserve a permanent warning label:

  • unauthenticated_role defaults to read-only. Roles are denied, read-only, exporter, ledger-operator and cluster-admin. On an exposed bind without required client authentication, an anonymous role of exporter or higher refuses to start; demo stacks widen it only behind loopback binds. Production is "denied" plus explicit grants.
  • wal_lane_count is 1, and 1 is the only supported value. A larger value refuses to start without an unsafe-acknowledgement flag, and changing lane count on a populated data directory orphans committed frames.