Autotune

Autotune lets a node size a few settings from the machine it actually runs on, instead of from a config written for a different one. It is deliberately narrow: it measures a little, changes only the knobs you hand it, writes down what it measured, and never overrides a value you set yourself.

autotune takes three values. off is the compiled default; the shipped safe, secure, run-tier, single-node production and cross-region profiles set apply, as does the AWS cluster reference config; the single-node AWS reference config and the bench and soak profiles don’t.

Mode	What it does
off	No probe, no record, nothing derived.
observe	Probes once, keeps the record, and reports what it would set — in log lines that name each knob’s current value and where it came from, and in metrics — but changes nothing.
apply	Everything observe does, then resolves each knob you set to "auto", and enforces the resource ceiling.

What it measures

The disks, once. On the first start of a fresh data directory, tuladbd probes the WAL directory and the data directory, under the data-dir lock and before recovery, bounded to 64 MiB and 2 s in total:

1. On the WAL directory: 4 KiB appends, each followed by fdatasync (F_FULLFSYNC on macOS), up to 512 samples — p50, p99 and max — then a sequential write for throughput. It uses O_DIRECT when wal_direct_io does.
2. The same on the data directory, buffered.
3. Interference: the WAL’s fsync p99 again while a background thread writes to the data directory.

It also records what the devices are — filesystem type, mount options, model and serial, rotational flag, block size, size — and whether the WAL and data share a disk (an autotune_shared_layout warning when they do). The probe is skipped when either filesystem is too close to its free-space reserve, and it cleans up its two probe files, including ones a killed probe left behind.

What it doesn’t measure. Autotune doesn’t probe CPU or memory. Two settings consult them: the SLO knob resolves only on a node with at least two effective CPUs, and each node advertises its usable memory (the machine’s limit less the memory-fit headroom) to the cluster, for the cluster plan.

The record

The result lands in autotune.json in the data directory (or the path in autotune_record): written to a temporary file, mode 0600, fsynced, renamed, and the directory fsynced. Later starts read the record instead of measuring again.

• It’s tied to the hardware. The record carries a fingerprint of the devices. If a hard field changes — a different disk, filesystem or mount options — the node logs autotune_record_stale and stops using the record for anything; a kernel or CPU change is noted (autotune_soft_change) and the record kept. Re-measure with tuladbd --autotune-reprobe.
• A bad record never stops a node. A corrupt record, or one from a newer release, logs autotune_record_unusable, and the node starts on built-in defaults. A newer record is never overwritten.
• An existing data directory isn’t probed. A directory that already holds engine state but no record — one upgraded from a release before autotune — logs autotune_no_record and is left alone.
• autotune.pending closes the gap. Before probing, the node writes an autotune.pending marker. If the probe runs but the record write then fails (autotune_record_write_failed), the next start sees the marker and probes again, even though the directory is no longer fresh. A successful write removes it.

What apply changes

Only knobs set to "auto" change. "auto" is accepted for exactly three keys, and "auto" without autotune = "apply" is a config error:

Knob	How it resolves
wal_group_flush_ns	From the WAL device’s class: local flash (fsync p50 under 0.5 ms) or network block storage → 1 ms. A rotational, tmpfs or unidentifiable device keeps the 2 ms default, with autotune_flush_unresolved.
write_latency_slo_ms	The node’s rated SLO, which turns the write SLO limiter on — only on a rated node with two or more effective CPUs. Otherwise it stays 0 (off), with autotune_slo_unresolved.
wal_segment_bytes	Sized so the WAL ring holds the larger of two snapshot intervals or ten minutes at the rated rate, from the rating’s measured WAL bytes per transfer and per batch, capped at half the WAL filesystem and clamped to 64 MiB–16 GiB per segment. wal_segment_count is never changed. A short ring logs autotune_ring_short.

The resolved values go through normal config validation; if that fails, the node logs autotune_apply_refused and keeps the defaults. The shipped profiles mostly set write_latency_slo_ms = "auto" and give wal_group_flush_ns an explicit value. The exceptions: cross-region pins the SLO at 200 ms for WAN latency, the AWS reference configs turn the guard off (0), and secure leaves wal_group_flush_ns at its default. Nothing else — capacity, dedup limits, timeouts — is tuned per node.

The rating

A probe doesn’t say how fast a node is. A rating does: the node’s sustained rate at a latency SLO, taken from 3 to 16 sustained-rate calibration runs and imported with tuladbd --autotune-rate. The rating is the slowest run, rounded down to two significant figures — and only if every run held its rate; otherwise the node is unrated, with the reason recorded. Rating a node whose record is missing, stale or unverifiable is refused, and a re-probe drops the rating.

Without a rating, the rate-based behaviour stays off: the SLO knob doesn’t resolve, the ring isn’t sized, and there is no resource ceiling.

A cluster’s rating is its slowest voter’s. Each node advertises its rating on the control plane, and the cluster is rated only when every voter is heard, rated, and rated at the same SLO; otherwise it’s unrated, and the reason names the first voter that broke it. Ratings more than 20% apart log autotune_cluster_rating_spread.

The cluster plan: tulactl autotune-plan

Some sizes must be the same on every node: max_accounts, and the dedup tier’s hot limits. Autotune doesn’t change those on its own. tulactl autotune-plan proposes them, read-only:

tulactl autotune-plan --nodes n1:7700,n2:7700,n3:7700 --config tuladb.toml

It reads each node’s usable memory and rating, then prints a snippet to set on every node before a rolling restart: the largest max_accounts whose memory plan fits the smallest node, and — when the cluster is rated — a dedup_hot_hard_limit_entries of at most the cluster rate × 60 s (--hot-window-s). It refuses (exit 1) if any node is unreachable, too old to report, or reports no memory. A node whose own config won’t fit the cluster’s smallest memory, or whose dedup limit is below the rating’s needs, warns at runtime (autotune_config_exceeds_cluster_memory, autotune_config_dedup_below_rating).

The resource ceiling

Under apply on a rated cluster, each node runs a controller that caps write admission as a transfers-per-second rate, starting at the rating. Every 5 s at most it reads four signals:

Signal	Degraded when	What it protects
Checkpoint multiplier — batches/s × the marginal checkpoint cost per batch	above 0.5	A checkpoint that can’t keep up with the write rate, so each one covers a longer range than the last
WAL ring headroom — time until the ring is full ÷ the time reclaim needs	under 3	The WAL ring filling before reclaim can free it
Dedup resident ratio — resident identities ÷ dedup_hot_hard_limit_entries	above 0.6	The dedup tier’s resident set
Dedup hand-off ratio — eviction capacity ÷ arrival rate	under 2	The evictor falling behind arrivals

A degraded or missing signal cuts the rate by 20%, never below 10% of the rating; all-healthy raises it 5% of the rating at a time, back up to the rating. Each node advertises its own ceiling, and the cluster bound is the quorum value — so one slow follower can’t throttle the cluster, but a majority under pressure does. A write over the bound is refused before it reaches the WAL with ERR_DEADLINE_EXCEEDED, retryable, with a reason that names the resource ceiling. Like the SLO limiter it’s enforced only where a latency SLO is set, and 2PC commits and rollbacks are never refused. Under observe the node computes and logs the ceiling (autotune_resource_ceiling … (observe) … not applied) and enforces nothing.

What you see

• Prometheus: tuladb_autotune_observe, tuladb_autotune_layout, tuladb_autotune_record_stale, tuladb_autotune_rated_tps, tuladb_autotune_rated_slo_ms, tuladb_autotune_applied_wal_group_flush_ns, tuladb_autotune_applied_write_latency_slo_ms, and tuladb_autotune_proposed.
• The metrics snapshot (runtime/metrics.json) carries the rest: the cluster rating and its reason, the resource ceiling’s own, quorum and applied rates, its refusal count, a bitmask of the signals that drove it, and each signal’s value. These aren’t exported to Prometheus in this release.
• Logs, under the autotune component: autotune_probed, autotune_explain, autotune_applied, autotune_resource_ceiling (each change, naming the signals), and the warnings named above.
• tuladbd --autotune-explain prints what the record would derive.

Tested by

make test-autotune (freshness, record round-trip and faults, fingerprint, the probe’s bounds, rating rules, apply and ring sizing, the pending marker), test-autotune-daemon (a real daemon: probed once, never on an upgraded directory, the pending re-probe, explicit values kept), test-autotune-plan, and test-resource-ceiling (the quorum rule, and queueing-model scenarios — including a checkpoint spiral that wedges without the ceiling) with test-resource-ceiling-daemon (apply refuses, observe doesn’t) and test-resource-ceiling-quorum (three daemons).

Not built yet

The ceiling’s only lever is admission: it doesn’t yet bring a checkpoint forward or pace eviction, and it has no CPU or I/O-pressure signals. Autotune doesn’t probe an archive volume, and cluster-uniform values are applied by you from autotune-plan, not distributed through replicated config.