Overload and self-protection

A ledger that accepts every write until something breaks fails in the worst way: late, all at once, and with acknowledged state at risk. TulaDB’s engine instead refuses early and says why. Most of the mechanisms below shed new work with a retryable status before it reaches the WAL; two stop the node outright, because continuing would mean serving balances nothing backs.

What a client sees

Status	Retry?	Raised by
ERR_DEADLINE_EXCEEDED (38)	yes	The write SLO limiter, the resource ceiling, the WAL-queue cap, the replication-lag gate, and a proxied write that outwaited its hold budget
ERR_CAPACITY (4)	yes	The free-space reserve, the disk quorum, the segment table, the memory budget, and dedup backpressure
ERR_PERMISSION (35)	no	The fifth failed MSG_AUTH on a connection (which is then closed) — or a real lack of role
ERR_CORRUPTION (17)	no	A halted engine (fail-stop)

A refused write was never executed and nothing was staged, so retry it with the same identity — the idempotency check makes that safe whichever attempt lands. In a batch, the refusal is per item. Through the REST gateway, every retryable status becomes HTTP 503 with Retry-After: 1 and retryable: true; a halted engine (17) is a 502. The capacity gates refuse only a transfer already proven new — a retry of one that committed is still answered with its original outcome — and none of these gates refuses 2PC commits or rollbacks, reads, or replication, unless a section below says otherwise.

Write SLO adaptive limiter

Protects against: queueing into collapse — accepted writes whose latency grows without bound under overload.

How it works. With write_latency_slo_ms > 0 and write_slo_mode = "reject", the leader caps the number of admitted, unfinished transfer and reserve writes. Each window — at least the target, and at least 50 ms — it compares the p99 of completed writes with the target. Over target and the limit was binding: halve it, never below the Little’s-law floor of throughput × target ÷ 4. Comfortably under target (below 90%) and binding: grow it by a sixteenth. A cut that costs more than a quarter of throughput is undone and held off, so a target the system can’t meet doesn’t collapse throughput — it is reported as unreachable instead. Before the limit, a WAL-queue cap (write_slo_wal_queue_max) refuses writes when the WAL queue is already deep. The limit is process-local load shedding — not per-tenant fairness, and not an SLA.

Keys. write_latency_slo_ms (default 0, off; must be below client_request_timeout_ms; "auto" under autotune resolves it to the node’s rated SLO), write_slo_mode, write_slo_max_inflight (the limit’s ceiling; 0 = max_inflight_batches ÷ 8, at least 64), write_slo_wal_queue_max (0 = max_inflight_batches ÷ 2, at least 128).

You see tuladb_server_write_slo_rejected_total, _limit, _limit_ceiling, _limit_decreases_total / _increases_total / _reverts_total, _unreachable, _pressure_active, and the tuladb_server_write_slo_admitted_latency_seconds histogram; log warnings write_slo_unreachable and write_slo_not_queueing.

Replication-lag gate. Separately, and even with the SLO off, write_admit_max_peer_lag > 0 refuses new writes when the quorum’s replication lag exceeds it, rather than let a follower fall unrecoverably behind (tuladb_write_admit_lag_rejected_total, write_admit_lag_rejected).

Tested by test-slo-limiter (queueing-model scenarios: overload settles under target, an unreachable target doesn’t collapse, recovery, no latch on one slow write, term changes, stalls) and test-cluster-write-slo-guard (three nodes: keeps at least 70% of guard-off throughput under overload, never a zero limit).

Resource ceiling

Protects against: the slow failures a latency target can’t see coming — a checkpoint that falls behind the write rate, a WAL ring that fills before reclaim frees it, a dedup tier that outgrows its budget. On a rated cluster under autotune = "apply", a transfers-per-second ceiling backs off when any of those degrade, and refuses the excess with ERR_DEADLINE_EXCEEDED. It’s enforced only where a latency SLO is set. Signals, rules and metrics: Autotune — the resource ceiling.

Memory budget

Protects against: being OOM-killed mid-write.

How it works. Every 250 ms the engine samples its anonymous resident memory (on Linux, resident minus shared; on macOS, the physical footprint). Admission closes when that plus the next checkpoint’s projected growth reaches the budget, and reopens below memory_budget_resume_pct of it. While closed, new transfers — plain, tiered and 2PC reserves — get ERR_CAPACITY.

Keys. memory_budget_bytes (0 = off; at least 64 MiB), or memory_budget_pct of the machine’s limit (cgroup, memory_limit_bytes, or physical RAM); memory_limit_bytes alone implies a 90% budget; memory_budget_resume_pct (default 95). Off by default. It is admission control, not a hard cap: keep a cgroup limit for that, with headroom for what the projection doesn’t cover.

You see tuladb_memory_budget_blocked, _bytes, _used_bytes, _projected_bytes, _rejected_total, _episodes_total; an ERROR memory_budget_admission_blocked once per episode and memory_budget_admission_resumed when it reopens.

Tested by test-memory-budget (arithmetic, hysteresis, projection, and that new transfers are refused while a retry and a 2PC commit or rollback still pass) and test-daemon-memory-budget (a real daemon under injected memory: refuses, one episode, reopens).

Disk-full free-space reserve

Protects against: a full disk — on Linux, a memory-mapped segment that can’t be backed kills the process with SIGBUS.

How it works. Every filesystem the node writes is watched (every 250 ms). Its reserve is the largest of disk_min_free_pct of its size, disk_min_free_bytes, and twice the largest recent checkpoint burst plus room for two segment activations — capped at half the filesystem. Then three levels:

• Below the reserve: the leader refuses new transfers with ERR_CAPACITY and asks history retention for a pass. It lifts at the reserve plus an eighth.
• Below half of it: the node flags itself in heartbeats and holds back its candidacy while a peer with room can lead; a leader that can’t find a quorum of voters with room also refuses new transfers.
• Below a quarter of it: a follower refuses replicated batches that carry transfers and lags rather than fill its disk; barrier, purge and config frames still apply.

Keys. disk_min_free_pct (default 1, at most 50) and disk_min_free_bytes (default 2 GiB). Both 0 turns the reserve off, with a disk_reserve_off warning — and a full disk then kills the process.

You see tuladb_disk_capacity_blocked, tuladb_disk_capacity_rejected_total, tuladb_disk_reserve_bytes, tuladb_disk_free_bytes, tuladb_disk_free_critical, tuladb_disk_quorum_short, tuladb_disk_replication_refused_total, tuladb_disk_free_bytes_min; ERRORs disk_capacity_admission_blocked, disk_quorum_admission_blocked, disk_replication_refused, and the warning disk_critical_candidacy_held.

Tested by test-disk-guard (reserve arithmetic, hysteresis, one watch per filesystem, unknown sizes never gate, and that new transfers are refused while a retry is answered).

Dedup backpressure

Protects against: the run-tier dedup index falling behind — too many resident identities, or more on-disk runs than lookups can afford.

How it works. New transfers are refused with ERR_CAPACITY when resident identities reach dedup_hot_hard_limit_entries (the run tier’s default is 600,000, with a target of 200,000), or when a shard’s run count reaches the compaction red line (72 runs; it clears below 60). Before red, at 48 runs or with compaction debt rising, one admission in four is shed. Duplicates are still recognised and answered throughout. A full indexer hand-off queue (dedup_index_queue_slots) doesn’t refuse writes — it costs throughput — but an indexer that stays stalled for a minute halts the node rather than answer from an index it can’t account for.

You see tuladb_dedup_backpressure, tuladb_dedup_capacity_rejected_total, tuladb_dedup_shed_occupancy_total / _red_total / _yellow_total, tuladb_dedup_index_handoff_dropped_total; warnings dedup_compaction_red, dedup_sweep_wedged, dedup_index_stalled.

Tested by test-dedup-backpressure, test-dedup-admission-bound, test-dedup-compaction-red, test-dedup-backpressure-outage and test-dedup-index-stall.

Fail-stop on stranded writes

Protects against: serving balances no log backs. If a node discovers it applied writes that the cluster’s log then truncated — after a quorum failure, a conflicting AppendEntries, or a chain repair — or that it is keeping a frame whose callers were already told it failed, continuing would mean answering from state that doesn’t exist anywhere else.

What happens. The WAL fails closed and the engine halts: a FATAL engine_halted log line naming the reason (stranded_halt, failed_then_kept_halt), tuladb_engine_halted = 1, and tuladbd exits with code 2. Requests in the meantime get ERR_CORRUPTION. A restart replays the log capped at what the cluster committed, so the node rejoins on the cluster’s history — which is why the engine expects a supervisor: the Kubernetes chart restarts the pod; elsewhere run it under systemd with Restart=on-failure. quorum_failstop_halt_on_stranded (default on) controls it; leave it on.

You see tuladb_engine_halted, tuladb_quorum_failstop_stranded_total, tuladb_chain_repair_stranded_total, tuladb_quorum_failed_then_kept_total, and the precursor ERRORs quorum_failstop_stranded, ae_conflict_stranded, chain_repair_stranded, quorum_failed_then_kept.

Tested by test-p24b-reachability, test-quorum-failstop-mixed-suffix and test-quorum-failed-then-kept (a kept frame with a failed caller halts; one without doesn’t).

WAL hole at boot

Protects against: a node that lost a stretch of its own log answering as if it hadn’t. A WAL with a gap in both segment ids and sequence numbers, left by an older build or an operator’s mistake, would otherwise report a retained floor it doesn’t have — and an export cursor inside the hole would be told “nothing here”.

What happens. At boot the engine raises its retained floor past the hole and deletes nothing (wal_boot_hole_floor_raised). If the hole sits above what the last snapshot covers: a single node refuses to start (replay_wal_hole_refused, with the missing sequence range in the error); a clustered node starts but latches — it refuses writes, can’t be promoted, keeps voting and following (replay_wal_hole_latched), until a snapshot install covering the hole clears it. An export cursor below the raised floor gets ERR_EXPORT_RANGE_RECLAIMED rather than an empty page. To recover a single node, restore the missing segments from the WAL archive, restore a backup, or re-seed from a peer — never delete segments to “fix” it.

Tested by test-wal-boot-hole (a hole raises the floor and deletes nothing; an id gap with contiguous sequences, and a torn tail, aren’t holes; a solo node refuses; a clustered node latches; a cursor below the floor is told so).

Authentication throttling

Protects against: credential guessing on an open connection. The fifth failed MSG_AUTH on a connection (auth_max_failures_per_conn, default 5) is answered ERR_PERMISSION and the connection is closed. On thread-per-connection ingress, which includes all TLS ingress, each earlier failure is delayed by auth_failure_delay_ms (default 50) times the failure count, at most one second, without slowing other connections. A success resets the count; 0 disables either limit. There is no per-source-IP limit. You see conn_authn_throttled warnings and tuladb_auth_rejections_total. Tested by case 14 of test-authn-msg-auth, on both ingress paths. More in Security.

Watching it

The observability table’s “Is the node shedding load, and why?” row is this page in one line. A rising rejected_total on any of these gates with flat client traffic is the system doing its job; a gate that stays closed is the signal — each section above names what clears it.