All notes

Performance ·

One hot account is a serialization point

A single account's throughput is bounded by the replicated write path, cluster-wide — and we learned the hard way what it takes to measure that number honestly.

An account is a serialization point. Every mutation of it must apply in sequence, on every node — that’s not an implementation shortcut, it’s what makes a balance a balance. So a single hot account has a throughput ceiling that is a property of the cluster’s replicated write path, not of any one node. This post is about what we’ve actually measured of that ceiling — including a correction, because the first number we measured turned out to need one.

Developer-preview note: the figures below come from the 2026-08-26 → 08-29 chaos campaign on 3× m8gd.2xlarge across three AWS availability zones, with durable dedup, query indexes, and mTLS on every hop, on a development build that predates 0.0.1-preview. Every campaign run sent 16 transfers per request. Per-account throughput is instance-class-, configuration- and batch-size-dependent — never quote a figure without all three attached.

What the current campaign establishes

Undegraded, single node (60-second canary): offered 6,667 TPS on one account, delivered 6,665 TPS — zero shed, zero errors, p99 12.3–14.3 ms. This is a target-met result, not a capacity limit: the canary never pushed past its target, so it establishes a floor for capacity, not a ceiling.

Through the full fault schedule (3 nodes, 20,001 TPS offered): sustained ~7,400–9,300 TPS aggregate on a single account across six independent runs — including every node kill, partition, stall, disk-pressure event, and double-fault in the schedule, and the shedding and catch-up they trigger. The throughput figure is derived from the money oracle (units actually moved ÷ window), so it counts only committed transfers — and money was conserved exactly in every run.

The 37–46%-of-offered ratio is deliberate: load is set above what the cluster can absorb so that shedding, queueing, and recovery are exercised continuously rather than occasionally. And one boundary stated plainly: a run under deliberate overload and continuous faults does not measure the cluster’s ceiling. Undegraded capacity on this hardware is not published yet.

The correction, because honesty is the product

An earlier campaign (2026-08-25, on smaller m8gd.xlarge instances at 15,000 TPS offered) measured a much lower delivered rate on one account, with pathological tail latency — and for a while we treated that as the definitive picture of hot-account collapse.

Two things were wrong with how that number was being used. First, it was quoted without its instance class, and per-account throughput doesn’t transfer across hardware — on the larger instances, a single node now delivers more than the older three-node figure. Second, later analysis attributed part of the collapse to something fixable: an internal dedup handoff queue sized at 1M slots overflowed more than seven million times at that offered rate, diverting traffic to a slower path and eventually latching backpressure that halted commits entirely. Overflow handling was correct — nothing was lost, and conservation held throughout — but the queue was undersized for the rate. The campaign’s profile has since sized it at 8M slots with wider sharding (dedup_index_queue_slots, dedup_run_shards), which is part of why the newer campaign sustains what it sustains.

The lesson we’ve internalized, and the reason our resilience page never quotes a figure without its hardware and fault posture attached: a throughput figure without its hardware, configuration, batch size and fault posture attached isn’t information — it’s a future correction.

What still holds, regardless of the number

The measurement got better; the design guidance never changed, because it follows from the architecture rather than from any specific figure:

  • A hot account’s budget is cluster-aggregate, not per-node. Pointing more nodes at the same account doesn’t multiply its throughput — every mutation still serializes through one replicated sequence.
  • Fan out hot balances. A merchant settlement account absorbing more than one account can serialize becomes N sub-accounts, swept periodically — the sweep is itself just transfers, so it stays balanced and auditable.
  • Shed early rather than queue into collapse. write_admit_max_peer_lag sheds client writes while a voter is behind — trading throughput you weren’t really getting for a cluster that keeps its ability to heal a member under sustained overload.
  • Measure on your own hardware. The published campaign figures are derived from the money oracle — conservation checked while throughput is measured — and a sizing number for your workload deserves the same standard.

In the docs: Limits and sizing carries the capacity guidance; How we verify correctness covers the campaign methodology these figures come from.

Evaluate it yourself

TulaDB 0.0.1-preview is a signed download. Run it locally, check the behaviour this note describes, then bring your workload to a pilot.