Docs/Deploy & operatev0.0.1-preview
Resilience
The EC2 chaos campaigns — exact conservation through kills, partitions, and double-faults, with the throughput data and its limits stated honestly.
The claim that matters for a ledger is not “fast” — it’s correct while things fail. So the evidence we publish first is a fault campaign, not a horse race: the full platform on EC2 across three availability zones, deliberately overloaded on the worst-case workload shape, while nodes are killed, partitioned, stalled, and starved of disk. These are our own internal campaigns, not an external audit.
Which build this is. Everything on this page comes from campaigns run in August 2026 on development builds that preceded the 0.0.1-preview release. The engine has changed substantially since — one write path, the account key, history retention, full-copy node joins — so read these results as evidence about the design and the method, not as validation of 0.0.1-preview. The release’s own chaos gate ran on the release build on 3 November 2026 and passed (the verdict); its six-hour multi-region soak has not run yet.
The verdict
From the consolidated campaign report of 2026-08-29 (which supersedes the earlier 08-25 and 08-26 reports):
36 scenario gates per fleet, 0 failures, reproduced on two independently provisioned fleets — 72 gates and four clean cycles in all, on fresh clusters that re-roll leader placement, AZ assignment, and timing. Prometheus covered 100% of both windows.
Correctness never moved. Across roughly a dozen hours of continuous faults, money was conserved exactly — ~31.9M units moved per window on each of the two fleets (31,915,520 and 31,897,568), debits equal to credits to the unit, zero acknowledged transfers lost. Every defect the campaigns found (and fixed) was an availability defect, never a correctness one.
The setup
| Value | |
|---|---|
| Ledger nodes | 3 × m8gd.2xlarge, one per AZ (us-east-1a/b/c) |
| Monitor | 1 × t4g.medium (Prometheus + Grafana, mTLS) |
| Security | Mutual TLS on every hop |
| Dedup | Run-tier backend, query indexes on |
| Offered load | 6,667 TPS per node → 20,001 aggregate, all against one account, 16 transfers per request |
| Window | 3,600 s, two full cycles of the S0–S8 fault schedule |
| Cost | ≈ $1.29/hour |
The workload is deliberately the pessimal shape — a single hot account is the one thing a distributed ledger cannot parallelize — and the offered rate is deliberately above what the cluster absorbs, so shedding, queueing, and catch-up are exercised continuously rather than occasionally.
The fault schedule
Each one-hour window cycles through the full schedule twice. What each scenario does, as the harness’s timeline script defines it today:
- S1 kill-follower —
SIGKILLa follower, then restart it - S2 kill-leader —
SIGKILLthe leader, then restart it - S3/S4 partition — cut a follower, then the leader, off the network with firewall drops, then heal
- S5 stall — freeze a node with
SIGSTOP, then resume it - S6 disk-pressure — fill a follower’s bulk (data) filesystem to 92% for two minutes, then release it
- S7 rolling restart — restart every node in turn
- S8 double-fault — kill the leader and partition a follower at once
After every scenario, the replicas’ state fingerprints are compared at a common position. Every one of the 36 gates per fleet passed, on both fleets.
Throughput data from the campaigns
Supporting data, not a benchmark — throughput here is derived from the money oracle (units moved ÷ window), so it counts only transfers that actually committed. Every campaign run sent 16 transfers per request:
| Measurement | Offered | Delivered | Conditions |
|---|---|---|---|
| Undegraded canary — one node, 60 s, no faults | 6,667 | 6,665 TPS | 0 shed, 0 errors, p99 12.3–14.3 ms |
| Full fault schedule — 3 nodes, 1 hour | 20,001 | 7,400–9,300 TPS sustained | across six runs, single account |
Read it with its limits:
- The single-node figure is a floor, not a ceiling. The canary met its target — it never pushed past 6,667 — so it does not measure the hardware’s capacity.
- The three-node figures are not a ceiling either. They were measured under deliberate overload and a continuous fault schedule, at 16 transfers per request. Undegraded capacity on this hardware is not published yet.
- A single account is the worst case by design. An account is a serialization point, and its budget is cluster-aggregate — see the hot-account rule. Aggregate throughput across many accounts is a different, much higher number.
- Figures are instance-class-dependent. The preceding campaign ran m8gd.xlarge at a different offered rate and measured materially lower figures. Never carry a figure across hardware.
- We publish no performance benchmark yet, deliberately. When a benchmark campaign is published, its numbers will come the same way — with the date, the build, the hardware, the workload shape, and what they don’t establish.
Stated plainly: what’s still open
One clean window has repeatedly proven worthless in these campaigns — a clean first cycle concealed a failing second one more than once, which is why the published verdict requires repeated cycles on independently provisioned fleets. And one defect was still open when that report was written: an election livelock seen once in nine runs, then unexplained — it froze commits until a third node returned, and money was still conserved exactly through it. The report’s 2026-09-25 update still lists its cause as unexplained.
Since August
The October 2026 runs on Google Cloud — at 50K TPS offered, with the Iceberg export and its lake-versus-ledger gates added — the 0.0.1-preview release gate of 3 November 2026 (CHAOS_CLEAN=YES on the release build), and the harness’s full gate list are on Chaos and soak testing.
Reproducing
The campaign harness — fleet provisioning, the S0–S8 fault timeline, a repeat driver for independent fleets, and an unattended-safety watchdog — ships with source access. Design partners can request the full campaign reports and evidence bundles; see design partners.