Chaos and soak testing

Resilience reports the August 2026 campaigns on AWS. This page covers the harness itself — what it does to a cluster and what a run must prove — and the October 2026 runs on Google Cloud, which added the Iceberg export to what is checked.

Which build this is. The release gate ran on the 0.0.1-preview release build itself and passed; the release’s six-hour multi-region soak has not run yet — see release criteria. Every other result here comes from development builds that preceded 0.0.1-preview, each named by its commit. These are our own internal runs, not an external audit, and the verdicts below are quoted from each run’s verdict file, including the ones that say NO or NOT_MEASURED.

The harness

One harness provisions a three-node cluster on either cloud (CLOUD=aws or CLOUD=gcp), drives sustained load from every node, runs a fault schedule against it, and scores the run. It is a test harness, not a deployment path — see Deployment for what it provisions — and it ships with source access.

• AWS — Graviton instances with local NVMe (default m8gd.xlarge), one per availability zone in us-east-1.
• GCP — Arm or x86 machines with local SSD (default c4a-standard-4-lssd), one per zone in us-east4; or one per region, across three regions, with GCP_NODE_REGIONS. On GCP the run also deploys the Iceberg exporter writing to BigLake over GCS, with history retention on, so that purge, export and the lake are judged together. The AWS path doesn’t run the exporter.

Every hop runs mutual TLS, with a certificate authority generated for the run. Load is a mix of transfer kinds — single-phase, 2PC commit and rollback, expiring holds, explicit rejects, duplicates and tiered transfers — each in its own account space, so that each kind’s counts can be checked separately. Every node must report and the canary must reach 90% of the total offered rate before the timed window starts, or the run is abandoned.

The fault schedule

Each cycle runs the schedule once; every scenario is followed by a replica-agreement barrier.

Scenario	Fault	What must hold
S0 baseline	none, 5 minutes	—
S1 kill-follower	SIGKILL a follower for 60 s	The leader keeps committing with no election; the node rejoins as a voter and catches up
S2 kill-leader	SIGKILL the leader	A new leader within 120 s; the old one restarts and catches up
S3 partition-follower	Drop a follower’s traffic for 180 s	The leader keeps committing; the follower reconverges after heal
S4 partition-leader	Isolate the leader	The majority elects; the old leader steps down within 90 s and rejoins
S5 stall	SIGSTOP a follower for 45 s	It catches up after SIGCONT
S6 disk-pressure	Fill a follower’s bulk (data) filesystem to 92% for 120 s	It recovers after release
S8 double-fault	Partition a follower and kill the leader	The survivor must not elect itself without a quorum; the cluster recovers after heal
S7 rolling-restart	Graceful restart of every node in turn	Each rejoins; a leader within 120 s

There’s no clock-skew, network-delay or snapshot-install fault in this schedule. A FAULTS=0 mode runs the same load and barriers with no scenarios, as a soak.

The gates

A run’s verdict ends in one line, CHAOS_CLEAN=YES or CHAOS_CLEAN=NO with every reason. A missing measurement is never a pass: it’s NOT_MEASURED, which makes the run NO.

• Money oracle (MONEY_ORACLE). Across a dedicated account pair: debits equal credits, the pair sums to zero, the flows agree, every replica answers byte-identically, no acknowledged transfer is missing, and enough money actually moved for the run to count.
• Final agreement (FINAL_AGREEMENT). After load stops and the cluster drains, every node computes digests at one common sequence — balances, history, holds, idempotency and the registries — and they must all match.
• Core health. No mid-apply gaps; the idempotency census exact (every history record has its key and no key lacks one; replicas equal); no recovery or checkpoint fault counters; the WAL hash chain valid and verifying.
• Availability and recovery. Every leader-loss or stall window under 1,000 ms of unavailability (UNAVAILABILITY), and every restart’s recovery under 10 s.
• Steady latency. Outside fault windows, p99 under 70 ms, no batch over 1 s, and no request left unanswered. A run that changes any threshold can only score YES(custom: …).
• Run validity. The load ran, money moved and every kind was accepted; the host settings the run asked for were really in force.
• Export gates (GCP, with the exporter):
  • EXPORT_LAKE — exact: every source row of a recent window is in the lake exactly once, with the same txn_id, checked against what the engine itself serves — a check that reads no source rows, or reaches less than 95% of its window, is NOT_MEASURED, never a pass. Count: per kind and outcome, lake rows fall between what committed and what committed plus what was ambiguous.
  • EXPORT_GROUP — the partitioned workers were credited as a group for at least 95% of the minutes outside fault windows, with no non-retryable group conflicts and no worker halts.
  • EXPORT_PURGE — history purge happened, and never ran ahead of any worker’s durable checkpoint.
  • EXPORT_HEALTH — the exporter passed its health check before the window.

The release gate

3 November 2026, core 2b2f8b96 — the build tagged v0.0.1-preview. 3 × c4a-highmem-8-lssd (Arm, local SSD, WAL and data on separate devices), one per zone in us-east4; 16,667 TPS offered per node, 50,001 in all, 64 transfers per request, the full transfer-kind mix; FAULTS=1; a 4,500 s window; the Iceberg exporter writing to BigLake over GCS with history retention on (one hour). Release build flags, io_uring on, fault injection off.

CHAOS_CLEAN=YES, at the default thresholds. As recorded:

Gate	Verdict
MONEY_ORACLE	PASS
FINAL_AGREEMENT	PASS at seq 1,531,374 on all three nodes: balances, history, holds, idempotency, registry and export credit
UNAVAILABILITY	OK all 5 within 1000ms; the leader-loss windows (S2, S4, and S8 from the heal) measured 0.37–0.38 s to the first commit, with a new leader elected 0.31–0.39 s in
Recovery	every restart under 10 s; the longest, 7.7 s (S8)
STEADY_LATENCY	outside fault windows: p50 4.7 ms, p99 7.3 ms, p99.9 14.0 ms, max 131.4 ms (n = 2,670,873 batches); STEADY_SPIKES=NONE, no batch ≥ 1 s
CORE_HEALTH, APPLY_GAP, IDEMPOTENCY, INDEX_INTAKE	PASS: the WAL hash chain valid on every node, no mid-apply gaps, the idempotency census exact (keys equal history on every node), no index intake event lost
FP_BARRIERS	0 of 10 unsettled
EXPORT_LAKE	PASS, 15 gates; the exact check reduced to the 11,229 batches the source still retained (purge reclaims the WAL behind the workers’ acknowledgements)
EXPORT_GROUP, EXPORT_PURGE, EXPORT_HEALTH	PASS (21, 3 and 1 gates)
HISTORY_PURGE	exercised: 166 segments purged, the floor equal on every node, exporter lag 0 batches
RUN_VALIDITY, KIND_COVERAGE	OK: 194,534,208 transfers accepted, every kind accepted transfers

What the verdict also records, and what it doesn’t establish:

• 2,035,136 answers (about 1%, against 194,534,208 accepted) were ERR_QUORUM_TIMEOUT, and they were the run’s only non-OK answers. A quorum timeout means the outcome isn’t known yet: a client settles it by retrying with the same identity, which can’t post twice (Idempotency). The money oracle and final agreement count only what committed, and both passed.
• One run, one region. Three zones of one region; the multi-region shape is the soak’s job, and it has not run on this build.
• Not a benchmark. The load is offered at a fixed rate; delivered capacity on this hardware was not measured.

GCP results, October 2026

The 50K TPS gate shape

All four runs: 3 × c4a-highmem-8-lssd (Arm) in us-east4; 16,667 TPS offered per node, 50,001 in all, 64 transfers per request, the full transfer-kind mix; FAULTS=1; a 4,500 s window; the exporter writing to BigLake with history retention on.

Date (UTC)	Core build	Verdict, as recorded
2026-10-25	6ff631b8	MONEY_ORACLE=PASS, FINAL_AGREEMENT=PASS, UNAVAILABILITY=OK; EXPORT_LAKE=NOT_MEASURED; EXPORT_GROUP=NOT_MEASURED; RUN_VALIDITY=INVALID on the first report pass, OK on a second; CHAOS_CLEAN=NO on both — recovery >= 10s on the S7 restarts of two nodes (20.7 s and 12.8 s).
2026-10-25	1a307f61	MONEY_ORACLE=PASS, FINAL_AGREEMENT=PASS, UNAVAILABILITY=OK, RUN_VALIDITY=OK; EXPORT_LAKE=PASS with the exact check reduced to the batches the source still retained; EXPORT_GROUP=NOT_MEASURED (credited in 33 of 36 minutes, against a 95% bar); CHAOS_CLEAN=NO. Re-scored offline afterwards with a corrected harness that changed only the group gate’s minute count: EXPORT_GROUP=PASS, CHAOS_CLEAN=YES. The original verdict is kept unaltered beside the re-score.
2026-10-26	8fc9b356	MONEY_ORACLE=PASS, FINAL_AGREEMENT=PASS, EXPORT_GROUP=PASS, UNAVAILABILITY=OK all 5 within 1000ms, RUN_VALIDITY=OK; EXPORT_LAKE=NOT_MEASURED — “the source served no transfer rows in the checked range: nothing compared”; CHAOS_CLEAN=NO (also: recovery not measured for one S8 restart).
2026-10-26/27 (started 23:11 on the 26th)	d3dd6112	MONEY_ORACLE=PASS, FINAL_AGREEMENT=PASS, EXPORT_GROUP=PASS, UNAVAILABILITY=OK all 5 within 1000ms, RUN_VALIDITY=OK; EXPORT_LAKE=NOT_MEASURED — the same “nothing compared”; CHAOS_CLEAN=NO.

Why the 10-26 and 10-27 exact checks measured nothing. One node’s WAL had a hole — left by an older build’s reclaim, which freed a later segment while an earlier one stayed — and after a restart the node reported a retained floor it didn’t have, answering the exact check’s cursor “OK, empty, complete”. So the check read zero source rows and correctly refused to call that a pass. It was a core bug: the reclaim is fixed (53a9807b), and a node now detects a hole at boot instead of trusting it (WAL hole at boot). The harness now also judges the exact check on what the source actually served. Both runs’ money oracle and final agreement passed.

Earlier, at a lower rate

Date (UTC)	Core build	Shape	Verdict, as recorded
2026-10-20	4b580d22	3 × c4a-highmem-4-lssd in us-east4; 2,500 TPS per node (7,500 offered); FAULTS=1; 4,500 s; exporter on	MONEY_ORACLE=PASS, FINAL_AGREEMENT=PASS, UNAVAILABILITY=OK, EXPORT_LAKE=PASS (exact check reduced to the batches retained); CHAOS_CLEAN=YES, noting 2 of 10 barriers unsettled and not compared

Multi-region, no faults

Three fault-free one-hour runs on 2026-10-26, core 1415742f: 3 × z3-highmem-16-highlssd (x86), one voter each in us-east4, us-east1 and us-central1; 50,001 TPS offered. They compared three settings of one replication parameter, so they’re experiments rather than gates. All three: MONEY_ORACLE=PASS and FINAL_AGREEMENT=PASS. Steady p99 was 294.9 ms, 147.5 ms and 229.4 ms; only the middle setting scored CHAOS_CLEAN=YES, and that against a relaxed 165 ms threshold for cross-region links (the default is 70 ms). One run per setting, so treat the differences as a lead, not a result.

Soaks

• 72-hour development soak. Launched 2026-10-21 on core 4b580d22: 3 × c4a-highmem-8-lssd in us-east4, FAULTS=0. It was stopped so its findings could be fixed, and wasn’t rerun. There is no final report, so we state no duration or result for it. No 72-hour run covers 0.0.1-preview.
• Six-hour multi-region soak. An attempt on 2026-10-25 (core 1a307f61, the three-region z3 shape) produced no verdict. It is not a result.
• Single-node soak. A separate GCP harness runs one node for hours under a nine-kind mixed load, then restarts it and re-reads every balance; its gates cover coverage of the target rate, money per account space, idempotent resubmits, mTLS posture, host memory pressure, WAL write and fsync tails, and the exporter. We publish no single-node soak result.

Release criteria

The criteria set for the 0.0.1-preview release are, on the release head itself: one 50K TPS chaos gate — FAULTS=1, a 4,500 s window, the exporter and its gates on — then one six-hour multi-region soak.

Criterion	Status
50K TPS chaos gate	passed, 3 November 2026 (above)
Six-hour multi-region soak	not run yet

When the soak runs, its verdict will be published here the same way, NO and NOT_MEASURED included. Until then, read the multi-region latency known issue before planning a cross-region cluster.