Analytics export

Operational reads belong on the query port; warehouse analytics don’t belong on the cluster at all. The TulaDB Iceberg exporter streams committed transaction history into Apache Iceberg tables, atomically committed, so finance, BI and data science query the ledger’s truth in the lakehouse without ever touching the ledger — and the dashboard can query those tables directly.

How it stays correct

The pipeline is built on the same discipline as the engine:

committed ledger history (replicated watermark)
  → export cursor on the read-only query port (7800)
  → micro-batch → Parquet data files
  → atomic Iceberg append commit (range + digest recorded in the snapshot)
  → checkpoint advances only after the commit succeeds
  → durable ack to the engine

• Only committed state is exported. The cursor reads applied, committed state on the query port — the exporter is downstream of commit and never participates in the write path. It is given every node’s query address, reads from the one that leads, and follows leadership changes on its own.
• Exactly-once into the table. Reading is at-least-once by construction: after a crash the exporter resumes from its durable checkpoint and may read a range again. Committing is idempotent and fail-closed: every commit records its range — keyed by the ledger’s own order (global_batch_seq, array_index) — and a SHA-256 digest of its rows in the Iceberg snapshot. Before appending, the exporter reloads the table: the same range with the same digest is already there and is skipped; the same range with a different digest, or any overlap, stops the exporter rather than write a duplicate or a gap.
• Writer fencing. Every commit checks a writer_epoch, and a commit is refused once a newer epoch has committed. Active/passive fencing by a lease (a local file, or a SQL table for a fleet) is opt-in ([fencing] enabled); the shipped Kubernetes example turns it on for its two replicas, and a writer that lost its lease fails closed rather than race.
• Retention coordination. The exporter registers a lease so WAL a lagging export still needs isn’t reclaimed underneath it. Leases are off on the engine until you set export_lease_ttl_ms > 0, and leasing needs the exporter role for the exporter’s principal. When a hold can’t be honoured the page still arrives, flagged: LEASE_NOT_DURABLE (the hold is in memory only and won’t survive a restart of that node), EXPORT_HOLD_RELEASED (WAL ring pressure or export_lease_max_hold_batches overrode it) or NO_LEASE (the lease table is full, so this page holds nothing). The exporter counts each — see metrics. If a cursor does fall below the reclaim floor, the answer is an explicit ERR_EXPORT_RANGE_RECLAIMED; the exporter prints the exact missing range and exits (code 87), rather than silently skipping records. Recovery is an operator decision: reseed at the floor into a new table generation, which makes the gap explicit, or bootstrap a fresh pipeline. A bootstrap seeds accounts and registries, not the missing transfers.
• Durable acks, not fetch positions. A pipelined exporter fetches ahead of its commits, so its read cursor isn’t an export position. It sends a separate durable ack of what it has committed to the table, only after its checkpoint is on disk, and only that ack credits history retention.

What lands in the table

Each row is one transaction record: the ledger coordinates (global_batch_seq, batch_seq, array_index), timestamp, transfer_id, txn_id, ledger_id, source_id, currency, both account ids, amount, outcome, and settlement columns for tiered ledgers. Rejected outcomes are rows too, so decline analytics come free; the caller’s reject reason code is filled in when full-stream mode (include_metadata = true) is on, and is empty otherwise. A full-record mode (full_txn_record) adds both accounts’ post-apply balances to each row — seven columns: dr_credits_posted, dr_debits_posted, cr_credits_posted, cr_debits_posted, dr_available, cr_available and digest_hex; lake reconciliation needs it. It requires include_metadata = true (the config is refused without it), and because it changes the table schema, the transactions table must be created with it on: turning it on for an existing table needs a new table (or an explicit schema evolution).

Layout (sink.transactions_layout) is fixed when a pipeline starts and recorded in its checkpoint, so it can’t silently change mid-stream:

• by_type (the default for a new pipeline) — one table per kind: single-phase, 2PC reserve, 2PC resolution, rejected, and the four tiered equivalents.
• by_mode — one table per ledger mode (default and position–settlement).
• shared — one table for everything.

Optional metadata changelogs (include_metadata) land account, ledger, currency, source and cluster changes as their own tables, in the same canonical order as the transfers. A watermark table records, per pipeline and partition, the position up to which the lake is complete. Tables are created unpartitioned and unsorted.

Where the lake lives

The exporter talks to catalogs through PyIceberg: the [iceberg.catalog] table is passed to PyIceberg as it is, and the warehouse URI picks the storage. Three targets are tested:

Target	Catalog	Storage	Credentials	How it’s been tested
AWS S3 Tables	S3 Tables’ Iceberg REST endpoint, SigV4-signed	The table bucket	The AWS default chain: instance role, IRSA, environment, shared config	Live, by a manual billable workflow; verified end to end on 2026-08-10
Google BigLake	BigLake’s Iceberg REST catalog	A GCS bucket (gs://…), or a BigLake catalog (bl://…)	Google Application Default Credentials: Workload Identity on GKE, the attached service account on GCE, GOOGLE_APPLICATION_CREDENTIALS, or a key file	Live, by a manual billable workflow, first on 2026-09-27; also the exporter in our GCP chaos gates
Local	A SQL catalog (SQLite)	A file:// warehouse	—	Every CI run

Other PyIceberg catalogs — AWS Glue, Nessie, JDBC, any other REST catalog — receive the same pass-through configuration, but we haven’t tested them, and the exporter has no Glue-, Nessie- or JDBC-specific checks. There is no MinIO or S3-compatible, Azure, or HMAC-key path in the test matrix. Exported S3 Tables data has been read back with PyIceberg, DuckDB and Athena (a live S3 Tables soak, 2026-08-02).

AWS: S3 Tables

[source]
hosts = ["node1.internal:7800", "node2.internal:7800", "node3.internal:7800"]
include_metadata = true          # account/ledger/currency/source changes, in stream order
register_export_lease = true     # needs export_lease_ttl_ms > 0 on the engine

[iceberg]
enabled = true
table_identifier = "ledger.transactions"
writer_epoch = 1

[iceberg.catalog]
type = "rest"
uri = "https://s3tables.us-east-1.amazonaws.com/iceberg"
warehouse = "arn:aws:s3tables:us-east-1:<account-id>:bucket/<your-table-bucket>"
"rest.sigv4-enabled" = "true"
"rest.signing-name" = "s3tables"
"rest.signing-region" = "us-east-1"
"s3.region" = "us-east-1"
"py-io-impl" = "pyiceberg.io.fsspec.FsspecFileIO"   # data files signed through boto3

Install with the s3tables extra. Set py-io-impl as shown: PyArrow’s own S3 client signs the S3 Tables backing bucket in a form it rejects, mid-commit. Leave the exporter’s own compact and cleanup off on S3 Tables — S3 Tables maintains its tables itself, and the two would contend.

GCP: BigLake over GCS

[iceberg.catalog]
type = "rest"
uri = "https://biglake.googleapis.com/iceberg/v1/restcatalog"
warehouse = "gs://<your-lake-bucket>"
"header.x-goog-user-project" = "<your-gcp-project>"   # required: the project billed for catalog calls

[iceberg.catalog.auth]
type = "google"                                        # Application Default Credentials

Install with the biglake extra. The principal needs BigLake editor (admin to create tables) on the project and storage.objectUser on the bucket. Two BigLake behaviours shape the deployment:

• Cleanup is mandatory, every minute. BigLake doesn’t expire snapshots and refuses a commit that would take a table’s metadata.json past 1 MB — about 700 export snapshots, roughly twelve minutes of commits under load. Schedule cleanup --execute every minute with min_snapshot_age_seconds of 300 or less; doctor warns about a policy that can’t keep up. If cleanup is missed, the exporter fails closed with a message naming the size limit, rather than retrying: run cleanup, restart, and it resumes from its checkpoint with nothing lost.
• Full snapshot history. BigLake returns only referenced snapshots unless asked; the exporter always asks for all of them on a REST catalog and refuses a config that doesn’t, because duplicate detection and reconciliation need the full history.

The exporter caches the Google OAuth token rather than fetching one per catalog request.

History retention needs an exporter

History retention (history_retention_hours > 0) is export-gated by default (history_purge_requires_export = true): the leader purges only what every registered export pipeline has durably acknowledged to it, in its current term. With no pipeline acking the leader, nothing is purged — history and the idempotency index keep growing. That is why the exporter reads from the leader by default; pointed at a follower (allow_follower_reads), its acks don’t count and purge fails safe by keeping history. Setting history_purge_requires_export = false purges by age alone, including history not yet exported.

Partitioned workers

One lake can be fed by N workers, each taking the committed positions where global_batch_seq % N equals its index (source.partition_count, source.partition_index) — the union is every record by construction, and N is recorded in each worker’s checkpoint and refused if it changes (exit code 88). Each worker writes its own tables, checkpoint and lease (suffixed _p<i>). Workers of one lake page as a partition group, so the engine credits the lake with the lowest of their durable acks, and only while every member is present; without the group a partitioned lake earns no purge credit at all. A refused group page answers ERR_EXPORT_GROUP_CONFLICT with a sub-code saying why. Partition groups turn on by themselves for 2 to 32 leased workers reading from the leader.

Running it

pip install '.[iceberg]'         # from the exporter distribution; [s3tables] or [biglake] for the clouds

tuladb-iceberg-exporter doctor    --config exporter.toml   # validate the config, flag common S3 Tables and BigLake mistakes
tuladb-iceberg-exporter bootstrap --config exporter.toml   # seed current state, then hand off to the stream
tuladb-iceberg-exporter run       --config exporter.toml   # continuous
tuladb-iceberg-exporter watermark --config exporter.toml   # the position up to which the lake is complete

doctor validates the resolved config and flags common S3 Tables and BigLake mistakes before you commit to a deployment; it makes no network calls, so the first once run is the connectivity test. The distribution includes a container image, hardened systemd units (the service, a metrics service, and cleanup and compaction timers) and Kubernetes templates (a two-replica active/passive Deployment, health endpoints, Prometheus rules, and a BigLake cleanup CronJob). No TulaDB installer or Helm chart deploys the exporter: you run it next to your cluster and grant its certificate the exporter role. Three exit codes mean “a person must look”, and the systemd unit doesn’t restart on them: 86 (a durability halt, cleared with clear-durability-halt --confirm), 87 (source range reclaimed), 88 (partition-group conflict or identity mismatch).

Verifying the lake

• reconcile compares the exporter’s own commit log with the snapshots in the table and with the table’s contents — record counts and amount sums must match exactly — and exits 0 (PASS) or 2 (FAIL). It checks every transactions table the layout writes (shared, by_type or by_mode), each against its own commit log. It proves the lake agrees with the exporter’s record of what it committed; comparing against the engine itself is repair’s job.
• repair --from --to re-reads a range from the engine, adds missing rows and removes duplicates in one commit, then re-checks that every expected row is present exactly once. --dry-run plans without writing. The dashboard can drive it automatically (lake reconciliation).
• watermark exits 0 only when the lake is complete up to a published position.
• In our chaos gates the lake is held to the source independently of the exporter: every source row of a recent window must be in the lake exactly once with the same txn_id, and per-kind row counts must fall between what committed and what was ambiguous — see chaos and soak testing.

Maintenance is deliberately conservative: both compact and cleanup are plan-only without --execute. Executing compact needs an explicit cold range by default and re-checks count and sums after the rewrite; opt-in auto-compaction (compaction.auto_interval_s) picks its own cold range between commits. cleanup expires snapshots; orphan-file deletion isn’t implemented.

Metrics

The exporter serves its own Prometheus metrics (metrics subcommand, default 127.0.0.1:9146; /metrics and /healthz, which answers 503 when the snapshot is stale). Every series is tuladb_iceberg_exporter_<name>, labelled by cluster, node and pipeline. The ones to alert on:

Question	Metrics
Is it running and current?	up, snapshot_stale, running, last_success_unix
How far behind is the lake?	export_lag_batches, checkpoint_global_batch_seq, committed_global_batch_seq
How close is it to losing its source?	export_headroom_batches, export_cursor_detached, export_range_reclaimed_total, export_gap_batches
Did it stop to protect the table?	export_fail_closed_total, export_errors_total
Is the WAL hold real?	export_lease_not_durable_total, export_no_lease_total, export_hold_released_total
Is fencing healthy?	fencing_active, fencing_lost_total
Are partitioned workers grouped?	partition_group_status, partition_group_members_present, partition_group_conflicts_total

export_lease_not_durable_total and export_no_lease_total count pages served with a hold that won’t survive the source’s restart, or with no hold at all. Both are advisory — the page is still exported, and a warning is logged at most once a minute — but a sustained rise means a lagging exporter can lose its source range. The shipped Prometheus rules don’t alert on these two yet; add your own.

Sizing honestly

The exporter’s committed measurements are all against a local Parquet sink, not a cloud catalog. In August 2026 a single pipeline drained full transaction records at 44,621 records/sec, sustained 39,755 records/sec for five minutes at 20K clearing transactions per second with lag flat at 3 to 6 batches (the report leaves an unexplained drop in its last 30 seconds open), and topped out near 59.5K records/sec in one process. (An earlier July figure of ~107K records/sec used the slim stream, before full records and per-type tables.) Throughput against your own catalog and storage can differ, so measure it; standby replicas add availability, not throughput. Above one worker’s rate, split the stream across partitioned workers — we haven’t published a scaling figure for them either. How far the lakehouse trails the ledger is the exporter’s lag metric, so watch it rather than assume it.