Querying the lake from the dashboard

The analytics exporter lands committed history in Apache Iceberg. The operations dashboard can then read those tables directly — no warehouse engine in between — so an operator can search years of history, run SQL against it, and check that the lake still agrees with the ledger, without putting that load on the cluster.

It is off by default. Nothing in the installers or the Helm charts turns it on: you point the dashboard’s backend at a catalog yourself, with the environment variables below.

How it reads Iceberg

The backend embeds DuckDB and attaches the lake with DuckDB’s own iceberg extension (with httpfs, and aws for S3 Tables). No PyIceberg, Trino, Athena or BigQuery is involved. Every query runs in a separate worker process with its own memory and thread caps and a per-query timeout; a worker that doesn’t answer is killed and replaced, and workers are recycled before their cloud credentials expire.

Three catalog types are supported (TULADB_LAKE_CATALOG_TYPE):

Type	What it attaches	Credentials
rest	An Iceberg REST catalog. The default endpoint is Google BigLake’s (https://biglake.googleapis.com/iceberg/v1/restcatalog), with data on GCS	TULADB_LAKE_AUTH = "google" (the default): Application Default Credentials, passed to DuckDB as a catalog token and a GCS bearer token. none sends nothing
s3tables	An AWS S3 Tables table bucket, by its ARN	The AWS default credential chain (an IAM role in production), frozen into a DuckDB S3 secret for the bucket’s region
duckdb	A local DuckDB file	None — for tests and offline demos

That is the whole list. With rest, the dashboard passes object-store credentials only in the Google form, so a REST catalog over S3 or another store that needs credentials isn’t a supported configuration; on AWS, use s3tables.

Read-only, in layers

The Iceberg side is read-only, and that isn’t a single check:

1. The lake is attached READ_ONLY.
2. The worker then limits file access to the warehouse alone (for S3 Tables, to each table’s own location, listed through the S3 Tables API), turns external access off and locks the configuration.
3. Only one statement per request, and it must be SELECT or EXPLAIN — classified by DuckDB’s own parser, never by text matching, and checked in both the API process and the worker.
4. Give the dashboard a read-only principal. On GCP: BigLake viewer plus storage.objectViewer. On AWS: s3tables:Get* and s3tables:List* on the table bucket. Layers 1–3 are defence in depth, not a reason to hand the worker write credentials.

Results are capped — 500 rows by default, at most 10,000 (TULADB_LAKE_DEFAULT_ROWS, TULADB_LAKE_MAX_ROWS) and 8 MiB — and a capped result is shown as incomplete, never as a total. Integers beyond JavaScript’s exact range, decimals and binary values come back as strings, so an amount or id is never silently rounded.

Read-only describes the lake connection, not the dashboard as a whole. The dashboard is also the ledger’s operator console — creating ledgers and accounts, posting transfers — over the engine’s own protocol with its client certificate, as on the local stacks in the quickstart. Two lake features also cause writes elsewhere, both opt-in: scheduled reports can upload CSV files to a bucket, and reconciliation can ask the exporter to repair the lake (below).

What it offers

• The Lake page — the table browser, the SQL terminal, and the watermark: the position up to which the lake is complete, as the exporter publishes it. Browsing is open to every dashboard role; running SQL needs operator or above (TULADB_LAKE_QUERY_ROLE), and every query goes to the dashboard’s audit log: the first 256 characters of its SQL plus a SHA-256 of the full text, the row count, and any refusal.
• Lake-backed history. The completed-transfers view of a ledger reads history from the lake and only the tail past the watermark from the engine, so a deep history search doesn’t scan the cluster. Account history and the ledger, source and registry lists fall back to the lake when the engine is unreachable.
• Reconciliation (/lake/reconcile). For every account of a ledger, the engine’s posted balance now, minus what it posted after the watermark, must equal the balance the lake holds at the watermark; and each lake row’s post-apply balance must equal the previous row’s plus that row’s posting, so a missing or duplicated row shows as a break. It needs the exporter’s full-record mode (post-apply balances on every row) and refuses a lake without them. Pending balances aren’t compared.
• Saved queries and scheduled reports — named, parameterised read-only SQL, run hourly, daily or weekly. A report is held, not delivered, while its ledger has an open reconciliation incident, while the lake doesn’t yet cover the period, or when a reconciliation run right before delivery fails. Delivery is a CSV kept on the server, optionally uploaded to TULADB_REPORTS_BUCKET (gs:// or s3://) and announced to a webhook.
• Automatic repair, only when you set TULADB_EXPORTER_CONTROL_URL (and its token) to the exporter’s control API: a confirmed gap makes reconciliation ask the exporter to re-export the affected range from the engine, then re-run reconciliation. A balance mismatch with continuous rows, an account missing on either side, or an unreachable exporter is reported, never auto-repaired.

Configuration

Variable	Default	Meaning
TULADB_LAKE_CATALOG_TYPE	(empty — lake off)	rest, s3tables or duckdb
TULADB_LAKE_ENDPOINT	BigLake’s REST endpoint	REST catalog URL
TULADB_LAKE_WAREHOUSE	—	gs://<bucket> or a BigLake catalog for rest; the table-bucket ARN for s3tables
TULADB_LAKE_NAMESPACE	—	The exporter’s namespace
TULADB_LAKE_AUTH	google (aws for s3tables)	Credential source
TULADB_LAKE_REGION	from the ARN	S3 Tables region override
TULADB_LAKE_QUERY_ROLE	operator	Minimum dashboard role for the SQL terminal
TULADB_LAKE_TIMEOUT_S, _MEMORY_LIMIT, _THREADS	30, 1GB, 2	Per-query timeout and worker caps
TULADB_LAKE_MAX_RESULT_BYTES	8388608 (8 MiB)	Result-size cap for one query (alongside TULADB_LAKE_DEFAULT_ROWS 500 and TULADB_LAKE_MAX_ROWS 10,000)
TULADB_LAKE_PIPELINE_ID	—	Which export pipeline’s watermark to read; required when several pipelines publish to the namespace
TULADB_LAKE_TABLE_SUFFIX	—	The exporter’s run suffix (e.g. _g10210137); the watermark table is then export_watermark<suffix>
TULADB_LAKE_WATERMARK_TABLE	—	The exact watermark table, no discovery; wins over TULADB_LAKE_TABLE_SUFFIX. With neither set, the dashboard uses the one watermark table it finds, and with several it lists them and claims no watermark
TULADB_EXPORTER_CONTROL_URL	—	The exporter’s control API, for automatic repair
TULADB_EXPORTER_CONTROL_TOKEN	—	Bearer token for that API — the same value the exporter reads

The backend image includes the lake dependencies and pre-installed DuckDB extensions by default.

The exporter’s control API is off until you give it an address in the exporter’s config ([control] listen; an address with no host binds 127.0.0.1). It needs a bearer token, read from the environment variable [control] token_env names — TULADB_EXPORTER_CONTROL_TOKEN by default — and refuses to start without one. It has four routes: POST /repairs with {"from_gseq": A, "to_gseq": B, "dry_run": false} queues a re-export of that range and answers 202 with a job, GET /repairs/<id> reports a job, GET /repairs lists recent ones, and GET /healthz. A repair runs on the exporter’s writer thread between commits, so it never races the pipeline’s own writes.

# GCP: BigLake catalog over GCS, ADC from the VM's or pod's service account
TULADB_LAKE_CATALOG_TYPE=rest
TULADB_LAKE_WAREHOUSE=gs://<your-lake-bucket>
TULADB_LAKE_NAMESPACE=<exporter-namespace>

# AWS: S3 Tables, credentials from the default chain (an IAM role)
TULADB_LAKE_CATALOG_TYPE=s3tables
TULADB_LAKE_WAREHOUSE=arn:aws:s3tables:<region>:<account-id>:bucket/<table-bucket>
TULADB_LAKE_NAMESPACE=<exporter-namespace>

What it’s been tested against

The unit tests run against local DuckDB files; the S3 Tables code paths are covered by mocks there. The cloud paths are covered by a live end-to-end script — a real engine, the real exporter and the dashboard — which is billable and doesn’t run in CI. Its recorded runs of 2026-09-27 passed every check on both supported clouds: 53 on Google BigLake and 30 on AWS S3 Tables. One phase that stops the engine was added afterwards and has only run offline.

Treat this as a developer-preview feature, like the rest of the platform: no installer or chart configures it, so a cloud deployment of it is yours to set up and validate.