Observability

You can’t run a system of record on vibes. Every TulaDB node continuously publishes its internal state, the deploy stacks ship a working Prometheus + Grafana pipeline, and the alert rules encode the operational judgment — what’s worth waking someone for — so you don’t have to rediscover it.

Metrics pipeline

The daemon atomically snapshots its metrics to <data_dir>/runtime/metrics.json (--metrics-snapshot PATH moves it); a sidecar exporter renders that file as several hundred Prometheus series on port 9145 (/metrics, /healthz; it binds 127.0.0.1 unless told otherwise). Collection is decoupled from exposition — a slow scraper can’t touch the engine, and a wedged engine is visible as a stale snapshot (tuladb_exporter_snapshot_age_seconds) rather than a silent scrape gap.

The exporter endpoint is intentionally unauthenticated and unencrypted — run it as a sidecar on a private network, never exposed. The engine can also export traces over OTLP/HTTP (otlp_endpoint); metrics reach an OpenTelemetry pipeline through the OTel Collector scraping the exporter, which the observability stack configures.

The metrics that answer real questions

Question	Metrics
Is the node alive and serving?	tuladb_up, tuladb_engine_halted, tuladb_cluster_node_ready (+ _reason: no leader / no quorum / metadata fault)
Who leads, and is leadership stable?	tuladb_cluster_leader_id, tuladb_node_role, tuladb_raft_current_term, tuladb_cluster_leader_changes_total
Is replication healthy?	tuladb_commit_index, tuladb_apply_index, tuladb_follower_lag_entries, tuladb_cluster_quorum_timeouts_total
Is durability keeping up?	tuladb_wal_fsync_total, tuladb_wal_fsync_errors_total, tuladb_wal_unflushed_entries, tuladb_wal_enospc_total, tuladb_disk_free_bytes_min
What’s the ledger doing?	tuladb_transfers_total, tuladb_duplicates_total, tuladb_rejected_total, tuladb_pending_reservations
End-to-end latency?	tuladb_server_e2e_latency_seconds histogram
Is the node shedding load, and why? (each gate)	tuladb_server_write_slo_rejected_total, tuladb_disk_capacity_rejected_total, tuladb_disk_reserve_bytes, tuladb_memory_budget_rejected_total, tuladb_segment_capacity_blocked
Is history retention working?	tuladb_history_floor_seq, tuladb_history_purge_blocked (+ _reason), tuladb_history_oldest_retained_age_seconds, tuladb_history_export_unacked_leader_pipelines
Anything growing without bound?	tuladb_idempotency_live_entries, tuladb_history_transfers_retained, tuladb_wal_archive_* backlog
Did recovery leave a hole?	tuladb_replay_hole_seq, tuladb_segment_emit_hole_present, tuladb_dedup_rebuild_missing — each should be zero
Will TLS bite us?	tuladb_tls_cert_days_remaining, tuladb_tls_cert_expired

A node that joined by snapshot install reports its history sync in the metrics snapshot (fullsync_state, join_history_ok) and through tulactl join-status; autotune decisions appear as tuladb_autotune_* (see Autotune).

tuladb_duplicates_total deserves a dashboard panel of its own: it’s a direct read on how often your delivery layer redelivers — a spike means a client fleet is retrying, and the ledger is absorbing it correctly.

Shipped dashboards and alerts

The six Grafana dashboards are a download of their own, grafana-dashboards (how to install them); the alert rules live in the deploy tooling and are not in a published download in 0.0.1-preview (design partners receive them on request). Both are written against the metrics the engine actually emits: a lint checks every metric name in the rules against the exporter source, and promtool unit tests check that the key alerts fire when they should.

What ships	Count
Prometheus alert rules	156 rules (146 distinct alerts; 10 have a warning and a critical tier) in 12 groups
Of those, page-worthy	64 critical
Runbook links	every rule carries a runbook_url
Grafana dashboards	6 dashboards, 209 panels
Recording rules	none: every alert reads raw engine series

The rule groups cover node health, leadership and quorum, replication lag, WAL durability, checkpoints, disk and capacity, load shedding and the latency budget, idempotency, history retention and export coverage, clock skew, TLS and auth, and the alert pipeline itself. One group, money integrity, holds 12 critical rules that should never fire on a healthy cluster: chain divergence between replicas, replay and apply holes, a checkpoint over a hole, a quorum fail-stop left stranded, and a replica refusing to apply.

The six dashboards are an overview (with “must be 0” integrity panels: rule violations, recount drift, cross-node agreement), cluster and consensus, durability and WAL (fsync p50 to p999, checkpoint age and failures), engine throughput and per-operation latency, runtime and security (certificate days left, auth rejections by reason), and the dedup run tier.

The alert pipeline watches itself: a Watchdog dead-man’s switch, plus alerts for a missing exporter target, a failed rule evaluation and a failed notification. The production Alertmanager configuration routes critical alerts to PagerDuty and warnings to Slack, and inhibits the noise around a halted node, so one dead node doesn’t page twenty times. The default local configuration notifies no one.

Where they come from:

• Compose observability stack: Prometheus, Alertmanager, node-exporter, an OpenTelemetry Collector and Grafana, all over HTTPS against the local CA, with the six dashboards provisioned.
• Helm chart: plugs into a Prometheus Operator you already run. It renders ServiceMonitors, a PrometheusRule with all 156 rules, an AlertmanagerConfig and a node-exporter DaemonSet; it does not install Prometheus or Grafana.
• Iceberg exporter: its lag, fail-closed and source-reclaimed alerts ship with the exporter, not in this set (Analytics export).

Start from these rules and tune thresholds to your SLOs.

Readiness for orchestrators

Kubernetes probes and load balancers should gate on tulactl ready (or the node_ready metric), which means “has a leader and is caught up” — not on a TCP port check. The engine image carries no tulactl, so inside a container read node_ready from <data_dir>/runtime/metrics.json instead — the engine chart’s readiness probe greps it for "node_ready": 1. This distinction is wired into the shipped charts already; keep it if you build your own manifests, or a rolling restart will happily route traffic to a node that’s still replaying its WAL.

Structured logs

Logs go to stderr, a file, TCP, or syslog (log_sink), in plain or JSON form (log_format = "json" for log pipelines). On SIGHUP the daemon reopens sinks for rotation. Fail-stop events log at FATAL with a machine-readable halt reason — the log line, the tuladb_engine_halted metric, and the process exit code all tell the same story.

The operations dashboard

Beyond metrics, the dashboard (bundled with the local and cluster stacks) is the operator console for the ledger itself: accounts with a balance-after-each-transaction chart and limit history, transfer legs and lifecycle, pending holds, rejected records, sources and units, and the node’s role, leader and commit and apply positions. Every screen has its own URL, so during an incident “what actually posted?” is a link, not a SQL session.

• Three roles, enforced on the server. A viewer reads; an operator also moves money (transfers, reserves, commits, rollbacks, rejects, funding); an admin also changes ledger shape (accounts, limits, caps, sources, units). A viewer who hand-crafts a POST gets a 403, not a transfer.
• Every change is on the record. Middleware writes every mutating request to the audit log (actor, role, action, redacted parameters, HTTP status and what the ledger answered), so a newly added route can’t skip it. With LEDGER_DASHBOARD_AUDIT_REQUIRED=1, an action whose record can’t be written is refused before it runs. The log answers who asked; the engine’s WAL stays the authority for what happened to money.
• Forms can’t double-post. Each money form carries a transaction id bound to its contents: an unchanged retry reuses it and the engine deduplicates, an edit makes a new one, and a double-click is blocked.
• Credentials stay server-side. Browser sessions use a signed HttpOnly, SameSite=Strict cookie bound to the password that opened it; the write-capable API key never reaches the browser. Users are local accounts (scrypt-hashed, managed by a CLI); there is no SSO yet.
• It refuses an unsafe start. It won’t listen beyond loopback outside production mode, and production mode needs an API key and a non-default password.

The same dashboard can also query the exported Iceberg tables directly: deep history, read-only SQL, scheduled reconciliation of the lake against the ledger and reports that wait until the data reconciles. It’s off until you point it at your catalog.