Docs/Get startedv0.0.1-preview

Integrated cluster

Run every TulaDB component together — a three-node mTLS cluster, tulactl, the Prometheus exporters, Prometheus and Grafana, the Iceberg exporter and the dashboard — and prove one transfer reached all of them.

On this page
  1. 1. Plan the machines
  2. 2. Download and verify
  3. 3. Certificates
  4. 4. The API-key file
  5. 5. Prepare the three nodes
  6. 6. Grant the clients, then start the cluster
  7. 7. A Prometheus exporter on each node
  8. 8. Prometheus and Grafana
  9. 9. The Iceberg exporter
  10. 10. The dashboard
  11. 11. Verify: one transfer, everywhere
  12. What this setup leaves out

This guide builds one evaluation deployment from the 0.0.1-preview downloads, in the order you would do it: certificates, three nodes, metrics, the lake exporter, the dashboard, and finally one transfer that you then find in the dashboard, in Grafana and in the Iceberg table. Every TulaDB port runs mutual TLS, and every client has its own certificate and its own role. Install the components describes each program on its own.

What you download. Every component in this guide is a signed 0.0.1-preview download on Downloads: core (the server, tulactl and the Prometheus exporter), sdk, dashboard, exporter (Iceberg), grafana-dashboards and, if you want the exporter on its own release line, prometheus-exporter. Verify each one as Install the components shows before you copy it to a host.

Just trying it out? The developer quick setup runs the same components on one machine with Docker or Podman, from the release’s container images, in a few minutes. This guide is the real deployment: separate machines and your own certificates.

1. Plan the machines

Machine Address (example) Runs
node 1, node 2, node 3 10.0.0.1, 10.0.0.2, 10.0.0.3 tuladbd, tulactl, the Prometheus exporter
ops host 10.0.0.10 Prometheus, Grafana, the Iceberg exporter, the dashboard

The nodes need a supported Linux and synchronised clocks. The ops host needs Linux x86_64 or arm64 with Python 3.12 and its venv module (Ubuntu 24.04 ships it as python3.12), because the dashboard bundle needs 3.12 and the exporter has a 3.12 wheelhouse.

Open these TCP ports, and nothing else from outside:

From To Ports
each node the other nodes 7700 (client), 7800 (query), 8700 (fabric), 9700 (heartbeat)
ops host each node 7700, 7800, 9145 (Prometheus exporter)

Replace the addresses with yours throughout.

2. Download and verify

Take each product into its own directory: every product has its own SHA256SUMS, and they must not overwrite each other.

Product Files to take Where
core tuladb-installer-0.0.1-preview-linux-x86_64 (or -linux-arm64) every node
exporter tuladb-iceberg-exporter-0.0.1-preview-wheelhouse-iceberg-cp312-manylinux_x86_64.tar.gz (or _aarch64) ops host
dashboard the whole directory for the ops host’s architecture ops host
grafana-dashboards tuladb-grafana-dashboards-0.0.1-preview.tar.gz ops host

With each, take SHA256SUMS, SHA256SUMS.cosign.bundle and trusted_root.json from the same directory, then verify. For the server installer:

mkdir core && cd core
B=https://downloads.tuladb.com/core/0.0.1-preview
for f in tuladb-installer-0.0.1-preview-linux-x86_64 SHA256SUMS SHA256SUMS.cosign.bundle trusted_root.json; do
  curl -fsSLO "$B/$f"
done
cosign verify-blob --offline --trusted-root trusted_root.json \
  --bundle SHA256SUMS.cosign.bundle \
  --certificate-identity https://github.com/TulaDB/tuladb-core/.github/workflows/release.yml@refs/tags/v0.0.1-preview \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
chmod +x tuladb-installer-0.0.1-preview-linux-x86_64

Repeat for the other three with their own directory and signer identity (the table). Copy the installer to every node.

3. Certificates

One CA signs everything. Each node gets a certificate for its own address with both serverAuth and clientAuth. Each client gets its own certificate with clientAuth: tuladb-operator (you, with tulactl), tuladb-dashboard and tuladb-iceberg-exporter.

A node names a client by its certificate’s principal: the first spiffe:// URI SAN, else the first urn: URI SAN, else the first URI SAN, else the first DNS SAN, else the subject CN. The client certificates below carry no SAN, so the principal is the CN, and that is the name the role grants in step 6 use.

For an evaluation, with OpenSSL (use your own PKI for anything else):

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 365 \
  -subj /CN=tuladb-ca -keyout ca.key -out ca.crt

for n in 1 2 3; do
  ip=10.0.0.$n
  openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -subj /CN=node$n \
    -keyout node$n.key -out node$n.csr
  printf 'subjectAltName=IP:%s\nextendedKeyUsage=serverAuth,clientAuth\n' "$ip" > node$n.ext
  openssl x509 -req -in node$n.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 \
    -extfile node$n.ext -out node$n.crt
done

for c in tuladb-operator tuladb-dashboard tuladb-iceberg-exporter; do
  openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -subj /CN=$c \
    -keyout $c.key -out $c.csr
  printf 'extendedKeyUsage=clientAuth\n' > $c.ext
  openssl x509 -req -in $c.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 \
    -extfile $c.ext -out $c.crt
done

Keep ca.key off every machine below. Put the files in place:

# on node N (here node 1): the CA, its own certificate and key
sudo groupadd --system tuladb 2>/dev/null || true
sudo install -d -m 0755 /etc/tuladb-pki
sudo install -m 0644 ca.crt node1.crt /etc/tuladb-pki/
sudo install -m 0640 -g tuladb node1.key /etc/tuladb-pki/

# on node 1, for you: the operator certificate, in your home directory
install -d -m 0700 ~/pki && install -m 0600 ca.crt tuladb-operator.crt tuladb-operator.key ~/pki/

# on the ops host: the CA and the two service certificates, owned by the user that runs them
sudo install -d -m 0755 /etc/tuladb-pki
sudo install -m 0644 ca.crt tuladb-dashboard.crt tuladb-iceberg-exporter.crt /etc/tuladb-pki/
sudo install -m 0600 -o "$USER" tuladb-dashboard.key tuladb-iceberg-exporter.key /etc/tuladb-pki/

4. The API-key file

Every node carries the same key file, so the admin key works on whichever node you reach:

openssl rand -hex 32 | tr -d '\n' > admin.key            # keep this secret
printf '# digest  principal\n%s  tuladb-admin\n' \
  "$(openssl dgst -sha3-256 -r < admin.key | awk '{print $1}')" > api_keys

Copy api_keys (digests only) to every node. Copy admin.key to node 1 only, as ~/pki/admin.key, mode 0600: you use it with tulactl there.

5. Prepare the three nodes

On node 1, with the other two as peers:

sudo ./tuladb-installer-0.0.1-preview-linux-x86_64 install --cluster-node \
  --node-id 1 --node-addr 10.0.0.1 \
  --peer 2=10.0.0.2:7700 --peer 3=10.0.0.3:7700 \
  --tls-cert /etc/tuladb-pki/node1.crt --tls-key /etc/tuladb-pki/node1.key \
  --tls-ca /etc/tuladb-pki/ca.crt \
  --api-key-file api_keys

Repeat on nodes 2 and 3 with their own --node-id, --node-addr and certificate, and the other two as peers. Add --wal-dir for a WAL on its own disk (Install TulaDB). Each run ends with:

tuladb-installer: PREPARED cluster node 1 (0.0.1-preview). It is NOT started and NOT enabled.

6. Grant the clients, then start the cluster

The installer wrote /opt/tuladb/node.toml with default_role = "denied": a certificate with no grant can do nothing. It granted two principals, the admin key and this node’s own certificate:

sudo grep -n '^role_grant_' /opt/tuladb/node.toml
role_grant_0 = "tuladb-admin -> cluster-admin"
role_grant_1 = "node1 -> read-only"

Grants are numbered from 0 with no gaps. On every node, add the three clients at the next indices, and turn on export leases, which the Iceberg exporter uses to hold the WAL it still needs:

sudo tee -a /opt/tuladb/node.toml >/dev/null <<'EOF'

# Integrated cluster: one principal per client (the certificate CN)
role_grant_2 = "tuladb-operator -> read-only"
role_grant_3 = "tuladb-dashboard -> ledger-operator"
role_grant_4 = "tuladb-iceberg-exporter -> exporter"
export_lease_ttl_ms = 300000
EOF
sudo -u tuladb /opt/tuladb/tuladbd --check-config --config /opt/tuladb/node.toml
Principal Role Why
tuladb-operator read-only your tulactl certificate; writes authenticate with the admin key, which raises the connection to tuladb-admin (cluster-admin)
tuladb-dashboard ledger-operator the dashboard sends no API key: its certificate is its identity; use read-only for a view-only dashboard
tuladb-iceberg-exporter exporter export leases and durable acks need it; read-only is refused

export_lease_ttl_ms is how long an exporter’s hold survives without renewal; the exporter renews it as it reads, and a few minutes is typical. With leases on, the node warns at start while export_lease_max_hold_batches is 0 (unbounded): set it to bound how far a stalled exporter may hold the WAL, sized to your WAL disk (Analytics export). Check that the peers, the grants and use_iso20022_ref_size match on all three nodes, then start them:

sudo systemctl enable --now tuladbd                                      # on each node
sudo ./tuladb-installer-0.0.1-preview-linux-x86_64 status               # on each node: exit 0 = ready

On node 1, point tulactl at the cluster with your certificate:

cat >> ~/.bashrc <<'EOF'
export TULADB_CLIENT_HOST=10.0.0.1 TULADB_CLIENT_PORT=7700 TULADB_CLIENT_TLS_ENABLED=1
export TULADB_CLIENT_TLS_CA_FILE=$HOME/pki/ca.crt
export TULADB_CLIENT_TLS_CERT_FILE=$HOME/pki/tuladb-operator.crt
export TULADB_CLIENT_TLS_KEY_FILE=$HOME/pki/tuladb-operator.key
EOF
. ~/.bashrc
KEY="$(cat ~/pki/admin.key)"
tulactl ready --json

7. A Prometheus exporter on each node

On each node, run tuladb_prometheus_exporter beside tuladbd. It reads the node’s metrics file and needs no certificate. Use the node’s own address in --listen:

sudo install -d -m 0755 /etc/tuladb
echo 'cluster_name = "ledger-eval"' | sudo tee /etc/tuladb/prometheus-exporter.toml
sudo tee /etc/systemd/system/tuladb-prometheus-exporter.service >/dev/null <<'EOF'
[Unit]
Description=TulaDB Prometheus exporter
After=tuladbd.service

[Service]
User=tuladb
Group=tuladb
ExecStart=/opt/tuladb/tuladb_prometheus_exporter --config /etc/tuladb/prometheus-exporter.toml --snapshot /var/lib/tuladb/runtime/metrics.json --listen 10.0.0.1:9145
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now tuladb-prometheus-exporter

The endpoint is plain HTTP with no authentication. Let only the ops host reach port 9145.

8. Prometheus and Grafana

On the ops host, install Prometheus and Grafana from their projects; neither is a TulaDB download. Add the three node exporters, and the Iceberg exporter’s metrics from step 9, to prometheus.yml:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: tuladb
    static_configs:
      - targets: ["10.0.0.1:9145", "10.0.0.2:9145", "10.0.0.3:9145"]
  - job_name: tuladb-iceberg-exporter
    static_configs:
      - targets: ["127.0.0.1:9146"]

The six TulaDB dashboards are in the grafana-dashboards download, with their provisioning files; they read a datasource whose uid is tuladb-prometheus. For a Grafana package install, whose provisioning directory is /etc/grafana/provisioning:

tar -xzf tuladb-grafana-dashboards-0.0.1-preview.tar.gz
G=tuladb-grafana-dashboards-0.0.1-preview
sed 's#http://prometheus:9090#http://127.0.0.1:9090#' $G/provisioning/datasources/datasource.yml \
  | sudo tee /etc/grafana/provisioning/datasources/tuladb.yml >/dev/null
sudo install -m 0644 $G/provisioning/dashboards/provider.yml /etc/grafana/provisioning/dashboards/tuladb.yml
sudo install -d /var/lib/grafana/dashboards
sudo install -m 0644 $G/dashboards/*.json /var/lib/grafana/dashboards/
sudo systemctl restart grafana-server

Check that Prometheus sees all three nodes: the query tuladb_up returns three series, each 1. No alert rules ship in 0.0.1-preview; see Grafana and Prometheus.

9. The Iceberg exporter

On the ops host, install it from the wheelhouse:

W=tuladb-iceberg-exporter-0.0.1-preview-wheelhouse-iceberg-cp312-manylinux_x86_64
tar -xzf $W.tar.gz
sudo python3.12 -m venv /opt/tuladb-iceberg-exporter
sudo /opt/tuladb-iceberg-exporter/bin/pip install --no-index --find-links $W/wheelhouse 'tuladb-iceberg-exporter[iceberg]'

This guide uses a local lake: a SQL catalog in SQLite and a file:// warehouse, the target every exporter CI run tests. For S3 Tables or BigLake, change [iceberg.catalog] as Analytics export shows. transactions_layout = "shared" writes every transaction to one table, default.transactions; the default for a new pipeline, by_type, writes one table per kind instead (default.transactions_single_phase and so on).

E=/var/lib/tuladb-iceberg
sudo install -d -o "$USER" -m 0750 $E $E/warehouse $E/state
cat > $E/exporter.toml <<'EOF'
[source]
hosts = ["10.0.0.1:7800", "10.0.0.2:7800", "10.0.0.3:7800"]
register_export_lease = true

[source.tls]
enabled = true
ca_file = "/etc/tuladb-pki/ca.crt"
cert_file = "/etc/tuladb-pki/tuladb-iceberg-exporter.crt"
key_file = "/etc/tuladb-pki/tuladb-iceberg-exporter.key"

[source.auth]
type = "none"                    # the certificate is the identity

[sink]
type = "parquet"
warehouse_path = "/var/lib/tuladb-iceberg/warehouse"
table_name = "transactions"
transactions_layout = "shared"

[iceberg]
enabled = true
catalog_name = "default"
table_identifier = "default.transactions"
writer_epoch = 1                 # iceberg.enabled requires writer_epoch >= 1

[iceberg.catalog]
type = "sql"
uri = "sqlite:////var/lib/tuladb-iceberg/warehouse/pyiceberg_catalog.db"
warehouse = "file:///var/lib/tuladb-iceberg/warehouse"

[checkpoint]
path = "/var/lib/tuladb-iceberg/state/checkpoint.json"

[service]
pid_file = "/var/lib/tuladb-iceberg/state/exporter.pid"
log_file = "/var/lib/tuladb-iceberg/state/exporter.log"

[observability]
metrics_snapshot_path = "/var/lib/tuladb-iceberg/state/metrics.json"
cluster_name = "ledger-eval"
metrics_listen = "127.0.0.1:9146"
EOF

X=/opt/tuladb-iceberg-exporter/bin/tuladb-iceberg-exporter
$X doctor --config $E/exporter.toml
$X start  --config $E/exporter.toml      # in the background; logs to state/exporter.log
$X status --config $E/exporter.toml
$X metrics --config $E/exporter.toml     # in its own terminal or service: /metrics and /healthz on 127.0.0.1:9146

The exporter finds the leader among the three hosts and follows it. If its log shows ERR_PERMISSION (35) on the export lease, the tuladb-iceberg-exporter grant from step 6 is missing on the node that answered. Exit codes 86, 87 and 88 mean a person must look (Analytics export).

10. The dashboard

On the ops host, install the bundle and start the dashboard with its certificate:

cd dashboard                      # the directory you downloaded the dashboard files into
bash ./tuladb-dashboard-0.0.1-preview-install.sh
umask 077 && cat > ~/dashboard.env <<EOF
LEDGER_DASHBOARD_ENV=prod
LEDGER_DASHBOARD_API_KEY=$(openssl rand -hex 32)
LEDGER_DASHBOARD_SESSION_SECRET=$(openssl rand -hex 32)
LEDGER_DASHBOARD_PASSWORD=<a password you choose>
TULADB_CLUSTER_NODES=10.0.0.1:7700,10.0.0.2:7700,10.0.0.3:7700
TULADB_QUERY_NODES=10.0.0.1:7800,10.0.0.2:7800,10.0.0.3:7800
TULADB_CLIENT_TLS_ENABLED=1
TULADB_CLIENT_TLS_CA_FILE=/etc/tuladb-pki/ca.crt
TULADB_CLIENT_TLS_CERT_FILE=/etc/tuladb-pki/tuladb-dashboard.crt
TULADB_CLIENT_TLS_KEY_FILE=/etc/tuladb-pki/tuladb-dashboard.key
EOF
set -a && . ~/dashboard.env && set +a
~/.local/share/tuladb-dashboard/bin/tuladb-dashboard --host 127.0.0.1 --port 8080

Replace the password placeholder before you start it: in production mode the dashboard refuses the default password, and refuses to start without LEDGER_DASHBOARD_API_KEY. It listens on loopback over plain HTTP. From your workstation, tunnel to it and to Grafana:

ssh -L 8080:127.0.0.1:8080 -L 3000:127.0.0.1:3000 ops-host

Then open http://localhost:8080 and sign in as admin with your password. For more than an evaluation, put an HTTPS reverse proxy in front instead.

11. Verify: one transfer, everywhere

On node 1, create a ledger and two accounts, then post one transfer with a transaction id you will search for:

L="--ledger 10 --currency 840"
tulactl --api-key "$KEY" ledger-create $L --yes
tulactl --api-key "$KEY" create-account --id 1001 $L --limit-low -1000000
tulactl --api-key "$KEY" create-account --id 1002 $L
tulactl --api-key "$KEY" transfer --from 1001 --to 1002 --amount 25 $L --txn integrated-check-1

The transfer answers ✓ OK phase: POSTED with its transfer_id. Now find it in each place:

  1. The cluster. tulactl --api-key "$KEY" balance --id 1002 $L shows a posted balance of 2500 (minor units: 25.00).

  2. The dashboard. Open Transfers, type integrated-check-1 in Txn / Transfer ID and click Search transfers. The transfer from 1001 to 1002 is listed; open it for its legs. Accounts → 1002 shows the balance.

  3. Grafana. Open http://localhost:3000, folder TulaDB → TulaDB · 0 Overview, with cluster set to ledger-eval. Transfers in history has grown by one, the per-node Transfers accepted / rejected / duplicate retries graph shows the write on every node, and in Fleet health Up is 1: the lowest tuladb_up across the nodes, so every node is up.

  4. The Iceberg table. On the ops host, with E and X set as in step 9, after a few seconds (the exporter commits in micro-batches):

    $X watermark --config $E/exporter.toml      # the position up to which the lake is complete
/opt/tuladb-iceberg-exporter/bin/python - <<'EOF'
from pyiceberg.catalog import load_catalog
from pyiceberg.expressions import EqualTo
cat = load_catalog("default", type="sql",
                   uri="sqlite:////var/lib/tuladb-iceberg/warehouse/pyiceberg_catalog.db",
                   warehouse="file:///var/lib/tuladb-iceberg/warehouse")
rows = cat.load_table("default.transactions").scan(row_filter=EqualTo("txn_id", "integrated-check-1")).to_arrow()
print(rows.select(["txn_id", "transfer_id", "debit_account_id", "credit_account_id", "amount", "outcome"]).to_pylist())
EOF
$X reconcile --config $E/exporter.toml      # exit 0: the table matches what the exporter committed

    One row, with the same transfer_id that tulactl printed and amount 2500.

Run the same transfer command again: it answers OK_DUPLICATE with the original transfer_id, the dashboard still lists one transfer, and the table still holds one row (Idempotency).

What this setup leaves out

  • Services. The dashboard and the Iceberg exporter run from a shell here. For anything longer-lived, run them under systemd as their own users.
  • HTTPS for the web endpoints. The Prometheus exporters, Prometheus and Grafana run over plain HTTP on a private network here, and the dashboard over a tunnel. Front them with TLS for real use.
  • Alerting. No alert rules ship in this release.
  • History retention. Off by default. When you turn it on (history_retention_hours), purges wait for the exporter’s durable acks (Analytics export).
  • Upgrades and node changes. Upgrade one node at a time with the new installer’s upgrade (Install TulaDB); add a node with Node join.