Docs/Deploy & operatev0.0.1-preview

io_uring and the WAL

How TulaDB's write-ahead log uses io_uring on Linux, which distributions allow it, how to enable it on RHEL 9, and how to check which backend a node runs.

On this page
  1. Which machines run it
  2. Check which backend a node runs
  3. Enable io_uring on RHEL 9
  4. Configuration
  5. Getting the most from it

On Linux, TulaDB’s write-ahead log (WAL) uses io_uring to take fdatasync off the writer thread’s critical path. The writer submits a group’s fsync to the kernel and goes on writing the next group while the kernel flushes. A transfer is still acknowledged only after its group is durable on disk: io_uring changes when the flush is waited for, not whether it is. Release builds always include io_uring support, and the package installs liburing as a dependency.

When the kernel refuses io_uring, the WAL uses blocking fdatasync instead. The node is fully correct either way; io_uring buys throughput and tail latency on a durability-bound workload.

Which machines run it

Distribution Kernel io_uring
Ubuntu 22.04 5.15 on
Ubuntu 24.04 6.8 on
Debian 12 6.1 on
RHEL 9, Rocky 9, AlmaLinux 9 5.14 off by default: the kernel ships kernel.io_uring_disabled = 2
macOS — not applicable: the WAL uses fsync

Inside a container, the runtime’s seccomp profile may block the io_uring system calls. The node then uses fdatasync, exactly as on RHEL.

Check which backend a node runs

The runtime metrics snapshot names it:

grep -o '"wal_backend": "[a-z_]*"' /var/lib/tuladb/runtime/metrics.json
# "wal_backend": "io_uring"     or     "wal_backend": "sync"

When the kernel refuses io_uring, the node logs this once at start:

WARN [wal] iouring_disabled_by_kernel: the kernel refuses io_uring (EPERM: kernel.io_uring_disabled is set; RHEL 9 ships 2). The WAL uses fdatasync. To allow it for this service only: kernel.io_uring_disabled = 1 and kernel.io_uring_group = the tuladb group's gid (sysctl.d)

Prometheus has two more metrics:

  • tuladb_wal_backend_fallback is 1 when a configuration asked for io_uring and got sync.
  • tuladb_wal_iouring_degraded is 1 while the io_uring backend cannot take fsyncs (a full submission queue, say). Groups then fall back to a blocking flush, still durable, and tuladb_wal_iouring_submit_fallbacks_total counts them.

Enable io_uring on RHEL 9

RHEL disables io_uring for every process, because of its history of kernel vulnerabilities. Allow it for the TulaDB service only. Setting kernel.io_uring_disabled = 1 permits io_uring for members of kernel.io_uring_group, so make that the tuladb group:

gid=$(getent group tuladb | cut -d: -f3)
printf 'kernel.io_uring_disabled = 1\nkernel.io_uring_group = %s\n' "$gid" \
  | sudo tee /etc/sysctl.d/60-tuladb-io-uring.conf
sudo sysctl --system
sudo systemctl restart tuladbd
grep -o '"wal_backend": "[a-z_]*"' /var/lib/tuladb/runtime/metrics.json   # "io_uring"

Every other process keeps io_uring disabled. To undo it, remove the file, set kernel.io_uring_disabled = 2 and restart tuladbd. Weigh the change with your security team: it widens the kernel surface reachable by the tuladb user.

Configuration

Key Default
wal_backend auto auto uses io_uring when the kernel allows it; io_uring asks for it (and reports tuladb_wal_backend_fallback when refused); sync never uses it
iouring_ring_depth 256 (512 in the production profiles) queue depth of the WAL’s ring; deeper lets more groups’ flushes overlap on NVMe

The shipped profiles use wal_backend = "auto", so there is nothing to set: the node takes io_uring wherever it is allowed.

Getting the most from it

io_uring overlaps flushes; it cannot make a slow disk fast. Put the WAL on its own local NVMe device (--wal-dir at install, or wal_dir in node.toml). A WAL sharing a disk with the data, or living on a network disk, is the most common cause of tail latency whichever backend runs. See Durability for what an acknowledgement promises, and Configuration for the WAL sizing keys.