Operate a sync fleet

Supervise many continuous syncs from one process — failure-isolated, observable, and reconfigurable without a restart.

Once you keep several cross-database syncs alive at once, running each sync start as its own pod or systemd unit gets unwieldy. sync run collapses that to one supervised process driven by a single fleet config: it runs N independent syncs, each with its own stream-id, and — the load-bearing property — fully failure-isolated, so one sync crashing, erroring, or even panicking can never take down its healthy peers (ADR-0122). This guide covers running a fleet, observing it, and reconfiguring it live.

The fleet config #

A fleet is a YAML file listing each sync as a curated subset of the sync start flags you already know, in kebab-case. A top-level restart block tunes the supervisor's bounded-backoff policy:

# syncs.yaml
syncs:
  - stream-id: orders
    source-driver: postgres
    source: postgres://user:pass@src-a:5432/app
    target-driver: mysql
    target: mysql://user:pass@dst:3306/app
    slot-name: orders           # distinct per Postgres source (see below)
    apply-concurrency: 4
    metrics-listen: :9101
  - stream-id: inventory
    source-driver: mysql
    source: mysql://user:pass@src-b:3306/inv
    target-driver: postgres
    target: postgres://user:pass@dst:5432/inv
    apply-delay: 60s
    metrics-listen: :9102
restart:
  backoff-base: 1s
  backoff-cap: 30s
  max-consecutive-failures: 0   # 0 = restart forever with capped backoff

Each sync's own retry (ADR-0093 re-snapshot, apply-retry backoff) is the inner loop; the supervisor's restart is the outer loop. A sync that drains cleanly (a sync stop or Ctrl-C) is left stopped; a sync that dies with the process still live is logged loudly, backed off, and restarted. The consecutive-failure counter resets once a sync has run longer than the healthy threshold, so a sync that ran for hours before dying carries no restart debt.

Run the fleet #

Validate the config first with --dry-run (it checks required fields, stream-id and slot-name uniqueness, and retry bounds, then prints the resolved plan without starting anything), then run it:

# validate + print the plan, start nothing
sluice sync run --config syncs.yaml --dry-run

# run the fleet, with a read-only dashboard on :9300
sluice sync run --config syncs.yaml --dashboard-listen :9300

The process blocks until every sync exits. Ctrl-C / SIGTERM stops all of them cleanly. A single-sync fleet that can never start exits non-zero, but a fleet with any healthy peer keeps running regardless of what its neighbors do.

Reload without a restart (SIGHUP) #

Edit syncs.yaml and send the running process a SIGHUP: sluice re-reads and re-validates the file, then reconciles the live fleet — starting newly-added syncs, draining and stopping removed ones, and restarting any whose spec changed (detected by a per-stream fingerprint, so unchanged syncs are left untouched):

kill -HUP "$(pgrep -f 'sluice sync run')"

See the whole fleet at once #

sync status --all rolls up every stream across every target named in the fleet config into one table — reading the target control tables directly, so no running supervisor is required. A target that can't be reached is reported inline and skipped rather than blanking the whole view:

sluice sync status --all --config syncs.yaml --summary

# live-refresh every 2s, machine-readable
sluice sync status --all --config syncs.yaml --format json --watch 2s

Observe it: Prometheus metrics and readiness #

Give each sync a metrics-listen address in the config (as above) and it binds a Prometheus-format /metrics endpoint plus /healthz (liveness) and /readyz (flips to 200 once the sync has finished its snapshot/warm-resume preamble and entered the apply loop). The exported sluice_* gauge families include:

Because /readyz is a real readiness signal, it wires straight into a Kubernetes probe on the sync's metrics port:

readinessProbe:
  httpGet:
    path: /readyz
    port: 9101
  periodSeconds: 10

Pre-emptive Postgres slot-health warnings #

For a Postgres source, a replication slot that outruns its retention budget gets evicted and the stream breaks. sluice watches for that ahead of time and emits severity-graded slog warnings: a WARN when retention pressure crosses 70% of max_slot_wal_keep_size, a CRITICAL at 85% (eviction imminent), and a WARN when the slot has been observed inactive for 30 minutes or more (ADR-0059). These are rate-limited and emit a "cleared" INFO when the condition resolves, so the alarm turns off visibly.

Live dashboards: web and terminal #

sync run --dashboard-listen ADDR serves a self-contained, auto-refreshing HTML page of the live fleet — per-sync state, restart count, consecutive failures, last error, uptime — backed by a stable GET /api/fleet JSON API (ADR-0124). It is strictly read-only: no stop/restart controls, no data path, and it exposes only what sync status --all already does (stream-ids, states, error strings — no DSNs, no row data). It has no authentication, so bind it to localhost or a trusted network.

For a terminal equivalent, sync tui --connect is a full-screen client that polls that same /api/fleet endpoint — so it works locally or over an SSH tunnel to the dashboard port, without disturbing the fleet process:

# terminal 1: run the fleet with the dashboard API exposed
sluice sync run --config syncs.yaml --dashboard-listen :9300

# terminal 2 (local or over an SSH tunnel): live terminal view
sluice sync tui --connect :9300 --refresh 2s

Threshold alerts (advisory) #

The fleet can push threshold alerts to a webhook or Slack. Set the sink URL via its env var (SLUICE_NOTIFY_WEBHOOK / SLUICE_NOTIFY_SLACK), then arm one or more thresholds. Alerts are edge-triggered, cooldown'd (--notify-cooldown, default 15m), and advisory + failure-isolated — a dead sink is logged and swallowed, never affecting the sync:

All the util / control-plane-lag / growth rules need a PlanetScale telemetry provider (--planetscale-org plus the metrics-token flags); only --notify-sync-lag-seconds works without it. The same alerter set is also available on the standalone metrics-watch probe.

Next steps #

• sync start reference — the full per-sync flag surface each fleet entry is built from.
• Sync to PlanetScale / Vitess — the telemetry credentials the util-threshold alerts and sluice_target_* gauges require.
• Zero-downtime migration — the single-stream cutover flow each fleet sync runs internally.