Operate a sync fleet

# syncs.yaml
syncs:
  - stream-id: orders
    source-driver: postgres
    source: postgres://user:pass@src-a:5432/app
    target-driver: mysql
    target: mysql://user:pass@dst:3306/app
    slot-name: orders           # distinct per Postgres source (see below)
    apply-concurrency: 4
    metrics-listen: :9101
  - stream-id: inventory
    source-driver: mysql
    source: mysql://user:pass@src-b:3306/inv
    target-driver: postgres
    target: postgres://user:pass@dst:5432/inv
    apply-delay: 60s
    metrics-listen: :9102
restart:
  backoff-base: 1s
  backoff-cap: 30s
  max-consecutive-failures: 0   # 0 = restart forever with capped backoff

A failure that is not restarted: UNFORWARDED-SCHEMA-CHANGE (v0.156.0+). A leg that stopped because its Postgres or MySQL/MariaDB binlog source took a constraint, row-level-security, policy or DEFAULT change the stream cannot carry (what that covers) is marked failed at once and logged at ERROR, naming the acknowledgement route; the other legs keep running. A restart could only refuse again — the refusal is recorded on the target — and this is the one refusal where a restart that somehow skipped the record would accept the change silently. Since v0.156.7, more refusals are not restarted either (listed below); every other failure, other terminal errors included, is still restarted exactly as above. There is deliberately no syncs.yaml key for the acknowledgement, because standing config would pre-accept refusals. Since v0.156.1 the same applies to an interrupted added-column backfill, ADD-COLUMN-BACKFILL-INCOMPLETE (its message begins UNFORWARDED-SCHEMA-CHANGE: ADD-COLUMN-BACKFILL-INCOMPLETE:), which the supervisor treats identically — but for that one, step 1 below is different: there is no DDL to apply, so instead copy the added column's values from the source for the rows that predate the ADD COLUMN, keyed by primary key (or re-copy by passing --restart-from-scratch on the acknowledged start). Since v0.156.2 the supervisor's ERROR line leads with that repair for this refusal; on v0.156.1 it said "apply the change to the target" (Bug 289). Stopping a leg, or the fleet, while a backfill runs produces it, so wait for the leg's backfill complete line first. To opt a leg out of the backfill, set no-backfill-added-column: true on it. To bring the leg back:

1. Apply the same change to the target (the leg is already stopped).
2. Start that stream once outside the fleet with the leg's own source, target and flags: sluice sync start --stream-id <id> … --accept-unforwarded-schema-change=<fingerprint>, using the fingerprint the replayed refusal prints.
3. Stop it with sluice sync stop --stream-id <id> … --wait.
4. Restart the fleet process. On Linux and macOS a SIGHUP also works: a reload starts a failed leg that is still in syncs.yaml.

These refusals are not restarted either (five since v0.156.7, one since v0.156.8, two since v0.156.9, two since v0.156.10, two since v0.158.0, two since v0.159.0, one since v0.161.1, one since v0.162.0, the last since v0.163.0). A leg that stops on one of these is marked failed and is not restarted, whatever max-consecutive-failures says; the other legs keep running. Each is a condition every restart would hit again:

• SLOT-ACKED-PAST-TARGET-POSITION: the slot and the persisted position are both durable, so a restart compares the same two values.
• SHARDED-TARGET-VINDEX-UPDATE: the refused change is after the persisted position, so a restart re-delivers it.
• APPLY-MARK-MISMATCH: the marks are durable, and a restart re-delivers the same transaction onto them.
• CHARSET-NOT-DECODABLE: a restart reads the same value again.
• DSN-TIME-ZONE-NOT-UTC: a restart parses the same DSN.
• SLUICE-E-CDC-KEY-MATCHED-MULTIPLE-ROWS (marker KEY-SCOPED-WRITE-MATCHED-MULTIPLE-ROWS, v0.156.8+): the target transaction holding the change is rolled back, so a restart replays the same source changes onto the same committed rows. It needs the re-copy the refusal names.
• CHANGE-LOG-WATERMARK-STALLED (v0.156.9+, postgres-trigger sources): a restart re-reads the same immutable change-log window through the same code. The halt means an unknown sluice bug, so whether every possible cause repeats is unverified; it is listed because restarting forever would turn the halt back into the silent stall it exists to end. Report it.
• HEARTBEAT-TABLE-NOT-SLUICES (v0.156.9+): the existing source table under the heartbeat's name is still not sluice's. A fleet spec has no heartbeat key today, so no fleet leg reaches it yet; it is listed so one never restart-loops on it.
• RESUME-SCHEMA-DIVERGENCE (v0.156.10+): the check compares the source's current table with the target's catalog and records nothing, so a restart refuses again until the target matches. Make the target match the source, then reload the fleet; a reload starts a failed leg still in syncs.yaml. A failure to read the target's catalog is not this marker and is still restarted.
• SCHEMA-CHANGE-REFUSED (v0.156.10+, on legs that forward no DDL: schema-changes: refuse, multi-database and multi-schema legs, --inject-shard-column with --no-coordinate-live-ddl): not restarted for the same reason. Apply the change on the target, then reload. A refusal that also carries AMBIGUOUS-SCHEMA-BOUNDARY is covered too; the fleet spec has no acknowledgement key, so after the checks start that leg once outside the fleet with --accept-unforwarded-schema-change=<fingerprint>.
• COLUMN-ORDER-CHANGED (v0.158.0+): the source flipped the declared order of two of a table's columns (a RENAME COLUMN swap or a reorder), and nothing proves the target's own order moved to match. The check records nothing, so a restart refuses again: apply the same change on the target, then reload. Where the message names a re-copy instead (no record for the table, or a target that cannot reorder in place), run sync start --reset-target-data --yes for that leg outside the fleet.
• DEFERRED-SCHEMA-CHANGE (v0.158.0+, MySQL-family binlog sources; under binlog_row_metadata=FULL, and since v0.159.2 under MINIMAL as well): a row carries a type or collation change that no schema boundary will describe before its rows are applied, and a restart re-reads the same binlog. Since v0.159.2 it is also raised marked COLLATION-CHANGE-UNANNOUNCED, under MINIMAL or FULL, for a character column whose collation changed with no logged DDL of the table (for example an ALTER run with sql_log_bin=0). The remedy is a re-copy (sync start --reset-target-data --yes), outside the fleet.
• COLUMN-IDENTITY-CHANGED (v0.159.0+, MySQL-family binlog sources, any metadata setting): the source's ALTERs left a column name holding a different column (a DROP COLUMN and re-ADD of the name, or a RENAME of another column onto it). The refused DDL is never crossed, so a restart re-reads it and refuses again. Also raised as COLUMN-IDENTITY-UNREAD for such an ALTER sluice could not parse. The remedy is a re-copy (sync start --reset-target-data --yes), outside the fleet; there is no acknowledgement flag.
• KEY-COLLATION-MERGE (v0.159.0+, every target): a CDC upsert, or since v0.159.1 a cold start's VStream COPY upsert, matched a target row through a primary or unique key whose target collation equates two values the source's own collation keeps apart, so applying it would merge two source rows. A restart re-delivers the same row and refuses again. Give the target key column the collation the message names, then reload: since v0.159.1 that is a binary collation such as utf8mb4_bin for a Postgres-source key on a MySQL-family target, and the NO PAD utf8mb4_uca1400_nopad_* form for a MySQL 8 key on MariaDB. Where the message names a re-copy (the identifying key itself is compared differently, or the table has no primary key), run it outside the fleet. A key armed to compare bytes (a Postgres source collation on a MySQL-family target) gets no re-copy offer, because a re-copy recreates the target's equating default; use the named collation. With KEY-COLLATION-GUARD-UNSUPPORTED in the message, the target lacks the collation the check compares under and none of the table's changes were applied: create the collation on the target or align the key column on both sides. KEY-COLLATION-CHANGED arrives inside RESUME-SCHEMA-DIVERGENCE or SCHEMA-CHANGE-REFUSED, already listed above.
• KEY-COLLATION-HISTORY-UNORDERED (v0.159.2+, inside a KEY-COLLATION-MERGE refusal, so classified with it): sync from-backup, or a chain restore that cannot place a window's schema versions between its changes (a trigger-CDC chain), refused a backup window that changes a key column's collation. It comes from those commands rather than from a sync start leg. Neither restarting nor changing the target key converges. The remedy is a new backup full after the key change, then sluice sync from-backup run --reset-target-data --yes against the new chain (destructive), run by hand. A KEY-COLLATION-MERGE from sluice restore whose message says the incremental changes a key column's collation inside its own window needs that fresh full backup too, since every restore re-applies the change ahead of the same rows. KEY-COLLATION-VERSION-NAMESPACE (v0.159.3+) also arrives inside a KEY-COLLATION-MERGE from chain restore and is classified with it; its remedy is the same fresh full backup.
• UNIQUE-KEY-CONFLICT-OTHER-ROW (v0.161.1+, every target): an upsert, CDC or copy, matched a different target row through a unique key the target has and the source evidently no longer does, so applying it would overwrite that row with the incoming one. A restart re-delivers the same row and refuses again. Drop that key on the target, then reload; or re-copy outside the fleet with sync start --restart-from-scratch (from a Postgres pgoutput source, after dropping the stream's replication slot). Both re-copy paths are destructive.
• SOURCE-UNIQUE-KEY-ABSENT (v0.162.0+, every source): the target has a unique key no source key implies, usually a source DROP INDEX the stream did not see, so the source's next duplicate of it would overwrite a different target row. Every start reads the same two catalogs and nothing is recorded, so a restart refuses again. Drop the key on the target, then reload; where the message says rows may already have merged, re-copy outside the fleet. A key the target is meant to have is accepted in the leg's spec with the standing accept-target-only-unique-key list (the value the refusal prints), then reload. A SOURCE-UNIQUE-KEY-RECHECK-FAILED stop (a catalog read failed) is not this marker and is still restarted.
• INCLUDE-TABLE-SCOPE-DRIFT (v0.163.0+, every source's warm resume): the leg's scope holds a table the stream never copied or stopped keeping up. The check reads the stream's record of copied tables and pending add-tables on its control row, which a restart does not change, so it refuses again. The same marker carries the shapes below; apply the remedy the message names: for LIVE-ADDED-TABLE-EXCLUDED (the leg's exclude-table names a table schema add-table brought in), narrow the exclusion in the spec or run sluice schema add-table --abandon <table> --yes, then reload; for TABLE-IDENTITY-CHANGED (a table swapped or re-created while stopped), re-copy outside the fleet; for a rename no engine id proves, rename the target table and start once outside the fleet with --accept-table-rename <old>=<new>; for COLD-START-INCOMPLETE (a cold start stopped after it began changing the target), re-run that cold start outside the fleet; for a dead add-table of a table since dropped or descoped, run sluice schema add-table --abandon <table> --yes, then reload. A failure to read the catalog or the control row is not this marker and is still restarted.

Not in the list: the Postgres reader's mid-stream refusal that carries PRE-ALTER-REPLAY-WEDGE. A wedged leg is restarted and refuses again each time (GC-44 F27, unchanged from v0.156.9), so alert on a leg that keeps restarting with that marker and apply the recovery outside the fleet. Nor is KEY-COLLATION-REPLAY-AMBIGUOUS (v0.159.1+), deliberately: a restart runs the copy again, which is its remedy. Nor, since v0.163.0, are INCLUDE-TABLE-ADD-IN-PROGRESS (every table the scope check refuses has a schema add-table still running; the restart after backoff resumes once the add completes, and an add that died goes stale and becomes the listed INCLUDE-TABLE-SCOPE-DRIFT) and INCLUDE-TABLE-NARROWING-UNRECORDED (a start that narrows the scope could not record the narrowing; a transient failure clears on a restart, while one that persists usually means a missing record column, so ship the ALTER it names). The supervisor logs each at ERROR with a marker= attribute naming the refusal, so alert on failed legs and on that attribute. Before v0.156.7 the first five were restarted after backoff and refused again, indefinitely under the default max-consecutive-failures: 0, while the fleet looked healthy. Other coded SLUICE-E-* refusals and other terminal errors are still restarted under the failure cap, because some clear without any change to the leg (for example SLUICE-E-CDC-REPLICATION-HEADROOM once another slot is freed). DEFERRED-KEY-CHECK-FAILED-AT-COMMIT is deliberately among them: one of its causes depends on arrival timing, and a restart can clear it.

Apply the remedy the message names. The fleet config has no key for --restart-from-scratch or for the one-shot acknowledgement flags, so a remedy that needs one runs outside the fleet. The exception is the standing accept-target-only-unique-key list (v0.162.0+): it is a fleet spec key, because an accepted target key stays on the target and must be accepted on every start, so put it in the leg's spec and reload rather than taking the leg out. --accept-table-rename (v0.163.0) has no fleet key and runs outside the fleet like the others. sluice schema add-table --abandon is a separate command, not a flag on the leg: run it against the leg's stream, then reload. For the others:

1. Remove the leg from syncs.yaml and reload the fleet (restart it, or SIGHUP on Linux and macOS). A reload starts every failed leg still in the config, so do not reload with the leg still listed until the fix is in.
2. Run that stream once with sync start, with the leg's own source, target and flags plus the remedy flag, for example --restart-from-scratch or --accept-slot-acked-past-position=<LSN>. Stop it with sluice sync stop --stream-id <id> … --wait once it is running cleanly.
3. Put the leg back in syncs.yaml and reload.

# validate + print the plan, start nothing
sluice sync run --config syncs.yaml --dry-run

# run the fleet, with a read-only dashboard on :9300
sluice sync run --config syncs.yaml --dashboard-listen :9300

kill -HUP "$(pgrep -f 'sluice sync run')"

sluice sync status --all --config syncs.yaml --summary

# live-refresh every 2s, machine-readable
sluice sync status --all --config syncs.yaml --format json --watch 2s

readinessProbe:
  httpGet:
    path: /readyz
    port: 9101
  periodSeconds: 10

# terminal 1: run the fleet with the dashboard API exposed
sluice sync run --config syncs.yaml --dashboard-listen :9300

# terminal 2 (local or over an SSH tunnel): live terminal view
sluice sync tui --connect :9300 --refresh 2s