Continuous sync from a backup chain (the broker)

Replay a backup chain into a target as a long-running broker — no direct source↔target connectivity required.

The broker (sluice sync from-backup run) replicates by reading a backup chain instead of connecting to the source's CDC stream directly. One sluice process produces the chain from the source; another tails it and applies the changes to a target. The backup store — S3 / GCS / Azure Blob / local FS — is the message log between them. Reach for this when the source and target can't (or shouldn't) talk directly: an air-gapped target, cross-region DR where the chain already crosses the boundary, or fanning one chain out to several targets.

1. Produce the chain #

On the source side, take a full backup to root the chain, then keep it fed with incrementals. On Postgres add --chain-slot to the full so it provisions the replication slot that anchors the chain (incrementals then chain with zero gap):

sluice backup full --source-driver postgres --source 'postgres://...source...' \
    --target s3://my-bucket/app-chain \
    --backup-endpoint https://<account>.r2.cloudflarestorage.com \
    --backup-region auto --backup-path-style \
    --chain-slot

Then feed it. Either run periodic incrementals from a scheduler:

sluice backup incremental --source-driver postgres --source 'postgres://...source...' \
    --target s3://my-bucket/app-chain \
    --backup-endpoint https://<account>.r2.cloudflarestorage.com \
    --backup-region auto --backup-path-style

…or run a continuous producer that commits rolling incrementals on a cadence (a long-lived process — run it under systemd / k8s). --rollover-window sets how often it commits an incremental; --retain-rotate-at-chain-length rotates into a fresh segment to keep segments compact for pruning:

sluice backup stream run --source-driver postgres --source 'postgres://...source...' \
    --target s3://my-bucket/app-chain \
    --backup-endpoint https://<account>.r2.cloudflarestorage.com \
    --backup-region auto --backup-path-style \
    --rollover-window 10s \
    --retain-rotate-at-chain-length 20

2. Replay it into the target #

On the consumer side, point the broker at the same chain. It reads the chain's catalog every --poll-interval, applies any incrementals newer than its persisted position in chain order, and persists progress in the target's sluice_cdc_state. The --stream-id is required so it can resume cleanly after a restart:

sluice sync from-backup run \
    --backup-target s3://my-bucket/app-chain \
    --backup-endpoint https://<account>.r2.cloudflarestorage.com \
    --backup-region auto --backup-path-style \
    --target-driver postgres --target 'postgres://...target...' \
    --stream-id app-broker \
    --apply-concurrency 4 \
    --poll-interval 10s

Replay safety: exactly-once resume, and when keyless tables are refused #

Resuming inside an incremental (v0.157.0+, ADR-0191). backup stream and backup incremental now record each change's apply identity (the source's transaction id and the change's ordinal in it) in the change chunk, and the incremental's manifest says so with apply_identity. The broker hands those identities to the same exactly-once apply marks sync start uses (sluice_cdc_apply_marks). Its position can also stand inside an incremental: in the same target transaction as the work, at each source-transaction boundary, it records the incremental's id, a digest of its chunk list and the event it has applied through. So when a run is interrupted partway through an incremental (an apply error, a chunk that failed to fetch, a crash, a SIGINT or SIGTERM, a supervisor restart), the next run re-reads that incremental, skips what is already applied and re-delivers from the first source transaction not durably applied. On serial apply that is the one in flight; on the lane apply mode it is the transactions whose lanes had not all committed. The marks then skip any re-delivered change that already landed, so the re-run is exactly-once, key changes and tables without a key included. Through v0.156.12 the broker re-applied the whole incremental instead.

• Changes without an identity. A change the source reader cannot name is recorded without one: a VStream COPY row or interleaved shard group, a MySQL transaction without a GTID or server identity, or a row the capture synthesized (such as an ADD COLUMN fill). On a keyed table it re-applies without marks and converges by key, except for a key moved onto another row inside the one interrupted transaction; the broker logs the WARN BROKER-UNIDENTIFIED-CHANGES once per such incremental, with the count. It is routine on VStream chains. On a keyless table it is refused (below).
• Incrementals without identities. A chain written by an older sluice records none, and neither does any incremental backup compact --smart-compaction rewrote, since collapsing changes across transactions drops them. There the in-flight transaction is re-applied without marks. A table with a PRIMARY KEY or NOT NULL UNIQUE index made of columns the replayed rows carry absorbs that, because the re-applied INSERT upserts on the key. A table with neither would gain a duplicate of every row of that transaction the interrupted run had committed, and so would a target table keyed only on a surrogate the rows do not carry (bigserial, identity, DEFAULT gen_random_uuid(), MySQL AUTO_INCREMENT or a DEFAULT-expression key), because every re-applied row draws a fresh key value and collides with nothing.
• A rewritten incremental. If the incremental the broker is inside was rewritten between the two runs (smart compaction), a different incremental now follows the last one fully applied, or it decodes to fewer events than the position counted, the skip would be counted over a different stream. The broker refuses with SLUICE-E-BROKER-INCREMENTAL-REWRITTEN (exit 3) before applying anything of it. Recover with --reset-target-data. Plain backup compact moves the same chunk bytes and resumes normally, so do not smart-compact a segment a broker still has to replay.

Keyless tables. The broker judges each incremental before applying any of it, for the tables that incremental touches, and refuses with SLUICE-E-BROKER-KEYLESS-TABLE (exit 3) when a table has no key the replayed rows carry and the replay cannot be made exactly-once. The message says which of these applies: the incremental records no identities; a change to the table inside it carries none (BROKER-KEYLESS-NO-IDENTITY); or the target's apply marks cannot cover the table — unusable by the role (APPLY-MARKS-UNAVAILABLE), a PlanetScale Postgres (Neki) target, a target engine without apply marks such as SQLite, or a target keyed only on a column the rows do not carry. The check also runs at start for the incremental a broker is inside or about to start, so a restart learns of a refusal before anything is written, and on a --reset-target-data cold start before the drop. A mark table that becomes unusable after the check refuses the apply rather than letting it run without marks. After an upgrade from v0.156.12 or older mid-incremental, the next incremental may have been partly applied with no marks to skip it, so a keyless table in it refuses with BROKER-CLASSIC-RESUME in the reason (recovering from it); the incrementals after it are judged normally. The broker has no table filter, so every table in the chain is in scope. v0.156.11 and v0.156.12 refused every keyless table in the chain at start, whatever the incremental. Each table's key is judged twice, and the message says which judgment failed:

• The chain's recorded schema — what the source declared: a PRIMARY KEY or a NOT NULL UNIQUE index. A partial UNIQUE index does not count, since a replayed row outside its predicate collides with nothing.
• The live target table — whether a re-written row actually collides. On Postgres, the ON CONFLICT arbiter the applier itself picks (the PRIMARY KEY before any UNIQUE index) must be made entirely of columns the rows supply, so a supplied UNIQUE index beside an unsupplied surrogate primary key does not count. On the MySQL family, some PRIMARY KEY or UNIQUE index of NOT NULL, non-generated columns must be fully supplied. On a multi-shard Vitess or PlanetScale keyspace, every primary-vindex column must also be supplied, because a key there is enforced per shard and a re-sent row without its routing column lands on another shard; the probe reads the keyspace's shards and the table's primary vindex (SHOW VSCHEMA VINDEXES ON) and fails closed if the target credentials cannot run them (measured on vttestserver; whether PlanetScale's restricted roles allow these reads is unverified). A generated key column never counts as supplied. A view, foreign table or materialized view under a chain table's name on a Postgres target is refused too.

The remedy depends on the reason the message gives. For an incremental without identities, --reset-target-data restores the chain through its tail, and the broker then judges only the incrementals that follow — which an upgraded writer records with identities, so upgrade the producer (backup stream / backup incremental) as well as the broker. For the target's apply marks, make them usable (a role that may create and write sluice_cdc_apply_marks). Otherwise put a key on the source and take a new full backup, so the chain's recorded schema carries it (then start on the new chain with --reset-target-data, or --at-chain-id after restoring it yourself); for a table keyless only on the target, give the target table the source's key rather than a surrogate; or replicate those tables with sync start instead.

After an upgrade: BROKER-CLASSIC-RESUME (v0.157.1 hint). None of the remedies above applies to this reason: the chain may already be one this sluice wrote, and re-running the same command refuses again, because the old position is replaced only by applying the incremental it refuses. Recover with --reset-target-data. When the classic position is the only reason the incremental is refused, the hint also offers an operator's assertion: only if you know that no broker run started applying the named incremental after that position was written, delete the stream's row from sluice_cdc_state and run again with --at-chain-id=<the row's last_applied_backup_id>. A later run that refused the incremental — a v0.156.11 or v0.156.12 broker, say — is no evidence: an earlier, older run may have been interrupted inside it and left the same position. If the assertion is wrong, the rows that run committed are duplicated, silently; if you cannot vouch for it, use --reset-target-data. When the incremental also records no identities, carries a change without one, or the target's marks cannot cover the table, that route would only refuse again, and the hint does not offer it. A key added to the target table is not a recovery here: a table keyless on the source may hold duplicate rows, which the key would merge. (v0.157.0's hint for this reason was the general keyless one above, and did not mention --reset-target-data.)

Severed transactions in the chain (v0.156.12+) #

Separately from re-applying an interrupted incremental, a chain an older backup stream wrote can carry one source transaction across two incrementals — a window that ended inside a source transaction (shape A), or, on Postgres chains written before v0.138.0, a resume that re-delivered the previous window's last transaction (shape B) — or an incremental whose EndPosition is past its last stored change, the old cancel-drain loss (shape C). On every tick that brings new incrementals, before applying them, the broker runs the same severed-transaction check chain restore runs, over the whole chain:

• A finding on a link the broker has not applied yet refuses (exit 3) — SLUICE-E-BACKUP-CHAIN-SEVERED-TRANSACTION for shapes A and B, SLUICE-E-BACKUP-INCOMPLETE for shape C — before anything of that tick is applied. There is no override; the remedy is a new full backup and a new chain.
• A finding on a link it already applied — by an older binary, a pre-v0.138.0 Postgres resume, or while that link was still the chain's tip, before the link that re-delivers its transaction arrived — cannot be undone by refusing, so it is logged once per run as the WARN CHAIN-APPLIED-SEVERED-TRANSACTION, naming the links, and the broker continues. Grep the broker log for that marker and compare the tables those links touched with the source (sluice verify --depth count), or rebuild the target from a new full backup.

Shape B is judged only when replaying into Postgres (it needs the source engine's position order). A current backup stream never writes a window that ends inside a source transaction — see who should check for chains written before v0.156.12, and this check for a stream cancelled on v0.156.12 through v0.159.2. Such a window is the case above: a broker that applied it while it was the chain's tip only WARNs CHAIN-APPLIED-SEVERED-TRANSACTION when the next link arrives, and applies the transaction again at exit 0 — a keyless row lands twice, and a key-reusing transaction is misapplied.

3. Cold-start vs warm-resume #

On its first launch against a chain, the broker has no sluice_cdc_state row for the chosen --stream-id, so it doesn't know where in the chain to begin and refuses loudly. There are two ways past that, mutually exclusive:

• --reset-target-data — drop the target's tables, run a chain restore (full + every incremental up to the tail), then transition to live polling. The full from-the-chain rebuild; suitable when the target is empty or you want a clean rebuild. Prompts (type reset) unless --yes.
• --at-chain-id <ID> — operator-asserted resume: tell the broker the target is already at chain ID <ID> (e.g. you just ran a manual sluice restore to bring the target up to a known checkpoint). It writes a fresh state row and tails forward from there — no re-bulking.

Over an existing position (v0.157.0+). Both flags are now read even when the target already holds this stream's broker position; through v0.156.12 the warm resume ran first and both were ignored. --reset-target-data is honoured over any broker-owned position, whether or not it decodes (fully true since v0.157.1; see below): after its pre-drop checks it clears the stream's sluice_cdc_state row and its sluice_cdc_apply_marks rows, then drops and restores as on a fresh target, so a crash mid-reset leaves no position to resume onto a half-restored target. --at-chain-id asserts what a target with no position holds. Over an existing one it is accepted only when it names the link the position records as last fully applied, with nothing in progress; any other value refuses with BROKER-AT-CHAIN-ID-CONFLICT (exit 1) before anything is applied, and repeats on every restart until you drop the flag, use --reset-target-data, or clear the position. If you restored the target to an earlier link yourself, delete the stream's row from sluice_cdc_state and its rows from sluice_cdc_apply_marks first, then pass --at-chain-id. Who owns a row (v0.157.1+). A row is the broker's when its position token's _engine marker says so (backup-broker-v2, or the classic backup-broker), whether or not the rest of it decodes. A broker row damaged from outside — a hand edit, a bad restore of the control table; no broker writes one — refuses a plain run with BROKER-POSITION-CORRUPT (exit 1, uncoded), with nothing applied, naming --reset-target-data (which discards the row and rebuilds the target) and, if you know which link the target holds, deleting the stream's sluice_cdc_state row and running with --at-chain-id. Through v0.157.0 such a row was reported as owned by a non-broker writer, naming the target applier's engine rather than the row's, and --reset-target-data was refused over it — so v0.157.0's claim that the flag was honoured over a broker row "decodable or not" held only for a row that decoded but was semantically corrupt. A row with no broker marker belongs to another writer, such as a sync start stream, and is refused with or without either flag; the refusal now says what the token carries (its _engine marker, or that it has none). Cold starts and chain restore clear the stream's apply marks before applying anything, so a mark left by an earlier run under the same stream id cannot skip a change the fresh target never received.

The common case is the post-restore cold-start: bulk-copy the chain once with sluice restore, then launch the broker with --at-chain-id set to that restore's tail manifest. Pass the flag only on the first launch; every subsequent restart warm-resumes from sluice_cdc_state automatically and needs neither flag:

# first launch after a fresh restore
sluice sync from-backup run --backup-target s3://my-bucket/app-chain \
    --target-driver postgres --target 'postgres://...target...' \
    --stream-id app-broker --apply-concurrency 4 --poll-interval 10s \
    --at-chain-id 9b12b8ccdc3e7fa9725825ab032e6d6d41d3db09

# every restart after that — warm-resume, no recovery flag
sluice sync from-backup run --backup-target s3://my-bucket/app-chain \
    --target-driver postgres --target 'postgres://...target...' \
    --stream-id app-broker --apply-concurrency 4 --poll-interval 10s

4. Stopping cleanly #

Stop the broker by writing a stop signal to the chain destination — the running process observes it on its next tick and exits cleanly. Because the signal lives in the store, you can stop a broker from a different host without process access (both sides agree on the chain, not on the host):

sluice sync from-backup stop --backup-target s3://my-bucket/app-chain

Exit codes on stop and cancel (v0.156.11+). sluice sync from-backup stop is observed only between ticks, so it never interrupts an incremental and the broker exits 0; use it to stop a broker deliberately. A SIGINT or SIGTERM — or q/ctrl+c on the live panel — cancels immediately:

• Between incrementals or between ticks — nothing of an unadvanced incremental was applied; exit 0.
• While an incremental is being applied — part of it may be committed and the position was not advanced, so the run returns an error carrying BROKER-INCREMENTAL-PARTIAL (exit 1) naming the incremental. Re-running the same command resumes at the source transaction that was in flight (v0.157.0+). When the incremental records identities that re-run is exactly-once, and it is the whole recovery whatever the transaction did. When it records none, the re-run converges only if that transaction changed no key value (see above); a source that changes key values recovers with --reset-target-data.
• During a --reset-target-data cold start, once its drop has begun — the target holds a partial restore and no position, so it returns BROKER-COLD-START-PARTIAL (exit 1). Re-run with --reset-target-data; never start that target with --at-chain-id.

Through v0.156.10 both of the latter exited 0 and the panel printed "stopped." — supervisors that treat a cancel's exit status as success should expect the change.

Separately, a broker that reaches a severed-transaction finding on a link it has not applied yet exits 3 (v0.156.12+) with SLUICE-E-BACKUP-CHAIN-SEVERED-TRANSACTION or SLUICE-E-BACKUP-INCOMPLETE; restarting repeats it, so alert on it rather than counting on the retry. A finding on a link it already applied only WARNs CHAIN-APPLIED-SEVERED-TRANSACTION and the broker keeps running.