sluice

Persist a resume cursor as JSON and it silently teleports

A resumable keyset walk persists its last-processed PK between runs, and the obvious store is JSON — which quietly rewrites database values on the way through. We saw half the class at design time ([]byte → base64, time.Time → RFC 3339) and normalized it. The half we missed fired live: Go's json.Marshal replaces invalid-UTF-8 bytes with U+FFFD, and numbers ride float64 past 2^53 — so a resumed walk skipped 73,100 of 100,000 rows at exit 0. Resume state is a codec too, and ours had zero coverage.

Landed in sluice · 2026-07-16 · Cross-cutting

Observed — in two acts. Act one caught at design time while building sluice backfill's resume path (ADR-0159, v0.99.244) and normalized before ship. Act two found by the 2026-07-15 repo audit (CRITICAL-2 and HIGH-1, both confirmed on live MySQL) and then exhibited on the shipped binaries by the v0.99.257 regression cycle with exact magnitudes — the numbers below are from real differential runs. Fixed in v0.99.257 with a typed envelope. The exposure was sluice's own: backfill resume since v0.99.244, and the migrate copy-resume cursor rode the same store.

Act one: the halves we saw #

A keyset walk — backfill, chunked copy, verify — persists a cursor between runs: the last-processed primary-key tuple, re-bound into the resume predicate WHERE (pk…) > (cursor…). The obvious store for a heterogeneous tuple is JSON, and Go's encoding/json rewrites two whole type families on the way through:

Both re-bound values are valid — no parse error, no refusal — just a walk that resumes from somewhere other than where it stopped. A misplaced cursor is the worst failure class for resumable work because the loss shows up as position, not as a message. The v0.99.244 design saw this and normalized at the scan boundary: []byte to its raw string form, time.Time to the engine's own comparable literal, pinned by unit tests in both engines.

Act two: the halves we missed #

The audit's finding was that the normalization itself handed encoding/json a Go string of raw bytes — and json.Marshal silently replaces every invalid-UTF-8 byte with U+FFFD (EF BF BD). A BINARY PK cursor 0x9F8041FE10 round-trips to 0xEFBFBDEFBFBD41EFBFBD10: 5 bytes become 11, and the mangled value sorts bytewise greater — so on live MySQL the resumed predicate (id) > (<mangled>) skips every PK range between the true cursor and the imposter. BINARY(16) UUID keys are mainstream; the run reports success; --verify is optional; and a following contract step (DROP the old column) makes the loss permanent.

The second missed half was the documented one: TableProgress decoded its JSON with a plain decoder — no UseNumber — so every number rode float64. A persisted cursor of 9007199254740995 decoded as …996 (+1: a row skipped forever); a snowflake-magnitude 1750000000000000123 drifted −123 (float64 granularity ≈256 up there), replaying past the documented at-most-one-chunk bound.

The regression cycle put exact magnitudes on all of it, running the shipped binaries against each other:

shape                          v0.99.256 (before)                    v0.99.257 (after)
---------------------------    ----------------------------------    -----------------------------
backfill, BINARY(16) PKs       resumes its own mangled cursor at     coded SLUICE-E-BACKFILL-
(every key leading 0x80)       rc=0, marks the run complete —        CORRUPT-CURSOR, rc=3, names
                               73,100 of 100,000 rows NEVER          --restart + the U+FFFD
                               visited (exactly 100,000 minus        fingerprint, writes nothing;
                               the 26,900 persisted as copied)       --restart heals, 100,000 exact
backfill, PG BIGINT dense      cursor 1152921504606879676 rounds     envelope cursor; resume exact
near 2^60                      up +68; EXACTLY the predicted 68
                               rows skipped, row-for-row
migrate --resume, bytea PK     its own writer persisted base64       WARN "persisted resume cursor
                               text; resume rc=0, complete,          is not trustworthy; truncating
                               46,000 of 200,000 rows never          and re-copying" → 200,000
                               copied                                exact, checksum == source

The float64 row deserves a second look: the skip window matched the drift arithmetic row-for-row. This class isn't flaky — it's deterministic corruption of position, which is why it survives every rerun.

The part that stings: we already owned the fix #

sluice's backup values had been protected from exactly these rewrites since v0.99.159 — Bug 172 introduced a tagged-envelope codec ({"_t":"i64"}-style) precisely because JSON mangles bytes and big integers. The value path also had a year of the family-matrix test discipline: pin every type family, not one representative. The cursor store got neither: it stayed a bare json.Marshal, and its one resume integration pin exercised only INT primary keys. Checkpoint state is a codec with all the same failure modes as the data path — it just corrupts where you are instead of what you have.

The v0.99.257 fix applies the owned remedy: cursor slices persist as typed envelopes ({"_t":"i64"|"u64"|"bytes"|"f64"|"time"}); valid-UTF-8 strings, bools, and nulls stay bare, and a string containing U+FFFD is enveloped as bytes so mangled and legitimate stay distinguishable. Legacy bare cursors parse exact-int64-first (including BIGINT UNSIGNED above MaxInt64), so pre-envelope integer cursors keep resuming losslessly — the cycle proved a v0.99.256-written cursor resumes under v0.99.257 with no refusal. Provably-mangled legacy cursors are handled where the PK types are known: backfill refuses with a coded error naming --restart; migrate self-heals through its existing truncate-and-redo disposition. And resume fidelity is now integration-pinned per orderable PK family — large-int, binary, composite, temporal, multibyte string — on real MySQL and Postgres.

Reproducing it #

The rewrites are visible in a few lines of Go — no database required:

cursor := map[string]any{
    "pk": string([]byte{0x9F, 0x80, 0x41, 0xFE, 0x10}), // raw bytes as string
    "n":  int64(9007199254740995),
    "ts": time.Date(2026, 7, 14, 10, 30, 0, 0, time.UTC),
}
b, _ := json.Marshal(cursor)
fmt.Println(string(b))
// "pk" is now U+FFFD-riddled (5 bytes -> 11), "ts" is a T/Z literal MySQL
// won't compare; decode "n" without UseNumber and it comes back ...996

The end-to-end shape, on sluice: create a table with a BINARY(16) PK whose keys lead with 0x80+ bytes, start sluice backfill (or an interrupted migrate) against v0.99.256, kill it mid-walk, and resume: rc=0, “complete,” and every row above the true cursor's mangled ghost untouched — the NULL-visible fixture (--set 'new = old + 1' --where 'new IS NULL') makes the skipped rows countable. The same states under ≥ v0.99.257 refuse coded or self-heal.

The transferable lesson #

JSON is a lossy wire format for database values — it base64s or replacement-characters bytes, reformats times, and floats integers — and checkpoint/resume state is where those rewrites do maximum damage, because a corrupted cursor doesn't corrupt a cell you might notice: it relocates the walk, silently, at exit 0. Treat resume state as a codec: give every type family crossing the boundary an explicit representation decision (a tagged envelope beats per-type normalization — our normalization was itself the vector for the missed half), decode integers exactly, and pin the round-trip per PK family on real engines. And when your data path already has a hardened codec and a testing discipline, ask what other stores in the system quietly round-trip the same values without either — ours was the one that decides which rows get processed at all.

Primary sources #