sluice

mydumper chunk numbers are PK ranges, not a sequence

Every consumer's instinct says numbered chunk files — table.00000.sql, 00001, … — form a sequence, so a missing file is detectable as a gap. Ground truth from real mydumper: the numbers are derived from primary-key ranges. Healthy dumps have gaps, -r dumps start at 00001, and a deleted trailing chunk leaves no gap at all — so contiguity is neither necessary nor sufficient. Meanwhile a deleted middle chunk streams silently short at exit 0, and the loss detector the format actually ships is the metadata everyone skips as informational.

Landed in sluice · 2026-07-16 · MySQL & Vitess

Observed — the 2026-07-15 repo audit (MED-D0-2: a dump with chunks 00000+00002 streamed 2 of 3 rows at exit 0, zero WARNs, and verify re-counted the same directory) plus a ground-truth probe against real mydumper v1.0.3 in Docker during the fix work. The silent short-stream was sluice's own gap — the reader explicitly dropped the dump's -metadata/-checksum companions as “informational only.” Two detection nets shipped in v0.99.258 and were proven both-sides by the regression cycle (numbers below). mydumper's numbering itself is by design; nothing to file upstream.

What happened #

Delete one data chunk from the middle of a mydumper dump and restore it. What should happen: an error, or at least a warning. What did happen through sluice v0.99.257: the remaining chunks stream, every row in them lands, exit 0, no signal anywhere. The regression cycle's fixture — three chunks, 15 rows total, rm the middle one — landed 10 rows with zero WARNs. A verifier re-scanning the same directory confirms the short count. Chunk-file-level loss is maximally quiet because each chunk is a complete, well-formed SQL file; nothing about the survivors reveals the absence.

The obvious net is a contiguity check on the chunk numbers. That's where the ground-truth probe earned its keep, because the obvious net is wrong:

So contiguity is neither necessary (healthy dumps have gaps) nor sufficient (trailing loss shows no gap). An alarm built on the assumed semantics would have both false-alarmed and under-detected.

The net was inside the dump the whole time #

mydumper records per-table row counts in its own metadata — rows = N entries in the modern dump-wide ini form, bare-integer -metadata companion files on older versions. The probe found the modern counts exact on v1.0.3. That's the free cross-check that catches chunk-level loss regardless of numbering: count what you streamed, compare with what the producer said it wrote. sluice's reader had been explicitly discarding those companions as “informational only” — forfeiting the only loss detector the format ships.

v0.99.258 added both nets, deliberately as WARNs rather than refusals (cross-producer metadata fidelity is unverified, and gaps can be legitimate): a non-contiguous chunk-number WARN at open naming the gap, and the decisive post-stream row-count tripwire on both the bulk-copy and verify-count doors. The regression cycle's differential, same deleted-middle-chunk fixture:

binary       outcome
---------    ----------------------------------------------------------
v0.99.257    rc=0, 10 of 15 rows, ZERO warns — the silent short-stream
v0.99.258    rc=0, same 10 rows, BOTH nets fire: the contiguity WARN
             (gap_after_chunk=0 next_chunk=2) and the count WARN
             naming metadata_rows=15 chunk_rows=10

One honest residual, stated because it's exactly the class this note is about: pscale-dump (the PlanetScale producer of the same format family) writes empty metadata companions, which sluice ignores leniently — so on that producer, a deleted trailing chunk leaves neither a gap nor a count mismatch. The net is only as good as the producer's metadata, per producer.

Reproducing it #

Real mydumper via Docker, any MySQL:

docker run --rm --network host mydumper/mydumper:v1.0.3-1 \
  mydumper -h 127.0.0.1 -u root -p secret -B mydb -o /dump -r 5   # small -r forces multiple chunks

ls dump/mydb.t.*.sql          # note: -r dumps start at 00001; sparse-PK tables can jump
grep -A2 '\[`mydb`.`t`\]' dump/metadata   # the dump's own per-table "rows = N"

rm dump/mydb.t.00002.sql      # delete a middle chunk

sluice migrate --source-driver=mydumper --source ./dump --target-driver=mysql --target '<dsn>'
# <= v0.99.257: rc=0, silently short, no signal
# >= v0.99.258: rc=0 + the gap WARN + "metadata records a different row count" naming both numbers

To see the numbering semantics directly: dump a table whose PKs live in two distant bands (say 1–1000 and 450000–451000) and observe the chunk numbers jump with the keyspace.

The transferable lesson #

Two lessons stacked. First: an identifier derived from data is not a sequence — before you alarm on a producer's numbering, verify what the numbers mean, on the real producer, or your check will reject healthy inputs and miss unhealthy ones. (The verification cost here was one Docker probe; the alternative was a contiguity refusal that false-alarms on every sparse-PK dump.) Second, the sharper one: the metadata your reader skips as “informational” may be the only loss detector the format ships. A dump format's row counts, checksums, and manifests exist because the producer knew exactly what it wrote — a consumer that discards them is choosing to verify nothing against the one party with ground truth.

Primary sources #