sluice

The alert cleared at the exact moment the slot died

Monitor a Postgres replication slot by WAL-retention pressure and you inherit a sign flip at the terminal event: when Postgres invalidates the slot, pg_replication_slots reports wal_status='lost' and the lag columns go NULL — not huge. Coerce NULL to zero, compute 0% pressure, and your threshold evaluator concludes the condition cleared. The operator is told the pressure resolved at precisely the moment it became fatal.

Landed in sluice · 2026-07-16 · Postgres

Observed — live on PG16, first by the 2026-07-15 repo audit (MED-D0-9, against sluice's own slot-health evaluator) and then reproduced as a staged before/after differential by the v0.99.258 regression cycle, exact log lines below. This was sluice's bug, not Postgres's: the catalog behavior is documented. Fixed in v0.99.258. One mitigating fact up front: the CDC streamer itself fails loudly on a lost slot, so this was an alerting-truth inversion, not a data-loss path — the feature that existed to give operators early warning gave them a false all-clear instead.

What happened #

sluice's slot-health watcher pages on WAL-retention pressure: percent of max_slot_wal_keep_size retained for the slot, WARN at 70%, CRITICAL at 85%. The natural implementation reads the slot's lag from pg_replication_slots, computes a percentage, and compares against thresholds — and that implementation contains the trap twice over:

So the observed sequence on the shipped binary was: page CRITICAL at 85%, keep paging as pressure climbs — then, at the exact tick the slot became irrecoverable, log “condition cleared” and go quiet. Terminal states in monitoring views often present as absent data rather than extreme data, and absent-means-clean defaults invert the alert exactly when it matters most.

The live differential #

The v0.99.258 cycle staged it on a throwaway postgres:16 with max_slot_wal_keep_size=1MB: stall the consumer realistically, burn WAL to ~92% of the cap (lag 969,120 bytes, identical on both binaries), take one in-condition retention pressure CRITICAL tick, then burn past the cap and CHECKPOINT so the slot invalidates live.

binary       at the terminal event
---------    ------------------------------------------------------------
v0.99.257    next tick emits `postgres: slot-health condition cleared`
             with wal_status=lost lag_bytes=0 — the false all-clear —
             and NO lost page ever (lost=0, cleared=1)
v0.99.258    exactly ONE terminal ERROR: `slot INVALIDATED
             (wal_status=lost) — terminal, re-snapshot required`;
             ZERO clears; the pre-invalidation retention WARN intact

A staging detail worth recording: a plain fast burn on v0.99.257 (clean straight to lost, no sampled in-condition tick) emits nothing at all — reproducing the audit's exact false-clear shape needs the staged burn, one retention tick first.

The fix's shape — and its two siblings #

The repair is instructive beyond the one bug. First, dispatch on wal_status before any percentage math: lost pages CRITICAL exactly once and latches — never repeats, never clears — because the state is terminal by definition. But unreserved pages CRITICAL and stays clearable, because Postgres documents that an unreserved slot can recover; latching a recoverable state would re-create the inversion in the other direction (a stale alarm the operator learns to ignore). Truthful alerting cuts both ways.

Two siblings shipped in the same pass, both variations on “the net must be truthful at the terminal event”:

Plus one-tick hysteresis on threshold downgrades, so a catch-up hovering at the 85% boundary doesn't page every 30 seconds in both directions.

Reproducing it #

Throwaway container, ~5 minutes (this is the regression-cycle recipe):

docker run -d --name slotpg -e POSTGRES_PASSWORD=x -p 5460:5432 postgres:16 \
  -c wal_level=logical -c max_slot_wal_keep_size=1MB

-- create a logical slot, attach a consumer (sluice sync or pg_recvlogical), then stall it
-- burn WAL past the cap and force recycling:
SELECT pg_switch_wal(); CHECKPOINT;   -- repeat with junk writes until:
SELECT active, wal_status, safe_wal_size FROM pg_replication_slots;
--  f | lost | NULL          <- the terminal row: status text, NULL numbers

Any monitor that computes pressure from the lag/safe_wal_size columns and treats NULL as zero will read that row as 0% — sluice ≤ v0.99.257 logged “condition cleared” on it; ≥ v0.99.258 pages terminal CRITICAL once. To see the false clear specifically, let the monitor sample one in-condition tick before the burn crosses the cap.

The transferable lesson #

When a monitored resource dies, its metrics don't spike — they vanish. Check the status column before the arithmetic, and treat absent data as “cannot assert health,” never as zero. Then audit the alert lifecycle at its edges: a terminal state should latch (one page, no clears, no repeats), a documented-recoverable state should not, a probe that can't see must say so, and a once-only notification must not mark itself sent until it was. Every one of those edges defaults to the quiet-and-wrong behavior if unexamined — and the quiet failure of an alerting net is indistinguishable from good news.

Primary sources #