# Object stores can now say "that changed since you read it" — the portability layer can't ask > Every writer of a backup chain shared one read-modify-write JSON catalog with no arbitration — last Put wins, loser's update silently vanishes. The fix wants compare-and-swap on the object, and the major stores now have it — but the portable object-store surface exposes only create-if-absent. You can build a CAS out of create-only (since ground-truthed live on real S3, GCS, and Azure Blob — three stores that answer an occupied-key conditional write with three different status codes). You can also be honest about the millisecond window it can't close — and about the fact that the primitive that would close it exists upstream, one abstraction layer away. Observed — hardening sluice's backup-chain catalog (ADR-0160, shipped v0.99.246), then ground-truthed against real AWS S3 (us-east-1), Google Cloud Storage (us-east1), and Azure Blob (eastus), all 2026-07-16: conditional-PUT enforcement, exactly-one-winner concurrency, and sluice's own coded conflict refusal observed end to end through the production path on all three. The unguarded race was real but never field-observed; the design constraint — no portable conditional overwrite — turned out to be a property of the portability layer, not the stores, and that reversal is the note. ## What happened A sluice backup chain has one structural record: a lineage catalog, a JSON object listing every segment and its parentage. Every writer — full-backup finalize, cron'd incrementals, stream rollovers, compaction, prune — loaded it, mutated it in memory, and Put it back whole. Two concurrent writers (two cron incrementals, a backup racing a prune, an operator double-start) interleave, the last Put wins, and the loser's structural update silently vanishes: a lost catalog append at best, a mis-parented chain at worst. The data chunks themselves are write-once at distinct paths; the catalog read-modify-write was the one unguarded shared object. The textbook fix is conditional overwrite — Put-if-ETag-matches, a true CAS on the object. The stores themselves have it: AWS S3 gained conditional writes in late 2024 — If-None-Match creates and, with them, If-Match ETag overwrites — GCS has always had generation-match preconditions, Azure has If-Match. But sluice's cloud backends ride a portable abstraction (gocloud.dev/blob), and the portable surface exposes exactly one conditional primitive: create-if-absent (mapping to If-None-Match: * on S3 and Azure, a generation-0 precondition on GCS, O_EXCL on local files). No If-Match — the writer options simply have no field for it (verified in source: nothing in v0.46.0's blob/ handles it, and no plan is visible). For years that was the fleet's fault — S3 had no conditional writes at all, and the portable layer tracked the floor of the fleet. The floor has moved; the abstraction hasn't. ## Building CAS from create-only The catalog write becomes a compare-and-swap on a chain write-generation, arbitrated by create-only claim markers at the chain root (lineage.gen/g-): - Observe, then read. At load, list the markers and record the max claimed generation — before reading the catalog. The order is load-bearing: reversed, you can observe past a competitor's just-landed update and silently clobber it; observe-first turns that window into a spurious-but-safe conflict. - Claim, then Put. At write, create marker g-. Create-if-absent guarantees exactly one concurrent writer wins the slot; the loser's create fails and the write refuses loudly — coded, with the marker's forensic body (host, pid, timestamp) pointing at the other writer — having changed nothing. - GC a trailing window of old markers after success. The liveness property is the part worth stealing: an orphaned marker from a crashed writer is not a stale lock. The next writer's observation lists markers, not catalog content, so the orphan simply becomes the new base and the next generation is claimed after it. No TTLs, no leases, no clock trust, no manual unlock. The rejected alternative makes the contrast sharp — storing the generation counter inside the catalog would turn a crashed claim into a permanent conflict with the recorded counter, a bricked chain needing manual repair. Listing markers as the observation source is what buys lock-free liveness. ## Verified on the real thing The scheme was originally pinned against MinIO and derived for AWS from documentation; a 2026-07-16 probe against real S3 (us-east-1) closed that gap, both at the raw API and through sluice's production path. Raw layer: If-None-Match: * PUT returns 200 on a fresh key and 412 PreconditionFailed on an occupied one; two concurrent conditional PUTs on one fresh key produce exactly one 200 and one 412, final content the winner's. Through sluice's own BlobStore (gocloud's s3blob), eight concurrent claim attempts produced exactly one winner and seven coded losers, and the full interleaved-writer scenario ended in the coded SLUICE-E-BACKUP-CHAIN-CONFLICT refusal naming the other writer's marker, with the loser having written nothing. The library genuinely sends the header — s3blob.go sets IfNoneMatch = "*" when asked, and a 412 on a PUT is only possible if a precondition was sent and enforced server-side. Real create-only CAS, end to end, no silent-ignore. One nuance the probe surfaced without reproducing: under truly simultaneous in-flight conditional PUTs, S3 can hand the loser a 409 ConditionalRequestConflict instead of the 412. gocloud maps only the 412 to the failed-precondition error the guard's conflict branch keys on, so a 409 loser would fall through to the guard's capability-degrade path — an unguarded write plus a WARN whose “the store may not support conditional PUTs” wording would be wrong in that instant. Safe direction (the loser is unguarded, never corrupted-by-guard, and the store's own arbitration already refused it), but it's a mapped-wrong degrade rather than the coded refusal; sluice v0.99.263 closed it by treating the 409 as a conflict, not a capability signal — a 409 means another conditional request was in flight and nothing was written, so the PUT is retried once (a clean retry means this writer won after all), and any still-contended outcome routes to the coded chain-conflict refusal, never the degrade path. The transferable half: when you build on a conditional primitive, audit the error mapping too — a store can refuse your precondition with a status your library doesn't translate. ## Three clouds, three loser shapes S3 was the first cloud row verified; GCS and Azure closed the matrix on the same day, and the interesting part is that all three answer “that key already exists” with a different status code — yet a portable guard survives all three, because the portability layer absorbs the divergence. The occupied-key create-only PUT returns: - S3: 412 PreconditionFailed — the normal loser, with the rare 409 ConditionalRequestConflict as an in-flight artifact under truly simultaneous PUTs (the one the v0.99.263 retry closes). - GCS: 412 conditionNotMet, uniformly — no 409-analogue exists. Under an eight-way simultaneous race GCS produced only 412s; it serializes conditional writes on the object generation, so there is no in-flight-conflict code to catch. The v0.99.263 S3-409 classifier never engages here — correct scoping, not a coverage gap. - Azure: 409 BlobAlreadyExists, uniformly — and this is Azure's normal occupied-key answer, not an anomaly. On Azure the codes are cleanly split: 409 is the create-only (If-None-Match: *) loser and 412 is the overwrite (If-Match) loser, where S3 folds both onto 412. Azure was the row that could have gone wrong. If gocloud mapped only the 412-family codes, every Azure conflict would have landed on the degrade-WARN path and the guard would be useless there. It doesn't: gocloud's azureblob driver explicitly routes BlobAlreadyExists (alongside the ConditionNotMet family) to the same failed-precondition error, with an in-source comment noting that the documented code is a variation of ConditionNotMet but BlobAlreadyExists “has also been observed” — the upstream maintainers hit exactly this divergence and absorbed it. Through sluice's own BlobStore, all three stores produced exactly one winner and the coded SLUICE-E-BACKUP-CHAIN-CONFLICT refusal for every loser, zero degrade-shaped errors, at both two-way and eight-way concurrency. Each store also has a residual throttling loser shape (GCS 429 rateLimitExceeded, Azure 503 ServerBusy) that maps to the degrade path rather than the coded refusal — safe direction, structurally unlikely against one-shot per-generation claim markers, not reproduced even at eight-way. The lesson the matrix carries: portability across object stores is real, but it lives in the error mapping, and the mapping is where a store's idiosyncratic status code either gets absorbed or silently breaks your guard. ## The residual, stated honestly A create-only CAS narrows the race; it cannot close it. Claim and Put are two operations, and a competitor whose observation lands inside another writer's claim-to-Put window — normally milliseconds, a marker PUT followed by the catalog PUT of an already-marshaled body — reads the pre-Put catalog, claims the next generation, and both writers Put unconditionally: last-write-wins again, undetected. The guard shrinks the silently-vulnerable window from the whole seconds-wide read-modify-write span to that millisecond gap. (The probe confirmed the window is real on live S3: the final catalog Put is an unconditional 200-overwrite.) True closure needs conditional overwrite on the catalog object itself — and the probes confirmed that primitive now exists on all three stores: an If-Match / generation-match PUT returned success against the current ETag/generation and a failed-precondition against a stale one, exactly the lost-update refusal the catalog wants (S3's If-Match, GCS's ifGenerationMatch=, Azure's If-Match). Closure no longer waits on any provider; it waits on the portability layer growing conditional overwrite, or on a per-provider SDK write (aws-sdk-go-v2's PutObject takes it today, GCS's storage.Conditions{GenerationMatch: N}, Azure's If-Match). That residual is documented in the ADR and accepted for v1 — a guard whose limits you can state is worth more than one you believe is airtight. ## The transferable lesson If your coordination lives on an object store, inventory the conditional primitives you actually have — and check which layer is withholding them. Portably it may be create-if-absent and nothing else, even when every store underneath now offers conditional overwrite; the constraint that shapes your design can belong to the client library, not the service. Know the two patterns create-only buys you: exactly-one-winner arbitration (claim markers) and lock-free liveness (observe the markers, not the guarded object, so a crashed claimant is a base, not a blocker). Order matters twice (observe before read; claim before put), and the honest accounting matters most: create-only CAS is a race-narrower, the claim-to-Put gap is the price of building on the one primitive your abstraction exposes — and when the missing primitive appears one layer up, say so, because that turns “accepted residual” into “closable, per-provider, whenever it's promoted.” This is the same family as our database-side note that CREATE … IF NOT EXISTS is not a lock — existence checks arbitrate creation, never modification. ## Primary sources - gocloud.dev/blob — WriterOptions.IfNotExist and its per-provider mapping (S3/Azure If-None-Match: *, GCS generation-0, local O_EXCL); no If-Match field in v0.46.0 (verified in source). - AWS S3 — conditional requests: If-None-Match creates and If-Match ETag overwrites (GA November 2024); 409 ConditionalRequestConflict on overlapping in-flight conditional writes. GCS generation-match preconditions (ifGenerationMatch); Azure Blob If-None-Match: * (returning 409 BlobAlreadyExists on an occupied key) and If-Match. - Real-cloud ground-truth probes (2026-07-16) — S3 (us-east-1), GCS (us-east1), and Azure Blob (eastus): the per-store loser-code matrix (S3 412+rare-409, GCS uniform 412, Azure uniform 409 BlobAlreadyExists), exactly-one-winner concurrency at two- and eight-way, and wire-level confirmation that gocloud sends the precondition and maps each store's loser code to the failed-precondition error through sluice's own BlobStore path. - sluice ADR-0160 — the backup-chain concurrent-writer guard: the marker scheme, the rejected alternatives, and the claim-to-Put residual. --- Canonical page: https://sluicesync.com/field-notes/object-store-create-only-cas/ · Full docs index: https://sluicesync.com/llms.txt