docs(wire-format): scope compressed-byte reproducibility per-vector (LAB-1751)

.github/workflows/verify.yml

@@ -27,6 +27,9 @@ jobs:
 
       - name: Python reference verify (stdlib only)
         run: |
+          # Mutation suite first, same doctrine as the version-floor guard below:
+          # prove the fail-closed guards still fail before trusting the verify.
+          python3 tools/test_wire_format_reference.py
           python3 tools/interop-reference.py verify
           python3 tools/interop-v2-reference.py verify
           python3 tools/encryption-verify.py
@@ -38,7 +41,7 @@ jobs:
           python3 tools/interop-reference.py verify
           python3 tools/interop-v2-reference.py verify
           python3 tools/encryption-verify.py --require-seal
-          python3 tools/wire-format-reference.py verify
+          python3 tools/wire-format-reference.py verify --require-extras
 
       - name: JS cross-check (independent encoder + @noble/hashes + WebCrypto)
         run: |

CHANGELOG.md

@@ -4,6 +4,119 @@ All notable changes to the CacheKit Protocol Specification.
 
 ## [Unreleased]
 
+### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751)
+
+- LZ4 compressed bytes are **not canonical** across conforming block encoders.
+  [`spec/wire-format.md`](spec/wire-format.md) now states this explicitly
+  (new "Compressed-byte reproducibility" section, mirroring interop v2's
+  doctrine): `compressed_data` conformance is read-side only, a **non-canonical**
+  writer is never judged non-conforming for differing from the pinned bytes
+  (byte-comparison as a declared-divergence tripwire remains allowed), and
+  only the canonical writer (`lz4_flex` via `cachekit-core`) has enforced
+  byte-reproducibility. The `large_compressible` / `large_compressible_bin`
+  pair is marked **known encode-divergent, decode-verified only** under the
+  spec's own reference liblz4 mapping — `lz4.block.compress(store_size=False)`
+  emits a 14 B block where the fixture pins `lz4_flex`'s 15 B. Found by
+  execution during the LAB-868 panel review; resolves the trust bug of a
+  fixture implying a reproducibility property the reference toolchain cannot
+  produce. Regeneration was rejected: every envelope-using SDK compresses
+  through `cachekit-core`'s `lz4_flex` (`cachekit-rs` writes plain MessagePack
+  with no envelope — spec 'SDK Storage Containers (auto mode)'), whose CI asserts re-encode byte-identity, so
+  re-pinning to liblz4 output would break the canonical writer and merely swap
+  which compressor diverges.
+- [`tools/wire-format-reference.py`](tools/wire-format-reference.py) `verify`
+  gains an optional `lz4` leg (the dependency was already installed in CI's
+  optional-deps step): liblz4 MUST decompress every pinned `compressed_data`
+  to the pinned input; encoder agreement is asserted only as a drift
+  tripwire against `LZ4_ENCODE_DIVERGENT`, never as a per-vector conformance rule. The CI invocation now passes `--require-extras`
+  (precedent: `encryption-verify.py --require-seal`) so a dependency drift
+  cannot silently turn the deeper checks off. Fixture bytes untouched
+  (version stays 1.1.1) — no downstream SDK re-vendors required.
+- Expert-panel hardening of the same verifier (crypto/protocol gate; every item
+  below was reproduced by poisoning the fixture and re-run after the fix):
+  - `original_size` is now checked against `len(input_hex)`, not just the
+    co-located `input_size` field. Both declared sizes live *in* the file under
+    test, so a regeneration bug that inflates them drifts them together and the
+    old check still passed — a vector declaring 100 MB for 16 bytes of input
+    verified green, and liblz4 did not catch it because
+    `decompress(uncompressed_size=…)` sizes the output buffer rather than
+    asserting the length. Runs on both CI legs (stdlib and optional-deps).
+  - **Both** commands refuse to run under `-O`/`PYTHONOPTIMIZE`: every conformance
+    check is an `assert`, so an optimised `verify` reported "all 7 vector pairs
+    verified" against a poisoned fixture, and an optimised `generate` rewrote the
+    fixture with its input checks stripped. The guard is at module scope, not in
+    `main()`, because a CLI-only guard is bypassed by importing the module and
+    calling `verify()` directly — which the regression harness's `importlib`
+    probe does, and which is how the sibling tools load each other's codecs.
+  - `generate` is now **append-only**: it refuses to write when the rebuild would
+    drop a committed vector. It previously rebuilt `vectors` from the legacy set
+    alone, so a bin vector with no legacy base was erased silently — and because
+    `verify`'s orphan FAIL names `generate` as the remedy, the documented repair
+    step completed the data loss. Reproduced end to end: dropping legacy
+    `width_boundary_bin16` (the fleet's only bin16 coverage) left `generate`
+    reporting success on a fixture two vectors smaller, with CI green.
+  - `--require-extras` is rejected outside `verify` (exit 2). It was accepted and
+    silently ignored on `generate`, the fixture-writing path — the same
+    accepted-and-dropped fail-open the unrecognised-argument check closes.
+  - Unrecognised arguments now exit 2 instead of being dropped, closing a
+    fail-open in the new flag itself: `verify --require-extra` (one character
+    short) exited 0 with the extras legs silently off.
+  - The set of vectors liblz4 fails to reproduce on encode is pinned in
+    `LZ4_ENCODE_DIVERGENT` and asserted, so a toolchain bump that changes it
+    fails CI instead of quietly making the new spec section's prose wrong.
+- Second expert-panel round on the remediated verifier (crypto/protocol gate
+  keys off current HEAD, not "a panel ran once"). Three whole-file fail-opens,
+  all reproduced by execution and all previously exit-0:
+  - **The base-vector set is now pinned in code** (`EXPECTED_BASE_VECTORS`).
+    Every other check iterates the fixture's own vector list and so is
+    structurally blind to a vector that is simply *absent*. Dropping a legacy
+    base **and** its `_bin` twin together — the realistic bad-merge shape, which
+    the orphan-twin refusal does not cover — netted to zero in `generate`'s
+    append-only diff: `verify` reported "all 6 vector pairs verified" and
+    `generate` wrote the 12-vector fixture, both exit 0. It also silently
+    disarmed `LZ4_ENCODE_DIVERGENT`, since the divergent vector was no longer
+    iterated. Same lesson as `original_size`/`input_size` one level up: a name
+    list derived from the artifact under test pins nothing.
+  - **The fixture's declared `limits` block is now compared against the spec's
+    Security Limits table.** SDKs read their bounds from that block and nothing
+    pinned it either way, so a fixture rewriting `max_uncompressed_size` to `1`
+    verified green while handing every downstream reader a wrong bound.
+  - **A declared-divergent vector's `compressed_data` is now byte-pinned.**
+    `assert diverges == (name in LZ4_ENCODE_DIVERGENT)` is a one-bit check that
+    any other valid LZ4 block satisfies, so re-pinning `large_compressible` to
+    an unrelated (valid, correctly-decompressing) block passed both CI legs. The
+    byte-pin sits outside the optional-deps gate, so the one vector this section
+    exists to document is enforced on the stdlib leg too — it has no
+    canonical-writer check anywhere else in the fleet.
+  - `tools/test_wire_format_reference.py` gains mutation cases for all three,
+    each verified non-vacuous by deleting the guard and confirming the case
+    fails. Its own invocations that can reach `generate` now run against a
+    scratch mirror rather than the repo's sha256-pinned fixture — with the
+    guard regressed, the suite (CI's first step) rewrote the vendored artifact.
+    Exit-code-only assertions gained guard-marker checks, because python itself
+    exits 2 on a bad script path and 1 on a traceback, which made an
+    exit-code-only case pass vacuously.
+  - Fixture-shape rejections now name the offending vector instead of exiting
+    via a bare traceback.
+- [`spec/wire-format.md`](spec/wire-format.md) corrections from the same panel:
+  the "MUST NOT byte-compare a writer's compressor output" rule is scoped to
+  **non-canonical** writers — unscoped, it forbade the `cachekit-core` re-encode
+  assertions that the very next paragraph relies on as the enforcement
+  mechanism, i.e. the fleet's only `lz4_flex` drift detector. The claim that
+  cachekit-core enforces canonical-writer reproducibility is now scoped to the
+  vectors that repo actually vendors: core pins `version == "1.1.0"`, so
+  `width_boundary_bin16` (added at 1.1.1) has no **canonical-writer
+  (`lz4_flex`) compressed-byte** check anywhere in the fleet, and its pinned
+  xxh3-64 checksum is recomputed nowhere. The earlier phrasing — "no
+  encode-side check anywhere" — was too broad and is corrected: this repo's
+  verifier does assert that vector's legacy and bin re-encode byte-identity on
+  every run, and liblz4 reproduces its compressed bytes on the optional leg. The
+  spec also now names what re-vendoring 1.1.1 into cachekit-core actually
+  requires: bump `FIXTURE_SHA256`, bump the version pin, **and** relax
+  `assert_eq!(twin_bytes[1], 0xc4)` to accept `0xc5` — that assertion demands
+  every twin be bin8, and `width_boundary_bin16_bin` is bin16, so a drop-in
+  re-vendor fails it. A remedy that fails on contact leaves the gap open longer.
+
 ### Interop v2 — compressed-values profile (DRAFT)
 
 - New [`spec/interop-v2.md`](spec/interop-v2.md) (LAB-1135, protocol#52):

spec/wire-format.md

@@ -40,7 +40,13 @@ This document specifies two layers:
    vendors the file sha256-pinned in `tests/wire_format_vectors.rs`, asserting
    decode byte-identity for every vector and re-encode byte-identity for the
    canonical `*_bin` vectors only — legacy array-of-integers vectors are
-   decode-only, retained as legacy-read proof.
+   decode-only, retained as legacy-read proof. That re-encode assertion covers
+   only the vectors the pinned file contains (core currently vendors 1.1.0, with
+   the resulting gap detailed below). Byte-canonicity scopes to the
+   envelope's MessagePack encoding and to the **canonical writer's** output:
+   the LZ4 bytes inside `compressed_data` are not reproducible across
+   conforming compressors — see
+   [Compressed-byte reproducibility](#compressed-byte-reproducibility-per-vector-scoping).
 2. **[SDK storage containers](#sdk-storage-containers-auto-mode)** — what each SDK
    *actually stores* in a backend in default (auto) mode. These differ per SDK, are
    **SDK-internal**, and are documented here so their bytes are identifiable — not so
@@ -248,6 +254,83 @@ the generic shortest-width selection property at fixture level, while
 > [!WARNING]
 > **PHP**: Standard `php-ext-lz4`'s `lz4_compress()` is **not compliant** — it prepends a proprietary 4-byte size header. Use `lz4_compress_raw()` from the forked extension at `27Bslash6/php-ext-lz4`.
 
+### Compressed-byte reproducibility (per-vector scoping)
+
+The LZ4 block **format** is fixed, but conforming **encoders** are not: the
+format constrains only what a block must decompress to, so two compliant
+compressors may legally emit different bytes for the same input. Compressed
+bytes are therefore
+**not canonical**, and conformance for `compressed_data` is **read-side**:
+
+- A conforming reader MUST decompress every pinned vector's `compressed_data`
+  to its pinned input, **and MUST enforce [Retrieve Flow](#retrieve-flow) steps
+  4, 5 and 9 while doing so.** Read-side conformance is not "the vectors pass":
+  every pinned vector is well-formed and declares a truthful `original_size`, so
+  they evidence **none** of those bounds, and a reader that omits all three
+  decompresses all of them successfully. The vectors prove decode
+  interoperability; the bounds in [Security Limits](#security-limits) are a
+  separate, non-negotiable obligation that no fixture can demonstrate.
+- A writer **other than the canonical `lz4_flex` writer** is NOT required to
+  reproduce the pinned compressed bytes, and MUST NOT be judged non-conforming
+  because its compressor output differs from the fixture — validate such a
+  writer by decoding its envelopes per the
+  [Retrieve Flow](#retrieve-flow) and checking its MessagePack encoding against
+  [Byte Layout](#byte-layout-canonical-encoding).
+- A writer MAY still byte-compare its compressor output against the pins as a
+  **drift tripwire**, provided the expected divergences are declared per vector
+  rather than treated as failures. This repo's own verifier does exactly that
+  (`LZ4_ENCODE_DIVERGENT` in `tools/wire-format-reference.py`), in two halves: a
+  set-level half that watches the reference **liblz4** mapping for
+  divergence-set drift, and a byte-level half that pins the exact
+  `compressed_data` of each declared-divergent vector. Both are needed —
+  "differs from liblz4's output" alone is a one-bit assertion that any other
+  valid LZ4 block satisfies, so it accepts a re-pin to unrelated bytes. Neither
+  half runs `lz4_flex`, so neither can detect an `lz4_flex` **behaviour** change;
+  that remains the job of the re-encode assertions in `cachekit-core` described
+  below, subject to the vendored-version gap noted there.
+
+This is the same doctrine [interop v2](interop-v2.md) records for its
+compressed-values profile. The pinned bytes are the **canonical implementation's**
+output (`lz4_flex` via `cachekit-core`), enforced by the re-encode byte-identity
+assertions in `cachekit-core/tests/wire_format_vectors.rs` — **but only for the
+vectors present in the fixture that repo vendors**. That matters today:
+cachekit-core vendors 1.1.0 and pins `version == "1.1.0"`, so
+`width_boundary_bin16` (added at 1.1.1) has **no canonical-writer (`lz4_flex`)
+compressed-byte check anywhere in the fleet**, and its pinned xxh3-64 checksum
+is recomputed nowhere. Its MessagePack encoding *is* covered: this repo's
+`tools/wire-format-reference.py verify` asserts legacy and bin re-encode
+byte-identity for it on every run, and liblz4 reproduces its compressed bytes
+on the optional `lz4` leg — so do not read this gap as "the vector is
+unverified". Closing it means re-vendoring 1.1.1 into cachekit-core, which
+requires three changes together, not one: bump `FIXTURE_SHA256`, bump the
+`version == "1.1.0"` pin to `1.1.1`, and relax
+`assert_eq!(twin_bytes[1], 0xc4)` to accept `0xc5` — that assertion currently
+requires *every* twin to be bin8, and `width_boundary_bin16_bin` is bin16
+(marker `0xc5`, 303-byte `compressed_data`), which is the whole point of the
+vector. A drop-in re-vendor fails that test. The reference liblz4 mapping
+above (`lz4.block`) is **decode-verified against every vector** in this repo's
+CI (`tools/wire-format-reference.py verify`, optional `lz4` leg); on encode it
+reproduces every pair except `large_compressible` byte-for-byte, which is an
+observation, not a guarantee — but one this repo's CI pins (see
+`LZ4_ENCODE_DIVERGENT`), so a toolchain change that alters the divergent set
+fails CI rather than quietly making this paragraph wrong.
+
+> [!NOTE]
+> **Known encode divergence — `large_compressible` / `large_compressible_bin`
+> (decode-verified only).** For this pair's input (1024 × `'A'`), liblz4
+> (observed at 1.9.4 via `python-lz4` 4.4.5) emits a **14-byte** block where
+> the fixture pins the **15-byte** block emitted by `lz4_flex` as shipped in
+> `cachekit-core` v0.2.0 (this vector's generator). Both sides are
+> version-stamped deliberately: encoder output is version-dependent, which is
+> the whole reason compressed bytes are not canonical. The blocks differ only in
+> the end-of-block match/literal split: `lz4_flex` ends the long match one byte
+> earlier and emits six trailing literals (`… e9 60` + `41`×6) where liblz4
+> emits five (`… ea 50` + `41`×5). Both are valid LZ4 blocks and both
+> decompress to the input; the divergence is encode-only. A third-party writer
+> following the Library Mapping will therefore produce a different — equally
+> conforming — envelope for this input. (LAB-1751; found by execution during
+> the LAB-868 panel review.)
+
 ---
 
 ## Checksum: xxHash3-64