diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml
index 319aabc..5911736 100644
--- a/.github/workflows/verify.yml
+++ b/.github/workflows/verify.yml
@@ -28,13 +28,15 @@ jobs:
- name: Python reference verify (stdlib only)
run: |
python3 tools/interop-reference.py verify
+ python3 tools/interop-v2-reference.py verify
python3 tools/encryption-verify.py
python3 tools/wire-format-reference.py verify
- - name: Python reference verify (optional deps — AES-GCM seal + msgpack third-encoder conformance)
+ - name: Python reference verify (optional deps — AES-GCM seal + msgpack third-encoder + lz4 C-implementation conformance)
run: |
- pip install cryptography==49.0.0 msgpack==1.2.1
+ pip install cryptography==49.0.0 msgpack==1.2.1 lz4==4.4.5
python3 tools/interop-reference.py verify
+ python3 tools/interop-v2-reference.py verify
python3 tools/encryption-verify.py --require-seal
python3 tools/wire-format-reference.py verify
@@ -44,6 +46,9 @@ jobs:
npm install --no-audit --no-fund --ignore-scripts @noble/hashes@2.2.0
node tools/interop-crosscheck.mjs
+ - name: Interop v2 JS cross-check (zero-dep independent container parser + LZ4 decoder + WebCrypto)
+ run: node tools/interop-v2-crosscheck.mjs
+
- name: Python-frame verify (stdlib only)
run: python3 tools/python-frame-reference.py verify
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aeecdc4..27235e5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,31 @@ All notable changes to the CacheKit Protocol Specification.
## [Unreleased]
+### Interop v2 — compressed-values profile (DRAFT)
+
+- New [`spec/interop-v2.md`](spec/interop-v2.md) (LAB-1135, protocol#52):
+ opt-in successor mode restoring the 2025-11-14 RFC's descoped
+ compressed+encrypted cross-SDK values. Values wrap in a `0xC1 0x02` +
+ msgpack `[method, original_size, payload:bin]` container (LZ4 block or
+ uncompressed), carried **inside** AES-256-GCM with a constant
+ four-component AAD reusing the frozen `"True"` token — deterministic
+ pre-AAD mode discrimination by configuration, no sniff-and-retry, and
+ cryptographic v1/v2 separation (cross-mode reads fail authentication).
+ Ships with a stdlib-only reference generator
+ ([`tools/interop-v2-reference.py`](tools/interop-v2-reference.py),
+ including a pure-Python LZ4 block codec), a zero-dependency independent
+ JS cross-check ([`tools/interop-v2-crosscheck.mjs`](tools/interop-v2-crosscheck.mjs)),
+ and [`test-vectors/interop-v2.json`](test-vectors/interop-v2.json)
+ (compressed, uncompressed, and non-canonical-widths round-trips,
+ compressed+encrypted round-trip, 16 structural + 2 cryptographic
+ must-reject vectors). Security limits
+ reuse the wire-format constants (512 MiB / 1000:1, enforced before
+ decompression); the CRIME/BREACH verdict is recorded in-spec (in threat
+ model, accepted with normative mitigations); the legacy array-of-ints
+ payload leniency is explicitly **not** inherited. Interop/v1 is
+ byte-for-byte untouched — its vectors and tools run unchanged beside the
+ new ones in CI. Status DRAFT until the vectors run in cachekit-py/ts/rs CI.
+
### SDK Feature Matrix
- Consolidated ten conflicting open matrix PRs into one code-verified end-state
diff --git a/README.md b/README.md
index a79fe5d..58388f6 100644
--- a/README.md
+++ b/README.md
@@ -69,6 +69,7 @@ layer's own store/retrieve flows are specified in
| [spec/encryption.md](spec/encryption.md) | AES-256-GCM encryption, HKDF-SHA256 key derivation, AAD v0x03, counter-based nonces, key rotation |
| [spec/saas-api.md](spec/saas-api.md) | REST API endpoints, binary wire protocol, error codes, metrics headers |
| [spec/interop-mode.md](spec/interop-mode.md) | Cross-SDK cache sharing — language-neutral key format, canonical argument normalization *(normative; shipped opt-in in all three SDKs — see the [feature matrix](sdk-feature-matrix.md#compliance-status) for per-SDK version floors)* |
+| [spec/interop-v2.md](spec/interop-v2.md) | Interop v2 compressed-values profile — opt-in LZ4-block + AES-256-GCM cross-SDK values *(DRAFT; no SDK implements it yet)* |
| [spec/file-backend-format.md](spec/file-backend-format.md) | Shared local File backend filename, header, expiry, and fail-closed flag negotiation |
| [sdk-feature-matrix.md](sdk-feature-matrix.md) | Feature parity tracking across Python, Rust, TypeScript, and PHP SDKs |
| [decisions/key-rotation.md](decisions/key-rotation.md) | Decision records — master-key rotation via client-side keyring (rationale, rejected options, operator runbooks) |
diff --git a/spec/interop-mode.md b/spec/interop-mode.md
index 3c190d6..6948dc2 100644
--- a/spec/interop-mode.md
+++ b/spec/interop-mode.md
@@ -14,6 +14,10 @@
> Design discussion: [Issue #1](https://github.com/cachekit-io/protocol/issues/1) ·
> Test vectors: [`test-vectors/interop-mode.json`](../test-vectors/interop-mode.json) ·
> Reference implementation: [`tools/interop-reference.py`](../tools/interop-reference.py)
+>
+> Compressed values: this mode deliberately has none. An opt-in successor
+> profile, **[interop/v2](interop-v2.md)** (DRAFT), adds an LZ4-compressed
+> value container as a distinct mode — nothing in v1 changes.
diff --git a/spec/interop-v2.md b/spec/interop-v2.md
new file mode 100644
index 0000000..1b9d81f
--- /dev/null
+++ b/spec/interop-v2.md
@@ -0,0 +1,499 @@
+**[Protocol](../README.md)** > **Interop v2 (Compressed Values)**
+
+
+
+# Interop v2 — Compressed-Values Profile
+
+**Opt-in LZ4-compressed (and optionally AES-256-GCM-encrypted) cross-SDK cache values, as a versioned successor mode to [interop/v1](interop-mode.md).**
+
+> **Status**: DRAFT (PROPOSED) — leaves DRAFT when the vectors below run in
+> cachekit-py, cachekit-ts, and cachekit-rs CI. No SDK implements this profile yet;
+> SDK work is follow-up, gated on ratification.
+> Design discussion: [Issue #52](https://github.com/cachekit-io/protocol/issues/52) ·
+> Test vectors: [`test-vectors/interop-v2.json`](../test-vectors/interop-v2.json) ·
+> Reference implementation: [`tools/interop-v2-reference.py`](../tools/interop-v2-reference.py) ·
+> Independent cross-check: [`tools/interop-v2-crosscheck.mjs`](../tools/interop-v2-crosscheck.mjs)
+
+
+
+---
+
+## Table of Contents
+
+- [Scope — What v2 Changes and What It Inherits](#scope--what-v2-changes-and-what-it-inherits)
+- [Mode Discrimination — No Sniffing, Ever](#mode-discrimination--no-sniffing-ever)
+- [The v2 Value Container](#the-v2-value-container)
+- [Compression Method Registry](#compression-method-registry)
+- [Security Limits (Decompression Bounds)](#security-limits-decompression-bounds)
+- [Reader Algorithm (Normative Order)](#reader-algorithm-normative-order)
+- [Encryption in Interop v2](#encryption-in-interop-v2)
+- [Threat Model: Compress-Then-Encrypt (CRIME/BREACH)](#threat-model-compress-then-encrypt-crimebreach)
+- [Per-SDK LZ4 Dependency Bill](#per-sdk-lz4-dependency-bill)
+- [SDK Implementation Requirements](#sdk-implementation-requirements)
+- [Design Decisions](#design-decisions)
+- [Test Vectors](#test-vectors)
+
+---
+
+## Scope — What v2 Changes and What It Inherits
+
+Interop/v1 deliberately shipped without compression: values are bare MessagePack,
+and the encryption AAD pins `compressed = "False"`
+([interop-mode.md → Encryption in Interop Mode](interop-mode.md#encryption-in-interop-mode)).
+That surface is **frozen** — nothing in this profile changes any v1 byte, constant,
+or vector, and every published v1 vector passes unchanged (proof: the v1 vector
+file and both v1 reference tools are untouched by this profile; CI runs them
+side-by-side with the v2 tools).
+
+Interop/v2 changes **exactly one thing**: the value format. Everything else is
+inherited from interop/v1 normatively and by reference:
+
+| Surface | Interop/v2 rule |
+| :--- | :--- |
+| Key format | **Identical to v1**: `{namespace}:{operation}:{args_hash}`, same segment grammar, same canonical argument array, same Blake2b-256. A v1 and a v2 deployment produce byte-identical keys for identical inputs. |
+| Argument data model & canonicalization | Identical to v1 ([interop-mode.md](interop-mode.md#the-interop-data-model)). |
+| Value **content** encoding | Identical to v1: the logical value is serialized as one plain-MessagePack document under the v1 value-profile rules (no number canonicalization, sentinel maps for temporals, exactly-one-document). |
+| Value **container** | **New**: the plain-MessagePack value bytes are wrapped in the [v2 value container](#the-v2-value-container), optionally LZ4-block-compressed. |
+| Encryption | AES-256-GCM + HKDF-SHA256 unchanged; the AAD `compressed` component is the frozen `"True"` token as a **per-mode constant** — see [Encryption in Interop v2](#encryption-in-interop-v2). |
+| SaaS | Same as v1: keys are opaque strings, values are opaque bytes; nothing here is visible to the backend. |
+
+The mechanism is the one interop/v1 itself sanctions
+([Design Decisions → "No version segment in the key"](interop-mode.md#design-decisions)):
+*versioning-by-mode-name*. Interop/v2 is a **distinct opt-in mode**, not a flag on
+v1 — and explicitly not a change to v1.
+
+---
+
+## Mode Discrimination — No Sniffing, Ever
+
+[spec/encryption.md](encryption.md#additional-authenticated-data-aad) forbids
+retrying decryption across AAD variants and forbids selecting the post-decryption
+container by sniffing bytes. A v2 entry must therefore be distinguishable from a
+v1 entry **before AAD construction**. In this profile that determination is made
+by **configuration, not by inspecting stored bytes**:
+
+- A cache namespace is interop/v1-valued or interop/v2-valued **by out-of-band
+ agreement across the deployment**, exactly like operation names and effective
+ argument lists already are in v1. One namespace, one mode.
+- A reader configured for interop/v2 builds the v2 AAD (`compressed = "True"`),
+ attempts decryption **once** per permitted keyring key, and interprets the
+ plaintext as a v2 container. A reader configured for interop/v1 does the same
+ with the v1 constants. Neither ever probes the other mode's AAD or container.
+
+**Why not a per-entry marker instead?** For unencrypted entries a magic prefix
+would work — but an encrypted interop/v1 entry is a *bare* `nonce ‖ ciphertext ‖
+tag` blob with no metadata, and the first nonce byte can take any value,
+including any magic byte. Per-entry discrimination on stored bytes would
+misclassify 1-in-256 legitimate v1 entries per magic byte and turn their reads
+into authentication failures. Per-entry marking is unsound on this surface;
+mode-level configuration is deterministic. (See [Design Decisions](#design-decisions).)
+
+**Misconfiguration is loud, never silent.** If a v1 reader and a v2 writer are
+accidentally pointed at the same namespace:
+
+- *Unencrypted*: a v2 container begins `0xC1`, the one byte the MessagePack
+ specification reserves as a **never-used marker** — it cannot be the *first
+ byte* of a well-formed MessagePack document (it can, of course, appear inside
+ payload bytes or multi-byte integer bodies; only the leading-byte property is
+ load-bearing, and nothing in this profile scans past byte 0). A v1 reader
+ MUST already reject it as malformed MessagePack. Conversely, a v2 reader reading a bare v1 value fails
+ the magic check below. Both directions error; neither silently decodes wrong
+ data.
+- *Encrypted*: the v1 and v2 AADs differ in the `compressed` component, so
+ AES-GCM authentication fails cross-mode — terminal per the existing
+ [no-retry rule](encryption.md#additional-authenticated-data-aad). The keyring
+ key-attempt rule is unchanged: retrying across keyring **keys** is permitted,
+ retrying across AAD **variants** is not.
+
+On a magic-check failure, a v2 reader SHOULD report a mode-mismatch diagnostic:
+leading byte a plausible MessagePack marker → *"possible interop/v1 value —
+namespace mode misconfiguration?"*; leading bytes `0x43 0x4B` → the same
+*"Python-SDK-internal auto-mode entry"* diagnostic v1 specifies. Diagnostics are
+error-message quality-of-life; the normative behavior is the hard error itself.
+
+**Migration** between modes is a namespace-level operation: agree the new mode
+out-of-band and either use a fresh namespace or accept read errors on stale
+entries until TTLs drain (the same cache-warm cost the v1 spec assigns to any
+canonicalization version bump). There is no in-place mixed-mode namespace.
+
+---
+
+## The v2 Value Container
+
+Every interop/v2 value — compressed or not, encrypted or not — is exactly one
+container:
+
+```text
+┌──────┬──────┬────────────────────────────────────────────────────────────┐
+│ 0xC1 │ 0x02 │ body: one canonical MessagePack document (see below) │
+└──────┴──────┴────────────────────────────────────────────────────────────┘
+ magic version
+```
+
+| Field | Size | Rule |
+| :--- | :---: | :--- |
+| Magic | 1 B | `0xC1` — reserved/never-used in MessagePack, so a container is **structurally not** a MessagePack document. MUST be exactly `0xC1`. |
+| Container version | 1 B | `0x02`, matching the mode name. Any other value MUST be rejected (a future interop/v3 container would carry `0x03` and its own spec). |
+| Body | var | One MessagePack document, a positional 3-element array: `[method, original_size, payload]` — canonically `fixarray(3)`; readers also accept the wider array headers (see Encoding rules). The container ends where this document ends; **trailing bytes MUST be rejected** (same exactly-one-document strictness as v1 values). |
+
+Body elements:
+
+| Element | MessagePack type | Rule |
+| :--- | :--- | :--- |
+| `[0] method` | **unsigned-family** int | Compression method — see the [registry](#compression-method-registry). Unknown values MUST be rejected. |
+| `[1] original_size` | **unsigned-family** int | Exact byte length of the plain-MessagePack value document. Load-bearing twice: it is the LZ4-block decompression size hint **and** the first decompression-bomb bound. Bounds in [Security Limits](#security-limits-decompression-bounds). |
+| `[2] payload` | **bin** (`0xc4`/`0xc5`/`0xc6`) | `method = 0`: the plain-MessagePack value bytes verbatim. `method = 1`: one raw LZ4 block. MUST be the msgpack `bin` family — see the encoding rules below. |
+
+Encoding rules:
+
+- **Writers MUST emit canonical MessagePack** for the body: shortest-form
+ headers, `bin` for the payload. This is the protocol-1.1 `bin` rule
+ ([decisions/envelope-bin-encoding.md](../decisions/envelope-bin-encoding.md),
+ [wire-format.md → Byte Layout](wire-format.md#byte-layout-canonical-encoding))
+ applied from birth: no v2 container ever pays the array-of-ints ~1.5× wire tax.
+- **Readers MUST accept any well-formed header width for each element type**:
+ `fixarray(3)`, `array16(3)` (`0xdc`), or `array32(3)` (`0xdd`) for the body
+ array; any *unsigned-family* width for the two integer elements — positive
+ fixint and uint8/16/32/64 (`0xcc`–`0xcf`) are all legal on read, canonical or
+ not (pinned by the `method0_noncanonical_widths` vector, whose body uses an
+ `array16(3)` header); and any `bin8/16/32` width for the
+ payload. Readers **MUST enforce element types at the marker level**:
+ signed-family integer markers (negative fixint, int8–int64,
+ `0xd0`–`0xd3`) MUST be rejected for `method` and `original_size` **even when
+ the carried value is non-negative**, and a payload that is *anything other
+ than `bin`* — including the msgpack `str` family and the legacy
+ **array-of-ints** shape — MUST be rejected. Marker-level enforcement makes a
+ negative `original_size` *structurally unrepresentable* (a negative int can
+ only be carried by a signed-family marker), closing the signed-size bounds
+ bypass class outright; conformant writers can never emit signed-family
+ markers here anyway (canonical encoding of a non-negative int is always
+ unsigned-family). Note the implementation consequence: a generic MessagePack
+ decoder surfaces values, not markers, so SDKs SHOULD hand-parse the body —
+ it is a three-field grammar, and both reference tools demonstrate the
+ ~50-line parser. Pinned by the `reject_method_signed_marker` and
+ `reject_negative_original_size` vectors.
+- **The legacy array-of-ints leniency is explicitly NOT inherited.** The
+ ByteStorage envelope permanently dual-reads array-of-ints because a deployed
+ installed base wrote it, and because rmp-serde happens to decode both shapes
+ for free. Interop/v2 has no installed base, and an independent hand-written
+ reader in a new language would need *extra* code to accept both shapes — the
+ opposite of interop's lowest-implementation-bar goal. Exactly one payload
+ encoding is legal: `bin`. Pinned by the `reject_payload_array_of_ints` vector.
+- A reader MUST validate any declared `bin` length header against the remaining
+ input **before** allocating for it (a 5-byte forged `bin32` header must not
+ cause a 4 GiB allocation — same rule as
+ [wire-format.md → Security Limits](wire-format.md#security-limits)).
+
+The container is deliberately **not** the ByteStorage envelope: no xxHash3-64
+checksum field (integrity comes from the AES-GCM tag when encrypted, and is
+absent when not — exactly v1's posture), no `format` field (the payload's
+decompressed content is always one plain-MessagePack value document). Rationale
+in [Design Decisions](#design-decisions).
+
+---
+
+## Compression Method Registry
+
+| `method` | Name | Payload content |
+| :---: | :--- | :--- |
+| `0` | none | The plain-MessagePack value bytes, verbatim. `original_size` MUST equal the payload byte length; readers MUST reject a mismatch. |
+| `1` | lz4-block | Exactly one raw **LZ4 block** ([wire-format.md → Compression](wire-format.md#compression-lz4-block-format)). LZ4 **frame** format (magic `0x184D2204`) is FORBIDDEN. No prepended size word of any kind — `original_size` in the container is the size hint (Python `lz4.block`: `store_size=False`; Rust `lz4_flex`: the plain `block::compress`/`decompress`, never the `_prepend_size` variants). |
+
+New methods (e.g. zstd) require a spec revision to this registry. A new
+container version byte is **not** required: the registry is versioned by this
+spec, and readers reject unknown method values, which is the safe failure.
+Writers MUST NOT emit unregistered methods.
+
+Writer rules:
+
+- Writers MUST be able to produce `method 0` (it is the escape hatch for the
+ [threat model](#threat-model-compress-then-encrypt-crimebreach) and for
+ incompressible values) and MAY choose the method per entry. Readers MUST
+ accept both methods; the choice is invisible above the container.
+- Writers SHOULD emit `method 0` when LZ4 does not strictly reduce the payload
+ (`lz4_len >= original_size`) — compressing high-entropy data buys wire
+ inflation for CPU. Non-normative threshold guidance: values under ~64 bytes
+ rarely benefit.
+- **Compressed bytes are NOT canonical.** Different conformant LZ4 encoders (and
+ levels) legally produce different bytes for the same input. Cross-SDK
+ conformance for `method 1` is defined on the *read side*: any conformant
+ reader MUST decompress any valid LZ4 block to the same bytes. Two SDKs
+ writing the same logical value MAY produce different stored bytes — interop
+ never required stored-value byte equality (only **keys** are byte-canonical),
+ and the published vectors pin *reference* compressed bytes for read-side
+ conformance, not as the only legal writer output.
+
+---
+
+## Security Limits (Decompression Bounds)
+
+A compressed container introduces a byte-level DoS axis — the decompression
+bomb — that bare-MessagePack v1 does not have. These bounds reuse the
+ByteStorage constants from
+[wire-format.md → Security Limits](wire-format.md#security-limits) so the fleet
+carries **one** set of numbers, and all of them MUST be enforced **before**
+decompressing (integer arithmetic only — no floating point):
+
+| Limit | Value | Applies to |
+| :--- | ---: | :--- |
+| Max `original_size` | 512 MiB (536,870,912 B) | both methods |
+| Max payload size | 512 MiB (536,870,912 B) | both methods |
+| Max compression ratio | 1000:1 | `method 1` |
+
+```text
+// original_size and method are unsigned by construction — signed-family
+// markers were already rejected at parse time (see The v2 Value Container).
+reject if original_size > MAX_UNCOMPRESSED // 512 MiB
+reject if payload.length > MAX_COMPRESSED // 512 MiB
+if method == 1:
+ reject if payload.length == 0 // zero-length compressed = bomb
+ max_allowed = 1000 * payload.length // MUST be computed in >= 64-bit integers
+ reject if original_size > max_allowed
+if method == 0:
+ reject if original_size != payload.length
+```
+
+The ratio product MUST be computed in **at least 64-bit unsigned integers**.
+After the two 512 MiB caps pass, both operands are < 2³⁰ and the product is
+< 2⁴⁰, so it can never overflow a u64 — but it *does* overflow 32-bit `usize`
+arithmetic (a real target: cachekit-ts ships a wasm32 build), where release-mode
+wrapping would silently corrupt the bound in both directions. Do not compute
+this in pointer-width arithmetic.
+
+After `method 1` decompression, the output length MUST equal `original_size`
+exactly — shorter or longer output is a hard error (the
+`reject_lz4_length_mismatch` vector). Any malformed LZ4 stream (invalid offset,
+truncated sequence, output overrun) is a hard error. A deployment MAY configure
+*stricter* limits (e.g. its max-value-size ceiling); it MUST NOT accept beyond
+these.
+
+**Relationship to [protocol#20](https://github.com/cachekit-io/protocol/issues/20)
+(one sentence, as promised):** #20 decides *element-count* bounds for the decoded
+value — the same value whether it sits inside or outside a v2 container — while
+this section bounds the container's *byte* axis; they are complementary
+protections on orthogonal axes, and this profile inherits whatever #20 ratifies,
+unchanged.
+
+---
+
+## Reader Algorithm (Normative Order)
+
+Given stored bytes for an interop/v2-configured cache:
+
+```text
+1. (encrypted caches only) Build the v2 AAD from configuration —
+ (tenant_id, cache_key, "msgpack", "True") — and AES-256-GCM-decrypt per
+ spec/encryption.md (keyring key attempts permitted; AAD variants never).
+ Authentication failure after permitted key attempts is terminal.
+ The plaintext is the container. For unencrypted caches the stored bytes
+ are the container.
+2. Check container[0] == 0xC1 and container[1] == 0x02; else hard error
+ (mode-mismatch diagnostics per Mode Discrimination).
+3. Decode exactly one MessagePack document from container[2..]; reject
+ trailing bytes; enforce element types (int, int, bin) and the
+ header-vs-remaining-input rule.
+4. Enforce every Security Limit above — before any decompression.
+5. method 1: LZ4-block-decompress the payload with original_size as the
+ exact output size; reject on any LZ4 error or output-length mismatch.
+ method 0: the payload IS the value bytes.
+6. Decode the resulting bytes as one plain-MessagePack value document under
+ the interop/v1 value rules (exactly-one-document, trailing bytes
+ rejected, sentinel-map temporal convention).
+```
+
+Step order is normative: bounds run before decompression (step 4 before 5), and
+in the encrypted path the AES-GCM tag is verified (step 1) before any container
+parsing or decompression — hostile bytes never reach the LZ4 decoder
+unauthenticated when encryption is on. In the **unencrypted** path the LZ4
+decoder does face untrusted bytes directly; that asymmetry is inherent (v1's
+unencrypted values have no integrity protection either), and it is why the
+bounds in step 4 and strict decoding in step 5 are MUSTs, not advice.
+
+---
+
+## Encryption in Interop v2
+
+Encryption composes exactly as in v1, with the container as the plaintext:
+
+1. **The AES-GCM plaintext is the entire v2 container** (magic, version, body).
+ Stored bytes are the bare `nonce ‖ ciphertext ‖ tag` blob — byte-shape
+ identical to an encrypted v1 entry, carrying no cleartext metadata.
+2. AAD components are always `format = "msgpack"`,
+ **`compressed = "True"`** — the [frozen ASCII token](encryption.md#compressed-tokens)
+ `54 72 75 65` ratified in
+ [protocol#12](https://github.com/cachekit-io/protocol/issues/12), reused
+ verbatim. Exactly four components; `original_type` is never included
+ (same rule as v1).
+3. Key derivation (HKDF-SHA256), nonces, ciphertext layout, and keyring
+ behavior are unchanged from [encryption.md](encryption.md).
+
+**Token semantics (normative).** In interop/v2, `compressed = "True"`
+authenticates *"the AES-GCM plaintext is an interop/v2 value container"* — the
+compression-capable container — as a per-mode constant, exactly as v1's
+`"False"` authenticates *"the plaintext is one bare MessagePack document"*. It
+does **not** vary with the per-entry `method`: a `method 0` entry still carries
+`compressed = "True"`. This keeps the AAD fully derivable from configuration
+(the no-sniffing requirement) and is consistent with
+[encryption.md](encryption.md#encryption-flow)'s rule that AAD inputs reflect
+what the plaintext actually is: the plaintext *is* a v2 container in every
+case. The per-entry `method` needs no AAD binding because it sits **inside**
+the GCM-authenticated plaintext — flipping it is tamper, and tamper fails the
+tag.
+
+**Cryptographic mode separation.** Because the v1 and v2 AADs differ in the
+`compressed` component, a v2 ciphertext cannot be verified with the v1 AAD or
+vice versa, even under the same master key, tenant, and cache key. The
+`reject_v2_ciphertext_with_v1_aad` and `reject_v1_ciphertext_with_v2_aad`
+vectors pin both directions. This is the encrypted-path guarantee that mode
+misconfiguration fails loudly instead of decoding wrong bytes.
+
+**What the backend sees** is unchanged in kind from v1: an opaque key and an
+opaque ciphertext blob. It is changed in one measurable respect — ciphertext
+length now tracks *compressed* size. That is the subject of the next section.
+
+---
+
+## Threat Model: Compress-Then-Encrypt (CRIME/BREACH)
+
+Compress-then-encrypt leaks plaintext redundancy through ciphertext length:
+if attacker-influenced bytes and a secret share one compression context, the
+attacker can confirm guesses of the secret by watching compressed size shrink
+as guesses converge (CRIME/BREACH class).
+
+**Where CacheKit stands.** The observer in CacheKit's zero-knowledge model is
+the backend (or anyone reading it) — explicitly untrusted for confidentiality,
+and it can measure ciphertext lengths precisely. So the observation channel is
+IN the threat model and cannot be waved off. The remaining preconditions are:
+
+1. **Shared context** — the secret and attacker-influenced bytes must be inside
+ the *same cache value*. Interop/v2 compresses each entry independently: no
+ cross-entry dictionary, no shared compression state, no compression of keys
+ or AAD. Cross-entry redundancy leaks nothing; the blast radius of any attack
+ is a single value's contents.
+2. **Adaptive iteration** — the attacker must be able to trigger repeated
+ re-encryptions of that value with varied guesses (a chosen-plaintext
+ pressure v1 already tolerates, but v1's length leak — serialized size — does
+ not respond to *content similarity*, only to length).
+
+**Verdict (recorded, per the mandate that silence is not a verdict): the
+CRIME-class attack is IN the threat model for interop/v2 and is accepted as a
+documented residual risk, bounded by per-entry compression granularity and
+governed by the following normative mitigations** — it is not "not applicable",
+and it is not fully eliminated:
+
+- Compression is **opt-in twice**: interop/v2 is a distinct mode a deployment
+ must choose, and `method 0` disables compression per entry within the mode.
+- Values that interleave secrets with attacker-influenced content in one entry
+ MUST NOT be written with `method 1` — use `method 0` or stay on interop/v1.
+ This is an application-layer obligation (an SDK cannot detect it
+ mechanically); SDK documentation for the v2 opt-in MUST state it.
+- No length padding is added. A padding scheme was considered and rejected:
+ fixed-bucket padding gives quantized-but-real leakage while inflating every
+ entry, and no cheap scheme eliminates the channel — documented honesty beats
+ a false sense of security. Deployments needing length secrecy against their
+ backend should not compress (which still leaks exact serialized length, as
+ v1 does) — length-hiding is out of scope for this protocol.
+
+BREACH-style amplification via shared dictionaries or cross-request compression
+contexts is structurally absent: there are none.
+
+---
+
+## Per-SDK LZ4 Dependency Bill
+
+Named per the mandate — the 2025-11-14 RFC's unpriced PHP fork
+(`27Bslash6/php-ext-lz4`) is the cautionary tale. All entries are LZ4 **block**
+format; every "frame-format-only" binding is non-conformant.
+
+| SDK | Library | Call | Cost |
+| :--- | :--- | :--- | :--- |
+| cachekit-py | [`lz4`](https://pypi.org/project/lz4/) (PyPI) | `lz4.block.compress(data, store_size=False)` / `lz4.block.decompress(data, uncompressed_size=original_size)` | One new runtime dependency (C extension, prebuilt wheels for all supported CPython targets). `store_size=False` is **critical** — the default prepends a 4-byte size word that would corrupt the payload. Alternative: expose `lz4_flex` from the already-shipped cachekit-core FFI (zero new PyPI deps, small core-binding diff). |
+| cachekit-rs | [`lz4_flex`](https://crates.io/crates/lz4_flex) | `lz4_flex::block::compress` / `block::decompress(input, original_size)` | Effectively free: pure Rust, no C toolchain, already in the dependency tree via cachekit-core. MUST use the plain block functions, never `compress_prepend_size`/`decompress_size_prepended`. |
+| cachekit-ts | cachekit-core NAPI + wasm32 bindings (preferred) | new exported `lz4BlockCompress` / `lz4BlockDecompress` wrapping `lz4_flex` | **Zero new npm dependencies** — core already links `lz4_flex`; the cost is a small binding export in the existing NAPI and wasm builds (both artifacts, so Workers keep parity). Pure-JS fallback if ever needed: [`lz4js`](https://www.npmjs.com/package/lz4js) (block format, no native build, slower) — verified block-capable, already the documented Node binding in [wire-format.md](wire-format.md#compression-lz4-block-format). |
+| Future SDK (stated cost) | a block-format LZ4 codec | — | The bill is: a conformant LZ4 *block* codec + these vectors passing in that SDK's CI. Most ecosystems have one (Go `pierrec/lz4/v4` `CompressBlock`; JVM `lz4-java`; .NET `K4os.Compression.LZ4`). **PHP remains the known expensive case**: stock `php-ext-lz4` emits a proprietary size-prefixed format; spec-compliant raw blocks need the `27Bslash6/php-ext-lz4` fork (`lz4_compress_raw()`) or FFI. Price that before promising a PHP SDK v2 implementation. |
+
+---
+
+## SDK Implementation Requirements
+
+An SDK implementation of interop/v2 MUST:
+
+1. Implement interop/v1 key generation, argument canonicalization, and the
+ value-content rules unchanged (pass the v1 vectors).
+2. Write every value as exactly one v2 container; emit canonical MessagePack
+ body encoding with a `bin` payload; support `method 0`.
+3. Reject, on read: bad magic/version, wrong element types (including non-`bin`
+ payloads — the array-of-ints shape included), unknown methods, trailing
+ bytes, every Security-Limits violation (before decompressing), LZ4 errors,
+ and output-length mismatches.
+4. When encryption is enabled: use exactly the four-component AAD with
+ `format = "msgpack"`, `compressed = "True"` (frozen bytes `54 72 75 65`);
+ never emit or accept `original_type` on this surface; never retry across
+ AAD variants (keyring key retries per [encryption.md](encryption.md) remain
+ permitted).
+5. Leave interop/v1 and auto mode byte-for-byte unchanged; expose v2 as a
+ distinct opt-in (per-namespace mode selection).
+6. Document the [threat-model guidance](#threat-model-compress-then-encrypt-crimebreach)
+ at the v2 opt-in surface.
+7. Pass every vector in
+ [`test-vectors/interop-v2.json`](../test-vectors/interop-v2.json), including
+ all `reject_*` vectors (which MUST error) and — when the SDK supports
+ encryption — the encrypted round-trip decrypt and both AAD cross-mode
+ rejections.
+
+---
+
+## Design Decisions
+
+| Decision | Alternative rejected | Why |
+| :--- | :--- | :--- |
+| **New mode name (interop/v2)** | Per-entry container marker inside interop/v1 | Sanctioned by v1's own versioning rule. The per-entry marker is unsound on the encrypted surface: a v1 encrypted entry is a bare `nonce‖ct‖tag` blob whose first byte is unconstrained, so any magic byte misclassifies 1-in-256 legitimate v1 entries into authentication failures. Configuration-determined mode is the only deterministic pre-AAD discriminator that keeps v1 frozen. |
+| **Container inside the encryption** (plaintext = container) | Cleartext container header wrapping the ciphertext | Constant four-component AAD derivable purely from configuration (no cleartext metadata to tamper or bind); stored encrypted shape stays a bare blob like v1; and `original_size` stays confidential — a cleartext header would hand the observer the exact compression ratio, materially worsening the CRIME analysis for free. |
+| **`compressed = "True"` as a per-mode constant, even for `method 0`** | Per-entry AAD flag tracking the method | A per-entry AAD input must be known pre-decrypt, which forces cleartext metadata and re-opens the tamper/oracle surface the no-retry rule exists to close. The method sits inside the GCM-authenticated plaintext, where tamper already fails the tag. Reuses the frozen `b"True"` constant ([protocol#12](https://github.com/cachekit-io/protocol/issues/12)) verbatim. |
+| **`0xC1` magic** | msgpack-decodable container (bare array) | `0xC1` is the single byte the MessagePack spec pins as "never used": a v2 container is structurally not a MessagePack document, so a misrouted container fails loudly in any v1 reader. A bare-array container would silently decode in a v1 reader as a plausible 3-element value — the silent-wrong-data failure this protocol family refuses on principle. |
+| **Minimal 3-element container** | Reuse the ByteStorage envelope | The envelope drags xxHash3-64 into every SDK — a second native dependency per language (the PHP-fork class of cost) duplicating integrity the AES-GCM tag already provides when encrypted, and exceeding v1's posture when not. Its `format` field is also dead weight here (the content is always one plain-MessagePack document). What *is* kept from the envelope experience: `bin` payload encoding as normative from birth (protocol 1.1, [decisions/envelope-bin-encoding.md](../decisions/envelope-bin-encoding.md)). |
+| **Array-of-ints payload rejected** | Inherit cachekit-core's permanent dual-read leniency | That leniency serves a deployed installed base and falls out of rmp-serde for free; interop/v2 has no installed base, and hand-written readers in new languages would pay extra code to be lenient. One legal encoding is the lowest implementation bar. Pinned by vector. |
+| **Compressed bytes non-canonical, read-side conformance** | Pin one canonical LZ4 output | LZ4 encoders legally differ (implementation, level, version). Pinning writer bytes would freeze one library's output as protocol law and break on its next release. Keys stay byte-canonical; values never needed to be. |
+| **Bounds reuse wire-format.md constants (512 MiB / 1000:1)** | Profile-specific numbers | One set of constants fleet-wide; the guards are already implemented, reviewed, and vector-tested in cachekit-core. Integer arithmetic rule carried over verbatim. |
+| **No length padding** | Bucketed padding vs CRIME | Quantized leakage at real cost is not elimination; documented guidance plus the `method 0` / stay-v1 escape hatches are honest. Length-hiding from the backend is explicitly out of protocol scope (v1 leaks exact lengths today). |
+
+---
+
+## Test Vectors
+
+[`test-vectors/interop-v2.json`](../test-vectors/interop-v2.json) contains:
+
+| Group (JSON key) | Verifies |
+| :--- | :--- |
+| `container_vectors` | Byte-exact containers: `method 0` wrap of the v1 `issue_example_object` value; `method 1` compressed round-trip of a compressible value (reference LZ4 bytes pinned; readers must decompress them to the pinned value bytes); a `method 1` container whose inner value is byte-identical to the published v1 `issue_example_object` value vector (asserted against `interop-mode.json` at generation time — the content profile is inherited unchanged); and a hand-built non-canonical-widths container (uint8/uint32 ints, `bin16` payload, `array16` header) that readers MUST accept. |
+| `aad_vectors` | The v2 AAD (`compressed = "True"`) over the same tenant and cache key as v1's `interop_key_aad` — the two AAD hex strings differ only in the final component (`"True"` vs `"False"`), pinned side-by-side. |
+| `encryption_vectors` | Full compressed+encrypted round-trip: HKDF-SHA256 (same master key and tenant as v1 / `encryption.json`, so the derived-key fingerprint `96179a9b…` is the published one), AES-256-GCM over the v2 container with the v2 AAD and a fixed nonce; decrypt-verified on every cross-check run. |
+| `reject_vectors` | Structural must-rejects, including: bad magic (a bare v1 value fed to a v2 reader), bad container version, unknown method, signed-family integer markers (incl. a negative `original_size`), non-`bin` payloads (the array-of-ints leniency decision, pinned, and a `str` payload), forged `bin32` length header (4 GiB declared, input ends), `method 0` size mismatch, trailing bytes, declared-size bomb, ratio bomb (1000:1), zero-length compressed payload, malformed LZ4 (zero offset), truncated LZ4, and decompressed-length mismatch. All MUST error before or during step 5 of the reader algorithm; the `error` text is a maintainer note, not normative. |
+| `crypto_reject_vectors` | `reject_v2_ciphertext_with_v1_aad` and `reject_v1_ciphertext_with_v2_aad` — both cross-mode AAD combinations MUST fail AES-GCM authentication (mode separation), pinned against real ciphertexts from this file and the v1 file. |
+
+Regenerate / verify:
+
+```bash
+python3 tools/interop-v2-reference.py verify # CPython stdlib reference (incl. pure-Python LZ4 block codec)
+node tools/interop-v2-crosscheck.mjs # independent JS container parser + LZ4 decoder + WebCrypto HKDF/AES-GCM (zero deps)
+```
+
+The vectors are produced by the stdlib-only Python reference (which implements
+its own LZ4 block codec so vector generation has no third-party dependency) and
+byte-verified by an independently written JavaScript implementation. When the
+optional `lz4` package is importable, `verify` additionally proves bidirectional
+conformance with the de-facto C implementation (our compressed bytes decompress
+under `lz4.block`, and `lz4.block`'s compressed output decompresses under our
+decoder); when `cryptography` is importable it re-verifies the AES-GCM seal
+(the JS cross-check always does, via WebCrypto). The interop/v1 vectors are
+untouched and continue to run beside these in CI
+(`.github/workflows/verify.yml`) — that co-existence is the standing proof that
+every v1 vector passes unchanged.
+
+---
+
+
+
+[Protocol](../README.md) · [Interop Mode (v1)](interop-mode.md) · [Encryption](encryption.md) · [Wire Format](wire-format.md) · [SaaS API](saas-api.md)
+
+
diff --git a/spec/saas-api.md b/spec/saas-api.md
index f1db967..386c39f 100644
--- a/spec/saas-api.md
+++ b/spec/saas-api.md
@@ -129,7 +129,7 @@ X-CacheKit-TTL: 3600
> **Migration:** The `X-TTL` header is deprecated. The server MUST accept both `X-CacheKit-TTL` and `X-TTL` during the transition period, preferring `X-CacheKit-TTL` when both are present. SDKs MUST send `X-CacheKit-TTL` only. The `X-TTL` header will be removed in protocol version 2.0 (targeted at SDK 1.0 milestone).
> [!IMPORTANT]
-> **Maximum value size:** A single cache value may be at most **25 MB**. Larger values are rejected with `413 Payload Too Large` — a **permanent** error: SDKs MUST NOT retry and SHOULD surface "value too large" to the caller. This ceiling MAY change, so SDKs MUST treat any `413` as "value too large" regardless of the exact byte count. It is unrelated to the SDK serializer's 512 MB in-memory safety bound (see `wire-format.md`) — that bound governs what the SDK will serialize, not what the service will store.
+> **Maximum value size:** A single cache value may be at most **25 MB**. Larger values are rejected with `413 Payload Too Large` — a **permanent** error: SDKs MUST NOT retry and SHOULD surface "value too large" to the caller. This ceiling MAY change, so SDKs MUST treat any `413` as "value too large" regardless of the exact byte count. It is unrelated to the SDK serializer's 512 MiB in-memory safety bound (see `wire-format.md`) — that bound governs what the SDK will serialize, not what the service will store.
| Status | Meaning |
| :---: | :--- |
diff --git a/spec/wire-format.md b/spec/wire-format.md
index 3d98225..b1a18c6 100644
--- a/spec/wire-format.md
+++ b/spec/wire-format.md
@@ -298,8 +298,8 @@ let checksum: [u8; 8] = xxh3_64(&original_data).to_be_bytes();
| Limit | Value | Purpose |
| :--- | ---: | :--- |
-| Max uncompressed size | 512 MB | Memory safety |
-| Max compressed size | 512 MB | Memory safety |
+| Max uncompressed size | 512 MiB (536,870,912 B) | Memory safety |
+| Max compressed size | 512 MiB (536,870,912 B) | Memory safety |
| Max compression ratio | 1000:1 | Decompression bomb protection |
### Decompression Bomb Detection
@@ -328,9 +328,9 @@ if original_size > max_allowed:
```
Input: raw_data (bytes), format (string, default "msgpack")
-1. Validate: raw_data.length <= 512 MB
+1. Validate: raw_data.length <= 512 MiB
2. Compress: compressed = lz4_block_compress(raw_data)
-3. Validate: compressed.length <= 512 MB
+3. Validate: compressed.length <= 512 MiB
4. Checksum: checksum = xxh3_64(raw_data).to_be_bytes() // Hash ORIGINAL
5. Envelope: StorageEnvelope {
compressed_data: compressed,
@@ -339,7 +339,7 @@ Input: raw_data (bytes), format (string, default "msgpack")
format: format
}
6. Serialize: envelope_bytes = msgpack_encode(envelope) // compressed_data as bin (1.1+)
-7. Validate: envelope_bytes.length <= 512 MB
+7. Validate: envelope_bytes.length <= 512 MiB
8. Return: envelope_bytes
```
@@ -355,11 +355,11 @@ Input: raw_data (bytes), format (string, default "msgpack")
```
Input: envelope_bytes
-1. Validate: envelope_bytes.length <= 512 MB
+1. Validate: envelope_bytes.length <= 512 MiB
2. Deserialize: envelope = msgpack_decode(envelope_bytes) as StorageEnvelope
// accept BOTH element[0] encodings: bin AND array-of-ints
-3. Validate: envelope.compressed_data.length <= 512 MB
-4. Validate: envelope.original_size <= 512 MB
+3. Validate: envelope.compressed_data.length <= 512 MiB
+4. Validate: envelope.original_size <= 512 MiB
5. Bomb check: (see Security Limits above)
6. Decompress: data = lz4_block_decompress(envelope.compressed_data, envelope.original_size)
7. Checksum: computed = xxh3_64(data).to_be_bytes()
diff --git a/test-vectors/interop-v2.json b/test-vectors/interop-v2.json
new file mode 100644
index 0000000..3069ce1
--- /dev/null
+++ b/test-vectors/interop-v2.json
@@ -0,0 +1,225 @@
+{
+ "version": "1.0.0",
+ "spec": "spec/interop-v2.md",
+ "generator": "tools/interop-v2-reference.py (CPython stdlib, incl. pure-Python LZ4 block codec)",
+ "cross_checked_by": "tools/interop-v2-crosscheck.mjs (independent container parser + LZ4 block decoder + WebCrypto HKDF/AES-GCM; zero dependencies)",
+ "container_format": "0xC1 0x02 + canonical msgpack [method:int, original_size:int, payload:bin]",
+ "security_limits": {
+ "max_uncompressed_size": 536870912,
+ "max_compressed_size": 536870912,
+ "max_compression_ratio": 1000,
+ "note": "All enforced BEFORE decompression, integer arithmetic only \u2014 spec/interop-v2.md#security-limits-decompression-bounds"
+ },
+ "compressed_bytes_note": "method-1 payload bytes are NOT canonical: conformant LZ4 encoders legally differ. Vectors pin the reference compressor's output for READ-side conformance; writers may produce different valid LZ4 for the same value.",
+ "error_vectors_note": "reject_* vectors MUST be rejected with an error; the 'error' text is a maintainer note, not a normative message.",
+ "container_vectors": [
+ {
+ "name": "method0_issue_example",
+ "description": "method 0 (no compression) wrap of the v1 issue_example_object value \u2014 original_size MUST equal the payload length",
+ "value": {
+ "name": "alice",
+ "age": 30
+ },
+ "value_msgpack_hex": "82a36167651ea46e616d65a5616c696365",
+ "method": 0,
+ "original_size": 17,
+ "payload_hex": "82a36167651ea46e616d65a5616c696365",
+ "container_hex": "c102930011c41182a36167651ea46e616d65a5616c696365"
+ },
+ {
+ "name": "lz4_roundtrip_compressible",
+ "description": "method 1 compressed round-trip of a redundant value. The payload is the REFERENCE compressor's output: readers MUST decompress it to the pinned value bytes; writers are NOT required to reproduce these compressed bytes (compressed bytes are non-canonical, read-side conformance only)",
+ "value": {
+ "events": [
+ "GET /api/users/42 200 OK",
+ "GET /api/users/42 200 OK",
+ "GET /api/users/42 200 OK",
+ "GET /api/users/42 200 OK",
+ "GET /api/users/42 200 OK",
+ "GET /api/users/42 200 OK",
+ "GET /api/users/42 200 OK",
+ "GET /api/users/42 200 OK"
+ ],
+ "source": "interop-v2-reference"
+ },
+ "value_msgpack_hex": "82a66576656e747398b8474554202f6170692f75736572732f343220323030204f4bb8474554202f6170692f75736572732f343220323030204f4bb8474554202f6170692f75736572732f343220323030204f4bb8474554202f6170692f75736572732f343220323030204f4bb8474554202f6170692f75736572732f343220323030204f4bb8474554202f6170692f75736572732f343220323030204f4bb8474554202f6170692f75736572732f343220323030204f4bb8474554202f6170692f75736572732f343220323030204f4ba6736f75726365b4696e7465726f702d76322d7265666572656e6365",
+ "method": 1,
+ "original_size": 237,
+ "payload_hex": "ff1382a66576656e747398b8474554202f6170692f75736572732f343220323030204f4b19009cf00da6736f75726365b4696e7465726f702d76322d7265666572656e6365",
+ "container_hex": "c1029301ccedc445ff1382a66576656e747398b8474554202f6170692f75736572732f343220323030204f4b19009cf00da6736f75726365b4696e7465726f702d76322d7265666572656e6365"
+ },
+ {
+ "name": "lz4_wraps_v1_value_vector",
+ "description": "method 1 container over the SAME plain bytes as v1's issue_example_object value vector \u2014 the inner value profile is inherited from v1 unchanged. Incompressible at this size: the LZ4 payload is a literals-only block LARGER than the value (writers SHOULD have used method 0; readers MUST still accept it)",
+ "value": {
+ "name": "alice",
+ "age": 30
+ },
+ "value_msgpack_hex": "82a36167651ea46e616d65a5616c696365",
+ "method": 1,
+ "original_size": 17,
+ "payload_hex": "f00282a36167651ea46e616d65a5616c696365",
+ "container_hex": "c102930111c413f00282a36167651ea46e616d65a5616c696365"
+ },
+ {
+ "name": "method0_noncanonical_widths",
+ "description": "Same value as method0_issue_example, but with deliberately NON-canonical header widths (array16, uint8 method, uint32 original_size, bin16 payload). Readers MUST accept any unsigned-family width; writers MUST NOT emit this.",
+ "value": {
+ "name": "alice",
+ "age": 30
+ },
+ "value_msgpack_hex": "82a36167651ea46e616d65a5616c696365",
+ "method": 0,
+ "original_size": 17,
+ "payload_hex": "82a36167651ea46e616d65a5616c696365",
+ "container_hex": "c102dc0003cc00ce00000011c5001182a36167651ea46e616d65a5616c696365"
+ }
+ ],
+ "aad_vectors": [
+ {
+ "name": "interop_v2_aad",
+ "description": "AAD v0x03 over the same tenant + interop key as v1's interop_key_aad; the two AADs differ ONLY in the final component (frozen tokens 'True' vs 'False') \u2014 this is the cryptographic mode separation",
+ "tenant_id": "cross-sdk-test",
+ "cache_key": "users:get_user:61598716255080080f6456eb065c2e51badfaa4320b0efe97469c29cffee8875",
+ "format": "msgpack",
+ "compressed": true,
+ "aad_hex": "030000000e63726f73732d73646b2d746573740000004f75736572733a6765745f757365723a36313539383731363235353038303038306636343536656230363563326535316261646661613433323062306566653937343639633239636666656538383735000000076d73677061636b0000000454727565",
+ "v1_aad_hex_for_comparison": "030000000e63726f73732d73646b2d746573740000004f75736572733a6765745f757365723a36313539383731363235353038303038306636343536656230363563326535316261646661613433323062306566653937343639633239636666656538383735000000076d73677061636b0000000546616c7365"
+ }
+ ],
+ "encryption_vectors": [
+ {
+ "name": "interop_v2_compressed_encryption_roundtrip",
+ "description": "Full v2 round-trip: HKDF-SHA256 per spec/encryption.md (same master key + tenant as interop/v1 and encryption.json, hence the same derived key), AES-256-GCM over the ENTIRE v2 container bytes (lz4_roundtrip_compressible) with the v2 AAD (compressed='True') and a fixed nonce.",
+ "master_key_hex": "6161616161616161616161616161616161616161616161616161616161616161",
+ "tenant_id": "cross-sdk-test",
+ "derived_key_fingerprint_hex": "96179a9bc881aa7ca83f04b78a66afd3",
+ "cache_key": "users:get_user:61598716255080080f6456eb065c2e51badfaa4320b0efe97469c29cffee8875",
+ "format": "msgpack",
+ "compressed": true,
+ "aad_hex": "030000000e63726f73732d73646b2d746573740000004f75736572733a6765745f757365723a36313539383731363235353038303038306636343536656230363563326535316261646661613433323062306566653937343639633239636666656538383735000000076d73677061636b0000000454727565",
+ "plaintext_hex": "c1029301ccedc445ff1382a66576656e747398b8474554202f6170692f75736572732f343220323030204f4b19009cf00da6736f75726365b4696e7465726f702d76322d7265666572656e6365",
+ "nonce_hex": "101112131415161718191a1b",
+ "ciphertext_hex": "101112131415161718191a1b043e651799eeb533c4ddf646124b57eabe59f9292ed9f2cc1412ce52f677fccc93f82f01894f619b13576eb790ea10a714a3afd85019a283a5240a602a170c9485575fe350ba5c414d1374f33216bd6b38eaf1f55503767f71229480e5",
+ "ciphertext_layout": "nonce(12) || ciphertext || auth_tag(16)"
+ }
+ ],
+ "reject_vectors": [
+ {
+ "name": "reject_bad_magic_v1_value",
+ "description": "A bare interop/v1 value fed to a v2 reader: first byte is a msgpack marker, not 0xC1",
+ "container_hex": "82a36167651ea46e616d65a5616c696365",
+ "error": "bad magic; reader SHOULD diagnose 'possible interop/v1 value'"
+ },
+ {
+ "name": "reject_bad_container_version",
+ "description": "Right magic, wrong version byte (0x03)",
+ "container_hex": "c103930011c41182a36167651ea46e616d65a5616c696365",
+ "error": "unsupported container version"
+ },
+ {
+ "name": "reject_unknown_method",
+ "description": "method 2 is not in the registry",
+ "container_hex": "c102930211c41182a36167651ea46e616d65a5616c696365",
+ "error": "unknown compression method"
+ },
+ {
+ "name": "reject_payload_array_of_ints",
+ "description": "Payload encoded as a msgpack array of integers instead of bin \u2014 the legacy ByteStorage leniency is explicitly NOT inherited by interop/v2",
+ "container_hex": "c10293000393010203",
+ "error": "payload must be msgpack bin"
+ },
+ {
+ "name": "reject_method_signed_marker",
+ "description": "method encoded with a signed-family marker (int8 0xd0 carrying value 0) \u2014 marker-level enforcement rejects the signed family even for non-negative values",
+ "container_hex": "c10293d00011c41182a36167651ea46e616d65a5616c696365",
+ "error": "signed-family int marker for method"
+ },
+ {
+ "name": "reject_negative_original_size",
+ "description": "original_size encoded as negative fixint -1 (0xff) \u2014 negative sizes are structurally unrepresentable once signed-family markers are rejected; a value-level reader that accepts -1 here bypasses every upper-bound check",
+ "container_hex": "c1029301ffc40410410000",
+ "error": "signed-family int marker for original_size"
+ },
+ {
+ "name": "reject_forged_bin32_length",
+ "description": "bin32 payload header declaring 4 GiB (0xffffffff) with no data following \u2014 readers MUST validate the length header against remaining input BEFORE allocating",
+ "container_hex": "c102930005c6ffffffff",
+ "error": "bin length header exceeds remaining input"
+ },
+ {
+ "name": "reject_payload_str",
+ "description": "Payload encoded as the msgpack str family instead of bin",
+ "container_hex": "c102930003a3616263",
+ "error": "payload must be msgpack bin"
+ },
+ {
+ "name": "reject_method0_size_mismatch",
+ "description": "method 0 with original_size != payload length",
+ "container_hex": "c102930012c41182a36167651ea46e616d65a5616c696365",
+ "error": "method 0 original_size mismatch"
+ },
+ {
+ "name": "reject_trailing_bytes",
+ "description": "Valid container followed by one extra byte",
+ "container_hex": "c102930011c41182a36167651ea46e616d65a5616c69636500",
+ "error": "trailing bytes after container body"
+ },
+ {
+ "name": "reject_declared_size_bomb",
+ "description": "original_size declares 1 TiB \u2014 exceeds the 512 MiB cap (checked BEFORE decompression)",
+ "container_hex": "c1029301cf0000010000000000c40410410000",
+ "error": "original_size exceeds max uncompressed size"
+ },
+ {
+ "name": "reject_ratio_bomb",
+ "description": "10-byte payload declaring 10001 output bytes \u2014 exceeds the 1000:1 ratio (checked BEFORE decompression)",
+ "container_hex": "c1029301cd2711c40a00000000000000000000",
+ "error": "compression ratio exceeds 1000:1"
+ },
+ {
+ "name": "reject_zero_length_compressed",
+ "description": "method 1 with an empty payload",
+ "container_hex": "c102930101c400",
+ "error": "zero-length compressed payload"
+ },
+ {
+ "name": "reject_lz4_zero_offset",
+ "description": "LZ4 sequence with match offset 0 (invalid in the block format)",
+ "container_hex": "c102930105c40410410000",
+ "error": "invalid LZ4 match offset 0"
+ },
+ {
+ "name": "reject_lz4_truncated",
+ "description": "The lz4_roundtrip_compressible payload with its last 3 bytes removed",
+ "container_hex": "c1029301ccedc442ff1382a66576656e747398b8474554202f6170692f75736572732f343220323030204f4b19009cf00da6736f75726365b4696e7465726f702d76322d726566657265",
+ "error": "truncated LZ4 block"
+ },
+ {
+ "name": "reject_lz4_length_mismatch",
+ "description": "Valid LZ4 block whose output is one byte short of original_size",
+ "container_hex": "c1029301cceec445ff1382a66576656e747398b8474554202f6170692f75736572732f343220323030204f4b19009cf00da6736f75726365b4696e7465726f702d76322d7265666572656e6365",
+ "error": "LZ4 output length != original_size (also fine to fail as overrun, depending on decoder structure)"
+ }
+ ],
+ "crypto_reject_vectors": [
+ {
+ "name": "reject_v2_ciphertext_with_v1_aad",
+ "description": "The v2 ciphertext MUST fail AES-GCM authentication under the v1 AAD (compressed='False')",
+ "master_key_hex": "6161616161616161616161616161616161616161616161616161616161616161",
+ "tenant_id": "cross-sdk-test",
+ "ciphertext_hex": "101112131415161718191a1b043e651799eeb533c4ddf646124b57eabe59f9292ed9f2cc1412ce52f677fccc93f82f01894f619b13576eb790ea10a714a3afd85019a283a5240a602a170c9485575fe350ba5c414d1374f33216bd6b38eaf1f55503767f71229480e5",
+ "aad_hex": "030000000e63726f73732d73646b2d746573740000004f75736572733a6765745f757365723a36313539383731363235353038303038306636343536656230363563326535316261646661613433323062306566653937343639633239636666656538383735000000076d73677061636b0000000546616c7365",
+ "error": "authentication failure \u2014 cross-mode read, terminal per the no-retry rule"
+ },
+ {
+ "name": "reject_v1_ciphertext_with_v2_aad",
+ "description": "interop/v1's published interop_encryption_roundtrip ciphertext MUST fail under the v2 AAD (compressed='True')",
+ "master_key_hex": "6161616161616161616161616161616161616161616161616161616161616161",
+ "tenant_id": "cross-sdk-test",
+ "ciphertext_hex": "000102030405060708090a0b033caf732820ce189e1506f842aebf8cdb6a242eb08c55b6f5a91eb9007b3bd657",
+ "aad_hex": "030000000e63726f73732d73646b2d746573740000004f75736572733a6765745f757365723a36313539383731363235353038303038306636343536656230363563326535316261646661613433323062306566653937343639633239636666656538383735000000076d73677061636b0000000454727565",
+ "error": "authentication failure \u2014 cross-mode read, terminal per the no-retry rule"
+ }
+ ]
+}
diff --git a/tools/interop-v2-crosscheck.mjs b/tools/interop-v2-crosscheck.mjs
new file mode 100644
index 0000000..11a2170
--- /dev/null
+++ b/tools/interop-v2-crosscheck.mjs
@@ -0,0 +1,367 @@
+#!/usr/bin/env node
+// Independent cross-check of test-vectors/interop-v2.json (spec/interop-v2.md).
+//
+// From-scratch JavaScript implementations — sharing no code with
+// tools/interop-v2-reference.py — of the pieces this profile ADDS: the v2
+// value container parser (strict types, bounds-before-decompress), an LZ4
+// *block* decompressor, the v2 AAD (compressed = "True"), and the encrypted
+// round-trip via Node's built-in WebCrypto (HKDF-SHA256 + AES-256-GCM),
+// including both cross-mode AAD rejections. The inherited v1 surface
+// (canonical value encoding, key generation) is exhaustively cross-checked by
+// tools/interop-crosscheck.mjs already and is not re-verified here.
+//
+// Run (zero dependencies):
+// node tools/interop-v2-crosscheck.mjs [path/to/interop-v2.json]
+
+import { readFileSync } from "node:fs";
+import { fileURLToPath } from "node:url";
+import { dirname, join } from "node:path";
+import { webcrypto } from "node:crypto";
+
+const MAX_UNCOMPRESSED = 512n * 1024n * 1024n;
+const MAX_COMPRESSED = 512 * 1024 * 1024;
+const MAX_RATIO = 1000n;
+
+// Strict hex decoder for every JSON-provided hex field. Buffer.from(s, "hex")
+// silently truncates at the first invalid character and drops a trailing odd
+// nibble — a corrupted vector file could otherwise "pass" against only the
+// valid prefix of a field.
+function fromHex(s, field) {
+ if (typeof s !== "string" || s.length % 2 !== 0 || /[^0-9a-fA-F]/.test(s)) {
+ throw new Error(`vector field ${field} is not valid even-length hex`);
+ }
+ return Buffer.from(s, "hex");
+}
+
+// --- LZ4 block decompressor (independent implementation) --------------------
+function lz4BlockDecompress(block, originalSize) {
+ const out = Buffer.alloc(originalSize);
+ let o = 0;
+ let i = 0;
+ const n = block.length;
+ if (n === 0) throw new Error("empty LZ4 block");
+ for (;;) {
+ if (i >= n) throw new Error("truncated LZ4 block: missing token");
+ const token = block[i++];
+ let litLen = token >> 4;
+ if (litLen === 15) {
+ let b;
+ do {
+ if (i >= n) throw new Error("truncated LZ4 block: literal-length extension");
+ b = block[i++];
+ litLen += b;
+ } while (b === 255);
+ }
+ if (i + litLen > n) throw new Error("truncated LZ4 block: literals overrun input");
+ if (o + litLen > originalSize) throw new Error("LZ4 output exceeds original_size");
+ block.copy(out, o, i, i + litLen);
+ o += litLen;
+ i += litLen;
+ if (i === n) break; // clean end: last sequence is literals-only
+ if (i + 2 > n) throw new Error("truncated LZ4 block: missing match offset");
+ const offset = block[i] | (block[i + 1] << 8);
+ i += 2;
+ if (offset === 0) throw new Error("invalid LZ4 match offset 0");
+ if (offset > o) throw new Error("LZ4 match offset beyond output start");
+ let matchLen = (token & 0x0f) + 4;
+ if ((token & 0x0f) === 15) {
+ let b;
+ do {
+ if (i >= n) throw new Error("truncated LZ4 block: match-length extension");
+ b = block[i++];
+ matchLen += b;
+ } while (b === 255);
+ }
+ if (o + matchLen > originalSize) throw new Error("LZ4 output exceeds original_size");
+ for (let k = 0; k < matchLen; k++) {
+ out[o] = out[o - offset]; // byte-wise: overlapping matches are legal
+ o++;
+ }
+ }
+ if (o !== originalSize) throw new Error(`LZ4 output length ${o} != original_size ${originalSize}`);
+ return out;
+}
+
+// --- v2 container parser (strict) --------------------------------------------
+// Minimal msgpack reader for the body grammar only: array header, two
+// non-negative ints (BigInt — original_size may exceed 2^53 in hostile
+// input), one bin. Anything else is a hard error, including the legacy
+// array-of-ints payload shape.
+function parseContainer(data) {
+ if (data.length < 2) throw new Error("truncated container (magic + version bytes required)");
+ if (data[0] !== 0xc1) {
+ throw new Error("bad container magic (0xC1 expected) — possible interop/v1 value");
+ }
+ if (data[1] !== 0x02) throw new Error(`unsupported container version 0x${data[1].toString(16)}`);
+ const body = data.subarray(2);
+ let pos = 0;
+ const need = (k) => {
+ if (pos + k > body.length) throw new Error("container body truncated");
+ };
+
+ const readArrayHeader = () => {
+ need(1);
+ const m = body[pos++];
+ if (m >= 0x90 && m <= 0x9f) return m & 0x0f;
+ if (m === 0xdc) {
+ need(2);
+ const v = body.readUInt16BE(pos);
+ pos += 2;
+ return v;
+ }
+ if (m === 0xdd) {
+ need(4);
+ const v = body.readUInt32BE(pos);
+ pos += 4;
+ return v;
+ }
+ throw new Error(`container body must be a msgpack array, got marker 0x${m.toString(16)}`);
+ };
+ const readUint = () => {
+ need(1);
+ const m = body[pos++];
+ if (m <= 0x7f) return BigInt(m);
+ if (m === 0xcc) {
+ need(1);
+ return BigInt(body[pos++]);
+ }
+ if (m === 0xcd) {
+ need(2);
+ const v = BigInt(body.readUInt16BE(pos));
+ pos += 2;
+ return v;
+ }
+ if (m === 0xce) {
+ need(4);
+ const v = BigInt(body.readUInt32BE(pos));
+ pos += 4;
+ return v;
+ }
+ if (m === 0xcf) {
+ need(8);
+ const v = body.readBigUInt64BE(pos);
+ pos += 8;
+ return v;
+ }
+ throw new Error(`expected non-negative msgpack int, got marker 0x${m.toString(16)}`);
+ };
+ const readBin = () => {
+ need(1);
+ const m = body[pos++];
+ let len;
+ if (m === 0xc4) {
+ need(1);
+ len = body[pos++];
+ } else if (m === 0xc5) {
+ need(2);
+ len = body.readUInt16BE(pos);
+ pos += 2;
+ } else if (m === 0xc6) {
+ need(4);
+ len = body.readUInt32BE(pos);
+ pos += 4;
+ } else {
+ throw new Error(`payload must be msgpack bin (0xc4/0xc5/0xc6), got marker 0x${m.toString(16)}`);
+ }
+ // header-vs-remaining-input rule: validate BEFORE consuming/allocating
+ if (pos + len > body.length) throw new Error("bin length header exceeds remaining input");
+ const p = body.subarray(pos, pos + len);
+ pos += len;
+ return p;
+ };
+
+ if (readArrayHeader() !== 3) throw new Error("container body must be a 3-element array");
+ const method = readUint();
+ const originalSize = readUint();
+ const payload = readBin();
+ if (pos !== body.length) throw new Error("trailing bytes after container body");
+ return { method, originalSize, payload };
+}
+
+// Normative reader algorithm steps 2-5: container bytes -> plain value bytes.
+function decodeContainer(data) {
+ const { method, originalSize, payload } = parseContainer(data);
+ if (method !== 0n && method !== 1n) throw new Error(`unknown compression method ${method}`);
+ // Security Limits — all BEFORE decompression, integer arithmetic (BigInt).
+ if (originalSize > MAX_UNCOMPRESSED) throw new Error("original_size exceeds max uncompressed size");
+ if (payload.length > MAX_COMPRESSED) throw new Error("payload exceeds max compressed size");
+ if (method === 1n) {
+ if (payload.length === 0) throw new Error("zero-length compressed payload");
+ if (originalSize > MAX_RATIO * BigInt(payload.length)) {
+ throw new Error("compression ratio exceeds 1000:1 — decompression bomb");
+ }
+ return lz4BlockDecompress(payload, Number(originalSize));
+ }
+ if (originalSize !== BigInt(payload.length)) {
+ throw new Error(`method 0 original_size ${originalSize} != payload length ${payload.length}`);
+ }
+ return payload;
+}
+
+// --- AAD v0x03 + HKDF-SHA256 (per spec/encryption.md) ------------------------
+function aadV3(tenantId, cacheKey, format, compressed) {
+ const chunks = [Buffer.from([0x03])];
+ for (const comp of [tenantId, cacheKey, format, compressed ? "True" : "False"]) {
+ const b = Buffer.from(comp, "utf8");
+ const len = Buffer.alloc(4);
+ len.writeUInt32BE(b.length);
+ chunks.push(len, b);
+ }
+ return Buffer.concat(chunks);
+}
+
+function constructSalt(domain, tenantSalt) {
+ const d = Buffer.from(domain, "utf8");
+ const t = Buffer.from(tenantSalt, "utf8");
+ const tLen = Buffer.alloc(2);
+ tLen.writeUInt16BE(t.length);
+ return Buffer.concat([Buffer.from("cachekit_v1_", "utf8"), Buffer.from([d.length]), d, tLen, t]);
+}
+
+async function deriveEncryptionKey(masterKeyHex, tenantId) {
+ const masterKey = await webcrypto.subtle.importKey(
+ "raw",
+ fromHex(masterKeyHex, "master_key_hex"),
+ "HKDF",
+ false,
+ ["deriveBits"],
+ );
+ const bits = await webcrypto.subtle.deriveBits(
+ {
+ name: "HKDF",
+ hash: "SHA-256",
+ salt: constructSalt("encryption", tenantId),
+ info: Buffer.from("encryption", "utf8"),
+ },
+ masterKey,
+ 256,
+ );
+ return Buffer.from(bits);
+}
+
+// --- run ---------------------------------------------------------------------
+const here = dirname(fileURLToPath(import.meta.url));
+const vectorsPath = process.argv[2] ?? join(here, "..", "test-vectors", "interop-v2.json");
+const doc = JSON.parse(readFileSync(vectorsPath, "utf8"));
+
+let failures = 0;
+const check = (name, kind, expected, actual) => {
+ if (expected !== actual) {
+ failures++;
+ console.error(`FAIL ${name} (${kind})\n expected ${expected}\n actual ${actual}`);
+ }
+};
+
+for (const v of doc.container_vectors) {
+ try {
+ const container = fromHex(v.container_hex, `${v.name}.container_hex`);
+ const value = decodeContainer(container);
+ check(v.name, "decoded value bytes", v.value_msgpack_hex, value.toString("hex"));
+ // Structural pins: method / original_size / payload fields agree with the parse.
+ const parsed = parseContainer(container);
+ check(v.name, "method", BigInt(v.method), parsed.method);
+ check(v.name, "original_size", BigInt(v.original_size), parsed.originalSize);
+ check(v.name, "payload_hex", v.payload_hex, parsed.payload.toString("hex"));
+ } catch (err) {
+ failures++;
+ console.error(`FAIL ${v.name} (container): ${err.message ?? err}`);
+ }
+}
+
+for (const v of doc.aad_vectors) {
+ const aad = aadV3(v.tenant_id, v.cache_key, v.format, v.compressed);
+ check(v.name, "aad_hex", v.aad_hex, aad.toString("hex"));
+ const aadV1 = aadV3(v.tenant_id, v.cache_key, v.format, false);
+ check(v.name, "v1_aad_hex_for_comparison", v.v1_aad_hex_for_comparison, aadV1.toString("hex"));
+}
+
+for (const v of doc.encryption_vectors ?? []) {
+ try {
+ const derived = await deriveEncryptionKey(v.master_key_hex, v.tenant_id);
+ const fpInput = Buffer.concat([Buffer.from("key_fingerprint_v1", "utf8"), derived]);
+ const fp = Buffer.from(await webcrypto.subtle.digest("SHA-256", fpInput)).subarray(0, 16);
+ check(v.name, "derived_key_fingerprint_hex", v.derived_key_fingerprint_hex, fp.toString("hex"));
+
+ const gcmKey = await webcrypto.subtle.importKey("raw", derived, "AES-GCM", false, ["decrypt"]);
+ const ct = fromHex(v.ciphertext_hex, `${v.name}.ciphertext_hex`);
+ const plaintext = Buffer.from(
+ await webcrypto.subtle.decrypt(
+ {
+ name: "AES-GCM",
+ iv: ct.subarray(0, 12),
+ additionalData: fromHex(v.aad_hex, `${v.name}.aad_hex`),
+ tagLength: 128,
+ },
+ gcmKey,
+ ct.subarray(12),
+ ),
+ );
+ check(v.name, "plaintext_hex (AES-GCM decrypt)", v.plaintext_hex, plaintext.toString("hex"));
+ check(v.name, "nonce_hex", v.nonce_hex, ct.subarray(0, 12).toString("hex"));
+ // End-to-end: the decrypted container must decode to the pinned value bytes
+ // of the container vector it wraps.
+ const inner = decodeContainer(plaintext);
+ const src = doc.container_vectors.find((c) => c.container_hex === v.plaintext_hex);
+ if (src) check(v.name, "decrypted container decodes", src.value_msgpack_hex, inner.toString("hex"));
+ } catch (err) {
+ failures++;
+ console.error(`FAIL ${v.name} (encryption): ${err.message ?? err}`);
+ }
+}
+
+for (const v of doc.reject_vectors) {
+ // Parse the hex OUTSIDE the expected-rejection block: a malformed vector
+ // file must fail the run, not masquerade as a passing rejection.
+ const container = fromHex(v.container_hex, `${v.name}.container_hex`);
+ try {
+ decodeContainer(container);
+ failures++;
+ console.error(`FAIL ${v.name}: expected rejection (${v.error}), but decoding succeeded`);
+ } catch (err) {
+ // Expected — but only a deliberate reader rejection (a plain Error thrown
+ // by the parser/decoder), never an unrelated crash such as a TypeError.
+ if (err?.constructor !== Error) {
+ failures++;
+ console.error(`FAIL ${v.name}: rejected by unexpected error type ${err?.constructor?.name}: ${err?.message ?? err}`);
+ }
+ }
+}
+
+// Cross-mode AAD rejections: AES-GCM authentication MUST fail both ways.
+for (const v of doc.crypto_reject_vectors ?? []) {
+ const derived = await deriveEncryptionKey(v.master_key_hex, v.tenant_id);
+ const gcmKey = await webcrypto.subtle.importKey("raw", derived, "AES-GCM", false, ["decrypt"]);
+ const ct = fromHex(v.ciphertext_hex, `${v.name}.ciphertext_hex`);
+ const aad = fromHex(v.aad_hex, `${v.name}.aad_hex`);
+ try {
+ await webcrypto.subtle.decrypt(
+ {
+ name: "AES-GCM",
+ iv: ct.subarray(0, 12),
+ additionalData: aad,
+ tagLength: 128,
+ },
+ gcmKey,
+ ct.subarray(12),
+ );
+ failures++;
+ console.error(`FAIL ${v.name}: cross-mode decrypt unexpectedly succeeded (${v.error})`);
+ } catch (err) {
+ // Expected — but only a WebCrypto authentication failure (OperationError),
+ // never an unrelated crash.
+ if (err?.name !== "OperationError") {
+ failures++;
+ console.error(`FAIL ${v.name}: expected AES-GCM authentication failure (OperationError), got ${err?.name}: ${err?.message ?? err}`);
+ }
+ }
+}
+
+if (failures > 0) {
+ console.error(`\n${failures} mismatch(es) — reference and cross-check DISAGREE`);
+ process.exit(1);
+}
+console.log(
+ `OK: ${doc.container_vectors.length} container, ${doc.aad_vectors.length} AAD, ` +
+ `${(doc.encryption_vectors ?? []).length} encryption, ${doc.reject_vectors.length} reject, ` +
+ `${(doc.crypto_reject_vectors ?? []).length} crypto-reject vectors verified independently`,
+);
diff --git a/tools/interop-v2-reference.py b/tools/interop-v2-reference.py
new file mode 100644
index 0000000..c0cd3c6
--- /dev/null
+++ b/tools/interop-v2-reference.py
@@ -0,0 +1,845 @@
+#!/usr/bin/env python3
+"""Reference implementation of CacheKit interop/v2 (spec/interop-v2.md).
+
+Stdlib-only (two optional extras, see below). Executable companion to the
+compressed-values profile spec:
+ - the v2 value container (0xC1 0x02 + msgpack [method, original_size, payload:bin])
+ - a pure-Python LZ4 *block* codec (compressor + strict decompressor), so
+ vector generation has no third-party dependency
+ - the normative reader algorithm, including every Security-Limits bound
+ - v2 AAD construction (compressed = "True", the frozen token)
+ - test-vector generator + self-verifier for ../test-vectors/interop-v2.json
+
+The interop/v1 surface (canonical MessagePack encoder, HKDF chain, AAD builder,
+published v1 vectors) is imported from tools/interop-reference.py — v1 is the
+single source of truth for everything this profile inherits, and importing it
+(instead of copying it) is the standing proof that v1 is untouched.
+
+Usage:
+ python3 tools/interop-v2-reference.py generate # rewrite test-vectors/interop-v2.json
+ python3 tools/interop-v2-reference.py verify # re-derive and compare against the JSON
+
+Optional-dependency checks deepen `verify` when importable (both run in CI):
+ - `lz4`: bidirectional conformance with the de-facto C implementation —
+ our compressed bytes decompress under lz4.block, and lz4.block's output
+ (store_size=False) decompresses under our decoder.
+ - `cryptography`: re-verifies the AES-256-GCM seal of the encryption vector
+ and both cross-mode AAD rejections (tools/interop-v2-crosscheck.mjs ALWAYS
+ verifies these via Node's built-in WebCrypto regardless).
+
+Cross-check with an independent implementation: tools/interop-v2-crosscheck.mjs
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+import logging
+import sys
+from pathlib import Path
+from types import ModuleType
+
+_HERE = Path(__file__).resolve().parent
+
+
+def _load_v1() -> ModuleType:
+ """Import tools/interop-reference.py (hyphenated filename) as a module."""
+ spec = importlib.util.spec_from_file_location("interop_reference", _HERE / "interop-reference.py")
+ if spec is None or spec.loader is None:
+ msg = "cannot load tools/interop-reference.py"
+ raise ImportError(msg)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+v1 = _load_v1()
+
+# Security Limits (spec/interop-v2.md), reusing the wire-format.md constants.
+MAX_UNCOMPRESSED = 512 * 1024 * 1024
+MAX_COMPRESSED = 512 * 1024 * 1024
+MAX_RATIO = 1000
+
+MAGIC = 0xC1
+CONTAINER_VERSION = 0x02
+METHOD_NONE = 0
+METHOD_LZ4_BLOCK = 1
+
+
+class V2Error(ValueError):
+ """Raised for any interop/v2 container the reader algorithm must reject."""
+
+
+def _bad_marker(clause: str, marker: int) -> V2Error:
+ """Reader rejection for a wrong MessagePack marker (diagnostic text, not normative)."""
+ return V2Error(f"{clause}, got marker 0x{marker:02x}")
+
+
+class SelfCheckError(Exception):
+ """A vector invariant failed. Unlike `assert`, never disabled by `python -O`."""
+
+
+def _require(cond: object, msg: str) -> None:
+ """Self-check guard: `assert` semantics that survive optimized mode."""
+ if not cond:
+ raise SelfCheckError(msg)
+
+
+# ---------------------------------------------------------------------------
+# Pure-Python LZ4 *block* codec (https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md)
+#
+# The compressor is a simple greedy hash-table matcher. Its output is valid
+# LZ4 (verified against lz4.block when importable) but deliberately NOT
+# canonical — the spec pins read-side conformance only; compressed bytes are
+# writer-dependent. The decompressor is strict: invalid offsets, truncation,
+# and any output-size disagreement with original_size are hard errors.
+# ---------------------------------------------------------------------------
+
+# LZ4 end-of-block restrictions: the last 5 bytes are always literals, and the
+# last match must start at least 12 bytes before the end of the block.
+_MFLIMIT = 12
+_LAST_LITERALS = 5
+_MIN_MATCH = 4
+_MAX_OFFSET = 0xFFFF
+
+
+def _emit_literal_run(out: bytearray, literals: bytes) -> None:
+ n = len(literals)
+ token_lit = 15 if n >= 15 else n
+ out.append(token_lit << 4)
+ if n >= 15:
+ rem = n - 15
+ while rem >= 255:
+ out.append(255)
+ rem -= 255
+ out.append(rem)
+ out += literals
+
+
+def _emit_sequence(out: bytearray, literals: bytes, offset: int, match_len: int) -> None:
+ n = len(literals)
+ ml = match_len - _MIN_MATCH
+ token = (15 if n >= 15 else n) << 4 | (15 if ml >= 15 else ml)
+ out.append(token)
+ if n >= 15:
+ rem = n - 15
+ while rem >= 255:
+ out.append(255)
+ rem -= 255
+ out.append(rem)
+ out += literals
+ out += offset.to_bytes(2, "little")
+ if ml >= 15:
+ rem = ml - 15
+ while rem >= 255:
+ out.append(255)
+ rem -= 255
+ out.append(rem)
+
+
+def lz4_block_compress(data: bytes) -> bytes:
+ """Greedy LZ4 block compressor. Valid output, not canonical output."""
+ n = len(data)
+ out = bytearray()
+ if n < _MFLIMIT + 1: # too short for any match: literals-only block
+ _emit_literal_run(out, data)
+ return bytes(out)
+ table: dict[bytes, int] = {}
+ i = 0
+ anchor = 0
+ match_start_limit = n - _MFLIMIT # matches may not start after this
+ match_end_limit = n - _LAST_LITERALS # matches may not extend past this
+ while i <= match_start_limit:
+ seq = data[i : i + _MIN_MATCH]
+ j = table.get(seq)
+ table[seq] = i
+ if j is not None and i - j <= _MAX_OFFSET:
+ length = _MIN_MATCH
+ while i + length < match_end_limit and data[j + length] == data[i + length]:
+ length += 1
+ _emit_sequence(out, data[anchor:i], i - j, length)
+ i += length
+ anchor = i
+ else:
+ i += 1
+ _emit_literal_run(out, data[anchor:])
+ return bytes(out)
+
+
+def lz4_block_decompress(block: bytes, original_size: int) -> bytes:
+ """Strict LZ4 block decoder; output MUST be exactly original_size bytes."""
+ out = bytearray()
+ i = 0
+ n = len(block)
+ if n == 0:
+ raise V2Error("empty LZ4 block")
+ while True:
+ if i >= n:
+ raise V2Error("truncated LZ4 block: missing token")
+ token = block[i]
+ i += 1
+ lit_len = token >> 4
+ if lit_len == 15:
+ while True:
+ if i >= n:
+ raise V2Error("truncated LZ4 block: literal-length extension")
+ b = block[i]
+ i += 1
+ lit_len += b
+ if b != 255:
+ break
+ if i + lit_len > n:
+ raise V2Error("truncated LZ4 block: literals overrun input")
+ out += block[i : i + lit_len]
+ i += lit_len
+ if len(out) > original_size:
+ raise V2Error("LZ4 output exceeds original_size")
+ if i == n:
+ break # clean end: last sequence is literals-only
+ if i + 2 > n:
+ raise V2Error("truncated LZ4 block: missing match offset")
+ offset = block[i] | (block[i + 1] << 8)
+ i += 2
+ if offset == 0:
+ raise V2Error("invalid LZ4 match offset 0")
+ if offset > len(out):
+ raise V2Error("LZ4 match offset beyond output start")
+ match_len = (token & 0x0F) + _MIN_MATCH
+ if (token & 0x0F) == 15:
+ while True:
+ if i >= n:
+ raise V2Error("truncated LZ4 block: match-length extension")
+ b = block[i]
+ i += 1
+ match_len += b
+ if b != 255:
+ break
+ if len(out) + match_len > original_size:
+ raise V2Error("LZ4 output exceeds original_size")
+ for _ in range(match_len): # byte-wise: overlapping matches are legal
+ out.append(out[-offset])
+ if len(out) != original_size:
+ raise V2Error(f"LZ4 output length {len(out)} != original_size {original_size}")
+ return bytes(out)
+
+
+# ---------------------------------------------------------------------------
+# The v2 value container
+# ---------------------------------------------------------------------------
+
+def encode_container(method: int, original_size: int, payload: bytes) -> bytes:
+ """Canonical container bytes: 0xC1 0x02 + canonical msgpack [method, size, bin]."""
+ body = v1.encode_canonical([method, original_size, payload], collapse_floats=False)
+ return bytes([MAGIC, CONTAINER_VERSION]) + body
+
+
+class _Reader:
+ """Minimal strict MessagePack reader for the container body only.
+
+ Accepts any well-formed header width (per spec) but only the types the
+ body grammar allows: array, non-negative int, bin. Validates every
+ declared length against the remaining input before consuming it.
+ """
+
+ def __init__(self, buf: bytes) -> None:
+ self.buf = buf
+ self.pos = 0
+
+ def _take(self, n: int) -> bytes:
+ if self.pos + n > len(self.buf):
+ raise V2Error("container body truncated")
+ chunk = self.buf[self.pos : self.pos + n]
+ self.pos += n
+ return chunk
+
+ def read_array_header(self) -> int:
+ marker = self._take(1)[0]
+ if 0x90 <= marker <= 0x9F:
+ return marker & 0x0F
+ if marker == 0xDC:
+ return int.from_bytes(self._take(2), "big")
+ if marker == 0xDD:
+ return int.from_bytes(self._take(4), "big")
+ raise _bad_marker("container body must be a msgpack array", marker)
+
+ def read_uint(self) -> int:
+ # Unsigned-family markers ONLY (spec: marker-level enforcement). The
+ # signed family (negative fixint, 0xd0-0xd3) is rejected even when the
+ # carried value is non-negative — which makes a negative original_size
+ # structurally unrepresentable.
+ marker = self._take(1)[0]
+ if marker <= 0x7F:
+ return marker
+ if marker == 0xCC:
+ return self._take(1)[0]
+ if marker == 0xCD:
+ return int.from_bytes(self._take(2), "big")
+ if marker == 0xCE:
+ return int.from_bytes(self._take(4), "big")
+ if marker == 0xCF:
+ return int.from_bytes(self._take(8), "big")
+ raise _bad_marker("expected an unsigned-family msgpack int marker", marker)
+
+ def read_bin(self) -> bytes:
+ marker = self._take(1)[0]
+ if marker == 0xC4:
+ n = self._take(1)[0]
+ elif marker == 0xC5:
+ n = int.from_bytes(self._take(2), "big")
+ elif marker == 0xC6:
+ n = int.from_bytes(self._take(4), "big")
+ else:
+ # The explicit non-inheritance of the array-of-ints leniency (and
+ # rejection of str-family payloads) lands here.
+ raise _bad_marker("payload must be msgpack bin (0xc4/0xc5/0xc6)", marker)
+ if self.pos + n > len(self.buf): # header-vs-remaining-input rule
+ raise V2Error("bin length header exceeds remaining input")
+ return self._take(n)
+
+ def expect_exhausted(self) -> None:
+ if self.pos != len(self.buf):
+ raise V2Error(f"{len(self.buf) - self.pos} trailing byte(s) after container body")
+
+
+def decode_container(data: bytes) -> bytes:
+ """Normative reader algorithm steps 2-5: container bytes -> plain value bytes."""
+ if len(data) < 2:
+ msg = "truncated container (magic + version bytes required)"
+ raise V2Error(msg)
+ if data[0] != MAGIC:
+ raise V2Error("bad container magic (0xC1 expected) — possible interop/v1 value or mode misconfiguration")
+ if data[1] != CONTAINER_VERSION:
+ raise V2Error(f"unsupported container version 0x{data[1]:02x}")
+ r = _Reader(data[2:])
+ if r.read_array_header() != 3:
+ raise V2Error("container body must be a 3-element array")
+ method = r.read_uint()
+ original_size = r.read_uint()
+ payload = r.read_bin()
+ r.expect_exhausted()
+
+ if method not in (METHOD_NONE, METHOD_LZ4_BLOCK):
+ raise V2Error(f"unknown compression method {method}")
+ # Security Limits — all BEFORE any decompression, integer arithmetic only.
+ if original_size > MAX_UNCOMPRESSED:
+ raise V2Error(f"original_size {original_size} exceeds max uncompressed size")
+ if len(payload) > MAX_COMPRESSED:
+ raise V2Error("payload exceeds max compressed size")
+ if method == METHOD_LZ4_BLOCK:
+ if len(payload) == 0:
+ raise V2Error("zero-length compressed payload")
+ if original_size > MAX_RATIO * len(payload):
+ raise V2Error("compression ratio exceeds 1000:1 — decompression bomb")
+ return lz4_block_decompress(payload, original_size)
+ if original_size != len(payload):
+ raise V2Error(f"method 0 original_size {original_size} != payload length {len(payload)}")
+ return payload
+
+
+# ---------------------------------------------------------------------------
+# Encryption vector constants. Master key + tenant match interop/v1 and
+# test-vectors/encryption.json, so the derived key (fingerprint 96179a9b...)
+# is the published ground truth. The ciphertext was produced with AES-256-GCM
+# (cryptography/OpenSSL) over the lz4_roundtrip_compressible CONTAINER bytes
+# with the v2 AAD (compressed="True") and the fixed nonce below; it is
+# re-verified by the optional `cryptography` check here and ALWAYS by
+# WebCrypto in interop-v2-crosscheck.mjs.
+# ---------------------------------------------------------------------------
+
+ENC_NONCE_HEX = "101112131415161718191a1b"
+ENC_CIPHERTEXT_HEX = (
+ "101112131415161718191a1b"
+ "043e651799eeb533c4ddf646124b57eabe59f9292ed9f2cc1412ce52f677fccc93f82f01894f61"
+ "9b13576eb790ea10a714a3afd85019a283a5240a602a170c9485575fe350ba5c414d1374f33216"
+ "bd6b38eaf1f55503767f71229480e5"
+)
+
+
+# ---------------------------------------------------------------------------
+# Vectors
+# ---------------------------------------------------------------------------
+
+# A value with real redundancy so method 1 actually compresses (and the
+# reference compressor emits real match sequences, not a literals-only block).
+COMPRESSIBLE_VALUE = {
+ "events": ["GET /api/users/42 200 OK"] * 8,
+ "source": "interop-v2-reference",
+}
+
+CONTAINER_VECTOR_DEFS: list[dict] = [
+ {
+ "name": "method0_issue_example",
+ "description": (
+ "method 0 (no compression) wrap of the v1 issue_example_object value — "
+ "original_size MUST equal the payload length"
+ ),
+ "value": {"name": "alice", "age": 30},
+ "method": METHOD_NONE,
+ },
+ {
+ "name": "lz4_roundtrip_compressible",
+ "description": (
+ "method 1 compressed round-trip of a redundant value. The payload is the "
+ "REFERENCE compressor's output: readers MUST decompress it to the pinned "
+ "value bytes; writers are NOT required to reproduce these compressed bytes "
+ "(compressed bytes are non-canonical, read-side conformance only)"
+ ),
+ "value": COMPRESSIBLE_VALUE,
+ "method": METHOD_LZ4_BLOCK,
+ },
+ {
+ "name": "lz4_wraps_v1_value_vector",
+ "description": (
+ "method 1 container over the SAME plain bytes as v1's issue_example_object "
+ "value vector — the inner value profile is inherited from v1 unchanged. "
+ "Incompressible at this size: the LZ4 payload is a literals-only block "
+ "LARGER than the value (writers SHOULD have used method 0; readers MUST "
+ "still accept it)"
+ ),
+ "value": {"name": "alice", "age": 30},
+ "method": METHOD_LZ4_BLOCK,
+ },
+]
+
+
+def _hex_container(method: int, original_size: int, payload: bytes) -> str:
+ return encode_container(method, original_size, payload).hex()
+
+
+def _build_reject_vectors(containers: dict[str, dict]) -> list[dict]:
+ """Structural must-reject cases; every container_hex MUST raise in decode."""
+ method0 = containers["method0_issue_example"]
+ value_bytes = bytes.fromhex(method0["value_msgpack_hex"])
+ lz4_payload = bytes.fromhex(containers["lz4_roundtrip_compressible"]["payload_hex"])
+ lz4_size = containers["lz4_roundtrip_compressible"]["original_size"]
+
+ return [
+ {
+ "name": "reject_bad_magic_v1_value",
+ "description": "A bare interop/v1 value fed to a v2 reader: first byte is a msgpack marker, not 0xC1",
+ "container_hex": value_bytes.hex(),
+ "error": "bad magic; reader SHOULD diagnose 'possible interop/v1 value'",
+ },
+ {
+ "name": "reject_bad_container_version",
+ "description": "Right magic, wrong version byte (0x03)",
+ "container_hex": (bytes([MAGIC, 0x03]) + v1.encode_canonical([0, len(value_bytes), value_bytes], collapse_floats=False)).hex(),
+ "error": "unsupported container version",
+ },
+ {
+ "name": "reject_unknown_method",
+ "description": "method 2 is not in the registry",
+ "container_hex": _hex_container(2, len(value_bytes), value_bytes),
+ "error": "unknown compression method",
+ },
+ {
+ "name": "reject_payload_array_of_ints",
+ "description": (
+ "Payload encoded as a msgpack array of integers instead of bin — the legacy "
+ "ByteStorage leniency is explicitly NOT inherited by interop/v2"
+ ),
+ "container_hex": (bytes([MAGIC, CONTAINER_VERSION]) + v1.encode_canonical([0, 3, [1, 2, 3]], collapse_floats=False)).hex(),
+ "error": "payload must be msgpack bin",
+ },
+ {
+ "name": "reject_method_signed_marker",
+ "description": (
+ "method encoded with a signed-family marker (int8 0xd0 carrying value 0) — "
+ "marker-level enforcement rejects the signed family even for non-negative values"
+ ),
+ "container_hex": (bytes([MAGIC, CONTAINER_VERSION, 0x93, 0xD0, 0x00, len(value_bytes), 0xC4, len(value_bytes)]) + value_bytes).hex(),
+ "error": "signed-family int marker for method",
+ },
+ {
+ "name": "reject_negative_original_size",
+ "description": (
+ "original_size encoded as negative fixint -1 (0xff) — negative sizes are "
+ "structurally unrepresentable once signed-family markers are rejected; a "
+ "value-level reader that accepts -1 here bypasses every upper-bound check"
+ ),
+ "container_hex": (bytes([MAGIC, CONTAINER_VERSION, 0x93, 0x01, 0xFF, 0xC4, 0x04]) + bytes.fromhex("10410000")).hex(),
+ "error": "signed-family int marker for original_size",
+ },
+ {
+ "name": "reject_forged_bin32_length",
+ "description": (
+ "bin32 payload header declaring 4 GiB (0xffffffff) with no data following — "
+ "readers MUST validate the length header against remaining input BEFORE allocating"
+ ),
+ "container_hex": bytes([MAGIC, CONTAINER_VERSION, 0x93, 0x00, 0x05, 0xC6, 0xFF, 0xFF, 0xFF, 0xFF]).hex(),
+ "error": "bin length header exceeds remaining input",
+ },
+ {
+ "name": "reject_payload_str",
+ "description": "Payload encoded as the msgpack str family instead of bin",
+ "container_hex": (bytes([MAGIC, CONTAINER_VERSION]) + v1.encode_canonical([0, 3, "abc"], collapse_floats=False)).hex(),
+ "error": "payload must be msgpack bin",
+ },
+ {
+ "name": "reject_method0_size_mismatch",
+ "description": "method 0 with original_size != payload length",
+ "container_hex": _hex_container(METHOD_NONE, len(value_bytes) + 1, value_bytes),
+ "error": "method 0 original_size mismatch",
+ },
+ {
+ "name": "reject_trailing_bytes",
+ "description": "Valid container followed by one extra byte",
+ "container_hex": _hex_container(METHOD_NONE, len(value_bytes), value_bytes) + "00",
+ "error": "trailing bytes after container body",
+ },
+ {
+ "name": "reject_declared_size_bomb",
+ "description": "original_size declares 1 TiB — exceeds the 512 MiB cap (checked BEFORE decompression)",
+ "container_hex": _hex_container(METHOD_LZ4_BLOCK, 1 << 40, bytes.fromhex("10410000")),
+ "error": "original_size exceeds max uncompressed size",
+ },
+ {
+ "name": "reject_ratio_bomb",
+ "description": "10-byte payload declaring 10001 output bytes — exceeds the 1000:1 ratio (checked BEFORE decompression)",
+ "container_hex": _hex_container(METHOD_LZ4_BLOCK, 10_001, bytes(10)),
+ "error": "compression ratio exceeds 1000:1",
+ },
+ {
+ "name": "reject_zero_length_compressed",
+ "description": "method 1 with an empty payload",
+ "container_hex": _hex_container(METHOD_LZ4_BLOCK, 1, b""),
+ "error": "zero-length compressed payload",
+ },
+ {
+ "name": "reject_lz4_zero_offset",
+ "description": "LZ4 sequence with match offset 0 (invalid in the block format)",
+ "container_hex": _hex_container(METHOD_LZ4_BLOCK, 5, bytes.fromhex("10410000")),
+ "error": "invalid LZ4 match offset 0",
+ },
+ {
+ "name": "reject_lz4_truncated",
+ "description": "The lz4_roundtrip_compressible payload with its last 3 bytes removed",
+ "container_hex": _hex_container(METHOD_LZ4_BLOCK, lz4_size, lz4_payload[:-3]),
+ "error": "truncated LZ4 block",
+ },
+ {
+ "name": "reject_lz4_length_mismatch",
+ "description": "Valid LZ4 block whose output is one byte short of original_size",
+ "container_hex": _hex_container(METHOD_LZ4_BLOCK, lz4_size + 1, lz4_payload),
+ "error": "LZ4 output length != original_size (also fine to fail as overrun, depending on decoder structure)",
+ },
+ ]
+
+
+def _build() -> dict:
+ tenant = v1.ENC_TENANT_ID
+ master_key_hex = v1.ENC_MASTER_KEY_HEX
+
+ # Reuse the v1 single_int key so the v1/v2 AAD pair is side-by-side comparable.
+ cache_key = v1.interop_key("users", "get_user", [42])
+ aad_v2 = v1.aad_v3(tenant, cache_key, compressed=True)
+ aad_v1 = v1.aad_v3(tenant, cache_key, compressed=False)
+
+ container_vectors = []
+ by_name: dict[str, dict] = {}
+ for d in CONTAINER_VECTOR_DEFS:
+ value = v1.from_tagged(d["value"])
+ value_bytes = v1.encode_canonical(value, collapse_floats=False)
+ payload = value_bytes if d["method"] == METHOD_NONE else lz4_block_compress(value_bytes)
+ entry = {
+ "name": d["name"],
+ "description": d["description"],
+ "value": d["value"],
+ "value_msgpack_hex": value_bytes.hex(),
+ "method": d["method"],
+ "original_size": len(value_bytes),
+ "payload_hex": payload.hex(),
+ "container_hex": _hex_container(d["method"], len(value_bytes), payload).lower(),
+ }
+ container_vectors.append(entry)
+ by_name[d["name"]] = entry
+
+ # Hand-built NON-canonical container: array16 header, uint8/uint32 ints,
+ # bin16 payload. Pins the reader MUST for non-canonical unsigned-family
+ # widths (writers MUST NOT emit this; readers MUST accept it).
+ nc_value = by_name["method0_issue_example"]
+ nc_bytes = bytes.fromhex(nc_value["value_msgpack_hex"])
+ nc_container = (
+ bytes([MAGIC, CONTAINER_VERSION])
+ + b"\xdc\x00\x03" # array16(3)
+ + b"\xcc\x00" # method 0 as uint8
+ + b"\xce" + len(nc_bytes).to_bytes(4, "big") # original_size as uint32
+ + b"\xc5" + len(nc_bytes).to_bytes(2, "big") # payload as bin16
+ + nc_bytes
+ )
+ nc_entry = {
+ "name": "method0_noncanonical_widths",
+ "description": (
+ "Same value as method0_issue_example, but with deliberately NON-canonical "
+ "header widths (array16, uint8 method, uint32 original_size, bin16 payload). "
+ "Readers MUST accept any unsigned-family width; writers MUST NOT emit this."
+ ),
+ "value": nc_value["value"],
+ "value_msgpack_hex": nc_value["value_msgpack_hex"],
+ "method": METHOD_NONE,
+ "original_size": len(nc_bytes),
+ "payload_hex": nc_bytes.hex(),
+ "container_hex": nc_container.hex(),
+ }
+ container_vectors.append(nc_entry)
+ by_name[nc_entry["name"]] = nc_entry
+
+ enc_container_hex = by_name["lz4_roundtrip_compressible"]["container_hex"]
+
+ return {
+ "version": "1.0.0",
+ "spec": "spec/interop-v2.md",
+ "generator": "tools/interop-v2-reference.py (CPython stdlib, incl. pure-Python LZ4 block codec)",
+ "cross_checked_by": "tools/interop-v2-crosscheck.mjs (independent container parser + LZ4 block decoder + WebCrypto HKDF/AES-GCM; zero dependencies)",
+ "container_format": "0xC1 0x02 + canonical msgpack [method:int, original_size:int, payload:bin]",
+ "security_limits": {
+ "max_uncompressed_size": MAX_UNCOMPRESSED,
+ "max_compressed_size": MAX_COMPRESSED,
+ "max_compression_ratio": MAX_RATIO,
+ "note": "All enforced BEFORE decompression, integer arithmetic only — spec/interop-v2.md#security-limits-decompression-bounds",
+ },
+ "compressed_bytes_note": (
+ "method-1 payload bytes are NOT canonical: conformant LZ4 encoders legally differ. "
+ "Vectors pin the reference compressor's output for READ-side conformance; writers "
+ "may produce different valid LZ4 for the same value."
+ ),
+ "error_vectors_note": (
+ "reject_* vectors MUST be rejected with an error; the 'error' text is a maintainer "
+ "note, not a normative message."
+ ),
+ "container_vectors": container_vectors,
+ "aad_vectors": [
+ {
+ "name": "interop_v2_aad",
+ "description": (
+ "AAD v0x03 over the same tenant + interop key as v1's interop_key_aad; the two "
+ "AADs differ ONLY in the final component (frozen tokens 'True' vs 'False') — "
+ "this is the cryptographic mode separation"
+ ),
+ "tenant_id": tenant,
+ "cache_key": cache_key,
+ "format": "msgpack",
+ "compressed": True,
+ "aad_hex": aad_v2.hex(),
+ "v1_aad_hex_for_comparison": aad_v1.hex(),
+ }
+ ],
+ "encryption_vectors": [
+ {
+ "name": "interop_v2_compressed_encryption_roundtrip",
+ "description": (
+ "Full v2 round-trip: HKDF-SHA256 per spec/encryption.md (same master key + "
+ "tenant as interop/v1 and encryption.json, hence the same derived key), "
+ "AES-256-GCM over the ENTIRE v2 container bytes (lz4_roundtrip_compressible) "
+ "with the v2 AAD (compressed='True') and a fixed nonce."
+ ),
+ "master_key_hex": master_key_hex,
+ "tenant_id": tenant,
+ "derived_key_fingerprint_hex": v1.ENC_KEY_FINGERPRINT_HEX,
+ "cache_key": cache_key,
+ "format": "msgpack",
+ "compressed": True,
+ "aad_hex": aad_v2.hex(),
+ "plaintext_hex": enc_container_hex,
+ "nonce_hex": ENC_NONCE_HEX,
+ "ciphertext_hex": ENC_CIPHERTEXT_HEX,
+ "ciphertext_layout": "nonce(12) || ciphertext || auth_tag(16)",
+ }
+ ],
+ "reject_vectors": _build_reject_vectors(by_name),
+ "crypto_reject_vectors": [
+ {
+ "name": "reject_v2_ciphertext_with_v1_aad",
+ "description": "The v2 ciphertext MUST fail AES-GCM authentication under the v1 AAD (compressed='False')",
+ "master_key_hex": master_key_hex,
+ "tenant_id": tenant,
+ "ciphertext_hex": ENC_CIPHERTEXT_HEX,
+ "aad_hex": aad_v1.hex(),
+ "error": "authentication failure — cross-mode read, terminal per the no-retry rule",
+ },
+ {
+ "name": "reject_v1_ciphertext_with_v2_aad",
+ "description": "interop/v1's published interop_encryption_roundtrip ciphertext MUST fail under the v2 AAD (compressed='True')",
+ "master_key_hex": master_key_hex,
+ "tenant_id": tenant,
+ "ciphertext_hex": v1.ENC_CIPHERTEXT_HEX,
+ "aad_hex": aad_v2.hex(),
+ "error": "authentication failure — cross-mode read, terminal per the no-retry rule",
+ },
+ ],
+ }
+
+
+# ---------------------------------------------------------------------------
+# Self-checks
+# ---------------------------------------------------------------------------
+
+def _deterministic_junk(n: int) -> bytes:
+ """Deterministic pseudo-random bytes (no RNG state, no seed drift)."""
+ import hashlib
+
+ out = bytearray()
+ counter = 0
+ while len(out) < n:
+ out += hashlib.sha256(counter.to_bytes(8, "big")).digest()
+ counter += 1
+ return bytes(out[:n])
+
+
+def _expect_structural_reject(rv: dict) -> None:
+ try:
+ decode_container(bytes.fromhex(rv["container_hex"]))
+ except V2Error:
+ return
+ msg = f"reject vector {rv['name']} did not raise"
+ raise SelfCheckError(msg)
+
+
+def _self_check(built: dict) -> None:
+ # LZ4 codec round-trips: repetitive, incompressible, and boundary sizes.
+ samples = [
+ b"a",
+ b"abcd" * 4,
+ bytes(range(13)),
+ b"the quick brown fox jumps over the lazy dog. " * 40,
+ _deterministic_junk(1),
+ _deterministic_junk(12),
+ _deterministic_junk(13),
+ _deterministic_junk(64 * 1024 + 17),
+ b"\x00" * 100_000,
+ ]
+ for s in samples:
+ _require(lz4_block_decompress(lz4_block_compress(s), len(s)) == s, f"LZ4 roundtrip failed for {len(s)}-byte sample")
+
+ # Container round-trips + pinned bytes.
+ for cv in built["container_vectors"]:
+ got = decode_container(bytes.fromhex(cv["container_hex"]))
+ _require(got.hex() == cv["value_msgpack_hex"], f"container {cv['name']} does not decode to its value bytes")
+
+ by_name = {c["name"]: c for c in built["container_vectors"]}
+ # The inherited-value-profile claim: identical inner bytes across the two wraps,
+ # AND byte-identical to the PUBLISHED v1 value vector in interop-mode.json —
+ # cross-file, so drift in either file breaks generation/verify loudly.
+ _require(
+ by_name["method0_issue_example"]["value_msgpack_hex"] == by_name["lz4_wraps_v1_value_vector"]["value_msgpack_hex"],
+ "the two method0/lz4 wraps of the v1 value no longer share inner bytes",
+ )
+ v1_path = _HERE.parent / "test-vectors" / "interop-mode.json"
+ try:
+ v1_doc = json.loads(v1_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as e:
+ msg = f"cannot read the published v1 vectors at {v1_path}: {e}"
+ raise SelfCheckError(msg) from e
+ v1_example = next(v for v in v1_doc["value_vectors"] if v["name"] == "issue_example_object")
+ _require(
+ by_name["method0_issue_example"]["value_msgpack_hex"] == v1_example["canonical_msgpack_hex"],
+ "inner value bytes no longer match the published v1 issue_example_object vector",
+ )
+ v1_enc = v1_doc["encryption_vectors"][0]
+ _require(
+ built["crypto_reject_vectors"][1]["ciphertext_hex"] == v1_enc["ciphertext_hex"],
+ "reject_v1_ciphertext_with_v2_aad no longer pins the published v1 ciphertext",
+ )
+ # The compressible vector must actually compress (real match sequences).
+ lz4_cv = by_name["lz4_roundtrip_compressible"]
+ _require(
+ len(bytes.fromhex(lz4_cv["payload_hex"])) < lz4_cv["original_size"],
+ "the 'compressible' vector did not compress — vector loses its point",
+ )
+
+ # Every structural reject vector must raise.
+ for rv in built["reject_vectors"]:
+ _expect_structural_reject(rv)
+
+ # AAD pair: v2 differs from v1 exactly in the final component.
+ aad = built["aad_vectors"][0]
+ v2b, v1b = bytes.fromhex(aad["aad_hex"]), bytes.fromhex(aad["v1_aad_hex_for_comparison"])
+ _require(v2b[: -len(b"\x00\x00\x00\x04True")] == v1b[: -len(b"\x00\x00\x00\x05False")], "AAD prefixes diverge")
+ _require(v2b.endswith(b"\x00\x00\x00\x04True") and v1b.endswith(b"\x00\x00\x00\x05False"), "frozen token suffixes wrong")
+
+ # HKDF ground-truth continuity (same chain as v1 / encryption.json).
+ key = v1.derive_encryption_key(bytes.fromhex(v1.ENC_MASTER_KEY_HEX), v1.ENC_TENANT_ID)
+ _require(v1.key_fingerprint(key) == v1.ENC_KEY_FINGERPRINT_HEX, "derived-key fingerprint diverges from the published chain")
+
+ # Optional: bidirectional conformance with the de-facto C implementation.
+ try:
+ import lz4.block # noqa: PLC0415
+ except ImportError:
+ logging.warning("note: `lz4` not installed — C-implementation conformance check skipped")
+ else:
+ for s in samples:
+ ours = lz4_block_compress(s)
+ _require(lz4.block.decompress(ours, uncompressed_size=len(s)) == s, "lz4.block rejects our compressor output")
+ theirs = lz4.block.compress(s, store_size=False)
+ _require(lz4_block_decompress(theirs, len(s)) == s, "our decoder rejects lz4.block output")
+ pinned = bytes.fromhex(lz4_cv["payload_hex"])
+ _require(
+ lz4.block.decompress(pinned, uncompressed_size=lz4_cv["original_size"]).hex() == lz4_cv["value_msgpack_hex"],
+ "lz4.block does not decompress the pinned payload to the pinned value bytes",
+ )
+
+ # Optional: AES-GCM seal + both cross-mode AAD rejections.
+ try:
+ from cryptography.hazmat.primitives.ciphers.aead import AESGCM # noqa: PLC0415
+ from cryptography.exceptions import InvalidTag # noqa: PLC0415
+ except ImportError:
+ logging.warning("note: `cryptography` not installed — AES-GCM checks run only in interop-v2-crosscheck.mjs (WebCrypto)")
+ else:
+ ev = built["encryption_vectors"][0]
+ ct = bytes.fromhex(ev["ciphertext_hex"])
+ pt = AESGCM(key).decrypt(ct[:12], ct[12:], bytes.fromhex(ev["aad_hex"]))
+ _require(pt.hex() == ev["plaintext_hex"], "v2 encryption vector does not decrypt to the pinned container")
+ resealed = ct[:12] + AESGCM(key).encrypt(ct[:12], pt, bytes.fromhex(ev["aad_hex"]))
+ _require(resealed.hex() == ev["ciphertext_hex"], "v2 encryption vector seal mismatch")
+ for rv in built["crypto_reject_vectors"]:
+ rct = bytes.fromhex(rv["ciphertext_hex"])
+ try:
+ AESGCM(key).decrypt(rct[:12], rct[12:], bytes.fromhex(rv["aad_hex"]))
+ except InvalidTag:
+ pass
+ else:
+ msg = f"{rv['name']}: cross-mode decrypt unexpectedly succeeded"
+ raise SelfCheckError(msg)
+
+
+def main() -> int:
+ vectors_path = _HERE.parent / "test-vectors" / "interop-v2.json"
+ cmd = sys.argv[1] if len(sys.argv) > 1 else "verify"
+ if cmd not in ("generate", "verify"):
+ logging.error("unknown command %r — use 'generate' or 'verify'", cmd)
+ return 2
+ if cmd == "generate":
+ # Generation (unlike verify) hard-requires `cryptography`: the pinned
+ # ciphertext constant is coupled to the compressor's exact container
+ # bytes, and without an AES-GCM reseal check a compressor change would
+ # silently write vectors whose ciphertext no longer decrypts.
+ try:
+ import cryptography # noqa: F401, PLC0415
+ except ImportError:
+ logging.error("generate requires the `cryptography` package (verify stays stdlib-only): pip install cryptography")
+ return 2
+ built = _build()
+ _self_check(built)
+
+ counts = (
+ f"{len(built['container_vectors'])} container, {len(built['aad_vectors'])} AAD, "
+ f"{len(built['encryption_vectors'])} encryption, {len(built['reject_vectors'])} reject, "
+ f"{len(built['crypto_reject_vectors'])} crypto-reject vectors"
+ )
+ if cmd == "generate":
+ vectors_path.write_text(json.dumps(built, indent=2, ensure_ascii=True) + "\n", encoding="utf-8")
+ logging.info("wrote %s (%s)", vectors_path, counts)
+ return 0
+
+ on_disk = json.loads(vectors_path.read_text(encoding="utf-8"))
+ if on_disk != built:
+ logging.error("MISMATCH: test-vectors/interop-v2.json does not match the reference implementation")
+ return 1
+ logging.info("OK: %s all verified", counts)
+ return 0
+
+
+if __name__ == "__main__":
+ # stdout, matching the pre-logging behaviour and the twin JS cross-check.
+ logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout)
+ sys.exit(main())