Skip to content

docs(spec): one >=64-bit ratio-product rule, guarded in both specs (LAB-2594) #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
27Bslash6 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

docs(spec): one >=64-bit ratio-product rule, guarded in both specs (LAB-2594) #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c9bb9be
docs(spec): one >=64-bit ratio-product rule, guarded in both specs (L…
27Bslash6 Sep 2, 2026
cd840aa
fix: address coderabbit review — correct the wrapping-impact figure
27Bslash6 Sep 2, 2026
aca7052
fix: apply expert-panel findings on the wrapping-impact rationale
27Bslash6 Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/verify.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,15 @@ jobs:
- name: Python-frame JS cross-check (zero-dep independent reader, full round-trip)
run: node tools/frame-crosscheck.mjs

# The >=64-bit ratio-product rule is stated in full in BOTH spec documents
# (an implementer reads one standalone), so the copies are guarded rather
# than trusted -- divergent normative text for this exact bound is the bug
# LAB-2594 closed. Mutation suite first, same doctrine as below.
- name: Ratio-product rule identical in both spec documents
run: |
python3 tools/test_check_spec_duplication.py
python3 tools/check-spec-duplication.py

# Narrow on purpose: catches one facet of ONE of the six matrix incidents
# listed in decisions/matrix-version-verification.md — a version in the SDK
# Overview table stated as a snapshot, true when written and false at the
Expand Down
70 changes: 70 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,76 @@ All notable changes to the CacheKit Protocol Specification.

## [Unreleased]

### Decompression bomb bound — one ≥ 64-bit rule, guarded in both specs (LAB-2594)

- [`spec/wire-format.md`](spec/wire-format.md) § Security Limits now carries the
same normative obligation as
[`spec/interop-v2.md`](spec/interop-v2.md) § Security Limits — *"The ratio
product MUST be computed in **at least 64-bit unsigned integers**"* — with an
identical rationale block (identical text, modulo each document's operand name,
`compressed_size` vs `payload.length`). The bound was previously specified
twice with divergent normative text: interop v2 bound the integer width
(LAB-1135, [#53](https://github.com/cachekit-io/protocol/pull/53)), wire format
said nothing, so an implementer reading only wire format could legally compute
the product in 32-bit pointer-width arithmetic.
- The rule now names the **operation**, not a property of the arithmetic: widen
the operand to ≥ 64-bit unsigned *before* multiplying. That is what defeats the
actual defect — `1000 * payload.len()` on `usize` is 64-bit on the author's
host and in 64-bit CI, and wraps only on the shipped wasm32 target, so a rule
phrased as "compute in ≥ 64 bits" never fires in the author's self-assessment.
Rust `u64`, Python `int` and JavaScript `Number` (exact below 2⁵³; the product
is < 2³⁹, so no `BigInt`) all satisfy it, so no fallback is offered.
- The `if max_allowed overflows: REJECT` pseudocode is **removed**, not reworded.
It is not an observable event as written (Rust release-mode arithmetic wraps
silently, JavaScript `Number` loses precision above 2⁵³ rather than
overflowing, Python integers are arbitrary-precision — the repo's own
`tools/interop-v2-reference.py` can never take that branch), and once the
operand is widened the branch is unreachable anyway: the two 512 MiB caps bound
the product below 2³⁹. Rejecting on overflow is explicitly **not** an accepted
substitute for widening — at 32-bit width it would refuse 99.2 % of the legal
compressed-size range. Both documents now also state the caps as a normative
precondition of the product bound, and wire format's pseudocode enforces them
inline instead of leaving them to a collapsed flow section.
- The bound MUST now be computed by **multiplication**; deriving it by division
is forbidden in any arithmetic. Narrowing the old blanket "no floating point"
rule to "no floating-point *ratio*" (needed to permit JavaScript `Number`)
would otherwise have sanctioned `original_size / compressed_size > 1000`, whose
truncation accepts up to `1000·compressed_size + (compressed_size − 1)` — a
looser bound than this spec permits.
- **`spec/interop-v2.md`'s rationale corrected**: 32-bit wrapping was said to
"silently corrupt the bound **in both directions**". It cannot. Wrapping begins
at `⌈2³²/1000⌉ = 4,294,968` B (~4.29 MB) and can only *tighten* the bound, so
the failure mode is **spurious rejection**, never a bomb bypass — a hard
error rather than a cache miss, and permanent for an affected entry. The
rejection is not uniform (a density of `2²⁹/2³² = 1/8` over the legal range),
but where it bites the collapse is near-total: 704 B at the threshold, 8 B at
its floor, 0 at the 512 MiB cap. The 99.2 % figure four bullets above measures
*reject-on-overflow*, a different mechanism, and is unaffected. Both documents
now say so, marked non-normative, and interop v2 records the corrected claim
in place.
- **New CI guard**: [`tools/check-spec-duplication.py`](tools/check-spec-duplication.py)
compares the two copies of the rule (delimited by sentinel comments) modulo the
operand rename, with a 9-case mutation suite
([`tools/test_check_spec_duplication.py`](tools/test_check_spec_duplication.py))
that re-arms the LAB-2594 divergence and asserts the guard fails. Stating the
rule twice is deliberate — an implementer reads one document standalone — so
the duplication is guarded rather than trusted.
- **No implementation change.** `cachekit-core/src/byte_storage.rs` already
computes `MAX_COMPRESSION_RATIO: u64 * (len as u64)` via
`checked_mul(…).ok_or(DecompressionBomb)?`, correct and fail-closed on every
target including wasm32. cachekit-ts was verified to have no JS-side ratio
check and to inherit that arithmetic through the Rust core on the envelope
decode path; a JS-only decode path, if one is ever added, would be unbounded
and is out of scope here. This was a spec trust bug — the documented rule was
weaker and vaguer than the shipped one. No limit values changed and no fixture
bytes were touched (no SDK re-vendors).
- **Known gap, not closed here**: no published conformance vector can distinguish
a 64-bit decoder from a 32-bit wrapping one. The largest `original_size` in
`test-vectors/interop-v2.json` is 237 B while divergence begins at a 4,294,968 B
compressed payload, so a wrapping decoder passes 100 % of the suite. Closing it
needs a limits-only vector shape and a fixture version bump (forcing py/ts
re-vendors), which is out of scope for a spec-text change — filed separately.

### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751)

- LZ4 compressed bytes are **not canonical** across conforming block encoders.
Expand Down
64 changes: 55 additions & 9 deletions spec/interop-v2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,8 @@ bomb — that bare-MessagePack v1 does not have. These bounds reuse the
ByteStorage constants from
[wire-format.md → Security Limits](wire-format.md#security-limits) so the fleet
carries **one** set of numbers, and all of them MUST be enforced **before**
decompressing (integer arithmetic only — no floating point):
decompressing (integer arithmetic only — no floating-point *ratio*; the ratio
product's integer-width requirement is stated below):

| Limit | Value | Applies to |
| :--- | ---: | :--- |
Expand All @@ -247,18 +248,63 @@ reject if original_size > MAX_UNCOMPRESSED // 512 MiB
reject if payload.length > MAX_COMPRESSED // 512 MiB
if method == 1:
reject if payload.length == 0 // zero-length compressed = bomb
max_allowed = 1000 * payload.length // MUST be computed in >= 64-bit integers
max_allowed = 1000 * uint64(payload.length) // widen BEFORE multiplying; see below
reject if original_size > max_allowed
if method == 0:
reject if original_size != payload.length
```

The ratio product MUST be computed in **at least 64-bit unsigned integers**.
After the two 512 MiB caps pass, both operands are < 2³⁰ and the product is
< 2⁴⁰, so it can never overflow a u64 — but it *does* overflow 32-bit `usize`
arithmetic (a real target: cachekit-ts ships a wasm32 build), where release-mode
wrapping would silently corrupt the bound in both directions. Do not compute
this in pointer-width arithmetic.
<!-- BEGIN shared-block: ratio-product-rule (guarded by tools/check-spec-duplication.py) -->
The ratio product MUST be computed in **at least 64-bit unsigned integers**:
promote `payload.length` to a ≥ 64-bit unsigned (or arbitrary-precision) integer *before*
the multiply. Multiplying in pointer width and widening the result afterwards
does not satisfy this, and is invisible on a 64-bit host and in 64-bit CI — it
is the wasm32 defect described below. Every target language has a conforming
path: Rust `u64` (on every target, `wasm32` included), Python's
arbitrary-precision `int`, and JavaScript `Number` — an IEEE-754 double
represents every integer below 2⁵³ exactly and this product is < 2³⁹, so no
`BigInt` is required. Because `payload.length` ≤ 2²⁹ once the two 512 MiB caps have
passed, the product is < 2³⁹ and cannot overflow 64 bits; that is why the
pseudocode above carries no overflow branch, and why rejecting on overflow is
**not** a substitute for widening — at 32-bit width it would refuse 99.2 % of
the legal `payload.length` range (see the note below).

The bound MUST be computed by **multiplication**. Deriving it by division, or
as a *ratio*, is forbidden in any arithmetic — integer or floating-point.
Truncating integer division (`original_size / payload.length > 1000`) accepts up to
`1000·payload.length + (payload.length − 1)`, which is looser than this specification permits, and a
floating-point ratio is the precision bypass the integer rule exists to prevent.

> [!NOTE]
> **Non-normative rationale — the *ratio product's* failure direction under
> pointer-width arithmetic is fail-closed, never a bypass.** (This covers the
> product only. The width of `original_size` itself is a separate obligation
> not bound by this rule.) 32-bit pointer width is a live
> target: cachekit-ts ships a `wasm32` build. (That build is *not* affected — it
> computes this bound through `cachekit-core`'s `u64`.) Wrapping begins at
> `payload.length ≥ ⌈2³²/1000⌉ = 4,294,968` B (~4.29 MB), and it can only ever *tighten*
> the bound: for any product `p ≥ 2³²`, `wrapped(p) = p mod 2³² < 2³² ≤ p`,
> while `original_size` (≤ 512 MiB < 2³²) cannot itself wrap, so the direction
> of the comparison is preserved. The failure mode is therefore **spurious
> rejection**, not a bomb bypass — but a hard error rather than a cache miss,
> and a permanent one: the wrapped bound is a pure function of
> `payload.length`, so an affected entry fails identically on every read. It
> does not bite everywhere — only where the wrapped bound falls below the
> 512 MiB cap, a density of `2²⁹/2³² = 1/8` over the legal range — but where it
> does, the collapse is near-total: 704 B at that first threshold, 8 B at
> `payload.length = 115,964,117` (the wrapped bound only ever lands on
> multiples of `gcd(1000, 2³²) = 8`), and 0 at the 512 MiB cap. Those payloads
> are legal under this specification; an implementation that refuses them is
> non-conforming. So is one that rejects on overflow instead of widening: that
> refuses the entire 99.2 % of the legal range above the same threshold, and it
> is unnecessary besides — once the two size caps have passed, the product is
> < 2³⁹ and cannot overflow 64 bits at all.
<!-- END shared-block: ratio-product-rule -->

*An earlier revision of this section claimed 32-bit wrapping would corrupt the
bound "in both directions". It cannot: wrapping is fail-closed, as derived above.
Recorded because the mis-stated failure direction, not the bound, was the part an
implementer would have acted on — a believed bypass mis-prioritises the fix.*

After `method 1` decompression, the output length MUST equal `original_size`
exactly — shorter or longer output is a hard error (the
Expand Down Expand Up @@ -454,7 +500,7 @@ An SDK implementation of interop/v2 MUST:
| **Minimal 3-element container** | Reuse the ByteStorage envelope | The envelope drags xxHash3-64 into every SDK — a second native dependency per language (the PHP-fork class of cost) duplicating integrity the AES-GCM tag already provides when encrypted, and exceeding v1's posture when not. Its `format` field is also dead weight here (the content is always one plain-MessagePack document). What *is* kept from the envelope experience: `bin` payload encoding as normative from birth (protocol 1.1, [decisions/envelope-bin-encoding.md](../decisions/envelope-bin-encoding.md)). |
| **Array-of-ints payload rejected** | Inherit cachekit-core's permanent dual-read leniency | That leniency serves a deployed installed base and falls out of rmp-serde for free; interop/v2 has no installed base, and hand-written readers in new languages would pay extra code to be lenient. One legal encoding is the lowest implementation bar. Pinned by vector. |
| **Compressed bytes non-canonical, read-side conformance** | Pin one canonical LZ4 output | LZ4 encoders legally differ (implementation, level, version). Pinning writer bytes would freeze one library's output as protocol law and break on its next release. Keys stay byte-canonical; values never needed to be. |
| **Bounds reuse wire-format.md constants (512 MiB / 1000:1)** | Profile-specific numbers | One set of constants fleet-wide; the guards are already implemented, reviewed, and vector-tested in cachekit-core. Integer arithmetic rule carried over verbatim. |
| **Bounds reuse wire-format.md constants (512 MiB / 1000:1)** | Profile-specific numbers | One set of constants fleet-wide; the guards are already implemented, reviewed, and vector-tested in cachekit-core. The integer-width rule is stated in full in both documents — edit this section and [wire-format.md → Decompression Bomb Detection](wire-format.md#decompression-bomb-detection) together; `tools/check-spec-duplication.py` fails CI if they drift. |
| **No length padding** | Bucketed padding vs CRIME | Quantized leakage at real cost is not elimination; documented guidance plus the `method 0` / stay-v1 escape hatches are honest. Length-hiding from the backend is explicitly out of protocol scope (v1 leaks exact lengths today). |

---
Expand Down
67 changes: 61 additions & 6 deletions spec/wire-format.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,7 +372,7 @@ let checksum: [u8; 8] = xxh3_64(&original_data).to_be_bytes();
## Security Limits

> [!IMPORTANT]
> All three limits below MUST be enforced by every implementation of the ByteStorage envelope. The decompression bomb check uses integer arithmetic — do not substitute floating-point.
> All three limits below MUST be enforced by every implementation of the ByteStorage envelope. The decompression bomb check uses integer arithmetic — do not substitute a floating-point *ratio*, and see [Decompression Bomb Detection](#decompression-bomb-detection) for the normative integer-width requirement.
> Additionally, a decoder MUST validate any declared MessagePack `bin`/array
> length header against the remaining input bytes **before** allocating for it —
> a 5-byte `bin32` header can otherwise declare a 4 GiB allocation from a
Expand All @@ -387,20 +387,75 @@ let checksum: [u8; 8] = xxh3_64(&original_data).to_be_bytes();

### Decompression Bomb Detection

The ratio check uses **integer arithmetic** to prevent floating-point precision bypass:
All three limits above are enforced here, and the step order is normative —
the ratio check relies on both size caps having already passed. The check uses
**integer-valued arithmetic** and never a floating-point *ratio*:

```text
if original_size > MAX_UNCOMPRESSED:
REJECT // 512 MiB cap

if compressed_size > MAX_COMPRESSED:
REJECT // 512 MiB cap

```
if compressed_size == 0:
REJECT // Zero-length compressed with non-zero original = bomb

max_allowed = MAX_COMPRESSION_RATIO * compressed_size
if max_allowed overflows:
REJECT // Overflow = bomb
// MAX_COMPRESSION_RATIO = 1000. Widen compressed_size to >= 64-bit unsigned
// BEFORE multiplying (see below); the two caps above bound the product < 2^39.
max_allowed = MAX_COMPRESSION_RATIO * uint64(compressed_size)

if original_size > max_allowed:
REJECT // Ratio exceeded
```

<!-- BEGIN shared-block: ratio-product-rule (guarded by tools/check-spec-duplication.py) -->
The ratio product MUST be computed in **at least 64-bit unsigned integers**:
promote `compressed_size` to a ≥ 64-bit unsigned (or arbitrary-precision) integer *before*
the multiply. Multiplying in pointer width and widening the result afterwards
does not satisfy this, and is invisible on a 64-bit host and in 64-bit CI — it
is the wasm32 defect described below. Every target language has a conforming
path: Rust `u64` (on every target, `wasm32` included), Python's
arbitrary-precision `int`, and JavaScript `Number` — an IEEE-754 double
represents every integer below 2⁵³ exactly and this product is < 2³⁹, so no
`BigInt` is required. Because `compressed_size` ≤ 2²⁹ once the two 512 MiB caps have
passed, the product is < 2³⁹ and cannot overflow 64 bits; that is why the
pseudocode above carries no overflow branch, and why rejecting on overflow is
**not** a substitute for widening — at 32-bit width it would refuse 99.2 % of
the legal `compressed_size` range (see the note below).

The bound MUST be computed by **multiplication**. Deriving it by division, or
as a *ratio*, is forbidden in any arithmetic — integer or floating-point.
Truncating integer division (`original_size / compressed_size > 1000`) accepts up to
`1000·compressed_size + (compressed_size − 1)`, which is looser than this specification permits, and a
floating-point ratio is the precision bypass the integer rule exists to prevent.

> [!NOTE]
> **Non-normative rationale — the *ratio product's* failure direction under
> pointer-width arithmetic is fail-closed, never a bypass.** (This covers the
> product only. The width of `original_size` itself is a separate obligation
> not bound by this rule.) 32-bit pointer width is a live
> target: cachekit-ts ships a `wasm32` build. (That build is *not* affected — it
> computes this bound through `cachekit-core`'s `u64`.) Wrapping begins at
> `compressed_size ≥ ⌈2³²/1000⌉ = 4,294,968` B (~4.29 MB), and it can only ever *tighten*
> the bound: for any product `p ≥ 2³²`, `wrapped(p) = p mod 2³² < 2³² ≤ p`,
> while `original_size` (≤ 512 MiB < 2³²) cannot itself wrap, so the direction
> of the comparison is preserved. The failure mode is therefore **spurious
> rejection**, not a bomb bypass — but a hard error rather than a cache miss,
> and a permanent one: the wrapped bound is a pure function of
> `compressed_size`, so an affected entry fails identically on every read. It
> does not bite everywhere — only where the wrapped bound falls below the
> 512 MiB cap, a density of `2²⁹/2³² = 1/8` over the legal range — but where it
> does, the collapse is near-total: 704 B at that first threshold, 8 B at
> `compressed_size = 115,964,117` (the wrapped bound only ever lands on
> multiples of `gcd(1000, 2³²) = 8`), and 0 at the 512 MiB cap. Those payloads
> are legal under this specification; an implementation that refuses them is
> non-conforming. So is one that rejects on overflow instead of widening: that
> refuses the entire 99.2 % of the legal range above the same threshold, and it
> is unnecessary besides — once the two size caps have passed, the product is
> < 2³⁹ and cannot overflow 64 bits at all.
<!-- END shared-block: ratio-product-rule -->

---

## Store Flow
Expand Down
Loading
Loading