diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index a267e98..b3b1246 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -32,6 +32,7 @@ jobs: python3 tools/test_wire_format_reference.py python3 tools/interop-reference.py verify python3 tools/interop-v2-reference.py verify + python3 tools/test_encryption_verify.py python3 tools/encryption-verify.py python3 tools/wire-format-reference.py verify @@ -40,6 +41,7 @@ jobs: pip install cryptography==49.0.0 msgpack==1.2.1 lz4==4.4.5 python3 tools/interop-reference.py verify python3 tools/interop-v2-reference.py verify + python3 tools/test_encryption_verify.py python3 tools/encryption-verify.py --require-seal python3 tools/wire-format-reference.py verify --require-extras diff --git a/CHANGELOG.md b/CHANGELOG.md index 7954064..4dee827 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,30 @@ All notable changes to the CacheKit Protocol Specification. ## [Unreleased] +### Encryption — keyring conformance vectors + status reconciliation (LAB-687) + +- [`test-vectors/encryption.json`](test-vectors/encryption.json) gains a `keyring` + block: two master keys (`k1`, `k2`) for tenant `keyring-conformance`, one entry + sealed under each, and per-vector `key_fingerprint_hex` — the fingerprint of the + HKDF-derived per-tenant encryption key, as cachekit-py stores it. Frozen names + `encrypted_with_k1` / `encrypted_with_k2`; append-only like the main set. +- [`tools/encryption-verify.py`](tools/encryption-verify.py) enforces + [`spec/encryption.md` § Key Rotation (Keyring)](spec/encryption.md#key-rotation-keyring): + `[k2, k1]` decrypts both at the declared entry, `[k2]` alone rejects the k1 + entry, and the stored fingerprint selects the derived key — a master-key + fingerprint cannot select. Entry derivation, metadata, AAD and selection run + in the stdlib lane; only the decrypt attempts need `cryptography`. +- New [`tools/test_encryption_verify.py`](tools/test_encryption_verify.py) mutation + suite runs ahead of the verifier in both CI lanes (same doctrine as the + wire-format guard): every keyring guard is proven to go red by poisoning a copy + of the fixture. Added after the LAB-687 panel found three vacuous passes in the + first revision. +- Status banners reconciled with shipped code: `spec/encryption.md` and + `decisions/key-rotation.md` no longer say "not yet implemented"; the + [feature matrix](sdk-feature-matrix.md#encryption) Key rotation row is ✅ for + Python, Rust and TypeScript (cachekit-py#261, cachekit-rs#63, cachekit-ts#103). + + ### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751) - LZ4 compressed bytes are **not canonical** across conforming block encoders. diff --git a/decisions/key-rotation.md b/decisions/key-rotation.md index abceea5..6b2ba80 100644 --- a/decisions/key-rotation.md +++ b/decisions/key-rotation.md @@ -4,11 +4,11 @@ | | | | :--- | :--- | -| **Status** | Proposed (accepted on merge) | +| **Status** | Accepted (protocol [#34](https://github.com/cachekit-io/protocol/pull/34), 2026-07-23); implemented in all three SDKs by 2026-08-15 | | **Date** | 2026-07-23 | | **Ticket** | LAB-516 (filed by the LAB-275 cross-SDK feature-gap audit) | | **Normative spec** | [`spec/encryption.md` → Key Rotation (Keyring)](../spec/encryption.md#key-rotation-keyring) — the spec section owns the rules; this record owns the rationale and runbooks. | -| **Implementation** | Not yet shipped in any SDK — tracked as LAB-516 sub-issues. `ZeroKnowledgeEncryptor::rotate_key()` returns `NotImplemented` (`cachekit-core/src/encryption/core.rs:492`). The [feature matrix](../sdk-feature-matrix.md#encryption) rotation row reads ❌ fleet-wide as of the LAB-1400 consolidation and flips per SDK only as each implementation ships. | +| **Implementation** | Shipped. Shared decrypt helper `Keyring` in cachekit-core 0.5.0 ([cachekit-core#67](https://github.com/cachekit-io/cachekit-core/pull/67), LAB-683 — also deleted the `rotate_key()` stub, `KeyRotationState` and `RotationAwareHeader`); SDK surfaces in [cachekit-py#261](https://github.com/cachekit-io/cachekit-py/pull/261) (LAB-684), [cachekit-rs#63](https://github.com/cachekit-io/cachekit-rs/pull/63) (LAB-686) and [cachekit-ts#103](https://github.com/cachekit-io/cachekit-ts/pull/103) (LAB-685). The [feature matrix](../sdk-feature-matrix.md#encryption) rotation row reads ✅ for all three; conformance is enforced by [`tools/encryption-verify.py`](../tools/encryption-verify.py) against the keyring vectors in [`test-vectors/encryption.json`](../test-vectors/encryption.json) (LAB-687). | --- diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index a1f64e9..a23af9f 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -6,9 +6,9 @@ **Feature parity and compliance status across all CacheKit SDK implementations.** -*Last updated: 2026-08-04 — LAB-1400 consolidation of ten open matrix PRs into one end-state. Every version-keyed claim is verified against the **published artifact** (registry metadata, and the `.crate`/`.tgz` contents where an embedded dependency version decides the answer), not against a repo branch — see [decisions/matrix-version-verification.md](decisions/matrix-version-verification.md) for why and how. Per-PR fold verdicts are in [CHANGELOG.md](CHANGELOG.md); per-row history is `git log sdk-feature-matrix.md`.* +*Last updated: 2026-09-02 — LAB-687 keyring conformance and documentation reconciliation (following LAB-1400's matrix baseline correction). Every version-keyed claim is verified against the **published artifact** (registry metadata, and the `.crate`/`.tgz` contents where an embedded dependency version decides the answer), not against a repo branch — see [decisions/matrix-version-verification.md](decisions/matrix-version-verification.md) for why and how. Per-PR fold verdicts are in [CHANGELOG.md](CHANGELOG.md); per-row history is `git log sdk-feature-matrix.md`.* -*__Cells that reversed — check these if you built on them:__ Key rotation (py/rs ✅ → ❌ fleet-wide), Rust `::secure` preset and Rust sync support (both ✅ → do not exist), Builder API (py/ts ✅ → ❌), Hardware acceleration (rs ✅ → not re-exported, ts N/A → ❌), TypeScript Arrow (🔜 → ❌), Python's encrypted read path (documented fail-closed → **fail-open by default**), and `cache.secure.wrap()` in TypeScript (implied encryption → **no guarantee**). The TypeScript protocol-1.1 `bin` rollout also reversed twice in two days: it is **not** shipped on either ts path (per-artifact evidence in the [cachekit-core architecture note](#architecture-notes)).* +*__Cells that reversed — check these if you built on them:__ Rust `::secure` preset and Rust sync support (both ✅ → do not exist), Builder API (py/ts ✅ → ❌), Hardware acceleration (rs ✅ → not re-exported, ts N/A → ❌), TypeScript Arrow (🔜 → ❌), Python's encrypted read path (documented fail-closed → **fail-open by default**), and `cache.secure.wrap()` in TypeScript (implied encryption → **no guarantee**). The TypeScript protocol-1.1 `bin` rollout also reversed twice in two days: it is **not** shipped on either ts path (per-artifact evidence in the [cachekit-core architecture note](#architecture-notes)).* @@ -66,19 +66,17 @@ | HKDF-SHA256 key derivation | ✅ via Rust FFI | ✅ | ✅ via NAPI (Rust) | 🔜 Planned | | Per-tenant key isolation | ✅ | ✅ | ✅ via TenantKeys NAPI | 🔜 Planned | | AAD v0x03 (cache_key binding) | ✅ | ✅ | ✅ | ❌ | -| Key rotation | ❌ mismatch detection only⁵ | ❌⁵ | ❌ — nonce-exhaustion detection exists but is **unobservable** to the caller⁵ ⁸ | ❌ | +| Key rotation | ✅ keyring; derived-key fingerprint selection⁵ | ✅ keyring; sequential attempts⁵ | ✅ keyring; sequential attempts⁵ | ❌ | | **Tamper / wrong-key failure mode** | ⚠️ **fail-OPEN by default** — warn + recompute; switchable with `CACHEKIT_ENCRYPTION_FAIL_CLOSED=true`¹⁸ | ✅ **Fails closed** — `decrypt(…)?` propagates (`client.rs:830`, `:847`), and `#[cachekit(secure)]` emits no fail-open arm (`cachekit-macros/src/lib.rs:439-451`) | ⚠️ **fail-OPEN on reads, silently drops writes, not switchable**⁸ | — | | **Does the `secure` API enforce encryption?** | ✅ Raises without a key | ✅ `secure()` returns `Err` | ❌ **`cache.secure.wrap()` is an unconditional alias for `wrap()`** — silently caches plaintext on any instance not built by `createCache.secure()` (LAB-513, CWE-311); see [Intent-preset semantics](#intent-preset-semantics-parity-not-presence) | — | | Hardware acceleration detection | ✅ surfaced (`hardware_acceleration_enabled()`) | ⚠️ core-internal, not re-exported⁶ | ❌ not exposed⁶ | N/A | | Counter-based nonces | ✅ via Rust | ✅ | ✅ via NAPI (Rust) | ❌ use random | > [!IMPORTANT] -> AAD v0x03 is required for protocol compliance. SDKs without it cannot safely interoperate with encrypted payloads from compliant SDKs — the auth tag will fail verification. Python, Rust, and TypeScript construction was code-verified byte-identical on 2026-07-21 (LAB-272); the normative byte layout and the frozen `True`/`False` compressed tokens (protocol#12) are defined in [spec/encryption.md](spec/encryption.md#additional-authenticated-data-aad). Python's auto serializers append the optional `original_type` fifth component; Rust, TypeScript, and interop mode always emit exactly four. When encryption is enabled but a stored entry claims plaintext, Python never returns it — the entry is converted to a miss and evicted rather than raising (LAB-241, [cachekit-py#215](https://github.com/cachekit-io/cachekit-py/pull/215)); see note ⁵ for what that means if you are monitoring for hard errors. +> AAD v0x03 is required for protocol compliance. SDKs without it cannot safely interoperate with encrypted payloads from compliant SDKs — the auth tag will fail verification. Python, Rust, and TypeScript construction was code-verified byte-identical on 2026-07-21 (LAB-272); the normative byte layout and the frozen `True`/`False` compressed tokens (protocol#12) are defined in [spec/encryption.md](spec/encryption.md#additional-authenticated-data-aad). Python's auto serializers append the optional `original_type` fifth component; Rust, TypeScript, and interop mode always emit exactly four. When encryption is enabled but a stored entry claims plaintext, Python never returns it — the entry is converted to a miss and evicted rather than raising (LAB-241, [cachekit-py#215](https://github.com/cachekit-io/cachekit-py/pull/215)). > [!WARNING] -> ⁵ **Key rotation ships in no SDK. Rotating a master key today invalidates every encrypted entry.** Corrected 2026-08-04 (LAB-1400) from a previous Python ✅ / Rust ✅ that no code supported. `cachekit-core`'s `ZeroKnowledgeEncryptor::rotate_key()` returns `EncryptionError::NotImplemented` (`src/encryption/core.rs:492`); cachekit-rs neither re-exports nor calls any rotation API; cachekit-ts classifies the core's nonce-exhaustion message into a `NonceExhaustedError` that **no caller can observe**, because it is raised inside the `set` callback and absorbed there (note ⁸). Note that nonce exhaustion does *not* make rotation mandatory — the core fails closed and a fresh encryptor instance clears it; reaching for a master-key rotation instead would destroy every existing entry, which is exactly the trap this note exists to prevent. -> -> **Do not build on cachekit-py's importable `KeyRotationState`.** cachekit-py ships a live PyO3 binding — `#[pyclass(name = "KeyRotationState")]` with `start_rotation()` / `complete_rotation()` / `is_rotating()` (`rust/src/python_bindings.rs:258`, `:283`, registered on `_rust_serializer` at `:394`) — so `from cachekit._rust_serializer import KeyRotationState` works today, `start_rotation(new_key)` **returns success**, and `is_rotating()` then reports `True`. Nothing in `src/cachekit/` consults any of it: no dual-key read path exists, so an operator who takes that success as a grace window and swaps `CACHEKIT_MASTER_KEY` renders every encrypted entry undecryptable. Removal is tracked under LAB-516. `cachekit-core` likewise exports `RotationAwareHeader` (`src/encryption/mod.rs:20`, aliased as the "canonical" `EncryptionHeader`) which no SDK ever writes to the wire. +> ⁵ **Keyring rotation ships in Python, Rust, and TypeScript.** Each SDK accepts one current encryption key plus up to three decrypt-only previous keys, rejects a configuration that repeats the current key in that list, writes only with the current key, and keeps reading old-key entries until the operator removes that key — retirement is manual, not TTL-driven. Python selects a keyring entry by the stored fingerprint of its HKDF-derived per-tenant encryption key; Rust and TypeScript attempt the current key followed by decrypt-only keys with identical AAD. The protocol reference verifier ([`tools/encryption-verify.py`](tools/encryption-verify.py)) checks the [keyring vectors](test-vectors/encryption.json): both entries decrypt under `[k2, k1]` at the declared entry, the k1 entry is rejected under `[k2]`, and each stored fingerprint selects the HKDF-derived (not master) key; no SDK consumes these vectors yet. Use the [key rotation runbook](https://docs.cachekit.io/concepts/key-rotation/) (normative source: [decisions/key-rotation.md § Runbooks](decisions/key-rotation.md#runbooks-normative-for-docs)) for the required three-phase rollout and compromise response. > > ¹⁸ **Python's encrypted read path is fail-OPEN by default.** `EncryptionWrapper(fail_closed=False)` (`serializers/encryption_wrapper.py:113`) from `encryption_fail_closed: bool = Field(default=False)` (`config/settings.py:225`): on a key-fingerprint mismatch the default is a warning plus a recompute, not a raise — that flag gates the fingerprint pre-check — and the same setting drives the AES-GCM authentication-failure policy through `handle_decrypt_failure` (`cache_handler.py:1307`, `:1317`; resolved at `:550-552`, L1 path `decorators/wrapper.py:1210`). Both default to recompute. Set `CACHEKIT_ENCRYPTION_FAIL_CLOSED=true` if you are relying on hard errors as your wrong-key or tamper alarm — otherwise the signal is a log line and a billable miss. (Separately, a stored entry that claims plaintext while encryption is enabled is never returned to the caller: it is converted to a miss and evicted, deliberately, for plaintext→encrypted migration — LAB-241, [cachekit-py#215](https://github.com/cachekit-io/cachekit-py/pull/215).) > @@ -88,7 +86,7 @@ > > - **Read** — a tampered payload or wrong key makes `cache.get()` return `null`, an ordinary miss. Through `wrap()` the function is re-executed and the result re-stored. > - **Write** — an encrypt, NAPI or key failure makes `set()` **resolve like success while storing nothing**; every later `wrap()` re-executes the origin, forever, silently. -> - **Nonce exhaustion** — `NonceExhaustedError` is raised *inside* `encrypt()` (`encryption/manager-core.ts`), i.e. inside the `set` callback, so it is absorbed too. **Nonce reuse is not the risk:** `cachekit-core` fails closed at 2³² operations per encryptor instance — `generate_nonce()` returns `NonceCounterExhausted` once `counter >= u32::MAX`, and the counter is an `AtomicU64` specifically so it *stays* exhausted rather than wrapping (`encryption/core.rs:168-185`, `:297-311`). The unobservable consequence is instead that **every** encrypted `set()` on that encryptor silently stores nothing for the remaining life of the process. An operator seeing only a rising miss rate has no way to reach the real remedy — a fresh encryptor instance — and per note ⁵ the instinct to rotate the master key would destroy every existing entry. +> - **Nonce exhaustion** — `NonceExhaustedError` is raised *inside* `encrypt()` (`encryption/manager-core.ts`), i.e. inside the `set` callback, so it is absorbed too. **Nonce reuse is not the risk:** `cachekit-core` fails closed at 2³² operations per encryptor instance — `generate_nonce()` returns `NonceCounterExhausted` once `counter >= u32::MAX`, and the counter is an `AtomicU64` specifically so it *stays* exhausted rather than wrapping (`encryption/core.rs:168-185`, `:297-311`). The unobservable consequence is instead that **every** encrypted `set()` on that encryptor silently stores nothing for the remaining life of the process. An operator seeing only a rising miss rate has no way to reach the real remedy — a fresh encryptor instance — follow the [key rotation runbook](https://docs.cachekit.io/concepts/key-rotation/) for the safe scheduled procedure. > > Degradation is on unless explicitly disabled (`degradationEnabled = config.degradation !== false`, `reliability/executor.ts:39`) and `createCache.secure()` / `.production()` / `.io()` all set it `true` (`intents-core.ts:186`); only `minimal` sets `false`, and `minimal` carries no encryption. **There is no `failClosed` option anywhere in cachekit-ts** — Python's `CACHEKIT_ENCRYPTION_FAIL_CLOSED` has no counterpart, so the only lever is `reliability: { degradation: false }`, which also gives up backend-outage degradation. If you rely on a thrown error as your tamper, wrong-key or nonce alarm, TypeScript raises none; the sole signal is the `errors_total` counter. > diff --git a/spec/encryption.md b/spec/encryption.md index dfb7714..6b92a44 100644 --- a/spec/encryption.md +++ b/spec/encryption.md @@ -342,13 +342,17 @@ by `tools/encryption-verify.py`. ## Key Rotation (Keyring) -> [!WARNING] -> **Status: specified, not yet implemented** (decision record: -> [decisions/key-rotation.md](../decisions/key-rotation.md), 2026-07-23; -> implementation tracked under LAB-516). Until an SDK ships this section, rotating -> its master key invalidates every encrypted entry — fail-open readers take misses, -> fail-closed readers take errors. The [feature matrix](../sdk-feature-matrix.md) -> reflects per-SDK implementation status. +> [!NOTE] +> **Status: shipped in Python, Rust, and TypeScript** (decision record: +> [decisions/key-rotation.md](../decisions/key-rotation.md), 2026-07-23; LAB-516). +> The shared decrypt helper is `cachekit_core::encryption::keyring::Keyring` +> ([cachekit-core#67](https://github.com/cachekit-io/cachekit-core/pull/67)); the SDK +> surfaces are [cachekit-py#261](https://github.com/cachekit-io/cachekit-py/pull/261), +> [cachekit-rs#63](https://github.com/cachekit-io/cachekit-rs/pull/63) and +> [cachekit-ts#103](https://github.com/cachekit-io/cachekit-ts/pull/103). An SDK that +> does not implement this section invalidates every encrypted entry when its master +> key rotates — fail-open readers take misses, fail-closed readers take errors. The +> [feature matrix](../sdk-feature-matrix.md) reflects per-SDK implementation status. > > Earlier revisions of this spec described a 32-byte `RotationAwareHeader` prepended > to ciphertext. **That header was never written by any SDK and is not part of the diff --git a/test-vectors/encryption.json b/test-vectors/encryption.json index 2960c67..658bb35 100644 --- a/test-vectors/encryption.json +++ b/test-vectors/encryption.json @@ -12,6 +12,45 @@ "master_key_note": "32 bytes of 0x61 (ASCII 'a' repeated 32 times)", "tenant_id": "cross-sdk-test", "derived_key_fingerprint_hex": "96179a9bc881aa7ca83f04b78a66afd3", + "keyring": { + "tenant_id": "keyring-conformance", + "entries": [ + { + "id": "k1", + "master_key_hex": "1111111111111111111111111111111111111111111111111111111111111111", + "derived_key_fingerprint_hex": "b6bfaa4119e9d72c87a393b17f1e117b" + }, + { + "id": "k2", + "master_key_hex": "2222222222222222222222222222222222222222222222222222222222222222", + "derived_key_fingerprint_hex": "e066ffdacfb5fb11a6ecc44c7a0ecb4e" + } + ], + "vectors": [ + { + "name": "encrypted_with_k1", + "encrypted_with": "k1", + "plaintext_hex": "6b312d706c61696e74657874", + "cache_key": "keyring:k1:entry", + "format": "msgpack", + "compressed": false, + "aad_hex": "03000000136b657972696e672d636f6e666f726d616e6365000000106b657972696e673a6b313a656e747279000000076d73677061636b0000000546616c7365", + "ciphertext_hex": "010101010101010101010101ebc2f0799d210e06f1ec990ec8d2b3b1b6876e1cdbecb09600b89292", + "key_fingerprint_hex": "b6bfaa4119e9d72c87a393b17f1e117b" + }, + { + "name": "encrypted_with_k2", + "encrypted_with": "k2", + "plaintext_hex": "6b322d706c61696e74657874", + "cache_key": "keyring:k2:entry", + "format": "msgpack", + "compressed": false, + "aad_hex": "03000000136b657972696e672d636f6e666f726d616e6365000000106b657972696e673a6b323a656e747279000000076d73677061636b0000000546616c7365", + "ciphertext_hex": "020202020202020202020202c9e7a0e25b70e67fc71a191af2d8ec795ce7fa3bf69d06e291ec1bce", + "key_fingerprint_hex": "e066ffdacfb5fb11a6ecc44c7a0ecb4e" + } + ] + }, "vectors": [ { "name": "basic_bytes", diff --git a/tools/encryption-verify.py b/tools/encryption-verify.py index f8b69a0..3184dc6 100644 --- a/tools/encryption-verify.py +++ b/tools/encryption-verify.py @@ -8,6 +8,8 @@ - Every vector's `aad_hex` reconstructs byte-for-byte from its fields (tenant_id, cache_key, format, compressed, optional original_type), including the frozen ASCII boolean tokens `True` / `False`. + - Keyring vectors: entry derivation, metadata, AAD, and fingerprint selection + pinned to the HKDF-derived per-tenant encryption key, never the master key. - The vector count never drops below the frozen floor — vectors are ground truth and may be added, never removed. @@ -17,6 +19,8 @@ `plaintext_hex`. (No re-seal comparison: AES-GCM is deterministic per (key, nonce, aad, plaintext), so a successful decrypt + plaintext match already proves the seal.) + - Keyring vectors decrypt under [k2, k1] at the declared entry, and the k1 entry + is rejected under [k2] alone (sequential-attempt path). Exit status is non-zero on any mismatch. Vectors are frozen ground truth and are never regenerated by this tool. @@ -53,6 +57,12 @@ # format registry per spec/encryption.md — vectors must not invent tokens. FORMAT_REGISTRY = frozenset({"msgpack", "orjson", "arrow"}) +# Keyring conformance vectors: frozen name -> the entry that sealed it. The +# conformance keyring is current=k2 with decrypt-only k1 (KEYRING_ORDER). +FROZEN_KEYRING_VECTORS = {"encrypted_with_k1": "k1", "encrypted_with_k2": "k2"} +FROZEN_KEYRING_VECTOR_NAMES = frozenset(FROZEN_KEYRING_VECTORS) +KEYRING_ORDER = ("k2", "k1") + def hkdf_sha256(ikm: bytes, salt: bytes, info: bytes, length: int = 32) -> bytes: """RFC 5869 HKDF-Extract + Expand with SHA-256.""" @@ -78,6 +88,20 @@ def key_fingerprint(key: bytes) -> str: return hashlib.sha256(b"key_fingerprint_v1" + key).digest()[:16].hex() +def decrypt_with_keyring(keys: list[bytes], ciphertext: bytes, aad: bytes) -> tuple[int, bytes] | None: + """Try each key once, in keyring order, with identical authenticated data.""" + from cryptography.exceptions import InvalidTag # noqa: PLC0415 + from cryptography.hazmat.primitives.ciphers.aead import AESGCM # noqa: PLC0415 + + nonce, sealed = ciphertext[:12], ciphertext[12:] + for index, key in enumerate(keys): + try: + return index, AESGCM(key).decrypt(nonce, sealed, aad) + except InvalidTag: + continue + return None + + def aad_v3(tenant_id: str, cache_key: str, *, fmt: str, compressed: bool, original_type: str | None = None) -> bytes: """AAD v0x03 per spec/encryption.md — 4-byte BE length-prefixed components. @@ -99,9 +123,95 @@ def aad_v3(tenant_id: str, cache_key: str, *, fmt: str, compressed: bool, origin return bytes(aad) -def main() -> int: - require_seal = "--require-seal" in sys.argv[1:] - doc = json.loads(VECTORS_PATH.read_text()) +def verify_keyring(keyring: dict | None, *, seal: bool) -> int: + """Return the number of failed keyring checks (spec/encryption.md § Key Rotation). + + Entry derivation, metadata, AAD reconstruction and fingerprint selection are + stdlib and always run; only the decrypt attempts need `cryptography` (`seal`). + """ + if keyring is None: + print("FAIL keyring vectors missing") + return 1 + failures = 0 + + entries = keyring.get("entries", []) + ids = [entry.get("id") for entry in entries] + if len(set(ids)) != len(ids) or not set(KEYRING_ORDER) <= set(ids): + print(f"FAIL keyring entries must have unique ids including {sorted(KEYRING_ORDER)}; got {ids}") + return failures + 1 + derived_keys: dict[str, bytes] = {} + for entry in entries: + derived = derive_encryption_key(bytes.fromhex(entry["master_key_hex"]), keyring["tenant_id"]) + derived_keys[entry["id"]] = derived + if key_fingerprint(derived) != entry["derived_key_fingerprint_hex"]: + print(f"FAIL keyring {entry['id']}: derived-key fingerprint mismatch") + failures += 1 + # Selection table for the per-entry-identity path: fingerprints of the + # HKDF-DERIVED per-tenant encryption keys only — a master-key fingerprint + # is deliberately not in here, so a vector carrying one cannot select. + derived_fingerprints = {key_fingerprint(derived): key_id for key_id, derived in derived_keys.items()} + + vectors = keyring.get("vectors", []) + missing = FROZEN_KEYRING_VECTOR_NAMES - {vec.get("name") for vec in vectors} + if missing: + print(f"FAIL frozen keyring vectors missing: {sorted(missing)}") + failures += 1 + if not seal: + print("note: keyring decrypt checks require cryptography — AAD, metadata and fingerprint selection only") + + full_keyring = [derived_keys[key_id] for key_id in KEYRING_ORDER] + current_only = [derived_keys[KEYRING_ORDER[0]]] + for vec in vectors: + name = vec["name"] + encrypted_with = vec.get("encrypted_with") + if encrypted_with not in KEYRING_ORDER or FROZEN_KEYRING_VECTORS.get(name, encrypted_with) != encrypted_with: + print( + f"FAIL keyring {name}: encrypted_with must be the entry the frozen name declares (one of {list(KEYRING_ORDER)}); got {encrypted_with!r}" + ) + failures += 1 + continue + if not isinstance(vec["compressed"], bool) or vec["format"] not in FORMAT_REGISTRY: + print( + f"FAIL keyring {name}: invalid metadata (compressed must be a JSON boolean, format must be in {sorted(FORMAT_REGISTRY)})" + ) + failures += 1 + continue + aad = aad_v3(keyring["tenant_id"], vec["cache_key"], fmt=vec["format"], compressed=vec["compressed"]) + if aad.hex() != vec["aad_hex"]: + print(f"FAIL keyring {name}: AAD mismatch\n expected {vec['aad_hex']}\n rebuilt {aad.hex()}") + failures += 1 + continue + if derived_fingerprints.get(vec.get("key_fingerprint_hex")) != encrypted_with: + print( + f"FAIL keyring {name}: stored key fingerprint must select the HKDF-derived key of {encrypted_with} (never a master key)" + ) + failures += 1 + continue + if not seal: + print(f"ok keyring {name} (AAD + fingerprint selection only)") + continue + + ciphertext = bytes.fromhex(vec["ciphertext_hex"]) + full_result = decrypt_with_keyring(full_keyring, ciphertext, aad) + if full_result is None or full_result[1].hex() != vec["plaintext_hex"]: + print(f"FAIL keyring {name}: {list(KEYRING_ORDER)} did not decrypt to plaintext_hex") + failures += 1 + continue + if full_result[0] != KEYRING_ORDER.index(encrypted_with): + print( + f"FAIL keyring {name}: decrypted with keyring entry {KEYRING_ORDER[full_result[0]]}, expected {encrypted_with}" + ) + failures += 1 + continue + if encrypted_with != KEYRING_ORDER[0] and decrypt_with_keyring(current_only, ciphertext, aad) is not None: + print(f"FAIL keyring {name}: [{KEYRING_ORDER[0]}] alone accepted the {encrypted_with} entry") + failures += 1 + continue + print(f"ok keyring {name}") + return failures + + +def verify(doc: dict, *, require_seal: bool) -> int: failures = 0 missing = FROZEN_VECTOR_NAMES - {v["name"] for v in doc["vectors"]} @@ -132,7 +242,9 @@ def main() -> int: for vec in doc["vectors"]: name = vec["name"] if not isinstance(vec["compressed"], bool) or vec["format"] not in FORMAT_REGISTRY: - print(f"FAIL {name}: invalid metadata (compressed must be a JSON boolean, format must be in {sorted(FORMAT_REGISTRY)})") + print( + f"FAIL {name}: invalid metadata (compressed must be a JSON boolean, format must be in {sorted(FORMAT_REGISTRY)})" + ) failures += 1 continue aad = aad_v3( @@ -162,13 +274,21 @@ def main() -> int: continue print(f"ok {name}") + failures += verify_keyring(doc.get("keyring"), seal=aesgcm is not None) + if failures: print(f"{failures} vector(s) FAILED") return 1 mode = "AAD + AES-GCM seal" if aesgcm is not None else "AAD-only" - print(f"all {len(doc['vectors'])} encryption vectors verified ({mode})") + keyring_count = len(doc.get("keyring", {}).get("vectors", [])) + print(f"all {len(doc['vectors'])} encryption vectors plus {keyring_count} keyring vectors verified ({mode})") return 0 +def main() -> int: + require_seal = "--require-seal" in sys.argv[1:] + return verify(json.loads(VECTORS_PATH.read_text()), require_seal=require_seal) + + if __name__ == "__main__": sys.exit(main()) diff --git a/tools/test_encryption_verify.py b/tools/test_encryption_verify.py new file mode 100755 index 0000000..c52c461 --- /dev/null +++ b/tools/test_encryption_verify.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +"""Mutation tests for encryption-verify.py's keyring guards. + +Same doctrine as test_wire_format_reference.py: a conformance gate is proven by +poisoning the fixture and watching it go red, not by reading it. Every case below +exited 0 on protocol#60's first revision (LAB-687 expert panel, 2026-09-02) — the +keyring block sat behind the `cryptography` guard, so the stdlib CI lane verified +nothing, and a blanked fingerprint selection printed `ok ... None`. + +Stdlib cases always run. Seal cases run only when `cryptography` imports, which +is the optional-deps CI lane. The fixture is loaded once and every case mutates a +deep copy; nothing is ever written. + +Run: python3 tools/test_encryption_verify.py (exit 1 on any failure) +""" + +from __future__ import annotations + +import contextlib +import copy +import importlib.util +import io +import json +import sys +from collections.abc import Callable +from pathlib import Path + +HERE = Path(__file__).resolve().parent +spec = importlib.util.spec_from_file_location("encryption_verify", HERE / "encryption-verify.py") +if spec is None or spec.loader is None: + sys.exit("cannot load encryption-verify.py as a module") +ev = importlib.util.module_from_spec(spec) +spec.loader.exec_module(ev) + +DOC = json.loads(ev.VECTORS_PATH.read_text()) +HAVE_SEAL = importlib.util.find_spec("cryptography") is not None + + +def run(mutate: Callable[[dict], None]) -> tuple[int, str]: + doc = copy.deepcopy(DOC) + mutate(doc) + out = io.StringIO() + with contextlib.redirect_stdout(out): + rc = ev.verify(doc, require_seal=HAVE_SEAL) + return rc, out.getvalue() + + +def k1(doc: dict) -> dict: + return next(v for v in doc["keyring"]["vectors"] if v["name"] == "encrypted_with_k1") + + +def k2(doc: dict) -> dict: + return next(v for v in doc["keyring"]["vectors"] if v["name"] == "encrypted_with_k2") + + +def master_fingerprint(doc: dict, key_id: str) -> str: + entry = next(e for e in doc["keyring"]["entries"] if e["id"] == key_id) + return ev.key_fingerprint(bytes.fromhex(entry["master_key_hex"])) + + +def swap_k1_payload_for_k2(doc: dict) -> None: + """k1-labelled vector carrying k2's sealed bytes: decrypts, but at the wrong entry.""" + a, b = k1(doc), k2(doc) + for field in ("cache_key", "aad_hex", "ciphertext_hex", "plaintext_hex"): + a[field] = b[field] + + +STDLIB_CASES: dict[str, Callable[[dict], None]] = { + "keyring block deleted": lambda d: d.pop("keyring"), + "fingerprint selection removed": lambda d: k1(d).pop("key_fingerprint_hex"), + "fingerprint selection blanked": lambda d: k1(d).__setitem__("key_fingerprint_hex", ""), + "fingerprint is the MASTER key's": lambda d: k1(d).__setitem__("key_fingerprint_hex", master_fingerprint(d, "k1")), + "fingerprint selects the other entry": lambda d: k1(d).__setitem__("key_fingerprint_hex", k2(d)["key_fingerprint_hex"]), + "encrypted_with unknown id": lambda d: k1(d).__setitem__("encrypted_with", "kx"), + "encrypted_with contradicts frozen name": lambda d: k1(d).__setitem__("encrypted_with", "k2"), + "duplicate entry ids": lambda d: d["keyring"]["entries"].insert(0, copy.deepcopy(d["keyring"]["entries"][0])), + "entry k1 missing": lambda d: d["keyring"]["entries"].pop(0), + "entry fingerprint corrupted": lambda d: d["keyring"]["entries"][0].__setitem__("derived_key_fingerprint_hex", "00" * 16), + "compressed as JSON int": lambda d: k1(d).__setitem__("compressed", 0), + "format off-registry": lambda d: k1(d).__setitem__("format", "pickle"), + "aad corrupted": lambda d: k1(d).__setitem__("aad_hex", "03" + k1(d)["aad_hex"][2:].replace("6b", "6c", 1)), + "cache_key substituted": lambda d: k1(d).__setitem__("cache_key", "keyring:attacker:entry"), + "frozen keyring vector renamed": lambda d: k1(d).__setitem__("name", "renamed"), +} + +SEAL_CASES: dict[str, Callable[[dict], None]] = { + "ciphertext corrupted": lambda d: k1(d).__setitem__("ciphertext_hex", k1(d)["ciphertext_hex"][:-2] + "00"), + "plaintext pinned wrong": lambda d: k1(d).__setitem__("plaintext_hex", "00"), + "decrypts at the wrong keyring entry": swap_k1_payload_for_k2, +} + + +def main() -> int: + bad = 0 + rc, out = run(lambda d: None) + if rc != 0: + print(f"FAIL baseline fixture does not verify:\n{out}") + return 1 + print(f"ok baseline verifies ({'seal' if HAVE_SEAL else 'stdlib'} lane)") + + cases = dict(STDLIB_CASES) + if HAVE_SEAL: + cases.update(SEAL_CASES) + else: + print(f"note: {len(SEAL_CASES)} seal cases skipped — cryptography not installed") + for name, mutate in cases.items(): + rc, out = run(mutate) + if rc == 0: + print(f"FAIL mutation '{name}' exited 0:\n{out}") + bad += 1 + else: + print(f"ok mutation '{name}' goes red") + if bad: + print(f"{bad} mutation(s) NOT caught") + return 1 + print(f"all {len(cases)} mutations caught") + return 0 + + +if __name__ == "__main__": + sys.exit(main())