docs: phase 14 plan — evasion by register (planned, not started) - #21
Merged
Merged
Conversation
Records a spike that REJECTED the CTF-framing hypothesis (mean drop 0.052 over five matched pairs, 0/5 crossed the shipped 0.8 threshold) and redirects the phase at the larger effect visible in phase 13's data: content register. The USB-malware example scores 0.467 with no framing trick at all, and 3 of the 4 sub-threshold misses are cyber-technical. Planned, not started. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deterministic LLM05 detector fired on 0/4 of moderation's misses, and 0/25 of the harmful corpus. A control check shows it is not inert: the same sentence plus a defanged shell artifact blocks at Critical. The corpus was authored non-operational, so it contains none of the artifacts the detector matches -- the 0/4 is a property of the corpus, not evidence about production. Narrows the real gap: the layers ARE complementary when a reply carries an executable artifact; the unmitigated hole is harmful operational PROSE with no artifact, where moderation is the only layer and is weakest. The worst case (USB malware, hospital network) scores 0.467 -- below both the shipped 0.8 and the sensitive 0.5, so no threshold setting catches it. Adds an artifact axis to the corpus design, which is the variable that actually moves the deterministic layer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Doc-only. Saves the phase 14 plan into the repo so the work can be picked up in a later session.
Records two spikes, both null results, which is why the plan is aimed where it is:
CTF-framing hypothesis — REJECTED. Matched pairs (technique clause held identical, only the wrapper varied): mean drop 0.052, 0/5 crossed the shipped 0.8 threshold. The classifier keys on what the text describes, not its presentation.
Layer-complement hypothesis — inconclusive, and the test was flawed. The deterministic LLM05 detector fired on 0/4 of moderation's misses and 0/25 of the harmful corpus — but a control check shows it is not inert (the same sentence plus a defanged
curl … | shblocks at Critical). The phase-13 corpus was authored non-operational, so it contains none of the artifacts the detector matches. The 0/4 is a property of the corpus, not evidence about production.What that narrowed down: the layers are complementary when a reply carries an executable artifact. The unmitigated gap is harmful operational prose with no artifact — where moderation is the only layer and is weakest. The worst case (USB malware / hospital network) scores 0.467, below both the shipped 0.8 and the sensitive 0.5, so no threshold setting catches it.
Design consequence: the corpus gains an artifact axis (
plain/technical_prose/technical_artifact/diluted) using defanged artifacts — real command shapes pointing at192.0.2.1, functional as patterns but not as attacks. Deliverable is a per-layer coverage table, not one blended number.No code changes; nothing to verify beyond CI.
🤖 Generated with Claude Code