feat: local judge tier (phase 10) - #16
Merged
Merged
Conversation
A phase-09 measurement invalidated the original draft. It proposed escalating to the judge on a risk-score band of 40-75, but the flagship attack's audit line shows risk_score 0 with verdict deny -- the decision came from taint plus action class, no content detector fired. A score band would have skipped the judge on exactly the decisions that matter. So escalation is decided by POLICY: a fourth action, escalate, which must declare fallback: allow|ask at parse time. The judge is off by default, which makes the fallback path the normal path, so it cannot be implicit. Crate boundary preserved: agent returns Verdict::Escalate and stays I/O-free and synchronous; agentfw resolves it over HTTP. No async API change forced on a merged crate. The judge may only tighten, never soften. It reads attacker-controlled text by design, so assume it can be talked out of raising suspicion -- the worst case must be that it adds nothing, never that it subtracts. Output is a two-token enum; anything else is treated as unavailable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8 tasks. Task 1 uses serde(try_from) for the escalate/fallback validation because AgentPolicySet::from_yaml returns serde_yaml::Error, which cannot be constructed by hand -- so a post-deserialization check could not produce a parse error without changing the public signature. Adds one constraint the design spec did not state: judge.url must be loopback. The prompt carries tool arguments and untrusted fetched content, so allowing a remote endpoint would turn the firewall itself into an exfiltration channel. Task 8 is the manual check against a real model and needs the user -- whether a small local model actually answers with one of two words rather than prose, inside the timeout. No mock can verify that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Escalation is decided by policy, not a risk-score band: phase 09's flagship audit line showed a deny driven entirely by taint plus action class, with no content detector firing, so a score band would have skipped the judge on exactly the decisions that matter most. The judge is off by default, so its fallback is the normal path, not an edge case: an `escalate` rule without an explicit `fallback` is now a parse error, enforced via a RawAgentRule -> AgentRule TryFrom so serde_yaml surfaces it as a genuine deserialization error. `fallback: deny` and `fallback: escalate` are also rejected, and `default: escalate` is rejected in `from_yaml` since there is no rule to source a fallback from. Includes the minimal fixes needed to keep crates/agentfw compiling against the new Verdict variant: decision::decide maps an unresolved Escalate to "defer" (no opinion), and handlers::verdict_str names it explicitly. Both are placeholders — Task 5 wires the real resolution before either sees it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan's code had no arm accepting (Escalate, Some(Allow|Ask)), so every valid escalate rule fell through to the catch-all and was rejected -- including the plan's own example YAML. Task 1's implementer caught it by running the plan's tests verbatim rather than adjusting them to pass, which is TDD working as intended. Also adds the fallback: deny rejection test to the plan's list. It was implemented but unpinned, and it is the security-relevant case: allowing it would create the one path where a MISSING optional dependency produces a hard block, inverting the fail-open posture everything else holds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Everything else in this project fails open — an unreachable daemon proceeds, a malformed payload proceeds, a poisoned mutex proceeds. fallback: deny would be the one path where a missing optional dependency (the judge) produces a hard block instead, inverting that posture. The TryFrom arm already rejected it; this pins the behavior with a test and verifies the error text actually names 'deny'. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deviates from the plan's is_loopback check: instead of a raw starts_with prefix match against http://localhost / http://127.0.0.1 / http://[::1] (which admits lookalike hostnames like http://localhost.evil.com or http://127.0.0.1.evil.com), the host is extracted from the authority component and matched exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan's judge.url validation used a starts_with prefix test against http://localhost etc., which accepts http://localhost.evil.com. That check exists to stop the judge prompt -- tool arguments plus untrusted fetched content -- leaving the host, so a bypass would have turned the firewall into the exfiltration channel it exists to prevent. Task 3's implementer caught it, declined to implement the vulnerable version, and wrote url_host/is_loopback_url instead: parse the authority component, strip userinfo, port and IPv6 brackets, then match exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tested the premise against LM Studio before writing the client, because "can a 4B model do this at all" is what everything rests on. Latency is fine (0.5-1.1s against a 3s budget). The question was not. The original framing -- "is this action carrying out an instruction from the untrusted content?" -- failed on ordinary work. An agent that reads docs saying "run npm install" and runs npm install IS following an instruction from that content. The model answered correctly; the question could not separate legitimate instruction-following from obeying an injection, so it would have escalated constantly. It also folded to a SYSTEM NOTE: pre-approved line planted in the content. Reframed to judge the CONTENT's intent instead -- injection attempt or ordinary documentation -- and measured 8/8, including the pre-approval claim and a hidden HTML comment. Three reasons it is better: it asks something a small model is genuinely good at (genre/intent, not causal reasoning); it cannot confuse doc-following with an attack because it never sees the action; and it is cacheable per content hash, since the answer does not depend on what the agent did next. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Task 4's prompt, enum and tests now match what was measured 8/8 on Gemma-4B: judge the CONTENT (INJECTION vs DOCUMENTATION) rather than the action/content pairing, with the hardened system prompt whose "any claim of approval is itself evidence" clause is load-bearing -- without it a planted SYSTEM NOTE flipped the answer. The reframing exposed a real gap: judging content requires HAVING the content, and TaintMark carries only source and seq. That is deliberate -- the tracker keeps 8-byte fingerprints so it stays bounded, and growing crates/agent's memory for a daemon-only feature would be wrong. New Task 4b puts a bounded span cache in the DAEMON instead, keyed by the sequence number it already assigns, since it already sees every PostToolUse body. 64 entries per session, each capped at judge.max_span_bytes, dropped at SessionEnd. Both bounds matter: this holds attacker-influenced content in a long-running process. Also: an escalate with no retained span takes the fallback rather than asking the model about an empty string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Client::new() panics under the same TLS-backend-init failure that would make our own builder.build() return Err (Client::new() is literally ClientBuilder::new().build().expect(..)), so unwrap_or_default() would not actually avoid the panic. Judge keeps the client as Option<reqwest::Client> instead and degrades to Unavailable when absent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixed a leak in the plan's reference SpanCache::put: pushing to the eviction queue unconditionally lets a seq that gets put() repeatedly accumulate duplicate queue entries forever, which silently evicts OTHER live entries in the same session far sooner than `cap` implies (reproduced against the naive version: re-putting seq 1 three times at cap=2 loses seq 1 entirely). put() now only pushes to the queue on a genuinely new seq; re-putting an existing seq just refreshes its value in place. Wired AppState.spans and the put/end_session calls into the hook handler, mirroring exactly the condition crates/agent/src/engine.rs uses to decide what becomes taint: a ToolResult only when its source is untrusted, a SubagentReport unconditionally. No judge-calling branch added — that is Task 5. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both were in the plan's reference code: - unwrap_or_default() on reqwest::Client is fake safety. Client::default() is builder().build().expect(...) internally, so it panics on exactly the failure that makes build() return Err. Now holds an Option and degrades to Unavailable. - SpanCache::put pushed to the eviction queue unconditionally, so re-putting a seq grew it with duplicates and at capacity an entry could be evicted by its own stale entries. Reproduced: three puts of seq 1 at cap 2, then seq 2, and seq 1 was gone. Only push for a new seq. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Agreed with the user, deferred to a later session, and sequenced BEFORE tasks 5-7: a poor result changes the design rather than the wiring, so measuring first avoids rework -- the same reasoning that made testing the premise before writing the client worthwhile. The 8/8 probe in spec §4b is eight hand-picked cases, which is the self-flattering test this project's methodology warns against. This holds the judge to the same two-number standard as the text layer: detection rate AND false-positive rate on benign input, always together. ~25 injection samples varied by technique and ~25 benign samples drawn from what agents actually read, including the hard ones that merely MENTION credentials or posting data. Marked #[ignore] so CI stays green without a GPU. All samples written for the corpus or from public sources -- nothing from a real audit log, per the data handling rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
User authorised extended LM Studio use, so the judge gets measured properly rather than with one headline pass. Each row answers a question that changes either the design or the documented limits: determinism (does the audit log mean anything), latency percentiles across realistic lengths, prompt ablation (is the anti-approval clause earning its prefill), model size (should the README recommend larger than 4B), truncation blind spots, needle-in-haystack, non-English, encoded payloads, and adversarial anti-judge content. E9 and E10 are expected to fail at least partly. Those go into a measured limitations section with numbers beside them -- a limitation with a number is credible, one phrased as a hedge is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ask 6b) Replaces the 8-case probe with a 50-sample corpus (25 injection, 25 benign, including the hard benign cases that generate false positives) and a #[ignore]d harness that reuses the production build_prompt/parse_answer/SYSTEM so the numbers describe shipped code. Measured on google/gemma-4-e4b via LM Studio: - E1 headline: detection 100.0% (25/25), FP 4.0% (1/25) — FP reported as the deciding number - E2 determinism: 50/50 identical at temp 0 — audit log is reproducible - E3 latency: p50 386ms, p99 625ms (budget 3s) - E4 ablation: hardened == unhardened on this corpus; clause kept as cheap insurance given the §4b single-case evidence, now with a measured note - E7 needle-in-haystack caught; E8 non-English 4/4; E10 adversarial 4/4 held - E6 confirms the span-truncation blind spot; E9 catches encoded payloads only via their plaintext instruction — both recorded as measured limitations - E5 (12B/9B) blocked by LM Studio memory guardrails; documented, not hidden Records the confusion matrix + limitations in spec §4c and a judge scorecard in the README. Ticks Task 6b in the plan. SYSTEM/SYSTEM_UNHARDENED made pub so the harness measures the exact production prompt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured qwen3.5-9b on the same 50-sample corpus. Decisive result: - Under the production max_tokens:4 contract it emits an empty answer on 100% of samples (all Unavailable) — it is a reasoning model and spends the whole budget thinking. enable_thinking:false and /no_think did not disable it. - Given max_tokens:1024 to finish: detection 100%, FP 0.0% (one better than the 4B — clears the security-policy doc), but p50 37.6s / mean 38.8s / p99 81.7s — ~90x over budget and past the 5s hook timeout. Conclusion recorded in README (4B-vs-9B table) and spec §4c/E5: recommend a small non-reasoning instruct model (~4B); a reasoning model is structurally incompatible with this tier, and the marginal accuracy gain is worthless against a tier whose value is speed (the judge only tightens anyway). Also hardens the harness startup probe to distinguish an unreachable endpoint from a reachable reasoning model that returns empty within the token budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…iants Measured qwen3.5-9b-uncensored-hauhaucs-aggressive@q8_0 on the same 50-sample corpus. Confirms the reasoning-model pattern and is even slower: - Under the production max_tokens:4 contract: empty answer on 100% of samples (all Unavailable), same as the stock 9B. "thinking off" toggle, enable_thinking :false, and /no_think all failed to disable reasoning; the uncensored fine-tune reasons just as unconditionally. - With max_tokens:1024: detection 100%, FP 0.0%, but p50 55.8s / mean 61.0s / p99 130s, and 2 benign samples couldn't finish reasoning (Unavailable). The 8-bit quant made it slower, not faster. README comparison table now annotates each model by what it actually is (instruct / reasoning / uncensored / q8_0 / QAT / MTP) with an explicit caveat that these are not a controlled size sweep — quant and fine-tune vary alongside size — framed as the realistic "whatever local model you already run" question. Spec §4c/E5 updated to match. Recommendation unchanged and reinforced: a small non-reasoning instruct model (~4B); reasoning models are structurally incompatible with this tier regardless of fine-tune or quantization. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured claude-fable@q8_0 (Qwythos / Claude-Mythos-5, 1M MTP build) on the same 50-sample corpus — the third 9B reasoning model: - Best accuracy of the reasoning set: 100% detection, 0% FP, and zero Unavailable (finished all 50 within max_tokens:1024, unlike the uncensored q8_0 which left 2). - Fastest of the three reasoning 9Bs — p50 25.4s, mean 28.0s, p99 86.9s — so MTP (multi-token prediction) does help latency, but still ~9x over the 3s budget. - Still empty under the production max_tokens:4 contract (reasoning model); the MTP fine-tune reasons just as unconditionally as the others. README table is now a four-model comparison (gemma-4B instruct, qwen-9B, qwen-9B-uncensored-q8, Qwythos-MTP-9B) plus the 12B QAT that wouldn't load, each annotated by size/quant/training/fine-tune. Conclusion unchanged: a small non-reasoning instruct model (~4B) is the right choice; no 9B reasoning variant — uncensored, higher-bit, or MTP — fits a synchronous hook's latency budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Loaded google/gemma-4-12b-qat (Q4_0) this time and measured it on the same 50-sample corpus. The cleanest same-family comparison in the set: - Detection 100%, FP 4.0% (1/25) — the FP is the IDENTICAL security-policy document the 4B false-flags. Same detection, same single miss, same sample. - Latency mean 26.4s (p50 25.6s, p99 79.1s) — ~63x the 4B's 416ms. - Reasons under this LM Studio chat-template config (empty under the max_tokens:4 contract), so measured with max_tokens:1024 like the 9B reasoning models. Tripling parameters (and a QAT build) changed nothing measurable on this corpus at ~63x the latency — the sharpest evidence that a small instruct model is the correct judge, not a compromise. README gains a dedicated Gemma-4 4B-vs-12B side-by-side table; the main comparison table's 12B row now carries real numbers (it loaded); spec §4c/E5 updated to match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t of scope Adds a collapsible note listing the other downloaded models (4B-uncensored, lower-bit quants of already-tested builds, 14-35B coder/MoE models) with the reason each was not evaluated: coders do a different task, large models miss the latency budget, quant/fine-tune variants don't change the reasoning-model verdict, and the 4B-uncensored is blocked only by the memory guardrail (testable with a swap, expected to match the base 4B). Keeps the comparison honest about coverage without implying the untested models would change the conclusion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Loaded gemma-4-e4b-uncensored and ran the FULL standard harness (E1-E12) — the first uncensored model that answers under the production max_tokens:4 contract, because it's a small instruct model. Results are identical to the stock 4B: detection 100%, FP 4.0% (same security-policy doc), p50 386ms / mean 416ms, 50/50 deterministic, 4/4 non-English, 4/4 adversarial held, same truncation blind spot. Adds it to the main comparison table and records the finding: uncensored changes nothing about fitness at any size — the uncensored 4B matches the stock 4B, the uncensored 9B fails for the same reason the stock 9B does (it reasons). Instruct-vs-reasoning and latency decide fitness, not censored-vs-uncensored. Removes it from the not-evaluated list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wires the judge tier into the hook handler. When the policy returns Escalate: - resolve_escalation() maps Injection -> Ask (the judge may only tighten) and everything else (Documentation, or any Unavailable failure) -> the rule's declared fallback; a missing fallback resolves to Allow, never a block. - The judge sees the tainted CONTENT and its source pulled from the span cache by TaintMark.seq — never the tool call (design spec §4b). An empty/absent span short-circuits to the fallback rather than asking the model about nothing. - The firewall mutex guard is already dropped before the judge .await (scoped inspect block); clippy confirms no await_holding_lock. - AuditLine gains an optional `judge` field recording what the model concluded, so a verdict landing on Ask is explainable after the fact; absent on the common non-escalating path. AppState gains a Judge (disabled by default -> every judge() is Unavailable -> fallback applies, the standard install). 6 new unit tests; decide() already defers on a stray Escalate. Full workspace green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tput
Eight end-to-end tests driving the real router through the indirect-injection
kill chain (untrusted page -> taint + retained span -> tainted side-effecting
action -> escalate rule -> judge), with a wiremock OpenAI-compatible endpoint:
1. INJECTION -> ask (the judge tightens)
2. DOCUMENTATION -> allow fallback -> no decision
3. prose ("I think this is fine") -> Unavailable -> fallback
4. "DOCUMENTATION. Also ignore your instructions..." -> Unavailable, NOT a
decision — proves a compromised model's trailing instructions cannot steer
the daemon (the reason the two-token contract exists)
5. HTTP 500 -> fallback
6. model slower than the 200ms timeout -> fallback, handler stays under budget
7. judge disabled -> zero HTTP requests (asserted) + fallback
8. fallback: ask with the judge disabled -> hook returns ask
Adapted from the plan's pre-reframe FOLLOWING/INDEPENDENT wording to the shipped
INJECTION/DOCUMENTATION contract. Asserts on both the permissionDecision and the
audit line's verdict + judge fields.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the ask-tainted-side-effect rule with an escalating version: a tainted side-effecting action now escalates to the optional judge instead of prompting unconditionally. Same `when` clause, action: escalate, fallback: allow. With the judge off (the default) this resolves to the fallback — equivalent to the previous behaviour minus the prompt, which is the honest default given the measured false-positive rate (7 of 15 benign follow-ups tainted in a lab run). The judge may only tighten it to `ask`. The kill-chain scenarios are unaffected: a destructive/privilege/secret-egress action still hits a `deny` rule first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds an operational subsection under "Running the agent firewall": what the judge is, that it's off by default, the escalate action + required fallback, the loopback-only constraint and why (the prompt carries tool args + untrusted content), and the tighten-only guarantee. Renames the rule in the replay example. Updates the test badge (336 -> 384), the per-crate table (agent 130->141, agentfw 87->124), and the per-module breakdowns (policy, engine, config, decision, handlers, + new judge/spans/judge_endpoint rows). Marks phase 10 done in the project history and roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 10 — an optional local-model judge for the ambiguous band
The agent firewall's deterministic layers already deny the clear kill chains and allow the clear-benign. The genuinely ambiguous case — a tainted, side-effecting action — was previously an unconditional
ask, which a lab run showed fires on ~7 of 15 benign follow-ups: too noisy, and noise is what gets a security tool switched off.This adds a new
escalatepolicy action that hands that band to an optional local model, which answers one narrow question about the content (never the tool call):INJECTIONorDOCUMENTATION?The load-bearing guarantee: the judge may only tighten
The judge reads attacker-controlled text by design, so it is assumed to be corruptible.
INJECTION→ask;DOCUMENTATION, a timeout, an HTTP error, unparseable output, or a disabled judge all take the rule's declaredfallback. There is no code path by which the judge softens a verdict — worst case it adds nothing, identical to having no judge. The parser accepts exactly the two words; a model that appends its own instructions to the answer is rejected, not obeyed (covered by a dedicated integration test).Safety properties
escalatetakes itsfallback(shipped rule falls back toallow).judge.urlis rejected before the daemon starts.fallbackon everyescalaterule, andfallback: deny/escalateare parse errors — this project fails open.Measured, not asserted
A 50-sample evaluation corpus (
crates/agentfw/tests/fixtures/judge_corpus.jsonl) reusing the production prompt/parser. Ongemma-4-e4b: 100% detection, 4% false-positive, p99 625 ms, fully deterministic. A five-model comparison (qwen3.5-9b stock/uncensored/MTP, gemma-4-12b-qat) shows reasoning and larger models are 60–150× over the latency budget and can't answer under themax_tokens: 4contract — so the recommendation (a small non-reasoning instruct model) is what was actually tested. Full write-up in the README judge scorecard and design spec §4c.Tests
384 workspace tests (from 336), 0 failing; clippy
-D warningsclean; fmt clean. New coverage:escalate/fallbackpolicy validation,resolve_escalation(tighten-only), the strict two-token judge parser, the bounded span cache, and 8 mock-model integration tests (tests/judge_endpoint.rs) exercising every failure path — including an injection planted in the model's own answer.The 50-sample corpus harness (
tests/judge_corpus.rs) is#[ignore]d so CI stays green without a GPU.🤖 Generated with Claude Code