feat: agent-attack benchmark & scorecard (phase 12) - #19
Merged
Merged
Conversation
…ark) Scores the real AgentFirewall over a hand-authored corpus of agent sessions, reporting detection rate + FPR + per-category to the text layer's two-number standard. Detection = any Deny/Ask (Ask is the agent layer's interruption). Honesty framing: hand-authored corpus measures known-shape coverage, not novel- attack generalization. 6 TDD tasks: session loader, replay guard, corpus evaluation, the corpus itself, the --agent scorecard mode, README + PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Session corpus (compact EventKind per event, wrapped into real AgentEvents; note Provenance is tagged 'origin'), a replay guard that flags on Deny/Ask/Escalate, and corpus evaluation reusing Confusion for detection rate + FPR + per-category. root_tools lets subagent-escalation sessions register the parent grant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
40 hand-authored sessions (19 attack across 7 categories, 21 benign incl. the hard tainted-but-benign / credential-path-read cases). The --agent mode prints detection rate + FPR + per-category + named misses/FPs. Guard resolves Escalate to its fallback (the shipped no-judge behavior), so a tainted benign side-effect is not a false positive. Secret-egress samples use high-entropy generic values (no real vendor tokens) so the corpus is safe to commit. Measured: 0.0% FPR (0/21), 73.7% detection (14/19). Misses are honest gaps (taint-fingerprint granularity; escalate-before-unknown-host ordering without a judge; weak-injection description coverage) — kept, not hidden. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 12 — the agent-attack benchmark
Completes the v0.3 agent-firewall arc: phases 08–11 built and wired the agent layer; this measures it, to the text layer's two-number standard.
What it is
A 40-session corpus (19 attack across 7 categories, 21 benign) replayed through the real
AgentFirewalland shipped policy. Each session is a sequence ofAgentEvents; a session is "caught" if any event would interrupt the agent (DenyorAsk).Escalateresolves to its fallback — exactly as the shipped daemon does with no judge configured (the default install).The numbers (no judge)
Per-category: secret-egress 4/4, unknown-host 3/3, subagent-escalation 2/2, pii-egress 1/1, indirect-injection 3/5, mcp-poisoning 1/2, destructive-from-taint 0/2.
Honesty framing (§4 of the spec)
Unlike the text layer's four public datasets, there is no established public agent-attack benchmark in this event form, so the corpus is hand-authored — it measures coverage of known attack shapes, not generalization to novel attacks. Stated plainly in the scorecard and README.
The misses are kept, not hidden (deleting them for a 100% figure is the self-flattering test the methodology warns against), and they're instructive:
escalate → allowwith no judge — precisely the band the judge tier (phase 10) exists to catch.The benign half — including the hard cases (fetch-then-act, credential-path reads without egress, allowlisted egress) — produced zero false alarms.
Tests
422 workspace tests (from 415), 0 failing; clippy
-D warningsclean; fmt clean. New: session loader, replay guard (incl. the tainted-benign / Escalate-fallback case), corpus evaluation.Note: secret-egress corpus samples use high-entropy generic values (no real vendor tokens) so the corpus is safe to commit.
Reproduce:
cargo run -p llm-firewall-bench -- --agent crates/bench/corpora/agent_sessions.jsonlDesign:
docs/superpowers/specs/2026-07-30-agent-firewall-12-benchmark-design.md. Plan:docs/superpowers/plans/2026-07-30-agent-firewall-12-benchmark.md.🤖 Generated with Claude Code