feat: model-agnostic output moderation (phase 13) - #20
Merged
Merged
Conversation
The reverse proxy as a model-agnostic output enforcement point: restrict harmful LLM responses regardless of backend (incl. uncensored models). Off by default, flag-first; a dedicated ml-gated moderation gate in the output path (no policy- YAML surgery); the DECIDING metric is the over-block FPR on benign SECURITY content (the user's own domain), headlined. 7 TDD tasks: config, gate, wiring, e2e test, harmful + benign-security corpora, over-block scorecard, README + PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
verdict_from_scores (pure, threshold + safe-label + category-allowlist logic) and ModerationGate (loads the binary harmful/safe classifier under the new `ml` feature; a clean no-op with a warning otherwise). 7 unit tests; both default and ml builds compile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…g audits) AppState gains the ModerationGate; both OpenAI and Anthropic non-streaming response paths run it after the existing output scan — block returns the refusal message, flag audits and forwards. No-op unless enabled + ml + model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The prompt-injection classifier is trained on user prompts that try to hijack a model. A model's own reply is not such a prompt, so running the rule on the response path is a category error. Measured on the phase-13 output corpus it caught 0% of harmful replies while over-blocking 16% of legitimate security answers -- defensive phishing guidance, password-policy advice and authorized pentest methodology were all blocked as 'ML-detected prompt injection'. Both injection rules needed scoping: 'detector: injection' matches 'injection.ml' by segment prefix, so leaving block-high-injection unscoped silently re-blocked what block-ml-injection was scoped to allow. Indirect injection is not lost -- retrieved content and tool results are projected as Direction::Input and remain covered. The agent layer has its own policy (crates/agent/policies/agent-default.yaml) and is untouched. New tests assert against the real shipped policies/default.yaml, so a change to the deployed defaults cannot pass silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… path CoreGuard hardcoded Direction::Input, so an output-moderation corpus was scored against the input rules -- it could not see direction-scoped policy at all. The builtin bench policy now mirrors the shipped policies/default.yaml for the same reason: a bench that measures rules nobody deploys is not a measurement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bench used the detector default (0.5) while the proxy's output_moderation.threshold ships at 0.8, so the scorecard described a configuration nobody deploys. Default stays 0.5 so the published JBB number is unchanged (re-verified: 58.0%). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Headlines the over-block rate rather than the catch rate, documents the injection-rule scoping defect the corpus exposed, and records the reproduce command with the two flags that make the measurement honest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the firewall a model-agnostic output enforcement point: harmful replies are refused at the proxy regardless of which backend produced them, so an operator can hold a policy line across models they don't control — including self-hosted or deliberately uncensored ones. Off by default,
flagbeforeblock.Measured
50-example corpus of model replies (25 harmful / 25 legitimate security answers), evaluated on the response path at the shipped
threshold: 0.8:The over-block column is the headline, not the catch rate. A filter for "hacking content" that blocks the pentest report, the CTF writeup and the incident-response runbook has made the tool useless to the people most likely to run it. The benign half is deliberately the hard half — authorized pentest methodology, OSINT procedure, CVE explanation, YARA rules, SQLi remediation, social-engineering scoping.
The defect this surfaced
Building the corpus exposed a real bug in the shipped policy. The prompt-injection rules were unscoped, so the DeBERTa injection classifier also ran on model replies — where it caught 0% of harmful content while over-blocking 16% of legitimate security answers:
The classifier is trained on user prompts that try to hijack a model; a model's own reply is not one. Both injection rules are now scoped to
direction: input—detector: injectionmatchesinjection.mlby segment prefix, so scoping one without the other changes nothing, which is why the first fix attempt did nothing. Pinned by tests that read the realpolicies/default.yamlrather than an inline fixture.Indirect injection is unaffected: retrieved content and tool results are projected as
Direction::Input. The agent layer has its own policy and is untouched. Input-path scorecard re-verified unchanged — safe-guard 84.3% @ 0.2% FPR, JBB moderation 58.0%.Bench honesty fixes
Two flags exist because without them this corpus was scored against rules nobody deploys:
--direction output—CoreGuardhardcodedDirection::Input, so it could not see direction-scoped policy at all.--moderation-threshold— the bench used the detector default (0.5) while the proxy ships 0.8. Default stays 0.5 so published numbers are unchanged.--threshold 255disables the bench's risk-score shortcut so the result reflects the policy decision a deployed proxy makes.Limits, stated plainly
25 harmful examples is a small hand-authored corpus measuring known-shape harm, not novel evasion — an attacker phrasing a harmful request as a CTF writeup is exactly the gap these numbers cannot see. Harm categories are broad, not exhaustive. No claim is made about illegal material. This is a policy layer, not a safety guarantee.
Verification
435 tests passing (0 failing), clippy clean in both default and
--all-featuresmodes, fmt clean,--features mlbuilds.Design record:
docs/superpowers/specs/2026-07-31-phase-13-output-moderation-design.md🤖 Generated with Claude Code