Skip to content

test: cover hook decisions with 65 behavioural tests - #17

Merged
skateddu merged 1 commit into
mainfrom
test/hook-behavior-suite
Aug 13, 2026
Merged

test: cover hook decisions with 65 behavioural tests#17
skateddu merged 1 commit into
mainfrom
test/hook-behavior-suite

Conversation

@skateddu

Copy link
Copy Markdown
Owner

Why

The hooks are the only enforced guardrails in this setup — everything else is advice Claude may or may not follow. Both hook bugs this project has shipped were logic errors, not shell errors:

Neither shellcheck nor scripts/validate_config.py can see that class of fault. Both scripts were syntactically perfect and structurally wired; they just decided the wrong thing.

Approach

Each test runs the real script in a subprocess against a crafted payload and asserts on the decision it emits. Nothing is mocked — "the hook silently stopped firing" is precisely the regression worth catching, and a mock cannot fail that way.

tests/hook_harness.py wraps the output in a HookResult exposing permission_decision, updated_command, decision, reason and is_silent, so tests read as statements about behaviour rather than JSON spelunking.

File Asserts
test_protect_main_hook.py 11 dangerous commands denied, 9 safe ones passed through
test_enforce_uv_hook.py 7 rewrites to uv run ..., 6 denials, 7 pass-throughs
test_guard_secrets_hook.py 7 credential shapes blocked, 5 prose-about-credentials allowed
test_session_start_hook.py Missing .venv/stale uv.lock reported, healthy project silent
test_verify_hook.py stop_hook_active loop guard, non-Python-project exit
test_auto_lint_hook.py Non-Python, deleted, and path-less payloads ignored

65 tests, ~10s.

Proof they work

A test suite that passes proves nothing on its own, so I mutated the hook back to the #11 bug — restoring the \b boundary — and re-ran:

8 failed, 14 passed
FAILED ...denies_dangerous_command[rm -rf /]
FAILED ...denies_dangerous_command[rm -rf ~]
FAILED ...denies_dangerous_command[rm -rf .]
FAILED ...allows_safe_command[rm -rf .git]
FAILED ...allows_safe_command[rm -rf ~/tmp-build-dir]

Failures in both directions, matching the original bug's profile exactly. The hook was then restored.

Gaps I chose to leave, and why

Stated in the README rather than papered over:

  • verify.sh's ruff/pytest body is untested. Running it from inside the suite would invoke pytest recursively; provisioning a throwaway uv project would need a network install on every CI run. Only its guard clauses are covered.
  • auto-lint.sh's formatting body is untested, because whether ruff acts on a file depends on the surrounding project's include config — an assertion here would test that config, not the hook.
  • Hook wiring in settings.json is still unverified. Nothing checks that a matcher and if condition route the events you expect. That is exactly what broke in fix(hooks): correct PreToolUse if-condition syntax and add explicit settings defaults #10, and it remains verifiable only by running Claude Code. Worth knowing this PR does not close that hole.

Note on the test credentials

test_guard_secrets_hook.py needs credential-shaped strings. They are fake — one is AWS's own published documentation example — and assembled at runtime from fragments so the repository never contains a contiguous string matching a real credential format, which would otherwise trip push protection and secret scanners.

Also

  • CI gains a pytest step.
  • ruff's include and known-first-party extended to cover tests/.

🤖 Generated with Claude Code

Both hook bugs this project has shipped were logic errors, not shell
errors: a PreToolUse condition that silently disabled two hooks, and a
regex boundary that let `rm -rf /` through while blocking `rm -rf .git`.
Neither shellcheck nor the config validator can see that class of fault.

Each test runs the real script in a subprocess against a crafted payload
and asserts on the decision it emits. Validated by mutation: restoring
the old \b boundary in protect-main.sh turns 8 tests red in both
directions, matching the original bug exactly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@skateddu
skateddu merged commit 6527815 into main Aug 13, 2026
1 check passed
@skateddu
skateddu deleted the test/hook-behavior-suite branch August 13, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant