sidetrack

When you ask a coding agent for one thing, it often does that thing plus four other things.

A Claude Code skill that diffs what an agent actually changed against what you asked for, and flags every out-of-scope edit before you accept the diff.

Across a 41-scenario hand-labeled benchmark (20 diffs where the agent did exactly what was asked, 21 with real scope drift: a deleted test, a smuggled dependency, a stripped try/except, a hardcoded secret), sidetrack's detector layer scored 1.000 precision and 1.000 recall on every out-of-scope edit and mechanical red flag, with zero false alarms on the clean diffs, while cutting what a reviewer has to read by a median of 34.3% versus the raw diff (up to 64.1% on messier ones).

The problem

Ask a coding agent for one change and it often makes that change plus a few more: a new dependency slipped into package.json, an unrelated file rewritten "while I was in there," a failing test deleted instead of fixed, a CI file quietly edited. None of that is malicious, it is just what an agent does when nothing is checking scope. By the time you review the diff, the extra work is buried in it and easy to rubber-stamp along with the part you actually wanted.

Sidetrack is not a linter and not a test suite. It is a scope check: given the request you made, does the diff match it, and if not, exactly where does it drift.

How it works

• scripts/scope_report.py is deterministic and does the mechanical work only: collect every changed file, compute churn, group by directory, and run pattern-based detectors for known red flags. It never judges intent and never calls a model. Same diff in, same output out, every time.
• SKILL.md does the judgment work: restating what you actually asked for as a scope boundary, classifying each file against that boundary, and deciding accept / accept with edits / reject and split. That judgment cannot be automated with regexes, so it stays with the model, grounded in the script's facts rather than a re-read of the raw diff.

What it detects

The script never says "this is wrong." It says "this file changed, here is where, here is why it's worth a second look." Severity is about how often the pattern signals a real problem, not about how bad this specific instance necessarily is.

Flag	Severity	What trips it
TEST_DELETED	high	A file matching a test naming/path convention (test_*.py, *_test.py, *.spec.js, tests/, spec/, __tests__/) was deleted. Also fires when a test is renamed to a non-test path, since rename detection is intentionally off (see Limitations).
TEST_SKIPPED	high	An added line matches a skip/focus marker: @pytest.mark.skip, @pytest.mark.xfail, unittest.skip, .skip(, .only(, xit(, fit(, xdescribe(, fdescribe(, t.Skip(.
ASSERTION_REMOVED	high	More assertion-shaped lines (assert, assertEqual, expect(, .should., etc.) were removed from a file than were added back.
ERROR_HANDLING_REMOVED	high	More error-handling lines (try:, except, catch(, .catch(, rescue, if err != nil) were removed than added.
CI_CONFIG_TOUCHED	high	A file under .github/workflows/, .circleci/, a Dockerfile*, or a known CI/deploy config (.gitlab-ci.yml, .travis.yml, Jenkinsfile, vercel.json, netlify.toml, docker-compose.yml) was touched.
SECRET_FILE	high	A credential-shaped filename was touched: .env*, credentials.json, secrets.yml, id_rsa, *.pem, *.key.
SECRET_PATTERN	high	An added line matches a secret-shaped pattern: a PEM private key header, an AWS access key (AKIA...), an OpenAI-style key (sk-...), a GitHub token (ghp_...), a Slack token (xox...), or a quoted literal assigned to something named api_key/secret/password/token.
DEPENDENCY_MANIFEST_CHANGED	medium	package.json, requirements.txt, pyproject.toml, go.mod, Cargo.toml, or Gemfile changed.
LOCKFILE_CHURN	medium	A lockfile changed (package-lock.json, yarn.lock, pnpm-lock.yaml, poetry.lock, Cargo.lock, Gemfile.lock, uv.lock, Pipfile.lock, composer.lock).
GOVERNANCE_FILE_CHANGED	medium	.gitignore or a LICENSE* file changed.
WHOLESALE_REWRITE	medium	A modified file (at least 10 lines before the change) had added+removed lines cover 70% or more of its original size.
LARGE_BLOB_ADDED	medium	A new file is binary, over 500KB, or lives under node_modules/, vendor/, dist/, or build/.
TODO_DELETED	low	A TODO/FIXME/XXX marker was removed from a file without an equivalent marker added back.

Benchmarks

bench/corpus.py is a hand-labeled, 41-scenario corpus of realistic agent-produced diffs, built the same way as this portfolio's other benchmarked tools (see safedeps/bench/): 20 clean scenarios where the agent did exactly what was asked (a one-line off-by-one fix, a docstring typo, a new test), and 21 scenarios with genuine scope drift modeled on real agent behavior (a test quietly skipped while fixing an unrelated bug, a dependency smuggled into requirements.txt, a whole module rewritten when one function needed a fix, a hardcoded key dropped into a new file). Every scenario carries a hand-verified ground-truth label: exactly which (file, flag_type) pairs scope_report.py should raise. bench/run_bench.py replays each scenario as a real temporary git repo and runs the real script against it, exactly as a user would. No model calls -- this benchmarks the deterministic detector layer only.

$ python3 bench/run_bench.py
precision=1.000 recall=1.000 f1=1.000 (tp=33 fp=0 fn=0 tn=799)
all scenario labels matched actual output

wrote bench/results.md

Precision: 1.000, Recall: 1.000, F1: 1.000 across all 13 flag types on this corpus (0 false positives, 0 false negatives, out of 33 true positives and 799 true negatives at the per-file-per-flag-type level). Every one of the 13 detectors scored a clean 1.000/1.000 individually too; the full per-detector table is in bench/results.md.

Token compression (the number that decides whether reading the report beats reading the diff): median raw git diff was 154 tokens, median scope report was 93 tokens, a 34.3% median reduction, ranging 6.7% to 64.1% depending on the diff (tiktoken cl100k_base; see bench/results.md).

Honest caveat: this corpus understates the compression on real sessions. Scenarios here are small (3-15 line, mostly single-file diffs) by design, because a bigger diff is harder to hand-verify a ground-truth label against. The report's overhead is close to fixed per file (one status/churn line, one directory-grouping line) while raw diff text scales with how many lines actually changed. On a genuinely large or multi-file agent session, the reduction should be larger than what is measured here, not smaller, but that is not a number this benchmark actually ran, so it is not claimed as one. Where compression helps least: a single one-line fix to a short file (the smallest scenario in this corpus reduces only 6.7%, because there is barely any diff to compress in the first place).

Wall-clock: median 117.0ms, p95 131.3ms per invocation (includes Python process startup), about 8.6 diffs/sec sustained on the machine this was run on (wall-clock varies run to run more than the other numbers here; re-run bench/run_bench.py for a fresh measurement). Full numbers in bench/results.md.

Reproduce: pip install -r bench/requirements.txt (optional, tiktoken only, bench-only dependency; falls back to a labeled chars/4 estimate without it), then python3 bench/run_bench.py.

Install

As a Claude Code skill (personal, all projects):

mkdir -p ~/.claude/skills/sidetrack
cp SKILL.md ~/.claude/skills/sidetrack/
cp -r scripts ~/.claude/skills/sidetrack/
cp -r .claude-plugin ~/.claude/skills/sidetrack/

Restart Claude Code (or start a new session) if ~/.claude/skills/ did not already exist. After that, ask "did the agent go out of scope" or "/sidetrack" in any project and Claude picks it up automatically. To install it for one project only, use .claude/skills/sidetrack/ inside that repo instead of ~/.claude/skills/.

The .claude-plugin/plugin.json in that copy is what makes the badge above literally true: it loads the same install as sidetrack@skills-dir, per Claude Code's skills-directory plugin loading. No marketplace, no separate install command. Skip that one cp if you only want the plain skill.

As a plain CLI, no Claude Code required:

python3 scripts/scope_report.py                  # staged + unstaged vs HEAD
python3 scripts/scope_report.py --base main       # main vs current working tree
python3 scripts/scope_report.py --base "main HEAD"  # two fixed commits, no working tree
python3 scripts/scope_report.py --json            # machine-readable

As a pre-commit hook or CI gate, using the exit code:

python3 /path/to/scope_report.py --fail-on high || exit 1

Exit 0 means no flag at or above the chosen severity; exit 1 means at least one was found. --fail-on accepts low, medium, or high (default high).

Requires Python 3 only. No pip install, no network call, no API key.

Worked example

Given a repo where a coding agent was asked to harden login() against silently swallowed request errors, and instead deleted the error handling, weakened a test assertion, skipped another test, added a dependency, touched CI, and dropped a hardcoded key into a new file:

Sidetrack scope report (base: HEAD)
Files changed: 8  (+16 / -10)

MODIFIED  .github/workflows/ci.yml                   (+0 / -4)
MODIFIED  .gitignore                                 (+1 / -0)
NEW       assets/bundle.min.js                       (+1 / -0)
NEW       package-lock.json                          (+5 / -0)
MODIFIED  requirements.txt                           (+1 / -0)
MODIFIED  src/auth/login.py                          (+5 / -5)
NEW       src/config/settings.py                     (+1 / -0)
MODIFIED  tests/test_login.py                        (+2 / -1)

By directory:
  .                        3 files  (+7 / -0)
  .github/workflows        1 file  (+0 / -4)
  assets                   1 file  (+1 / -0)
  src/auth                 1 file  (+5 / -5)
  src/config               1 file  (+1 / -0)
  tests                    1 file  (+2 / -1)

RED FLAGS (4 high, 5 medium, 1 low)
  [HIGH  ] CI_CONFIG_TOUCHED          .github/workflows/ci.yml:3          CI/deploy config touched
  [HIGH  ] ERROR_HANDLING_REMOVED     src/auth/login.py:5                 removed error handling: try:
  [HIGH  ] SECRET_PATTERN             src/config/settings.py:1            line matches a secret-like pattern
  [HIGH  ] TEST_SKIPPED               tests/test_login.py:9               added pytest.mark.skip: @pytest.mark.skip(reason="flaky in CI")
  [MEDIUM] GOVERNANCE_FILE_CHANGED    .gitignore:2                        .gitignore changed
  [MEDIUM] LARGE_BLOB_ADDED           assets/bundle.min.js                large/generated/vendored file added (600000 bytes)
  [MEDIUM] LOCKFILE_CHURN             package-lock.json:1                 package-lock.json lockfile churned
  [MEDIUM] DEPENDENCY_MANIFEST_CHANGED requirements.txt:2                 requirements.txt changed
  [MEDIUM] WHOLESALE_REWRITE          src/auth/login.py                  10 lines churned of ~14 original (71%)
  [LOW   ] TODO_DELETED               src/auth/login.py:13                marker removed without resolving

This is real output from scripts/scope_report.py (the same scenario also lives in bench/corpus.py as harden_login_error_handling). Claude then reads this report, classifies each file against the actual request, and lands on a verdict (in this case REJECT AND SPLIT, since none of the eight changed files match "harden error handling," and several actively invert it). The full worked classification and verdict format is in SKILL.md.

Reproduce the self-checks

python3 scripts/scope_report.py --selftest

Runs 14 named tests, each covering one detector in both directions (fires / does not fire) plus one integration check that the git-diff parser assigns correct line numbers, and prints PASS/FAIL per test with a final count. Exits non-zero on any failure.

Limitations

• It cannot know intent. A try/except removal that fixes a real bug looks identical to the script as one that reintroduces a bug. Severity reflects how often a pattern is worth a second look, not a verdict on this specific instance.
• It is heuristic pattern matching, not a parser. All detectors work on the literal text of added/removed diff lines and file paths. There is no AST, no type information, no cross-file reasoning.
• Legitimate refactors will trip it. A genuine large-scale rename, a real test suite cleanup, or a planned dependency bump will show up as WHOLESALE_REWRITE, TEST_DELETED, or DEPENDENCY_MANIFEST_CHANGED exactly like an unwanted one. The skill's job in SKILL.md is to read the flag against the actual request and decide, not to treat every flag as an automatic rejection.
• The benchmark's 1.000/1.000 is real but on a corpus I wrote and hand-verified against this same tool. It is not a blind, third-party-labeled dataset, and it is small (41 scenarios). Treat it as evidence the detectors do what they are documented to do on realistic-shaped diffs, not as a claim that nothing in the wild will slip through or false-positive. In plain terms: the score measures detector consistency against scenarios I hand-labeled myself, not measured performance on unseen, real-world diffs from someone else's sessions. A perfect score on your own test set is weaker evidence than a perfect score on someone else's; weigh it accordingly.
• Token compression is measured on deliberately small diffs (median reduction 34.3%, low end 6.7%) -- see the Benchmarks caveat above for why real, larger sessions should compress more, not less, and why that larger number is not claimed here without having run it.
• Rename detection is deliberately off (git diff --no-renames). A file rename shows up as a delete of the old path plus an add of the new one. This is intentional: it is what makes "a test renamed away from a test-looking name" show up as TEST_DELETED on the old path, without a separate rename-tracking detector. The tradeoff is that a large, legitimate rename-only commit shows as churn on two paths instead of one clean rename. Upgrade to rename-aware diffing (git diff -M) if that false-positive rate becomes a real problem.
• The secret pattern detector is a heuristic, not a secret scanner. It will miss secrets that do not match its patterns and can false-positive on long test fixtures or example strings that happen to look key-shaped. Use a real secret scanner (gitleaks, trufflehog) for anything security-critical; this flag exists to catch the obvious case during a scope review, not to replace one.
• Assertion/error-handling detection is a line-count heuristic, not control-flow analysis. Moving a try/except to wrap a larger block, or replacing one broad assert with several narrower ones, can register as "removed" even when the change is a genuine improvement.

License

MIT, see LICENSE.