Skip to content

Harden CLI resolver regression checks - #29

Merged
raghubetina merged 1 commit into
mainfrom
codex/harden-cli-resolver-tests-20260812
Aug 12, 2026
Merged

Harden CLI resolver regression checks#29
raghubetina merged 1 commit into
mainfrom
codex/harden-cli-resolver-tests-20260812

Conversation

@raghubetina

Copy link
Copy Markdown
Contributor

Summary

  • make the resolver test inspect every First Draft CLI command block, not only blocks already containing the resolver
  • reject bare operational CLI invocations across the packaged Skill and references
  • align evaluation language with the project-wrapper resolver contract

Scope

This is a test-and-evaluation-only follow-up to #28. It does not change Skill, package, catalog, or published plugin bytes, and Drawing Board does not need a new Skills pin.

Verification

  • npm run check (64/64)
  • node --test test/repository.test.mjs (23/23)
  • git diff --check

Independent review is in progress.

Select every CLI subcommand block before asserting the project resolver
so a future bare command cannot evade the guard. Keep model-graded eval
expectations aligned with the resolver-backed Skill language.
@raghubetina
raghubetina merged commit 263326a into main Aug 12, 2026
2 checks passed
@raghubetina
raghubetina deleted the codex/harden-cli-resolver-tests-20260812 branch August 12, 2026 18:43
@raghubetina

Copy link
Copy Markdown
Contributor Author

Post-merge review, hosted CI green; test-and-evaluation-only follow-up to #28, no packaged bytes changed. The
fix targets a real regression-test blind spot: the resolver test previously inspected only command blocks that
already contained the resolver wrapper, so a newly added block calling bare firstdraft would sail past the
very test meant to catch it. Inverting the iteration (inspect every First Draft CLI command block, then assert
the wrapper) closes the loophole, and the accompanying rejection of bare operational invocations across the
packaged Skill and references makes the contract uniform. Evaluation language aligned to the same contract.
This is the correct shape for a guard: enumerate the population first, then check the property, never the
reverse.

@raghubetina

Copy link
Copy Markdown
Contributor Author

There is a classic way to write a test that always passes, and #28's original resolver test fell into it: it
found all the places that already used the safe pattern and checked that they used it. Any new code that
skipped the pattern was invisible, because the test's search was keyed on the pattern itself.

Abstractly: if the property is "every X must have Y," your test must enumerate the Xs independently of Y.
Enumerate by Y and you have proved "everything that has Y, has Y." This PR flips the resolver test to walk
every First Draft CLI command block in the packaged Skill and references, then assert each one goes through
the project-owned wrapper. Now a contributor adding a bare firstdraft plan push to any reference file gets
a red build.

You have this bug in Rails test suites more often than you would think. A spec that loops over
Payment.processors.select(&:retryable?) asserting retry behavior proves nothing about a processor that
forgot to declare retryable. The fix is always the same: derive the collection from the source of truth
(all files, all subclasses, all routes) and assert the property, letting the test discover the cases you have
not thought about yet.

The PR also shows the discipline of scope labeling: it changes tests and evaluation prose only, states that
packaged bytes are untouched, and therefore needs no release machinery. Knowing which of your changes are
inert to the shipped artifact, and saying so, is what lets reviewers calibrate their attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant