fix(claims): a nonzero exit does not refute an observation the claim promised - #193
Merged
Conversation
…promised `gradeFor` consulted the exit code before the expectation, so a negative control that correctly refuses, exits nonzero, and prints the refusal it promised was graded `contradicted` and `expect` was dead code on every such claim (#192, two refusal gates in one director run). When the output contains the expectation, the claim is verified whatever the exit status. Two guards keep their precedence: an unreached-input signature in the output still grades `unrunnable`, and an expectation that itself names a signature keeps the existing refutation rule, so a run that never happened is not rescued by a coincidental substring. Verification: vitest src/claim-evidence.test.ts (57 passed; the one failure, 'grades a check that finishes in budget', fails identically on main on macOS because `wc -c` pads its output); tsc --noEmit; biome. Closes #192
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gradeForread the exit code before the expectation, so a refusal gate that correctly refuses, exits nonzero, and prints the promisedREFUSED=1was gradedcontradicted;expectwas dead code for every negative control.unrunnable, and an expectation that itself names a signature keeps the existing refutation rule (thestill refutes a claim that PREDICTED the errortest is unchanged).Closes #192
Verification
vitest run src/claim-evidence.test.ts: 57 passed, 1 failed. The failure (grades a check that finishes in budget) fails identically on main on macOS becausewc -cpads its output; it passes on Linux CI.tsc --noEmit,biome check