fix(testing): flag tests that can't fail and test-only production seams - #1779
Merged
Merged
Conversation
The testing reviewer now asks whether a test would still pass with the code under test broken, which covers expected values computed by the code under test, mocks or fixtures that supply the result, and negative cases rejected by a different guard. It also flags exports or hooks that exist only so a test can reach an internal the real entry point could exercise, and new tests that duplicate a contract an existing test already owns. ce-work holds new and changed tests to the same bar while writing them.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6ea8578638
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Address PR review feedback (#1779): the example rejected any call-shape assertion, which would forbid the only focused test for code whose contract is to call an external client or enqueue a job.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ce-code-review's testing reviewer now flags tests that would still pass if the code under test were broken, and production exports that exist only so a test can reach an internal.ce-workapplies both rules to the tests it writes.Before this, the reviewer's false-confidence check named only no-throw and truthiness assertions and heavy mocking. It missed expected values computed by the code under test, mocks or fixtures that supply the result the code should produce, and negative tests rejected by a different guard than the one they name. It had no check for test-only seams.
Design decisions
Validation
Before/after eval with
bun run test:skill-eval-cell, Claude and Codex, one run per host per arm.Testing reviewer, run against a staged diff with planted problems:
__testexport used only by testsformatPriceas the entry pointmoney.test.jsownsThe duplicate-coverage item did not fire. The planted case is arguably a legitimate higher-level check, so the fixture may be too weak to show it.
ce-work, given a feature where one option is exporting a private helper to test it: neither host added a test-only export in either arm. All four runs tested through the public function with literal expected values. That shows no regression, and no measurable effect on these hosts.bun run testandbun run release:validatepass.Security Disclosure
No security-relevant changes.
Agent Disclosure
Claude Code · claude-opus-5-5[1m]