Skip to content

fix(testing): flag tests that can't fail and test-only production seams - #1779

Merged
tmchow merged 2 commits into
mainfrom
tmchow/explore-test-audit-skill
Sep 24, 2026
Merged

tmchow merged 2 commits into
mainfrom
tmchow/explore-test-audit-skill

Conversation

@tmchow

@tmchow tmchow commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

ce-code-review's testing reviewer now flags tests that would still pass if the code under test were broken, and production exports that exist only so a test can reach an internal. ce-work applies both rules to the tests it writes.

Before this, the reviewer's false-confidence check named only no-throw and truthiness assertions and heavy mocking. It missed expected values computed by the code under test, mocks or fixtures that supply the result the code should produce, and negative tests rejected by a different guard than the one they name. It had no check for test-only seams.

Design decisions

  • The false-confidence item is restated as one condition (would the test still pass with the code broken?), with the failure shapes as examples, instead of one new item per shape.
  • The test-only seam check excludes hooks that control what no entry point can, such as time or randomness, because the same reviewer asks for fake clocks.
  • A new duplicate-coverage item flags a new test that restates a contract an existing test already owns, unless it covers a risk the owning test can't reach.

Validation

Before/after eval with bun run test:skill-eval-cell, Claude and Codex, one run per host per arm.

Testing reviewer, run against a staged diff with planted problems:

Planted problem Before After
Expected value computed by the code under test Both hosts caught it Both caught it
Negative test passing through the wrong guard Both caught it Both caught it
__test export used only by tests Neither flagged it as its own finding Both flagged it and named formatPrice as the entry point
Injected clock (legitimate, should not be flagged) Not flagged Not flagged
New test duplicating a contract money.test.js owns Not flagged Not flagged

The duplicate-coverage item did not fire. The planted case is arguably a legitimate higher-level check, so the fixture may be too weak to show it.

ce-work, given a feature where one option is exporting a private helper to test it: neither host added a test-only export in either arm. All four runs tested through the public function with literal expected values. That shows no regression, and no measurable effect on these hosts.

bun run test and bun run release:validate pass.

Security Disclosure

No security-relevant changes.

Agent Disclosure

  • Model: Claude Code · claude-opus-5-5[1m]

The testing reviewer now asks whether a test would still pass with the code
under test broken, which covers expected values computed by the code under
test, mocks or fixtures that supply the result, and negative cases rejected
by a different guard. It also flags exports or hooks that exist only so a
test can reach an internal the real entry point could exercise, and new
tests that duplicate a contract an existing test already owns.

ce-work holds new and changed tests to the same bar while writing them.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-24T17:36:45.908090Z 6ea8578 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6ea8578638

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/ce-work/references/implementation-loop.md Outdated
Address PR review feedback (#1779): the example rejected any call-shape
assertion, which would forbid the only focused test for code whose
contract is to call an external client or enqueue a job.
@tmchow
tmchow merged commit 868da03 into main Sep 24, 2026
6 checks passed
@github-actions github-actions Bot mentioned this pull request Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant