ci: measure scripts/ in the coverage gate — .coveragerc source (only file) - #818
Conversation
.coveragerc set 'source = tests', so the coverage run in .github/workflows/unit-tests.yml measured only tests/. scripts/ was invisible to 'coverage report --fail-under=75' and to the codecov patch target, even though 8 of the 10 scripts already have unit tests under tests/unit/. A coverage regression in a CI gate script — or deleting its test file outright — would not move the gated number. Adding scripts to the source list closes that hole. Measured with the workflow's own command: scripts/ is at 81% and the repo-wide total is 84%, so the existing --fail-under=75 threshold is unchanged and still passes. 'coverage xml' now also reports scripts/ to Codecov. Closes #817 Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
82fd64a to
f6f7a2f
Compare
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
castrojo
left a comment
There was a problem hiding this comment.
Backlog triage pass. I have not approved and have not merged or enqueued anything: .github/copilot-instructions.md says agents never approve, merge, or enqueue, and AGENTS.md:136,163 + docs/skills/meta/human-gates/SKILL.md make merge a human gate. This is a review comment only.
Checked the thing that could plausibly break here, which is whether widening .coveragerc from source = tests to source = tests,scripts drops the total below the --fail-under=75 in unit-tests.yml:46.
It cannot. unit-tests.yml runs coverage run -m pytest -n auto tests/unit/ with COVERAGE_PROCESS_START: .coveragerc, so the change is exactly in scope, and tests/unit/ already covers scripts/ — test_assert_kde_passed.py, test_quarantine_age.py, test_e2e_summary.py, test_generate_badges.py, test_parse_results.py, test_validate_docs.py, test_install_kde_webdriver.py, test_update_coverage_snapshot.py and test_generate_skill_index.py are all present. Even the impossible worst case — scripts/ at 0% across ~800 of ~17.5k statements — lands at ~80.6%, above the gate.
One-line change, no skill-doc required (.coveragerc is not tests/**, .github/workflows/**, .github/actions/** or scripts/**). All six checks green.
Outstanding: one more approving review, plus the repo-wide ghost-lab outage.
Gate status for this PR
- Ruleset
main — merge queuerequires 2 approving reviews (required_approving_review_count: 2,dismiss_stale_reviews_on_push: true,require_last_push_approval: true, andbypass_actorsis empty — nobody can bypass). Required checks:Lint & syntax,Behave dry-run,pytest. ghost-labis red, and that is not this PR's fault. Every lab workflow in the Argo namespace isFailed/Errorsince 2026-09-15 acrosstestsuite,common,knuckleandbluefin-lts; the last success anywhere was testsuite #790 on 2026-09-15.docs/skills/meta/human-gates/SKILL.md:80-107still makesghost-laba merge gate, so I am naming it as outstanding rather than waiving it.
Danathar
left a comment
There was a problem hiding this comment.
What I checked: one-line change, source = tests → source = tests,scripts in .coveragerc. The concern with widening the measured set is that the 75% gate might start failing; the pytest job on this PR ran coverage report --fail-under=75 with scripts/ included and reports TOTAL 85%, so the gate holds with headroom. From here on a coverage regression in a CI-gate script (or deleting its test file) moves the number, which is the point. Closes #817.
castrojo
left a comment
There was a problem hiding this comment.
Approve — second review to reach +2.
One line: .coveragerc source = tests → source = tests,scripts.
The gap is real. unit-tests.yml runs coverage run -m pytest tests/unit/ with COVERAGE_PROCESS_START=.coveragerc, then coverage report --fail-under=75. With source limited to tests, coverage of everything under scripts/ was collected and then discarded. The consequence is the part worth naming: scripts/ holds the CI gate implementations — validate_docs.py, update_coverage_snapshot.py, check_quarantine_age.py, generate_skill_index.py — so a regression in a gate script, or deleting its test file outright, could not move the gated number. The gates were the least-measured code in the repo.
The threshold claim is verified, not taken on trust. .coveragerc is the only file in this PR, and the pytest check is green on head f6f7a2f — that check is coverage report --fail-under=75 running under the new config. So the 84% total in the PR body is confirmed by the gate itself rather than by a pasted table. No threshold is lowered, no file is excluded, and --fail-under=75 is untouched.
Not padding the number: eight of the ten scripts already had tests; this makes their coverage count rather than adding new tests to inflate a total. generate_skill_index.py enters at 0% and still clears the gate, which is the honest way round — the number now reflects reality including the uncovered script.
Checks: six green on f6f7a2f. No open PR touches .coveragerc; I re-verified that against the current open set, not the list in the PR body, which is now stale.
Same two caveats as #811, neither blocking: ghost-lab is failure on this head and cannot be attributable to a coverage-config change that never reaches a VM; and the commit is unattributed, so require_extra_approval_for_unattributed_changes may hold this at +2 and require a third reviewer.
Test Improvement
Coverage-reporting configuration only. Single file, single line:
.coveragercsource = tests→source = tests,scripts..github/workflows/unit-tests.ymlrunscoverage run -m pytest tests/unit/withCOVERAGE_PROCESS_START=.coveragercand thencoverage report --fail-under=75. Because.coveragerclimitedsourcetotests, the gate never measuredscripts/— so a coverage regression in a CI gate script, or deleting its test file outright, could not move the gated number, andcodecov.yml'spatch: target: 70%never applied to a script-only patch.Eight of the ten scripts already have unit tests (
test_assert_kde_passed.py,test_quarantine_age.py,test_e2e_summary.py,test_generate_badges.py,test_parse_results.py,test_update_coverage_snapshot.py,test_validate_docs.py, plus the two shell wrappers). Their coverage was simply being discarded.Measured with the workflow's own command
Total is 84%, so
--fail-under=75is unchanged and still passes. No threshold is lowered or waived. No workflow file and no production code is touched.Claimed ground
Files:
.coveragerc(only file in this PR). Cluster: the unit-test coverage gate configuration.Checked against every open PR in this repo: #777 (
container/Containerfile.runner), #811 (SECURITY.md), #796 (tests/shared/timing.py), #812 (.github/actions/gnome-e2e/action.yml), #815 (docs/skills/.../suite-map/SKILL.md), #788 (tests/unit/test_generate_skill_index.py), #816 (tests/unit/test_screenshot_capture.py), and the smoke/software/ssh test PRs (#786, #787, #790, #798, #799, #806, #807, #808). None touches.coveragerc. Disjoint.Interaction with #788: that PR raises
scripts/generate_skill_index.pyoff 0%. The two are independent in either merge order — the total clears the gate with or without it.Related Issue
Closes #817
Filed by quality agent (hold-gated mode). Human review required.
— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.78