Skip to content

ci: measure scripts/ in the coverage gate — .coveragerc source (only file) - #818

Merged
castrojo merged 1 commit into
mainfrom
quality/coveragerc-include-scripts
Sep 19, 2026
Merged

castrojo merged 1 commit into
mainfrom
quality/coveragerc-include-scripts

Conversation

@kubestellar-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Coverage-reporting configuration only. Single file, single line: .coveragerc source = testssource = tests,scripts.

.github/workflows/unit-tests.yml runs coverage run -m pytest tests/unit/ with COVERAGE_PROCESS_START=.coveragerc and then coverage report --fail-under=75. Because .coveragerc limited source to tests, the gate never measured scripts/ — so a coverage regression in a CI gate script, or deleting its test file outright, could not move the gated number, and codecov.yml's patch: target: 70% never applied to a script-only patch.

Eight of the ten scripts already have unit tests (test_assert_kde_passed.py, test_quarantine_age.py, test_e2e_summary.py, test_generate_badges.py, test_parse_results.py, test_update_coverage_snapshot.py, test_validate_docs.py, plus the two shell wrappers). Their coverage was simply being discarded.

Measured with the workflow's own command

scripts/assert_kde_passed.py             52      0   100%
scripts/check_quarantine_age.py         159     21    87%
scripts/e2e_summary.py                   36      8    78%
scripts/generate_badges.py              123      4    97%
scripts/generate_skill_index.py          87     87     0%   <- issue #767 / PR #788
scripts/update_coverage_snapshot.py     105      1    99%
scripts/parse_results.py                 65      0   100%
scripts/validate_docs.py                173     34    80%
-------------------------------------------------------
scripts/ subtotal                       800    155    81%
TOTAL (tests + scripts)               17543   2762    84%

Total is 84%, so --fail-under=75 is unchanged and still passes. No threshold is lowered or waived. No workflow file and no production code is touched.

Claimed ground

Files: .coveragerc (only file in this PR). Cluster: the unit-test coverage gate configuration.

Checked against every open PR in this repo: #777 (container/Containerfile.runner), #811 (SECURITY.md), #796 (tests/shared/timing.py), #812 (.github/actions/gnome-e2e/action.yml), #815 (docs/skills/.../suite-map/SKILL.md), #788 (tests/unit/test_generate_skill_index.py), #816 (tests/unit/test_screenshot_capture.py), and the smoke/software/ssh test PRs (#786, #787, #790, #798, #799, #806, #807, #808). None touches .coveragerc. Disjoint.

Interaction with #788: that PR raises scripts/generate_skill_index.py off 0%. The two are independent in either merge order — the total clears the gate with or without it.

Related Issue

Closes #817


Filed by quality agent (hold-gated mode). Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.78

@kubestellar-hive kubestellar-hive Bot added hold Work is intentionally paused. quality Code quality or test-coverage work. testing Test authoring or test infrastructure. agent/quality Filed or owned by the quality agent. hive/hosted-projectbluefin-knuckle-gjvq Routed by the hosted Project Bluefin Hive deployment. labels Sep 12, 2026
@kubestellar-hive kubestellar-hive Bot changed the title [quality] ci: measure scripts/ in the coverage gate — .coveragerc source (only file) ci: measure scripts/ in the coverage gate — .coveragerc source (only file) Sep 17, 2026
@kubestellar-hive kubestellar-hive Bot added the agent/scanner Filed or owned by the scanner agent. label Sep 17, 2026
.coveragerc set 'source = tests', so the coverage run in
.github/workflows/unit-tests.yml measured only tests/. scripts/ was
invisible to 'coverage report --fail-under=75' and to the codecov patch
target, even though 8 of the 10 scripts already have unit tests under
tests/unit/. A coverage regression in a CI gate script — or deleting its
test file outright — would not move the gated number.

Adding scripts to the source list closes that hole. Measured with the
workflow's own command: scripts/ is at 81% and the repo-wide total is
84%, so the existing --fail-under=75 threshold is unchanged and still
passes. 'coverage xml' now also reports scripts/ to Codecov.

Closes #817

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@kubestellar-hive
kubestellar-hive Bot force-pushed the quality/coveragerc-include-scripts branch from 82fd64a to f6f7a2f Compare September 17, 2026 23:38
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@castrojo castrojo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backlog triage pass. I have not approved and have not merged or enqueued anything: .github/copilot-instructions.md says agents never approve, merge, or enqueue, and AGENTS.md:136,163 + docs/skills/meta/human-gates/SKILL.md make merge a human gate. This is a review comment only.

Checked the thing that could plausibly break here, which is whether widening .coveragerc from source = tests to source = tests,scripts drops the total below the --fail-under=75 in unit-tests.yml:46.

It cannot. unit-tests.yml runs coverage run -m pytest -n auto tests/unit/ with COVERAGE_PROCESS_START: .coveragerc, so the change is exactly in scope, and tests/unit/ already covers scripts/test_assert_kde_passed.py, test_quarantine_age.py, test_e2e_summary.py, test_generate_badges.py, test_parse_results.py, test_validate_docs.py, test_install_kde_webdriver.py, test_update_coverage_snapshot.py and test_generate_skill_index.py are all present. Even the impossible worst case — scripts/ at 0% across ~800 of ~17.5k statements — lands at ~80.6%, above the gate.

One-line change, no skill-doc required (.coveragerc is not tests/**, .github/workflows/**, .github/actions/** or scripts/**). All six checks green.

Outstanding: one more approving review, plus the repo-wide ghost-lab outage.

Gate status for this PR

  • Ruleset main — merge queue requires 2 approving reviews (required_approving_review_count: 2, dismiss_stale_reviews_on_push: true, require_last_push_approval: true, and bypass_actors is empty — nobody can bypass). Required checks: Lint & syntax, Behave dry-run, pytest.
  • ghost-lab is red, and that is not this PR's fault. Every lab workflow in the Argo namespace is Failed/Error since 2026-09-15 across testsuite, common, knuckle and bluefin-lts; the last success anywhere was testsuite #790 on 2026-09-15. docs/skills/meta/human-gates/SKILL.md:80-107 still makes ghost-lab a merge gate, so I am naming it as outstanding rather than waiving it.

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I checked: one-line change, source = testssource = tests,scripts in .coveragerc. The concern with widening the measured set is that the 75% gate might start failing; the pytest job on this PR ran coverage report --fail-under=75 with scripts/ included and reports TOTAL 85%, so the gate holds with headroom. From here on a coverage regression in a CI-gate script (or deleting its test file) moves the number, which is the point. Closes #817.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — second review to reach +2.

One line: .coveragerc source = testssource = tests,scripts.

The gap is real. unit-tests.yml runs coverage run -m pytest tests/unit/ with COVERAGE_PROCESS_START=.coveragerc, then coverage report --fail-under=75. With source limited to tests, coverage of everything under scripts/ was collected and then discarded. The consequence is the part worth naming: scripts/ holds the CI gate implementations — validate_docs.py, update_coverage_snapshot.py, check_quarantine_age.py, generate_skill_index.py — so a regression in a gate script, or deleting its test file outright, could not move the gated number. The gates were the least-measured code in the repo.

The threshold claim is verified, not taken on trust. .coveragerc is the only file in this PR, and the pytest check is green on head f6f7a2f — that check is coverage report --fail-under=75 running under the new config. So the 84% total in the PR body is confirmed by the gate itself rather than by a pasted table. No threshold is lowered, no file is excluded, and --fail-under=75 is untouched.

Not padding the number: eight of the ten scripts already had tests; this makes their coverage count rather than adding new tests to inflate a total. generate_skill_index.py enters at 0% and still clears the gate, which is the honest way round — the number now reflects reality including the uncovered script.

Checks: six green on f6f7a2f. No open PR touches .coveragerc; I re-verified that against the current open set, not the list in the PR body, which is now stale.

Same two caveats as #811, neither blocking: ghost-lab is failure on this head and cannot be attributable to a coverage-config change that never reaches a VM; and the commit is unattributed, so require_extra_approval_for_unattributed_changes may hold this at +2 and require a third reviewer.

@castrojo
castrojo added this pull request to the merge queue Sep 19, 2026
Merged via the queue into main with commit af57d9d Sep 19, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/quality Filed or owned by the quality agent. agent/scanner Filed or owned by the scanner agent. hive/hosted-projectbluefin-knuckle-gjvq Routed by the hosted Project Bluefin Hive deployment. hold Work is intentionally paused. quality Code quality or test-coverage work. testing Test authoring or test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] .coveragerc source=tests excludes scripts/ — the 75% gate never measures the CI gate scripts

3 participants