chore: upload coverage.xml artifact from CI - #108
Open
runpod-Henrik wants to merge 5 commits into
Open
Conversation
The test job already runs `make test-coverage` on every push, but the result only ever went to the terminal. Emit Cobertura XML and upload it so the weekly coverage report can read the number from the newest successful run on main instead of it being collected by hand. `if-no-files-found: error` so a silently-missing report fails the step rather than publishing an empty artifact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The coverage report was only readable by downloading the artifact. Write a Test Results + Coverage Summary table to $GITHUB_STEP_SUMMARY so the numbers show up on the run page, matching what the ai-api component workflow does. Adds --junitxml so the run's test counts can be reported alongside coverage; pytest-results.xml is gitignored. Stdlib only, so there is no extra install step, and `if: always()` means the summary still renders when tests fail — which is when it is most useful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
runpod-Henrik
requested review from
KAJdev and
jhcipar
and
a lite review from Copilot
August 25, 2026 23:40
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves CI observability by generating a Cobertura coverage.xml report during the existing make test-coverage run and publishing it as a per-Python-version GitHub Actions artifact, enabling automated downstream reporting from the latest successful main run.
Changes:
- Extend
make test-coverageto emit Cobertura XML (and a JUnit XML test report file). - Add a CI step that renders a coverage/test summary into the GitHub Actions run summary.
- Upload
coverage.xmlas a uniquely named artifact per matrix Python version.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
Makefile |
Adds --cov-report=xml (and --junitxml=pytest-results.xml) to the test-coverage target so CI can publish machine-readable reports. |
.gitignore |
Ignores the newly generated pytest-results.xml file. |
.github/workflows/ci.yml |
Adds a run-summary renderer and uploads coverage.xml as a per-matrix artifact. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| test-coverage: # Run tests with coverage report (parallel) | ||
| uv run pytest tests/ -v -n auto --dist loadscope --cov=handler --cov=remote_execution --cov-report=term-missing | ||
| uv run pytest tests/ -v -n auto --dist loadscope --cov=handler --cov=remote_execution --cov-report=term-missing --cov-report=xml --junitxml=pytest-results.xml |
Comment on lines
+111
to
+117
| - name: Upload coverage report | ||
| uses: actions/upload-artifact@v4 | ||
| if: always() | ||
| with: | ||
| name: coverage-${{ matrix.python-version }} | ||
| path: coverage.xml | ||
| if-no-files-found: error |
Review feedback from #373. Three defects, all reachable because this step
runs under `if: always()`:
* the coverage-XML parse was unguarded while the junit parse beside it
was. A report truncated by a timeout, OOM or crashed xdist worker made
ET.parse raise, so the summary step exited non-zero and stacked a
spurious failure on top of the real one. Verified: the old script exits
1 on a truncated report, the new one exits 0 and says so in the summary.
* hits/lines attributes were parsed with bare int(), so a malformed value
raised rather than degrading.
* the status cell treated `0 failures` as passing even when no tests ran
at all. A suite dying at import reports errors>0 with tests possibly 0,
so the check is now `no failures AND at least one test`.
Also switch the artifact upload to `if-no-files-found: warn`. With
`error` a run that never produced coverage.xml — pytest erroring at
collection, before pytest-cov writes anything — failed the upload step too,
red-flagging the job and masking the root cause. This upload hangs off the
PR-gating test job, so it should not be able to fail a PR on its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Branch coverage was never collected, so the job summary could only ever show line coverage. Adds --cov-branch. Purely additive: line coverage is unchanged (verified per repo), so the weekly trend, which reads line coverage, is unaffected. The Cobertura report now carries branches-valid/covered, which the summary renders as its own row. Note --cov-fail-under gates on coverage.py total, which now blends lines and branches, so that number drops even though line coverage does not. Measured before committing: this repo stays comfortably above its gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The
testjob already runsmake test-coverageon every push, but the result only ever went to the terminal. This emits Cobertura XML and uploads it as an artifact.Why
We're automating the weekly Test Coverage Progress report. It currently relies on hand-collected numbers, which has produced real errors — figures reported as improvements when they were regressions, and at least one number that matched no CI run at all.
With this artifact published, the report reads the number straight from the newest successful run on
maininstead of anyone retyping it.Changes
Makefile: add--cov-report=xmltotest-coverage(keeps the existingterm-missingoutput).github/workflows/ci.yml: uploadcoverage.xmlascoverage-${{ matrix.python-version }}The artifact name is per-matrix-version because
upload-artifact@v4requires unique names within a run.if-no-files-found: errorso a silently-missing report fails the step rather than publishing an empty artifact.Verification
Ran
make test-coveragein a clean worktree at this commit:make format-checkandmake lintboth clean. The generatedcoverage.xmlparses to 942/1160 lines (81.2%), matching coverage.py's ownlines-covered/lines-validattributes exactly.No production code touched — CI config and a Makefile flag only.
🤖 Generated with Claude Code