Skip to content

Refuse a skip, and a suite that shrank - #111

Merged
estebanzimanyi merged 1 commit into
MobilityDB:masterfrom
estebanzimanyi:ci/refuse-a-suite-that-shrinks
Sep 1, 2026
Merged

Refuse a skip, and a suite that shrank#111
estebanzimanyi merged 1 commit into
MobilityDB:masterfrom
estebanzimanyi:ci/refuse-a-suite-that-shrinks

Conversation

@estebanzimanyi

Copy link
Copy Markdown
Member

The suite's own summary states two things a job's conclusion does not. A skipped
test carries no assertion while the job still reports success — the eleven this
repository ran unnoticed are the measured case, and one flag is enough to reach
it elsewhere: MobilityFlink's binding module under -Dmeos.enabled=false reports
Tests run: 7, Failures: 0, Errors: 0, Skipped: 7 and BUILD SUCCESS, its whole
MEOS surface disabled with a green build over it.

The second is what the first cannot see. A test the run never collects appears in
no count at all, so a surefire <excludes>, a -Dtest= filter, a class renamed
out of *Test or a deleted file all leave the skip number at zero. Once skipping
is refused, removing a test is the remaining way to stop running it, so the total
carries a floor that may rise and may not fall.

tools/check-test-outcome.py states both rules as the one runnable definition
the check-test-outcome action and a developer both call. It reads the surefire
and pytest dialects and SUMS every module's summary rather than taking the last:
a multi-module build prints one per module, and reading only the last understates
the total — MobilityKafka prints 7 and 4, so its suite is 11 rather than the 4 a
tail reports. A log carrying no summary in either dialect fails rather than
passing quietly, since an extraction that finds nothing describes the parser
until it shows it found its input.

This repository runs the script from the checkout under test rather than through
the action, so a change to the rules is exercised by the pull request that makes
it; consumers use the action. Its floor is 268, the suite's current size.

Both directions are exercised, because a check proven only to refuse is half
proven. It REFUSES a log reporting 257 passed, 11 skipped against the floor
(raising both errors), 267 passed (the floor error alone), a log with no
summary at all, and a missing file. It ALLOWS a clean surefire log, a clean
pytest log, and a two-module log summing to 11 beside a per-class line it must
not double-count. Both deny messages name an action that clears them: supply the
precondition the guard reads, or account for the removal in the floor.

The suite's own summary states two things a job's conclusion does not. A skipped
test carries no assertion while the job still reports success — the eleven this
repository ran unnoticed are the measured case, and one flag is enough to reach
it elsewhere: MobilityFlink's binding module under `-Dmeos.enabled=false` reports
`Tests run: 7, Failures: 0, Errors: 0, Skipped: 7` and `BUILD SUCCESS`, its whole
MEOS surface disabled with a green build over it.

The second is what the first cannot see. A test the run never collects appears in
no count at all, so a surefire `<excludes>`, a `-Dtest=` filter, a class renamed
out of `*Test` or a deleted file all leave the skip number at zero. Once skipping
is refused, removing a test is the remaining way to stop running it, so the total
carries a floor that may rise and may not fall.

`tools/check-test-outcome.py` states both rules as the one runnable definition
the `check-test-outcome` action and a developer both call. It reads the surefire
and pytest dialects and SUMS every module's summary rather than taking the last:
a multi-module build prints one per module, and reading only the last understates
the total — MobilityKafka prints 7 and 4, so its suite is 11 rather than the 4 a
tail reports. A log carrying no summary in either dialect fails rather than
passing quietly, since an extraction that finds nothing describes the parser
until it shows it found its input.

This repository runs the script from the checkout under test rather than through
the action, so a change to the rules is exercised by the pull request that makes
it; consumers use the action. Its floor is 268, the suite's current size.

Both directions are exercised, because a check proven only to refuse is half
proven. It REFUSES a log reporting `257 passed, 11 skipped` against the floor
(raising both errors), `267 passed` (the floor error alone), a log with no
summary at all, and a missing file. It ALLOWS a clean surefire log, a clean
pytest log, and a two-module log summing to 11 beside a per-class line it must
not double-count. Both deny messages name an action that clears them: supply the
precondition the guard reads, or account for the removal in the floor.
@estebanzimanyi
estebanzimanyi merged commit 3d69912 into MobilityDB:master Sep 1, 2026
2 checks passed
@estebanzimanyi
estebanzimanyi deleted the ci/refuse-a-suite-that-shrinks branch September 1, 2026 06:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant