Refuse a skip, and a suite that shrank - #111
Merged
estebanzimanyi merged 1 commit intoSep 1, 2026
Merged
Conversation
The suite's own summary states two things a job's conclusion does not. A skipped test carries no assertion while the job still reports success — the eleven this repository ran unnoticed are the measured case, and one flag is enough to reach it elsewhere: MobilityFlink's binding module under `-Dmeos.enabled=false` reports `Tests run: 7, Failures: 0, Errors: 0, Skipped: 7` and `BUILD SUCCESS`, its whole MEOS surface disabled with a green build over it. The second is what the first cannot see. A test the run never collects appears in no count at all, so a surefire `<excludes>`, a `-Dtest=` filter, a class renamed out of `*Test` or a deleted file all leave the skip number at zero. Once skipping is refused, removing a test is the remaining way to stop running it, so the total carries a floor that may rise and may not fall. `tools/check-test-outcome.py` states both rules as the one runnable definition the `check-test-outcome` action and a developer both call. It reads the surefire and pytest dialects and SUMS every module's summary rather than taking the last: a multi-module build prints one per module, and reading only the last understates the total — MobilityKafka prints 7 and 4, so its suite is 11 rather than the 4 a tail reports. A log carrying no summary in either dialect fails rather than passing quietly, since an extraction that finds nothing describes the parser until it shows it found its input. This repository runs the script from the checkout under test rather than through the action, so a change to the rules is exercised by the pull request that makes it; consumers use the action. Its floor is 268, the suite's current size. Both directions are exercised, because a check proven only to refuse is half proven. It REFUSES a log reporting `257 passed, 11 skipped` against the floor (raising both errors), `267 passed` (the floor error alone), a log with no summary at all, and a missing file. It ALLOWS a clean surefire log, a clean pytest log, and a two-module log summing to 11 beside a per-class line it must not double-count. Both deny messages name an action that clears them: supply the precondition the guard reads, or account for the removal in the floor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The suite's own summary states two things a job's conclusion does not. A skipped
test carries no assertion while the job still reports success — the eleven this
repository ran unnoticed are the measured case, and one flag is enough to reach
it elsewhere: MobilityFlink's binding module under
-Dmeos.enabled=falsereportsTests run: 7, Failures: 0, Errors: 0, Skipped: 7andBUILD SUCCESS, its wholeMEOS surface disabled with a green build over it.
The second is what the first cannot see. A test the run never collects appears in
no count at all, so a surefire
<excludes>, a-Dtest=filter, a class renamedout of
*Testor a deleted file all leave the skip number at zero. Once skippingis refused, removing a test is the remaining way to stop running it, so the total
carries a floor that may rise and may not fall.
tools/check-test-outcome.pystates both rules as the one runnable definitionthe
check-test-outcomeaction and a developer both call. It reads the surefireand pytest dialects and SUMS every module's summary rather than taking the last:
a multi-module build prints one per module, and reading only the last understates
the total — MobilityKafka prints 7 and 4, so its suite is 11 rather than the 4 a
tail reports. A log carrying no summary in either dialect fails rather than
passing quietly, since an extraction that finds nothing describes the parser
until it shows it found its input.
This repository runs the script from the checkout under test rather than through
the action, so a change to the rules is exercised by the pull request that makes
it; consumers use the action. Its floor is 268, the suite's current size.
Both directions are exercised, because a check proven only to refuse is half
proven. It REFUSES a log reporting
257 passed, 11 skippedagainst the floor(raising both errors),
267 passed(the floor error alone), a log with nosummary at all, and a missing file. It ALLOWS a clean surefire log, a clean
pytest log, and a two-module log summing to 11 beside a per-class line it must
not double-count. Both deny messages name an action that clears them: supply the
precondition the guard reads, or account for the removal in the floor.