ci: fail the build when a policy has no test or no empty-input test - #60
Merged
Conversation
CONTRIBUTING.md has always required both. Nothing enforced it, and the gap
reached 22 of 96 policies with no test at all — including both Article 5
prohibited-practice policies, the two gating the practices the EU AI Act bans
outright.
An external reviewer showed what that meant rather than asserting it. In
prohibited_practices/social_scoring.rego he changed
default allow := false -> default allow := true
turning the Article 5 social-scoring gate into a default allow, and `opa
test` still reported 604/604. Nothing went red. A green suite and a per-file
coverage count were disagreeing, and the files sitting uncovered were the
prohibitions rather than the paperwork.
Every policy has had a test since v1.3.0, so this gate passes today. The
point is that it is a gate: the previous state was a snapshot that a single
new file could silently undo, which is exactly how the gap opened.
Both conditions are checked because they fail differently. No test file means
nothing is verified. A test file without an empty-input assertion usually
means the fail-open case specifically is unverified, and that is the one that
matters most here — in Rego an undefined value is not `false`, so a policy
that reaches no conclusion returns no decision, which at a gate reads as a
policy that has quietly stopped saying no.
The seven libraries under global/v1/common/ and helper_functions/ are exempt.
They define helpers rather than decisions and have no `allow` to hand an
empty input to, which is `is_library` in the coverage data.
Verified by breaking it both ways: removing social_scoring_test.rego exits 1
naming the file, and replacing `with input as {}` with a non-empty input
exits 1 naming the missing empty-input assertion. Restored, it exits 0 with
"All 91 policies have a test and an empty-input test (7 libraries exempt)."
Also corrects two docs claims that this makes false. docs/coverage/eu-ai-act.md
told contributors that "not every policy has a sibling *_test.rego" and asked
for tests on untested files; all 98 have tests, so the useful contribution is
now a test for a *case*, and the text says so. Credit for both findings to
@GeiserX in GeiserX/awesome-europe#40.
kmadan
added a commit
to cw-zhou/gopal
that referenced
this pull request
Aug 28, 2026
CI installed `latest` for both tools, so an upstream release could turn the
build red with no change on our side. On 2026-08-28 one did.
OPA 1.20.0 panics with `panic: illegal value` on any ordering comparison
against a float produced by a terminal `else :=` branch:
f(x) := 1.0 if { x.a } else := 0.0
g(x) if f(x) >= 0.7 # panics when the else branch is taken
Integers are unaffected, `==` is unaffected, and the value renders correctly
on its own — only the numeric ordering path rejects it. The panic is in
ast.NumberCompare, reached from builtinGreaterThanEq.
Four helpers in global/v1/common use that shape as a score fallback, so
`opa test` aborted with exit code 2 across the whole suite rather than
failing a test. `opa check` passes, so nothing else caught it.
Worth being precise about what happened, because the merge order made it look
like our doing: main went red on the merges of Principled-Evolution#58, Principled-Evolution#59 and Principled-Evolution#60, and none of
them caused it. 4e09411, the commit before all three, fails identically under
1.20.0 and passes under 1.19.1. The suite is 806/806 on 1.19.1 today.
Reported upstream as open-policy-agent/opa#9098 with a nine-line reproducer.
Regal is pinned to v0.42.0 for the same reason rather than any known problem.
Both pins should be raised deliberately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
CONTRIBUTING.mdhas always required a sibling test and an empty-input test perpolicy. Nothing enforced it, and the gap reached 22 of 96 policies with no test
at all — including both Article 5 prohibited-practice policies, the two gating
the practices the EU AI Act bans outright.
An external reviewer demonstrated the consequence rather than asserting it. In
prohibited_practices/social_scoring.rego:That turns the Article 5 social-scoring gate into a default allow, and
opa teststill reported 604/604. Nothing went red.Credit to @GeiserX, in
GeiserX/awesome-europe#40.
What this adds
scripts/check-test-coverage.sh, run in CI, reading the coverage data that CIalready keeps current. It fails if a non-library policy:
*_test.rego, orBoth are checked because they fail differently. No test file means nothing is
verified. A test file without an empty-input assertion usually means the
fail-open case specifically is unverified — and in Rego an undefined value is not
false, so a policy that reaches no conclusion returns no decision, which at agate reads as a policy that has quietly stopped saying no.
The seven libraries under
global/v1/common/andhelper_functions/are exempt:they define helpers rather than decisions and have no
allowto hand an emptyinput to. That is
is_libraryin the coverage data, derived from whether the filedeclares decision rules.
This passes today — that is the point
Every policy has had a test since v1.3.0. The previous state was a snapshot that
one new file could silently undo, which is exactly how the gap opened in the first
place. Now it is a gate.
Verified by breaking it
mv social_scoring_test.regoawaywith input as {}with a non-empty inputAlso worth recording: the reviewer's mutation no longer survives. On current
main, flippingdefault allow := falsetotruefails 4 tests insocial_scoringand 5 inemotion_recognition.Docs corrections
This makes two claims false, and they are fixed here:
docs/coverage/eu-ai-act.mdtold contributors "not every policy has a sibling*_test.rego" and asked for tests on untested files. All 98 have tests. Theuseful contribution now is a test for a case — a policy whose only test proves
it denies an empty input is verified against the fail-open class and nothing
else — and the text says that instead.
docs/coverage/README.mdsaid "most of them also assert that the policy deniesan empty input". All of them do, and it is now enforced.