Skip to content

ci: fail the build when a policy has no test or no empty-input test - #60

Merged
kmadan merged 1 commit into
mainfrom
feat/gate-test-coverage
Aug 28, 2026
Merged

ci: fail the build when a policy has no test or no empty-input test#60
kmadan merged 1 commit into
mainfrom
feat/gate-test-coverage

Conversation

@kmadan

@kmadan kmadan commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Why

CONTRIBUTING.md has always required a sibling test and an empty-input test per
policy. Nothing enforced it, and the gap reached 22 of 96 policies with no test
at all
— including both Article 5 prohibited-practice policies, the two gating
the practices the EU AI Act bans outright.

An external reviewer demonstrated the consequence rather than asserting it. In
prohibited_practices/social_scoring.rego:

-default allow := false
+default allow := true

That turns the Article 5 social-scoring gate into a default allow, and
opa test still reported 604/604. Nothing went red.

Credit to @GeiserX, in
GeiserX/awesome-europe#40.

What this adds

scripts/check-test-coverage.sh, run in CI, reading the coverage data that CI
already keeps current. It fails if a non-library policy:

  • has no sibling *_test.rego, or
  • has one with no empty-input assertion.

Both are checked because they fail differently. No test file means nothing is
verified. A test file without an empty-input assertion usually means the
fail-open case specifically is unverified — and in Rego an undefined value is not
false, so a policy that reaches no conclusion returns no decision, which at a
gate reads as a policy that has quietly stopped saying no.

The seven libraries under global/v1/common/ and helper_functions/ are exempt:
they define helpers rather than decisions and have no allow to hand an empty
input to. That is is_library in the coverage data, derived from whether the file
declares decision rules.

This passes today — that is the point

Every policy has had a test since v1.3.0. The previous state was a snapshot that
one new file could silently undo, which is exactly how the gap opened in the first
place. Now it is a gate.

Verified by breaking it

Change Result
mv social_scoring_test.rego away exit 1, names the file
Replace with input as {} with a non-empty input exit 1, names the missing empty-input assertion
Restored exit 0, "All 91 policies have a test and an empty-input test (7 libraries exempt)"

Also worth recording: the reviewer's mutation no longer survives. On current
main, flipping default allow := false to true fails 4 tests in
social_scoring and 5 in emotion_recognition.

Docs corrections

This makes two claims false, and they are fixed here:

  • docs/coverage/eu-ai-act.md told contributors "not every policy has a sibling
    *_test.rego" and asked for tests on untested files. All 98 have tests. The
    useful contribution now is a test for a case — a policy whose only test proves
    it denies an empty input is verified against the fail-open class and nothing
    else — and the text says that instead.
  • docs/coverage/README.md said "most of them also assert that the policy denies
    an empty input". All of them do, and it is now enforced.

CONTRIBUTING.md has always required both. Nothing enforced it, and the gap
reached 22 of 96 policies with no test at all — including both Article 5
prohibited-practice policies, the two gating the practices the EU AI Act bans
outright.

An external reviewer showed what that meant rather than asserting it. In
prohibited_practices/social_scoring.rego he changed

    default allow := false   ->   default allow := true

turning the Article 5 social-scoring gate into a default allow, and `opa
test` still reported 604/604. Nothing went red. A green suite and a per-file
coverage count were disagreeing, and the files sitting uncovered were the
prohibitions rather than the paperwork.

Every policy has had a test since v1.3.0, so this gate passes today. The
point is that it is a gate: the previous state was a snapshot that a single
new file could silently undo, which is exactly how the gap opened.

Both conditions are checked because they fail differently. No test file means
nothing is verified. A test file without an empty-input assertion usually
means the fail-open case specifically is unverified, and that is the one that
matters most here — in Rego an undefined value is not `false`, so a policy
that reaches no conclusion returns no decision, which at a gate reads as a
policy that has quietly stopped saying no.

The seven libraries under global/v1/common/ and helper_functions/ are exempt.
They define helpers rather than decisions and have no `allow` to hand an
empty input to, which is `is_library` in the coverage data.

Verified by breaking it both ways: removing social_scoring_test.rego exits 1
naming the file, and replacing `with input as {}` with a non-empty input
exits 1 naming the missing empty-input assertion. Restored, it exits 0 with
"All 91 policies have a test and an empty-input test (7 libraries exempt)."

Also corrects two docs claims that this makes false. docs/coverage/eu-ai-act.md
told contributors that "not every policy has a sibling *_test.rego" and asked
for tests on untested files; all 98 have tests, so the useful contribution is
now a test for a *case*, and the text says so. Credit for both findings to
@GeiserX in GeiserX/awesome-europe#40.
@kmadan
kmadan merged commit 12172d3 into main Aug 28, 2026
2 checks passed
kmadan added a commit to cw-zhou/gopal that referenced this pull request Aug 28, 2026
CI installed `latest` for both tools, so an upstream release could turn the
build red with no change on our side. On 2026-08-28 one did.

OPA 1.20.0 panics with `panic: illegal value` on any ordering comparison
against a float produced by a terminal `else :=` branch:

    f(x) := 1.0 if { x.a } else := 0.0

    g(x) if f(x) >= 0.7    # panics when the else branch is taken

Integers are unaffected, `==` is unaffected, and the value renders correctly
on its own — only the numeric ordering path rejects it. The panic is in
ast.NumberCompare, reached from builtinGreaterThanEq.

Four helpers in global/v1/common use that shape as a score fallback, so
`opa test` aborted with exit code 2 across the whole suite rather than
failing a test. `opa check` passes, so nothing else caught it.

Worth being precise about what happened, because the merge order made it look
like our doing: main went red on the merges of Principled-Evolution#58, Principled-Evolution#59 and Principled-Evolution#60, and none of
them caused it. 4e09411, the commit before all three, fails identically under
1.20.0 and passes under 1.19.1. The suite is 806/806 on 1.19.1 today.

Reported upstream as open-policy-agent/opa#9098 with a nine-line reproducer.

Regal is pinned to v0.42.0 for the same reason rather than any known problem.
Both pins should be raised deliberately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant