feat(tooling): derive each policy's input fields from its AST - #59
Merged
Conversation
`has_valid_consent` was labelled "placeholder logic" in a comment and did
what the label said: it read a status flag and a scope list.
student.consent.status == "active"
every item in requested_data { item in student.consent.scope }
34 CFR §99.30 does not treat consent as a flag. A written consent is valid
only if it is signed and dated by the parent or the eligible student
(§99.30(a), §99.5) and, per §99.30(b), specifies the records that may be
disclosed, states the purpose of the disclosure, and identifies the party or
class of parties who may receive it.
The practical difference: a consent permitting a transcript to go to a named
university for admissions also cleared a request to send that transcript to
a data broker for marketing, because neither purpose nor recipient was ever
looked at. That is the disclosure FERPA exists to prevent, and this policy
approved it.
All four elements are now checked, and the function is total, so a consent
record missing any of them is denied rather than left undefined.
Also fixes two vacuous-truth fail-opens found while writing the tests. Both
the consent branch and the directory-information branch used `every` over
`input.data_requested`, and `every` over an empty collection is true, so a
request for no records at all was approved by both. Both now require a
non-empty request.
This narrows what the policy accepts, so callers must supply the §99.30
fields. That is the fail-closed direction, and a consent rule that cannot
tell admissions from marketing is not worth being compatible with.
Tests: 23 new, one per element and one per branch, so a failure names the
element that stopped being enforced. Suite 782 → 806. Every guard was
mutation-verified by removing it and confirming the suite fails, including
the `default` on the function, which needed a test asserting `== false`
rather than `not ...` to be load-bearing at all.
"What does this policy need me to supply?" was answered by a hand-written
`# RequiredMetrics:` comment block. Twenty-two of 98 policies read `input`
and declared nothing at all, so anything consuming that answer got an empty
list and no signal that it was wrong. In the playground, two EU AI Act
policies presented no questions whatsoever and quietly passed on values the
visitor could neither see nor change.
The same drift the coverage figures once had, in a different field.
scripts/extract-input-fields.sh derives the fields from the AST instead, via
`opa parse --format json`. Two access forms have to be recognised, and
missing the second is what a regex over the source would do:
input.system.high_risk a plain ref
object.get(input, ["system", "sources"], []) a path as an argument
object.get(input.params, "threshold", 0.8) a ref plus a literal key
The second is used 284 times in this library, precisely because it is how a
field is read without the whole rule going undefined when it is absent.
It also infers the kind of each field from the literal the policy measures it
against — the default handed to `object.get`, or the constant on the other
side of a comparison. That is firmer evidence than a sample document, which
can omit the field entirely.
The comment block stays, and consumers union the two. Neither is complete
alone: a field name that is computed, as in `object.get(input, ["datasets",
field], false)` where `field` is bound by a loop, cannot be recovered from
the AST, and there the comment is the only record. Where the AST truncates
such a path to a bare `datasets`, prefixes are pruned so an object is not
offered as if it were a question, and the kind found on the prefix is
available for the leaves to inherit — which resolves 51 of the 53 EU fields
that carry no evidence of their own.
Playground manifest, before and after:
EU AI Act 142 declared, 2 policies with no fields at all
151 declared, 0 policies with no fields, 123 typed
UK 31 declared
33 declared, all typed
`input.params.*` thresholds are now a separate `parameter_fields` list. They
tune how strict a check is and each has a default in the Rego; listing
"toxicity threshold" among the obligations would invite someone to answer
their way to a pass.
All gates pass: opa check, regal lint (196 files), 806 tests, coverage
--check, version refs, and playground sample verification — the verdicts are
unchanged, which is the point: this adds knowledge about the policies without
altering what any of them decides.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
"What does this policy need me to supply?" was answered by a hand-written
# RequiredMetrics:comment block at the top of each policy. That block haddrifted: 22 of 98 policies read
inputand declared nothing at all.Consumers got an empty list and no way to know it was wrong. Concretely, in the
playground two EU AI Act policies (
transparency,risk_management) renderedno questions at all and passed on values the visitor could neither see nor
change. This is the same class of drift that the coverage figures had before
--checkexisted, in a different field.The fix
scripts/extract-input-fields.shderives the fields from the policy's AST viaopa parse --format json. Three access forms are recognised:The second form is used 284 times in this library — it is how a field is
read without the whole rule going undefined when the field is absent. A regex or
a naive ref walk reports almost nothing for those policies.
It also infers the kind of each field from the literal the policy measures it
against: the default handed to
object.get, or the constant on the other side ofa comparison.
input.system.x == trueis a yes/no question;object.get(input, ["logs","months"], 0) >= nis a number. That is firmer evidence than reading asample document, which can omit the field entirely.
Why the comments stay
Neither source is complete alone, so consumers union them.
The AST cannot recover a field name that is computed:
Here the names live in the
criteriamap, and the comment is the only staticrecord of them. The AST contributes a bare
datasetsfor this call, so:datasetsis an objectrather than a question and is answered by any leaf beneath it; and
falseas theobject.getdefault makes everydatasets.*leaf a yes/no. This resolves 51of the 53 EU AI Act fields that have no evidence of their own.
Result
Two fields remain untyped:
logs.role(covered by the sample) andlogs.sectoral_minimum_months, whoseobject.getdefault is a variable ratherthan a literal. Consumers fall back to a text input, which is honest.
input.params.*thresholds move to a separateparameter_fieldslist. They tunehow strict a check is and each has a default in the Rego; listing "toxicity
threshold" among the obligations would invite someone to answer their way to a
pass.
Verification
opa check,regal lint(196 files, no violations), 806 tests,generate-coverage.sh --check,check-version-refs.shbuild-playground.sh --verify: all sample verdicts unchanged. That is themain safety property — this adds knowledge about the policies without
altering what any of them decides.