Skip to content

feat(tooling): derive each policy's input fields from its AST - #59

Merged
kmadan merged 2 commits into
mainfrom
feat/derive-input-fields-from-ast
Aug 28, 2026
Merged

feat(tooling): derive each policy's input fields from its AST#59
kmadan merged 2 commits into
mainfrom
feat/derive-input-fields-from-ast

Conversation

@kmadan

@kmadan kmadan commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

The problem

"What does this policy need me to supply?" was answered by a hand-written
# RequiredMetrics: comment block at the top of each policy. That block had
drifted: 22 of 98 policies read input and declared nothing at all.

Consumers got an empty list and no way to know it was wrong. Concretely, in the
playground two EU AI Act policies (transparency, risk_management) rendered
no questions at all and passed on values the visitor could neither see nor
change. This is the same class of drift that the coverage figures had before
--check existed, in a different field.

The fix

scripts/extract-input-fields.sh derives the fields from the policy's AST via
opa parse --format json. Three access forms are recognised:

input.system.high_risk                          # a plain ref
object.get(input, ["system", "sources"], [])    # a path passed as an argument
object.get(input.params, "threshold", 0.8)      # a ref plus a literal key

The second form is used 284 times in this library — it is how a field is
read without the whole rule going undefined when the field is absent. A regex or
a naive ref walk reports almost nothing for those policies.

It also infers the kind of each field from the literal the policy measures it
against: the default handed to object.get, or the constant on the other side of
a comparison. input.system.x == true is a yes/no question; object.get(input, ["logs","months"], 0) >= n is a number. That is firmer evidence than reading a
sample document, which can omit the field entirely.

Why the comments stay

Neither source is complete alone, so consumers union them.

The AST cannot recover a field name that is computed:

some label, field in criteria
object.get(input, ["datasets", field], false) != true

Here the names live in the criteria map, and the comment is the only static
record of them. The AST contributes a bare datasets for this call, so:

  • prefixes are pruned against the fuller list, since datasets is an object
    rather than a question and is answered by any leaf beneath it; and
  • the kind found on the prefix is inheritable by the leaves — false as the
    object.get default makes every datasets.* leaf a yes/no. This resolves 51
    of the 53 EU AI Act fields that have no evidence of their own.

Result

Framework Before After
EU AI Act 142 declared, 2 policies with no fields 151 declared, 0 with no fields, 123 typed
UK 31 declared 33 declared, all typed

Two fields remain untyped: logs.role (covered by the sample) and
logs.sectoral_minimum_months, whose object.get default is a variable rather
than a literal. Consumers fall back to a text input, which is honest.

input.params.* thresholds move to a separate parameter_fields list. They tune
how strict a check is and each has a default in the Rego; listing "toxicity
threshold" among the obligations would invite someone to answer their way to a
pass.

Verification

  • opa check, regal lint (196 files, no violations), 806 tests,
    generate-coverage.sh --check, check-version-refs.sh
  • build-playground.sh --verify: all sample verdicts unchanged. That is the
    main safety property — this adds knowledge about the policies without
    altering what any of them decides.

kmadan added 2 commits August 27, 2026 22:25
`has_valid_consent` was labelled "placeholder logic" in a comment and did
what the label said: it read a status flag and a scope list.

    student.consent.status == "active"
    every item in requested_data { item in student.consent.scope }

34 CFR §99.30 does not treat consent as a flag. A written consent is valid
only if it is signed and dated by the parent or the eligible student
(§99.30(a), §99.5) and, per §99.30(b), specifies the records that may be
disclosed, states the purpose of the disclosure, and identifies the party or
class of parties who may receive it.

The practical difference: a consent permitting a transcript to go to a named
university for admissions also cleared a request to send that transcript to
a data broker for marketing, because neither purpose nor recipient was ever
looked at. That is the disclosure FERPA exists to prevent, and this policy
approved it.

All four elements are now checked, and the function is total, so a consent
record missing any of them is denied rather than left undefined.

Also fixes two vacuous-truth fail-opens found while writing the tests. Both
the consent branch and the directory-information branch used `every` over
`input.data_requested`, and `every` over an empty collection is true, so a
request for no records at all was approved by both. Both now require a
non-empty request.

This narrows what the policy accepts, so callers must supply the §99.30
fields. That is the fail-closed direction, and a consent rule that cannot
tell admissions from marketing is not worth being compatible with.

Tests: 23 new, one per element and one per branch, so a failure names the
element that stopped being enforced. Suite 782 → 806. Every guard was
mutation-verified by removing it and confirming the suite fails, including
the `default` on the function, which needed a test asserting `== false`
rather than `not ...` to be load-bearing at all.
"What does this policy need me to supply?" was answered by a hand-written
`# RequiredMetrics:` comment block. Twenty-two of 98 policies read `input`
and declared nothing at all, so anything consuming that answer got an empty
list and no signal that it was wrong. In the playground, two EU AI Act
policies presented no questions whatsoever and quietly passed on values the
visitor could neither see nor change.

The same drift the coverage figures once had, in a different field.

scripts/extract-input-fields.sh derives the fields from the AST instead, via
`opa parse --format json`. Two access forms have to be recognised, and
missing the second is what a regex over the source would do:

    input.system.high_risk                        a plain ref
    object.get(input, ["system", "sources"], [])  a path as an argument
    object.get(input.params, "threshold", 0.8)    a ref plus a literal key

The second is used 284 times in this library, precisely because it is how a
field is read without the whole rule going undefined when it is absent.

It also infers the kind of each field from the literal the policy measures it
against — the default handed to `object.get`, or the constant on the other
side of a comparison. That is firmer evidence than a sample document, which
can omit the field entirely.

The comment block stays, and consumers union the two. Neither is complete
alone: a field name that is computed, as in `object.get(input, ["datasets",
field], false)` where `field` is bound by a loop, cannot be recovered from
the AST, and there the comment is the only record. Where the AST truncates
such a path to a bare `datasets`, prefixes are pruned so an object is not
offered as if it were a question, and the kind found on the prefix is
available for the leaves to inherit — which resolves 51 of the 53 EU fields
that carry no evidence of their own.

Playground manifest, before and after:

    EU AI Act   142 declared, 2 policies with no fields at all
                151 declared, 0 policies with no fields, 123 typed
    UK           31 declared
                 33 declared, all typed

`input.params.*` thresholds are now a separate `parameter_fields` list. They
tune how strict a check is and each has a default in the Rego; listing
"toxicity threshold" among the obligations would invite someone to answer
their way to a pass.

All gates pass: opa check, regal lint (196 files), 806 tests, coverage
--check, version refs, and playground sample verification — the verdicts are
unchanged, which is the point: this adds knowledge about the policies without
altering what any of them decides.
@kmadan
kmadan merged commit 762c884 into main Aug 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant