test: walk the schema so a new digest site cannot join unlisted - #91
Conversation
test_every_known_site_is_listed asserted len(SCHEMA_SITES) == 6 and len(COMPILED_COPIES) == 5. Both are constants in this module, so no change to schemas/trace-claim.json could move either one. The module docstring and the commit message of f65e7c3 both said a seventh digest field appearing later would fail that test instead of joining silently. It would not. lywinged showed it by execution on the pull request that added the file: a seventh digest field carrying a drifted pattern left the suite green. This adds their test unchanged. It walks the schema for digest-shaped patterns and compares the walked set to SCHEMA_SITES, so a digest site present in the schema and absent from the list now fails, and dropping a site from the list is caught by the same assertion, and a listed entry naming no schema site fails from the other side. The list stays named, so adding a site is still a decision. The schema-side count is removed rather than kept beside the walk, because set equality against the schema is what that count stood in for. The compiled-side count stays. Nothing discovers a compiled copy: enum_drift.py walks the AST for set literals and a compiled regex is invisible to it, and a walk of the schema cannot see one either. The docstring now states that gap instead of implying the file closes both halves. Full suite green on this branch, and the plant reverts clean. Co-authored-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: opento-suggestions <opentosuggestionsofficial@gmail.com>
imran-siddique
left a comment
There was a problem hiding this comment.
Merging. Finding this in your own test, a week after writing it, is the part worth naming.
test_every_known_site_is_listed asserts len(SCHEMA_SITES) == 6 and len(COMPILED_COPIES) == 5, and both are constants in the same module. So the guard compares the test's beliefs against the test's beliefs, and no change to schemas/trace-claim.json can move either. The docstring said a seventh digest field would fail it. That was the claim, and it was wrong.
@lywinged tested the claim rather than reading it: added a seventh digest field to the schema with a deliberately drifted pattern and reported the suite stayed green. That is the only way this class gets found. A guard that cannot fail passes every review, including mine, because passing is what it does.
Taking the test they supplied in that review unchanged, and saying so, is the right call twice over: it is their work, and a test written by the person who broke the guard is more likely to break it again if it regresses.
Walking the schema for digest-shaped patterns and asserting the walked set equals SCHEMA_SITES makes the assertion depend on the artifact rather than on the list, which is what the docstring always claimed and now does.
This lands the same day as trace-spec#270, which found the identical shape in the schema-versus-models parity test: five of ten patterns unreachable because the fixture never set the optional fields carrying them. Two repositories, two tests meant to catch drift, neither able to. Both found by contributors auditing coverage rather than by CI. I am recording that on trace-spec#247.
tests/test_digest_parity.pynames its six schema sites and checks the fivecompiled copies against them. Its guard on that list,
test_every_known_site_is_listed, assertslen(SCHEMA_SITES) == 6andlen(COMPILED_COPIES) == 5. Both are constants in the test module, so nochange to
schemas/trace-claim.jsoncan move either.The module docstring said otherwise. It said a seventh digest field appearing
later would fail that test instead of joining silently, and that is not true
of the test as written. Reviewing #87, lywinged added
a seventh digest field to the schema with a deliberately drifted pattern and
reported the suite stayed green.
This applies the test they supplied in that review, unchanged. It walks the
schema for digest-shaped patterns and asserts the walked set equals
SCHEMA_SITES, so a digest site in the schema and absent from the list fails,and dropping a site from the list is caught by the same assertion, and a
listed entry naming no schema site fails from the other side. The list stays
named, so adding a site remains a decision rather than a default.
The schema-side count is removed rather than kept beside the walk: set
equality against the schema is what that count stood in for. The compiled-side
count stays, and the docstring now states the reason it has to. Nothing
discovers a compiled copy.
measurement/scripts/enum_drift.pywalks the ASTfor set literals, so a compiled regex is invisible to it, and a walk of the
schema cannot reach one either. That half of the file is still a named list
with a count beside it, and the docstring no longer implies otherwise.
Full suite green on this branch. The schema plant reproduces the failure and
reverts to a clean tree.
The drift figures in that review and the ones in the merged description
disagree. Neither set is asserted here; reconciling them under controls is
separate from this correction.