Skip to content

fix(ingestion): #884 extraction_notice records every condition, not the strongest - #892

Merged
jasonssdev merged 1 commit into
mainfrom
fix/extraction-notice-carries-every-condition-884
Aug 27, 2026
Merged

fix(ingestion): #884 extraction_notice records every condition, not the strongest#892
jasonssdev merged 1 commit into
mainfrom
fix/extraction-notice-carries-every-condition-884

Conversation

@jasonssdev

Copy link
Copy Markdown
Owner

Closes #884

The defect

extraction_notice held one value, so when a run tripped several disclosure conditions the strongest token overwrote the weaker at write time. lint — the audit surface — then rendered Unevidenced objects: and Staging-dropped candidates: as independent sections while the second silently masked the first.

The issue's cheapest proposed remedy was "make lint disclose the masking". Implementing it showed that is not possible: the displaced token is destroyed at ingest, so by the time lint reads the document there is nothing left to disclose. That remedy collapses into this one.

The masking was pervasive, not rare

Making the field plural turned ten existing ingest tests red with this shape:

['judge-selection-unavailable',   'objects-without-evidence'] == 'judge-selection-unavailable'
['sole-object-restates-source',   'objects-without-evidence'] == 'sole-object-restates-source'
['objects-without-evidence', 'candidates-dropped-in-staging'] == 'candidates-dropped-in-staging'

Those fixtures had been tripping two conditions and losing one all along — invisible, because the tests only asserted the winner. This was not a quirk of the E2E corpus.

Also fixed: the in-batch collision is no longer a staging loss

When two candidates of one run slugify alike, the first was already staged, so the content is on disk and the bundle represents the source — the identical reasoning the code already applied to the create-only skip. Counting it also left debt no command could clear, since the redo that marker prescribes (--re-extract) reproduces the same collision deterministically.

Precedence moved rather than disappeared

It was a rule that destroyed data in the frontmatter. It is now a presentation rule in the batch summary, which counts a file once however many conditions it carries. lint enumerates them all.

On disk, a single condition stays a bare scalar — byte-identical to earlier releases. Only a genuine co-occurrence widens to a list, and readers accept both shapes, so existing Sources keep their disclosure.

Two false claims already in the code, corrected

  • lint.py stated "disjoint by construction: a Source carries at most one token" — true of the tokens, false of the conditions. That sentence was the justification for the broken design.
  • A comment added during this change promised a single condition would write the bytes it always did, while the code wrote a list. The code was changed to match the comment.

Tests

Four tests named ..._outranks_... were renamed and strengthened: after the change they passed while asserting only membership, so their names claimed more than their bodies checked.

The headline lint test uses two Sources with the tokens in opposite order. One alone does not discriminate — whichever token sits at index 0 is still found by a reader inspecting only the first element. Mutating either lint reader to doc.extraction_notices[:1] now fails; with a single fixture, neither did.

Specs

ingestion/spec.md's precedence requirement rewritten, its two precedence scenarios replaced, and two scenarios added: an in-batch collision without a marker, and scalar-on-disk for a single condition.

Full suite 5659 passed, 1 skipped. ruff clean. mypy clean over 272 files.

Found during the 0.2.10 manual E2E.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MZz1zEMqg4X5mDEwwfC6dj

…he strongest

extraction_notice held ONE value, so when a run tripped several disclosure
conditions the strongest token overwrote the weaker at write time. lint --
the audit surface -- then rendered "Unevidenced objects:" and
"Staging-dropped candidates:" as independent sections while the second
silently masked the first.

lint could not paper over that from its own side: the displaced token is
destroyed at ingest, so by the time lint reads the document there is nothing
left to disclose. Recording every condition is the only place the fix can
live. (The issue proposed "make lint disclose the masking" as the cheapest
remedy; implementing it showed it is not implementable, and collapses into
this one.)

The masking was PERVASIVE, not the rare case the issue described. Making the
field plural turned ten existing ingest tests red with the shape

    ['judge-selection-unavailable', 'objects-without-evidence'] == 'judge-...'

-- those fixtures had been tripping two conditions and losing one all along,
invisible because the tests only asserted the winner.

Also fixed, per the same issue: an in-batch slug collision no longer counts
as a staging loss. When two candidates of one run slugify alike the FIRST
was already staged, so the content is on disk and the bundle represents the
source -- the identical reasoning the code already applied to the create-only
skip. Counting it also left debt no command could clear, since the redo that
marker prescribes (--re-extract) reproduces the same collision.

Precedence did not disappear; it moved to where it belongs. It was a rule
that DESTROYED data in the frontmatter, and is now a presentation rule in the
batch summary, which counts a file once however many conditions it carries.

On disk a single condition stays a bare scalar, byte-identical to earlier
releases; only a genuine co-occurrence widens to a list, and readers accept
both shapes so existing Sources keep their disclosure.

Two false claims already written in the code were corrected:
- lint.py stated "disjoint by construction: a Source carries at most one
  token" -- true of the TOKENS, false of the CONDITIONS. That sentence was
  the justification for the broken design.
- A comment added in this change promised a single condition would write the
  bytes it always did while the code wrote a list. The code was changed to
  match the comment.

Four tests named "..._outranks_..." were renamed and strengthened: after the
change they passed while asserting only membership, so their names claimed
more than their bodies checked.

The headline lint test uses TWO Sources with the tokens in OPPOSITE order.
One alone does not discriminate -- whichever token sits at index 0 is still
found by a reader inspecting only the first element. Mutating either lint
reader to doc.extraction_notices[:1] now fails; with a single fixture,
neither did.

Specs updated: ingestion's precedence requirement rewritten, its two
precedence scenarios replaced, two scenarios added (collision without a
marker, scalar-on-disk for one condition).

Full suite 5659 passed, 1 skipped; ruff clean; mypy clean over 272 files.

Closes #884
@jasonssdev
jasonssdev merged commit 9f13aa7 into main Aug 27, 2026
7 checks passed
@jasonssdev
jasonssdev deleted the fix/extraction-notice-carries-every-condition-884 branch August 27, 2026 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

extraction_notice holds one token, so the staging marker hides unevidenced objects from lint

1 participant