The expression values never reached the model, so it invented them - #286
Merged
adamjohnwright merged 2 commits intoSep 21, 2026
Merged
Conversation
Spec 011 Phase 9, less the two tasks that need a deploy. T034 and T035 were implemented in Phase 3 by copying the answer endpoint's patterns, and neither had a test. Written code with no test is not the same as working code, and both of these only run on paths nobody exercises deliberately. The bound now has one: a stuck model, the timeout patched to 0.25s, and the stream must still end with a terminal `failed` rather than holding the connection. The stand-in hangs for a finite five seconds so a regression fails quickly instead of hanging the suite. And a second test the task did not ask for: a timed-out generation must not store its partial text. `put` already refuses empty text, but a truncated summary is not empty -- it is worse, a plausible fragment ending mid-sentence that would then be served as the summary forever. The abandoned-stream log has one too, driven through the response iterator rather than a client, because the point is to close it mid-stream and a TestClient will not. An abandoned stream is identical to a healthy one in every other signal: the request 200s, tokens flow, and then nothing happens. Also, at the website's request, expression summaries now refer to columns as `column 1`, `column 2` and in no other form. They hold the real labels and we never do, so they substitute them into our prose -- which only works if our wording is fixed. They had decided to show labels alongside rather than splice on brittle matching; pinning the form is the better half of that trade, and it is in the contract now. All three verified by sabotage, with each sabotage asserted to have applied. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adversarial review of the polish work, and the finding is the worst of this feature so far. `exp` -- each pathway's value per column -- was allow-listed in `disclosure.py` and then dropped by `prompt_input`, which builds its own pathway dict. So the expression reading asked the model to describe behaviour across columns using data it had never been given. It complied. Against a real three-column analysis on beta it wrote that values "rise from column 1 to column 4" and described trends in a fourth column that does not exist. Every quantitative claim about expression was fabricated, which is precisely what FR-002 forbids. Both layers had tests and both passed. The allow-list test proved `exp` survives `aggregate()`; the type test proved the instruction says "across". Neither tested the path between them, and that is where the data was lost. Fixed by carrying `exp` into the prompt input, and by stating `expression_columns` rather than leaving the count to be inferred from an array -- the same correction as D9, for the same reason: a number a model can derive wrongly should be derived here. Absent when the pathways disagree on length, because inventing a count from inconsistent data is the same error one level up. Re-run three times against the real result: `column 1`, `column 2`, `column 3` in every one, no invented column, and the values read off correctly -- "strong expression in column 1 (0.7) but drops to 0.25 in both column 2 and column 3", which matches the actual array. Two process failures worth recording alongside it. The served-path assertion I added first could not fail: the payload is JSON inside JSON, so the escaped form never matched the unescaped string I looked for. It passed while testing nothing. And my check chain printed "pytest ok" over a failing suite. I cannot reproduce it and will not guess at a cause; `set -e` behaves correctly when I test it directly. So the checks now run from a script that prints each exit code and one verdict, with nothing inferred from absent output. It found a ruff failure on its first run that the chain had hidden. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adamjohnwright
deleted the
011-expression-values-never-reached-the-model
branch
September 21, 2026 00:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adversarial review of #285. Merge #285 first — this builds on it.
The finding
exp— each pathway's value per column — was allow-listed indisclosure.pyand then dropped byprompt_input, which builds its own pathway dict. So the expression reading asked the model to describe behaviour across columns using data it had never been given.It complied. Against a real three-column analysis on beta:
There is no column 4. Every quantitative claim about expression was fabricated — precisely what FR-002 forbids, in the story whose entire purpose is reading those values.
Why the tests missed it
Both layers had tests and both passed. The allow-list test proved
expsurvivesaggregate(). The type test proved the instruction says "across". Neither tested the path between them, which is where the data was lost.The website session named this shape earlier today: when a check and the thing it checks share an assumption, the check confirms the assumption rather than the thing.
The fix
Carry
expinto the prompt input, and stateexpression_columnsrather than leaving the count to be inferred from an array — the same correction as D9, for the same reason: a number a model can derive wrongly should be derived here. Absent when pathways disagree on length, because inventing a count from inconsistent data is the same error one level up.Re-run three times against the real result:
column 1,column 2,column 3every time, no invented column, values read off correctly — "strong expression in column 1 (0.7) but drops to 0.25 in both column 2 and column 3", matching the actual array.Two process failures, recorded
The served-path assertion I added first could not fail. The payload is JSON inside JSON, so the escaped form never matched the unescaped string I searched for. It passed while testing nothing.
My check chain printed "pytest ok" over a failing suite. I cannot reproduce it and will not guess at a cause —
set -ebehaves correctly when tested directly. The checks now run from a script that prints each exit code and one verdict, with nothing inferred from absent output. It caught a ruff failure on its first run that the chain had hidden.🤖 Generated with Claude Code