Run the quickstart against beta, and fix the check that proved nothing - #287
Merged
Merged
Conversation
T036, on image cc4315c, inside the container: the deployed code, the installed bundle and the real Analysis Service. Six of nine scenarios pass. First token 1.8s. Three cannot be constructed here and the quickstart says so rather than marking them skipped. A null result needs an identifier list that hits nothing past correction, and every list tried produced significant hits -- faking one by editing a result would test the code against a fixture rather than the service. A `gone` token needs one issued before release 97, which has already deleted them. A ReactomeGSA token needs a different service. All three are unit-tested; what is missing is live confirmation, and that distinction is in the record. The adversarial pass before this went up found that scenario 4 -- the one the quickstart itself calls the one worth distrusting -- proved nothing. The disclosure check ran before the expression summary, so the payload carrying `Patient_001_tumour`, the strongest disclosure risk in the feature, was not among those examined. It passed by ordering rather than by evidence. It now runs last, over every aggregate payload produced, and asserts in the other direction as well: the disclosing tier's payload must contain the names. Without that, a clean aggregate check is equally consistent with nothing having been sent at all. Three aggregate payloads checked, no forbidden field and no label in any. Scenario 3's expectation is corrected in the quickstart too: it asked for a proportion, and the aggregate result has no submitted-total to compute one from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
T036 recorded `gone` as unverifiable because release 97 has deleted the tokens that would show it. That was wrong, and the assumption behind it was that only the service can produce a token. A token is base64 of `YYYYMMDDHHMMSS_counter`, so one can be made for any past moment. The service distinguishes three ages, not two: timestamps to 2026-09-12 give 404, and 2026-09-14 onward give 410. There is a retention window in which a result deleted by a release is still remembered as deleted, and a token inside it produces a real 410 that the deployed pipeline maps to `gone`. Seven of nine scenarios pass now. It matters beyond the count: `gone` would otherwise have been first exercised during a release, which is the worst moment to find a handling bug in it -- and the website has to render that state as an action rather than an error. The window moves, so the recorded timestamp will eventually return 404. The quickstart says to find the current boundary rather than reuse it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
My recorded boundary was a day late, and the website session caught it by checking independently -- and found production behaves the same as beta. Re-derived on beta, it is finer than either of us wrote. The same calendar day gives both answers: 2026-09-13 00:00:01 is 404 and 2026-09-13 12:00:00 is 410. So it is a rolling cutoff of roughly eight days that slides forward continuously, not a date boundary. Which makes hardcoding a token actively harmful rather than merely fragile. A fixed timestamp eventually falls out of the window and the test does not fail -- it quietly starts asserting not_found while still passing. Bisect for the boundary at run time. Also recorded: the assumption that cost more than the boundary did. This scenario was written off as untestable because only the service can mint a token, which was never decided, is written nowhere, and was load-bearing for a plan where gone would first be exercised by a curator during a release. Nothing about it looked like a question. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The website session and I each corrected the other on the gone cutoff and both of us were still wrong. One said 09-13, the other 09-14; there is no boundary day at all. Neither had tested within a day, because both were picturing a date cutoff and so both asked which day rather than which shape. The agreement felt settled and was not. Two measurements taken the same way do not corroborate each other -- a shared assumption survives any number of them, and the thing they agree on is the assumption. It took testing at a finer grain than the question assumed. Recorded next to the recipe, because anyone re-deriving the boundary will be tempted to stop as soon as two answers match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The quickstart recorded "first token 1.8s" as a bare number. A re-run of the same token gave 2.0s, so it was one sample presented as a measurement, in a document written to be trusted later. This repository already knows better: the latency figure for the answer endpoint has been wrong three times, and the rule that came out of it was to quote a distribution over the tracked set and never one question. I quoted one run of one token, and put it in a table of pass results where it reads as established. Now stated as both samples with what they are -- two runs of a single token, not a distribution, and not comparable to the answer endpoint medians it sits near in the specs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
T036, on image
cc4315c, inside the container — the deployed code, the installed bundle and the real Analysis Service. Six of nine scenarios pass. First token 1.8s.The adversarial pass found scenario 4 proving nothing
Scenario 4 is the one the quickstart itself calls "the one to automate first and the one worth distrusting" — the assertion that the user's identifiers, filename, sample name and column labels appear in no outbound request.
My first run checked it before the expression summary ran. The expression payload is the one carrying
Patient_001_tumour— the strongest disclosure risk in the feature — and it was not among those examined. The check passed by ordering rather than by evidence.It now runs last, over every aggregate payload produced, and asserts in the other direction too: the disclosing tier's payload must contain the names. Without that, a clean aggregate result is equally consistent with nothing ever having been sent.
Result: three aggregate payloads checked, including the expression one. No forbidden field, no label. Disclosing payload does carry the names.
What passed
cached: true, byte-identicalnot_foundrefused/no_caller, HTTP 200Plus: an
EXPRESSIONresult usedcolumn 1,column 2,column 3and no other form; theidentifierstier named the unmatched identifiers.What could not be run, and why that is not "skipped"
gone— needs a token issued before release 97, which has already deleted them.All three are unit-tested. What is missing is live confirmation, and the record says so rather than implying coverage.
Scenario 3's expectation is also corrected: it asked for a proportion, and the aggregate result has no submitted-total to compute one from.
🤖 Generated with Claude Code