OATH is a contract, not a detector — and the contract does not hold: the pilot headline withdrawn, and the verifier swearing oaths to command-line flags - #42
Open
fathomlab wants to merge 81 commits into
Conversation
…-FAILED All 178 committed OATH certificates were on documents this lab wrote. Twelve cycles of instrument work, v0.1 through v0.11, were scored entirely against a corpus the instrument's authors also wrote. That was the largest unexamined assumption in the OATH lane and it is the first thing a hostile reader finds. Pointed at 12 public repositories selected mechanically by GitHub code search -- their own README against their own committed summary-metric receipts -- the shipped v0.11 verifier abstains on 0.9408 of 507 tokens, verifies 17, and accuses 13. Not one accusation is a catch. Every one traces to trigger vocabulary read out of context: `rate` firing on "learning rate" in a limitations paragraph, `mean` firing on the left margin of a pasted describe() table, `score` firing on console output, and -- the purest specimen this lab has produced -- `delta` firing on \Delta inside a LaTeX formula, accusing the literal 1 in \left(1 \pm ...\right). A mathematical constant inside a rendered formula, accused of being an unmet claim. That is the exact class v0.11 just spent a full preregistration retracting four instances of, found in minutes on a document nobody here wrote. The verification column is no better: 16 of 17 come from one repository, 5 of 17 are structurally coincident by the dogfood definition, and hand inspection immediately exceeds that floor -- a bare 3 sworn to `aggregated.standard.final_loss` on two separate lines. Stated fairly: this tests OATH outside its own stated contract. certify.py scopes itself to "the doc's own cited result JSONs"; external READMEs cite none, so the harness supplied them by filename heuristic. This is not a failure to keep a promise. It is a measurement of whether the instrument TRANSFERS to documents not written to be certified -- which is what "trust becomes a measurement" would require, and which had never been measured. What it clarifies is worth more than what it costs. Proof-carrying code does not verify arbitrary binaries; it requires a compiler that emits the proof. The same is true one level up. OATH is not a lie detector aimable at arbitrary prose -- on arbitrary prose it is nearly silent and where it speaks it is wrong. What the certified corpus does demonstrate is a CONTRACT: documents written to carry receipts can be mechanically held to them by anyone, without trusting the author. That is narrower than "trust is a measurement" and considerably more defensible, and it is what the evidence here actually supports today. Second finding, free: this note is itself OATH-FAILED, and is published that way rather than reworded until it passes. It is accused on the tokens it QUOTES as examples of false accusations. OATH cannot tell a mention from a use. The v0.10 panel brushed against this and recorded it as its only MEDIUM-confidence cases; it is named here as a residual, not fixed, because fixing it inside a RECON with no frozen prereg is the move this lab does not make. This is the first OATH-FAILED certificate in the corpus. The ledger now reads `OATH-HELD` x 178, `OATH-FAILED` x 1. RECON. Licenses no claim. Sizes a class and reports a boundary; its numbers are inputs to a successor's bars, not results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…as a loss
papers/LEDGER.md is the document every other claim in this repository is
collateralised against. Under a heading reading "cycles where a preregistered
gate returned INVALID__*" it listed:
cycle 133 - SHIPPED
cycle 134 - PRODUCT
cycle 152 - DO
cycle 154 - REWRITTEN
cycle 156 - BUILT
Two defects, one root cause. The selection test was `"INVALID__" in verdict` --
a substring match over a free-prose verdict blob -- so any cycle whose commentary
discussed an earlier invalid was counted as a machinery refusal. The renderer
then printed that blob's FIRST WORD, which is what made it visible. Cycle 156 is
the cycle that BUILT this ledger, counted as a loss because its verdict text
quotes the ledger's own negatives count.
That is mention-versus-use: a predicate that reads a line cannot tell you what the
line claims. It is the same defect RECON_oath_external_reach_2026_08_26.md
documents in the OATH verifier, in a second instrument written months apart for an
unrelated job. Two tools, one root cause, and the convergence is the finding.
Fixed: a cycle is a machinery refusal when its LEADING verdict token is an
INVALID__*, or when its verdict opens by announcing invalids ("TWO HONEST
INVALIDS"). The renderer now prints the actual INVALID__ token(s) rather than the
first word, so cycle 110 reads INVALID__pipeline_broken instead of `TWO` and
cycle 115 shows both of its invalids. 16 -> 9 machinery refusals, and the 7
mention-only cycles are disclosed in the section rather than dropped silently.
The 62-of-163 negatives ratio is NOT changed. Same broken method -- keyword
matching over prose -- and head-scoping the identical keywords gives 28, but
neither number is right because the field is prose and no keyword test over prose
is a classifier. It is left standing with a disclosure naming the defect, because
it has already been cited in a frozen preregistration that can never be edited,
and restating it without adjudicating all 163 cycles would replace a disclosed
error with an undisclosed one. The honest fix is a machine-readable verdict token
per cycle. That is now owed and named.
Found by an adversarial program audit, not by its author -- who read this list
this morning and went straight past `cycle 133 - SHIPPED`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… had drifted `tests/test_certificate_reproduces.py` is the only thing standing between this repository and a published document that quietly stops earning its certificate. It resolved receipts by looking only NEXT TO the certificate, so any document citing a receipt from another folder came back `missing` and was skipped entirely. That was 36 of 178 -- 20% of the corpus, and disproportionately the cross-arc syntheses most likely to drift. `_resolve_receipts` has taken a `search_root` for exactly this reason since the cross-directory defect was fixed once before; its own docstring records that fix. This caller never got it. `corpus_audit` passes it. The test did not, and neither does any census or baseline script in papers/ -- including the v0.11 baseline written this morning, which is why the v0.11 "certified frame" is 140 documents and not 178. Passing search_root turned the guard red on FINDING_behavioral_sycophancy_blackbox_2026_06_09.md: committed OATH-HELD (34/27/0) in June, OATH-FAILED (34/27/1) at HEAD with all four receipts resolved and byte-matching. Newly VISIBLE, not newly drifted -- it has been failing behind the resolver the whole time. The accusation looks correct. Line 13 of that FINDING is TRUNCATED in the source; it ends mid-sentence at "n=48 -> 43 scored (27 HELD, 16 CAVED, 4". The dangling `4` is in the receipts (behavioral_sycophancy_result.json:n_nogate = 4) but the words that would bind it were cut off with the rest of the sentence, while 27 and 16 still bind through HELD/CAVED. The verifier is reporting a real defect in a published FINDING. Recorded in KNOWN_VERDICT_DRIFT with that diagnosis rather than repaired: editing and re-certifying a published document is its own cycle with its own prereg, and the entry exists so that cycle cannot be forgotten. The v0.11 cycle emptied that set this morning on the strength of a drift census that ran on the same narrow frame. Empty meant the guard could not see, not that there was nothing to find. REPLICATIONS.md now prints the exact expected output of the corpus-audit target, including both disclosed failures. It previously promised "certificate verdicts match exactly" -- so the first outside replicator would have opened with a divergence that was really our own undisclosed known state, wasting the first thing an outside checker ever does for us. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RECON measured that the verifier does not transfer: pointed at twelve
repositories it had never seen it abstained on 94% of tokens and every accusation
it made was false. The conclusion was not that the instrument is broken but that
it is the wrong SHAPE of thing to describe as a detector. Proof-carrying code does
not verify arbitrary binaries; it requires a compiler that emits the proof.
That reframing is worth nothing unless someone outside this lab can act on it, and
until now nothing here told an outside author how to write a certifiable document
or whether they had. Two artifacts close that:
`styxx/oathready.py` (+ `styxx-oathready`) — the author-facing check. It lists
every numeric token, and instead of a verdict it gives an action. Four kinds that
partition the ledger exactly:
bound grounds at a leaf whose path relates to the line. A kept contract.
coincident grounds at a POSITION -- an index, a seed, a step counter. Reported
SEPARATELY and never counted as success: this is the v0.8
CLOSED_NEGATIVE channel, and a coincidence is worse than an
abstention because it looks like evidence.
accused blocks a certificate, and always names the vocabulary that caused
it -- an author cannot act on an accusation they cannot trace to a
word on their own line.
abstained no oath taken either way. Never a failure, never a non-zero exit.
Run on this repo's own flagship v0.11 RESULT it independently reproduces the nine
coincident bindings the dogfood instrument found, with per-token advice.
`OATH_CONTRACT.md` — nine rules, each one learned by getting it wrong, each naming
the cycle that taught it: cite and ship your receipts; persist summary fields
rather than bulk arrays (v0.1); name the quantity on the line (v0.3); never let a
claim sit on a truncated line (the drift found today); know that QUOTING a number
is treated as claiming it; keep configuration off measurement lines; numbers in
formulas are not exempt; full precision obligates regardless (v0.7); and check
what your numbers bind TO, not just that they bind. With an explicit "what this
does not buy you": a document can keep the contract perfectly and be completely
wrong.
Also: the README's front-door claim of "zero false accusations" is WITHDRAWN. It
was true of two frozen corpora and written in the present tense, so it kept
asserting itself as the corpus grew. Our own committed harness at 7.46.0 reports 4
contradictions, and hand-adjudicating all four finds every one is a false
accusation of the same family -- a filename MENTIONED in a commit message treated
as a file the diff must contain. One commit says a candidate file is "nothing
alike"; one describes a fix in someone else's repository; one is a commit from
today whose message discusses the document it reports a defect in. They are stated
openly, are not closed, and are owed a fix with its own prereg.
That makes four instruments in this repository with one root defect -- the OATH
obligation predicate, OATH on quotations, the ledger's refusal classifier, and now
diffgate's claim extractor. All read a line and call it a claim.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion from a use
Three unrelated investigations today each ended at the same sentence, and a fourth
turned up while checking the third.
1. The OATH obligation predicate, outside this corpus: 13 accusations across 12
public repositories, not one a catch. `rate` fires on "the learning rate of
the low-rank matrices"; `mean` on the left margin of a pasted describe()
table; `delta` on \Delta inside a LaTeX formula, accusing a mathematical
constant of being an unmet claim.
2. The OATH verifier on quotation: the document reporting that is itself
OATH-FAILED, accused on the tokens it quotes as examples.
3. The ledger's refusal classifier: substring-matching a free-prose verdict
field, so `SHIPPED`, `PRODUCT`, `DO`, `REWRITTEN` and `BUILT` were listed as
runs a preregistered gate refused. Cycle 156 -- the cycle that BUILT the
ledger -- was counted as a loss because its verdict quotes the ledger's own
negatives count.
4. diffgate's claim extractor: 4 contradictions at 7.46.0, all four false
accusations, a filename MENTIONED in a commit message treated as a file the
diff must contain.
The common cause: every one infers CLAIMHOOD FROM CO-OCCURRENCE. They read what
sits near a token and conclude what the token asserts. That proxy holds inside a
corpus whose authors share an idiom and does not degrade gracefully when the idiom
changes.
The uncomfortable part is where it breaks hardest: writing that is ABOUT claims
rather than making them. A recon quoting false accusations, a ledger describing its
own negatives, a commit message reporting a defect in a file it does not touch, a
limitations paragraph discussing hyperparameters. That is a genre, and it is the
genre this laboratory produces most of. Four tools calibrated for the papers we
would write if we were not doing the thing we say makes us different.
This synthesis is itself OATH-FAILED, accused on digits inside the formula it
quotes as the canonical specimen. An earlier draft quoted that formula twice to
explain why quoting it gets accused, which doubled the count; removing the second
quotation halved it. Both states are in this file's history. Under this verifier
the cost of explaining a defect scales with how carefully you explain it.
v0.11 is the one place this was solved, and its method is the template: it did not
write a cleverer regex, it asked "is this token a claim at all?", put it to a panel
with ties resolved AGAINST the clause, re-checked it blind, and shipped a
structural value-blind predicate over one enumerated class. Claimhood needs its own
predicate, adjudicated rather than inferred from neighbours.
Licenses no fix. Three of the four are unrepaired on purpose with their failing
cases recorded where they re-run; patching four instruments the same afternoon you
notice them is not this lab's method. Owed and named: a mention/use predicate with
its own prereg and panel; a machine-readable verdict token per cycle; and a
standing rule that any new predicate deciding a token by reading nearby text is
assumed to have this defect until adjudicated otherwise.
Second OATH-FAILED certificate in the corpus. The ledger reads HELD x 178,
FAILED x 2.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ourhood is occupied
Twelve preregistered cycles of instrument work, and nobody asked who else had
built one. Searching this repository for the obvious neighbours returns nothing:
statcheck, arXiVeri, Proof-Carrying Numbers and Agent-Native Research Artifacts
appear in no document here. RECON_landscape_2026_08_21 surveys the SILENT-PASS
lane and is untouched by this; the OATH lane simply had no equivalent.
Four sources opened and read:
statcheck (Nuijten et al., deployed at journal scale in psychology since the
mid-2010s) extracts statistical results from a paper and recomputes p-values
from the reported statistic and df. It checks a claim against ITSELF, needs no
contract, and therefore works on documents nobody wrote for it -- the exact
regime our own external recon shows we fail.
arXiVeri (Shin, Xie & Albanie, 2023) matches a target table to its cited source
and matches cells, GPT-4 baseline. Benchmark, not certificate.
Proof-Carrying Numbers (arXiv 2509.06902) is close enough to be uncomfortable:
numeric spans emitted as claim-bound tokens, a verifier in the RENDERER rather
than the model, a declared policy of exact equality / rounding / aliases /
tolerance, fail-closed so an absent mark means unverified, and the stated
principle that trust is earned only by proof. Same architecture, same slogan,
same name family, published earlier.
Agent-Native Research Artifacts (arXiv 2604.24658, 37 authors, April 2026):
structured research packages with evidence grounding every claim AND an
exploration graph preserving failed experiments. Evidence bound to every claim
with the negatives kept first-class is a fair description of this repository.
What appears to remain, stated as NOT FOUND rather than does-not-exist, because
this is one reader and one afternoon:
1. Preregistration compiled into the scorer. protocol.py refuses to emit a
verdict unless the prereg is committed in git history. `preregr` gives
machine-readable preregs; nothing found makes it a runtime precondition.
This is the claim that best survives a hostile expert, because the expert
can run it and watch the refusal fire.
2. A persisted certificate that can later STOP holding -- doc hash, receipt
hashes, verifier hash -- which is what makes drift detectable. One such
drift was found in this corpus today after months hidden. PCN verifies at
render time, ARA at review time; neither keeps an artifact whose later
failure is the signal.
3. The instrument audited by its own standard at scale, including a cycle whose
whole content was retracting four of its own accusations.
The honest read: the category is not new, and the north star's "for the whole of
history no mind has been able to prove its own sincerity" is a poor opening for a
neighbourhood containing a decade-old deployed tool. That gap is now measured and
it is wide.
What we do have that this literature needs is today's other recon. PCN and ARA
both ASSUME the contract is kept. Nobody surveyed here measured what the machinery
does pointed at prose never written to carry receipts -- 94% abstention and 13 of
13 accusations false. That is a negative, and negatives are what this lab is
supposed to be good at.
Owed: a related-work section on the north star, a rewritten opening claim that
survives contact with statcheck, and a real systematic survey with a frozen search
protocol before anything from this lane goes outward.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Yesterday's survey was one reader and one afternoon, and I said so. This freezes the protocol for the systematic pass: 13 verbatim queries in three families, inclusion/exclusion rules, extraction fields, and a pre-committed outcome table. Two things in it are load-bearing. The pilot is DISCLOSED CONTAMINATION. statcheck, arXiVeri, Proof-Carrying Numbers and Agent-Native Research Artifacts are already known, are excluded from the novelty accounting, and cannot be rediscovered and counted as evidence the search works. The pass is scored on what it ADDS. And the outcome table cannot return "novel". Its terminal row is SEARCH_UNDERPOWERED: a null across thirteen queries run by language models is evidence about the search, not about the world, and may not be reported as an absence of prior art. A survey whose search terms are chosen after seeing what they return is a literature review of the reviewer's priors. The hallucination bar is the leg most likely to fail, so every source must be FETCHED by the agent recording it and independently RE-FETCHED by an agent that did not, with dropped sources counted and published. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
I changed how papers/build_ledger.py classifies a machinery refusal and shipped it with nothing asserting the classification is right. tests/test_ledger.py regenerates LEDGER.md and fails on a changed character -- which proves the committed file agrees with its generator, and never that the generator is correct. A self-consistency check will accept a wrong classifier forever; that is precisely how the refusal list came to print `SHIPPED`. Extracted the classifier to papers/ledger_verdicts.py so it can be imported (build_ledger.py executes at import, so it was untestable in place). Regeneration is byte-identical after the extraction. tests/test_ledger_classifier.py asserts it against the real CYCLE_LOG in both directions: the nine genuine refusals must classify as refusals, the seven mention-only cycles must not, and the refusal set is asserted EXACTLY so a new one fails and a repaired one fails too. Each mention-only entry records why it is not a refusal. Plus renderer assertions -- the ledger must never print a verdict's first word again, and cycle 115 must show both of its independent invalids rather than `TWO`. Same shape as the fetch-depth fix and the resolver fix: a guard that was running but could not see what it was guarding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…erday's best claim Scored under PROTOCOL_oath_prior_art_survey_2026_08_26.md, frozen before the pass ran. 13 verbatim queries, 18 sources recorded each with a URL its recorder fetched, 18 survived independent re-fetch by an agent that did not record them, 0 dropped, 20 further leads named as could-not-verify and not counted. Q3 IS OCCUPIED. Yesterday's recon concluded the strongest defensible claim this lab had was "preregistration compiled into the scorer" -- protocol.py refusing to emit a verdict unless the prereg is committed in git -- and named it as the one mechanism no search had found. github.com/alexcard3/honest-signal implements it. firewall.py reads git history and exits non-zero, wired as a required status check so the PR does not merge, when the preregistration is missing, does not strictly precede the result commit, has been edited since, or names a vacuous kill criterion. It has refused a real PR on those grounds. On two counts it is stronger than ours: it gates the MERGE rather than the scoring, and it tests the kill criterion for VACUITY -- a check we do not have and have needed, since "a leg that cannot fail must not gate" is a sentence in our own preregistrations. Q1 is occupied thoroughly (ESCIMate, JATSdecoder, Aletheia, sciwrite-lint, EviBound, Deterministic Integrity Gates). Q2 is heavily neighboured by mature supply-chain standards plus Self-Verifying Measurement Records; whether any of them records the CHECKER's identity so a certificate can be seen to stop holding is unresolved and is claimed by nobody, including us, until 2606.27934 is read in full. One honest null preserved: query 8, run verbatim, returned only ML model-drift material. That is a fact about the phrase, not about the world. A judgement call stated so it can be disputed: EviBound arguably touches all three sub-questions, which would fire the more severe OATH_NOT_NOVEL row. I did not fire it, because its binding is a live tracking-server query rather than a self-contained hashed artifact. Score it differently and the harder row fires. What remains, at the strength that survives: not the instrument, not the certificate, not the enforcement mechanism. The negative result -- every system found assumes the contract is kept, and nobody pointed such machinery at prose never written to carry receipts -- and the scale of self-application. Both are smaller than what was being said yesterday. The document was also improved BY the tool shipped this morning: oathready reported 0 bound / 10 abstained, because its load-bearing numbers were spelled as words and carried none of its receipt. Rewritten to bind, it accused a `20` on a line containing "rate limits" -- `rate` firing on a rate limit, the same mention-versus-use defect, a fifth instrument-hour after the synthesis. Both fixes are recorded in the note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The defining document of this programme opened by claiming that no mind in
history had been able to prove its own sincerity, and that styxx was the first
time trust could be a measurement instead of a gamble. The frozen prior-art
protocol run today found the neighbourhood occupied on every axis the document
reaches for, so the claim is withdrawn in the text rather than quietly softened.
Adds a related-work section, stated in the body and not a footnote, because a
north star corrected by a reviewer is worth less than one that corrects itself:
- checking numbers in documents is a populated field with deployed tools, several
of which need no cooperation from the author -- a regime our instrument does
not reach;
- binding a claim to hashed evidence for later third-party checking is solved
engineering in software supply chains, with at least one close scientific
implementation;
- "proof-carrying numbers" already names and builds the architecture this
document borrows its metaphor from;
- preregistration enforced as a runtime precondition -- the mechanism this lab
believed was its own -- is implemented by honest-signal, whose vacuity check
on the kill criterion is one we lack and should adopt.
What it leaves is narrower and real: a measured account of what happens when the
contract is NOT kept, which every system found assumes away, and the instrument
turned on itself at scale -- including a survey that retired this document's
strongest claim the day after it was made.
The rung ladder is untouched. It was always scoped by experiment rather than by
rhetoric, and that scoping is exactly why this correction costs a paragraph and
not the programme.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ath to being real Two of the four items owed after the prior-art survey. VACUITY. `honest-signal` refuses a merge when the preregistration's kill criterion is vacuous; the survey found it occupying the enforcement mechanism this lab believed was its own, and that check is the half we did not have -- despite "a leg that cannot fail must not gate" appearing in our own preregs, and despite the v0.11 drafting record naming a BLOCKER where "the warrant gate as first drafted could not fail". Priority is not the useful question when someone has a check you need. `Experiment.vacuous_gates` names every gate no outcome row depends on: computed, hashed, displayed and scored while no verdict turns on it. Same contract as `require_power_basis` -- reported by default, raising only under `require_nonvacuous_gates=True` -- because a new refusal must not retroactively invalidate 40 frozen preregistrations whose bars never move. Carried on Verdict so a reader sees it without re-parsing the prereg. Deliberately narrow, and the narrowness is disclosed: single-polarity mention is NOT flagged (a gate appearing once as true, with a wildcard row catching false, genuinely decides the verdict, and the totality check already refuses the case where nothing catches it), and an unfailable BAR needs domain knowledge this parser does not have. Census: 1 of 40 gated preregs trips it, and that one is DISCLOSED rather than hidden. PREREG_c6_derived_bar's G3 carries `power_basis`: "RECORDED, NOT GATED -- value 0 makes this unfailable by construction and that is deliberate." So the finding is not a lie in the corpus, it is a gap in the schema: there is no way to say "compute and report this, but it is not a bar", so an honest author put it in `gates` with an unfailable bar and explained why in prose. A `records` block, or an explicit "gated": false, is now owed. VERDICT TOKEN. The flagship 62-of-163 negatives ratio is keyword-matching over a free-prose field, so it is not a measurement -- the same mention-versus-use defect as everything else found today. The mechanism to fix it now exists: `verdict_token` on a cycle record carrying verbatim the string the scorer emitted, plus `classify()` and `adjudication_coverage()`. `classify` returns None for a token it does not recognise, and the ledger reports unadjudicated cycles as unadjudicated. Defaulting unknown either way would flatter in one direction or the other. Cycle-scoped tokens like V11_BATTERY_VOID are deliberately unrecognised: widening the rule to match VOID anywhere inside a token is exactly the substring move that put SHIPPED in the refusal list, and the test pins that they stay unknown until a human registers them. LEDGER now prints "0 of 163 cycles carry one" beside the disclosure, so the ratio's own maturity is visible next to it and progress is a number rather than a promise. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e designs die on the numbers The last of the four items owed after the prior-art survey. The synthesis named a mention/use predicate as the largest known defect in the OATH lane; this sizes it and freezes a preregistration for the one sliver that is closeable. THE CENSUS scores five candidate markers for "quoted rather than asserted" over the 181-document / 7,665-token frame, on what each REACHES and what it DESTROYS. The column that decides is not `destroys` but `nominal`, because destroying a verification sworn to an array index or a seed is a gain wearing a loss's clothes: quoting verb on the line reaches 0 destroys 540 nominal 501 inline code span reaches 4 destroys 46 nominal 31 blockquote reaches 0 destroys 17 nominal 17 fenced code block reaches 0 destroys 1 nominal 1 LaTeX span on the line reaches 3 destroys 8 nominal 0 Four die. The quoting-verb rule -- the design a reasonable person reaches for first -- is the broad-detector catastrophe in its purest form: 501 genuine verifications destroyed to retract ZERO accusations. LaTeX span is the only candidate that reaches accusations and destroys nothing genuine: all 8 it silences are coincident, three sworn to a `seed`, one to `per_item[90].i`, five to a token string inside a receipt. THE PREREG freezes V12_FORMULA_CONSTANT on that sliver: a status-level ABSTAIN with reason `formula_constant` at the is_spec tier, fired only inside a delimited math span containing a backslash command, value-blind, never non-extraction. Nine gates. G4 is the bar the design lives on -- a single NOMINAL verification destroyed FAILS -- and G7's coverage is checked before any bar is read, because v0.11's equivalent gate passed vacuously on an empty list until an adversarial audit found it. What it deliberately does NOT fix is stated before the gates: 4 of the 8 standing accusations carry no quotation marker at all, including the DePT hyperparameters quoted in prose and the truncated-line `4`. A number quoted in running prose is the general case, this clause reaches none of it, and no candidate in the census does. Calling this a solution to mention/use would be the overclaim this repository exists to prevent. The prereg is a MEMBER of the class it proposes to close -- OATH-FAILED, accused on the exact digits of the formula it quotes as the specimen, third document in a row to be accused on its own example. So the repair carries a pre-committed test transition: when the clause lands, this document's verdict and the synthesis's must both flip to OATH-HELD, regenerated and not hand-edited, or the outcome is V12_UNDERREACH regardless of the other gates. Stated against myself in the text: that makes the most legible success criterion a document by the clause's own author, which is why G7 is blind, its ties resolve against the clause, and the corpus-wide firing surface carries the gate instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI went red on PR #42 claiming FINDING_behavioral_sycophancy_blackbox had been REPAIRED and must leave KNOWN_VERDICT_DRIFT. It had not been repaired. It had become invisible. Receipt JSONs are stored in git as LF and checked out as CRLF on Windows, and every `receipts_sha256` in this corpus was recorded from a Windows working tree -- so the pins are CRLF hashes. On Linux the same committed bytes hash differently, the cross-directory branch of `_resolve_receipts` finds no match, the receipt reports as `missing`, and `_resolvable()` drops the document. Never examined, so never in `drifted`, so the `repaired` assertion fires. Verified by construction: on LF bytes the old comparator matched 0 of 4 receipts and the new one matches 4 of 4 with zero drift. A guard that reports "fixed" when it means "could not look" is the defect class this repository exists to document, and it is the third instance found today -- after the shallow checkout that skipped the ledger test, and the resolver that skipped 36 of 178 certificates. It is also not new. `.gitattributes` carries a note about this exact bug hitting styxx/centroids/*.json -- "the pin was a CRLF-rendered hash, so the LF Linux CI checkout failed to verify" -- fixed there with `-text`, and nowhere else. `_receipt_sha_matches` compares raw bytes first and falls back to both newline normalisations. A receipt is a JSON document; its meaning does not depend on its line endings, so a hash that does is pinning the wrong thing -- it makes a certificate platform-dependent, which defeats the promise that anyone can re-run it. Disclosed weakening, stated in the docstring: two files differing ONLY in line endings now resolve as the same receipt, so the sha certifies content-identity-modulo-newlines rather than byte-identity. The stricter alternative is re-recording every certificate from normalised bytes, which is a corpus migration and needs its own prereg. 8 tests construct the bytes both platforms actually produce rather than trusting whichever one is running them, including that real content drift is still caught and that a genuinely absent receipt is still missing. REPLICATIONS.md's expected corpus-audit output is updated to the current four failures -- one real drift and three deliberate -- with a note for Linux replicators. Publishing an expected output and then letting it go stale is the same trap it was written to close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… its own prereg quotes The frozen prereg went to RUN and died on its own second gate. Published because a negative is the product too, and this one is unusually clean. G1 PASS: the frame reconstructed exactly at its pin. G2 FAIL: the clause fires on 6 tokens where the frozen roster is 11. Under the pre-committed outcome table an under-fire is V12_UNDERREACH -- revert and publish. No gate after G2 was scored, and the battery result names each unscored gate rather than leaving a reader to assume it passed. THE CAUSE IS A DRAFTING DEFECT AND IT IS MINE. The prereg froze G2's roster from the census's LINE-level marker (`latex_on_line`, true of any line carrying a backslash command) and then specified a SPAN-level clause (the token's column must fall inside a $...$ or inline-code span). Those are different populations. The census measured honestly; the prereg quoted its number as though it described a clause that had not been written yet. WHAT SETTLES IT is the pre-committed proof of repair. Both the SYNTHESIS and the PREREG ITSELF had to flip OATH-FAILED -> OATH-HELD, "or the cycle under-reached regardless of the other gates". SYNTHESIS flips. The PREREG does not -- still accused on line 23, on the digits of the formula it quotes as the specimen the clause exists to silence. The reason is entirely structural: the SYNTHESIS writes that formula in inline-code backticks and the prereg writes it as an indented code block. Same characters, same defect, different markdown. Two independent legs say the same thing. A SECOND PREREG DEFECT, disclosed rather than found later: G7 asked a blind adjudicator for 10 non-roster tokens from lines carrying a LaTeX span, and the frame does not contain 10. The population was unsatisfiable the moment it was frozen. G7 was not run for that reason and for the ordinary one -- a warrant panel adjudicates whether a retraction is DESERVED and cannot rescue a clause that does not reach its class. V12_FORMULA_CONSTANT stays in the tree set to False, as V05_APPROX_NOTATION and V08_FLOAT_FIELD_BINDING did after their kills, so the measurement re-runs and the negative is not re-attempted from memory. OFF arm verified inert. NOT licensed: widening conjunct 1 to catch indented code blocks and re-running. The fix looks like four characters and taking it would convert a frozen bar into one that moves when it is inconvenient. A successor needs a SPAN-level census, so the roster it freezes is the population its clause will actually see. Battery hardening worth keeping: the frame reconstruction is now declared in FRAME_EXCLUDE and done in place, because the first version required holding a certificate aside -- and that certificate was the document whose proof-of-repair had to be scored. A reconstruction that hides its own subject answers a question nobody asked. REPLICATIONS.md's expected corpus-audit output updated again (184 certificates, four disclosed failures). Letting a published expected-output go stale is the trap it was written to close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… own machinery
styxx.absence describes itself in one line -- "find the places where NOT MEASURING
reads as a good result." Over 25-26 Aug, four defects answering exactly that
description were found in this repository's own verification machinery. Every one
was found by accident. None was found by the detector.
VP-A tests/test_ledger.py skips on a shallow clone; CI checks out shallow, so
the LEDGER's regeneration guarantee has never run in CI. A skipped test
reads GREEN on a PR.
VP-B test_certificate_reproduces yields only fully-resolvable documents; the
rest are dropped rather than reported.
VP-C a sha mismatch is classified as absent, and every receipt hash here is a
CRLF hash, so on Linux the documents vanished from the guard.
VP-D the v0.11 battery's G4'b used all() over a list that could be empty, so a
panel artifact containing nothing cleared the bar.
Both detectors run on the real pre-fix source at each commit, driven exactly as
benchmarks/silent_pass drives them: 0 of 4 caught. absence returns no finding on
any of the four; loops returns findings on three, none within six lines of the
defect.
THE CONTROLS ARE THE POINT. The first version of this census reported zero
everywhere -- because scan_source returns a LIST while scan_path returns a report,
and the harness read .findings off a list and measured nothing. It was caught
before publication by asking the obvious question of a detector that finds
nothing: would it find something it should? A positive control is flagged, a
negative control stays clean. A census that skipped that step would have published
a far more dramatic and completely false result -- which is itself a silent pass,
in the instrument built to study silent passes.
THE SHARED SHAPE, which is not the shape the detector hunts: none of the four is
"a healthy value returned on a crash" (SP-1/SP-5, which absence does catch and the
positive control confirms). All four are SUCCESS BY EMPTY POPULATION -- the thing
to be checked is filtered away upstream by a skip, a generator guard, or a
mismatch reclassified as absence, and a downstream check then passes over nothing,
reporting the same green it would report on a full population. all() over [] is
not a bug in all(). The code is locally correct; the composition lies.
Licenses nothing. Explicitly NOT that absence is broken -- its recall is published
as a plateau and its LIMITS string says a clean run is not a certificate. NOT that
the class is four. NOT that a detector is feasible: nothing was built, and the
reached-vs-destroyed arithmetic that would decide it has not been done, which is
exactly what v0.12 died of.
Owed: SP-9 VACUOUS_PASS as a subtype IF the arithmetic supports one, added under a
prereg rather than as a side effect (it will LOWER a published recall score, which
is the correct direction and should be deliberate); a population-level census
before any design; and the cheap partial fix that is not a detector at all --
assert the population is non-empty before asserting anything about it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while writing the RECON: CAPSTONE_universal_mind cites mind_v0_validation.json, that file is PRESENT in the tree, and its content differs from what was certified. Not a newline artifact -- genuine receipt drift. The audit reports `receipt-drift 0` over it. VP-C's shape exactly. The cross-directory branch of _resolve_receipts accepts only a sha match and otherwise falls through to `missing`, so a CHANGED receipt is reclassified as an ABSENT one and the whole document drops out of the drift guard. A guarantee prints a zero over it. THE OBVIOUS REPAIR WAS WRITTEN AND THEN REVERTED. Resolving the lone same-named candidate and flagging it as drift keeps the document examinable and makes the drift visible -- and it is wrong. tests/test_corpus_audit.py already pinned why, in a sentence written long before today: "a same-named file with DIFFERENT content must NOT satisfy the receipt -- the search is stricter than location-trust, not looser." This repository is full of files called *_result.json. Resolving one whose content does not match would certify a document against ANOTHER EXPERIMENT'S DATA while reporting success, which is a worse failure than invisibility and is the precise failure this programme exists to prevent. I had the change in, the suite red on the repo's own test, and the honest read was that the repo was right and I was not. Reverted. The resolver is byte-identical to before except for a comment recording why the tempting fix is refused. A test I wrote this morning went with it: it had asserted the reversed behaviour for about ten minutes. Restored to the strict form with the round trip recorded in its docstring rather than quietly deleted, because a test that flips to match the code is worth less than no test. The visibility defect stays OPEN. It needs a REPORTING change -- a channel that distinguishes "no file of that name anywhere" from "a file exists and has changed" -- not a resolution change, and it is owed its own cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… clean one The reporting fix the previous commit said was owed, done without touching the resolver's strictness by one character. The gap was worse than described. `corpus_audit` does not drop a document whose receipts partly fail to resolve -- it certifies it against whatever DID resolve and prints the verdict. CAPSTONE_universal_mind cites twelve receipts, eleven resolve, the twelfth (mind_v0_validation.json) is present in the tree with content that is not what was certified, and the audit printed `[OATH-HELD]` with nothing whatsoever saying the evidence was partial. A verdict from eleven-twelfths of the evidence looked exactly like a verdict from all of it. `classify_missing` says WHY an unresolved receipt did not resolve -- absent, or present-and-changed, or ambiguous when several same-named candidates exist and none matches. `audit_document` records `incomplete_receipts` and `receipt_changed`; `audit_corpus` counts them; the CLI prints them and surfaces the document. It now reads: incomplete 1 receipt-changed 1 [OATH-HELD] INCOMPLETE-RECEIPTS(changed) CAPSTONE_universal_mind_2026_06_10.md Nothing about resolution changed. A changed receipt still does not satisfy a certificate, for the reason tests/test_corpus_audit.py has pinned all along: this repo is full of *_result.json files and accepting a mismatch would certify a document against another experiment's data while reporting success. The previous commit tried the resolution fix and reverted it; this is the half that was actually safe to build. Six tests, including that a COMPLETE receipt set is not flagged -- an alarm that fires on everything is the same defect wearing the opposite coat. REPLICATIONS.md carries the new expected line with an explanation, so a replicator seeing INCOMPLETE-RECEIPTS knows it is disclosed rather than a divergence. VP-C in RECON_vacuous_pass_2026_08_27.md now has a fix for its reporting half. Its resolution half was never broken. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ector SP-9 VACUOUS_PASS was next on the list. It is now cancelled, on measurement, before a preregistration was frozen -- which is the whole point of doing the census first. Five naive syntactic candidates scored over 1,482 Python files on what each REACHES among the catalogued instances versus how many sites it fires on at all: all()/any() over a comprehension 559 sites reaches 1 559 per reach if not x: return 677 sites reaches 1 677 per reach filtered comprehension 3398 sites reaches 0 skip() call 44 sites reaches 1 44 per reach except: pass|continue 327 sites reaches 1 327 per reach Every candidate reaches at most ONE of three and costs hundreds of sites doing it. The cheapest flags every skip() in the suite and reaches only the shallow-clone case; that is a lint rule, not an instrument. None of these is a detector. The best is a highlighter with a good excuse. Scored out of the THREE instances with a syntactic defect site, not the five catalogued: VP-C is a classification decision and VP-E is a data fact, and no AST pattern reaches either. Scoring out of five would have flattered by choice of denominator. The structural reason is the same fact that makes the class dangerous: the emptiness is manufactured UPSTREAM of the check that passes, so a local pattern cannot see it. A composition is not visible at a node. Any future attempt should be runtime -- instrument the check to record the size of the population it examined -- which is a different instrument needing a different census. TWO MEASUREMENT ERRORS made and corrected in producing that table, recorded because they are the same error this lab keeps making. First run scored REACH against the CURRENT tree while the line numbers came from PRE-FIX code, so three repaired defects read as zeroes -- measuring against the wrong population, inside the census written to stop v0.12's exact mistake recurring. Second: VP-D's line was copied from an audit finding's header without being checked; at the catalogued commit that line is `if not PANEL.exists():` and the vacuous bars are forty lines further down. A census that trusts a cited line number is measuring the citation. The RECON's owed-list is corrected accordingly: not a detector. What may still be owed is the seed corpus -- adding these instances to the SILENT-PASS benchmark lowers a published recall score, which is the correct direction, done deliberately under a prereg and never as a side effect of a note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
….12 should have had v0.12 froze its bar against a LINE-level marker and specified a SPAN-level clause. The populations differed, the clause under-reached its own motivating example, and the RESULT's one-line lesson was: freeze the bar against the thing you are going to build. This is that measurement, taken before anything is designed. Five span definitions scored over the 184-document frame on what each REACHES (currently-accused tokens) and DESTROYS (currently-verified), with destroy split by whether the binding is nominal or merely coincident -- the column every design killed in v0.11 and v0.12 died on: S1 dollar span reaches 0 S2 inline code + command reaches 3 nominal 0 coincid 3 corpus 37 / 13 docs S3 indented block + command reaches 3 nominal 0 coincid 0 corpus 3 / 1 doc S4 fenced block + command reaches 0 S5 right of a backslash command reaches 6 nominal 0 coincid 3 corpus 236 / 31 docs Two findings. First, S1 and S4 reach NOTHING. The `$...$` delimiter that v0.12's conjunct 1 led with does not appear around a single accused token in this corpus, which is a large part of why that clause fired on six tokens instead of eleven. Second, and this is what the corpus column bought: S5 reaches all six but touches 236 tokens across 31 documents corpus-wide, while S2 union S3 reaches the SAME six for 40 tokens across at most 14. Same reach in frame, one sixth the regeneration surface, and S3 destroys literally nothing anywhere. Without the corpus-wide measurement S5 would have looked like the obvious winner on the in-frame table alone. Nominal verifications destroyed: ZERO for every surviving candidate. That is the shape v0.11's winning clause had. Licenses nothing and proposes nothing. Reaching an accusation is not the same as being RIGHT to silence it -- that needs a blind adjudication with ties resolved against the clause, and the accusations "look false to their author", which is exactly the judgement a panel exists to replace. In-frame n is 6, and five of the six sit in this lab's own documents from the last two days, which is small and self-referential and any successor must say so. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The span census scored five candidate span definitions and reported
destroys_nominal = 0 for all of them, calling it "the column that decides".
It decides nothing. A permissive control -- no span test at all, every bare
numeral on a line carrying a backslash command -- scores reaches 6,
destroys_nominal 0, identical to the best candidate and to the worst rule
available. A column that reads the same for the best and worst rule
discriminates between neither.
It is vacuous for a reason worth more than the fact: destroys_nominal can
only see the 184 certified documents, and every genuine measurement these
rules would silence lives in one of the ~935 markdown files outside that
frame. The zero was a property of coverage, not of any rule. This is VP-D
from RECON_vacuous_pass_2026_08_27.md, committed the same day, by that
RECON's author, and found by an adversary rather than by me.
Three design lenses and three adversaries returned 11 blockers, unanimous
against freezing. Beyond the vacuous column: both designs silence genuine
claims (a measured cosine, a stated bound, six numerals under a literal
"Testable predictions" heading, and one token the shipped verifier itself
obligates via the AUC trigger); the whole affirmative case is one formula
this lab wrote three times in 24 hours, and the sole externally-authored
specimen is missed by the narrow candidates; and shipping would have
required flipping the author's own certificates FAILED -> HELD.
formula_span_census.py permissive control added; the_column_that_decides
retracted in the receipt with the reason
RECON_v13_not_frozen the full record; OATH-HELD, 15 verified
SYNTHESIS addendum extended 7 -> 8 instances; the 8th is a proxy for
a COST rather than for a class
The synthesis's new 184 was UNGROUNDED on first certification -- the v0.3
count-binding rule, working exactly as designed, because the sentence named
neither `frame` nor `documents`. Repaired by naming the field, not by
weakening the oath. It is back to its three deliberate contradictions, all
on the formula it quotes as its own specimen.
The cycle produced no clause, retracted a column, and cost less than
shipping would have.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…about your frame
protocol.require_nonvacuous_gates closes the STRUCTURAL half of "a leg that
cannot fail must not gate" -- it refuses a prereg whose gate no outcome row
depends on. Its own comment discloses the half it cannot reach:
An unfailable BAR needs domain knowledge this parser does not have and
is not attempted; that residual is disclosed rather than silently
implied to be covered.
That residual is exactly what killed the v0.13 span census. An outcome row
DID name destroys_nominal, so the structural check passed; the bar was
unfailable for a reason no parser could infer from the prereg text. The
technique this cycle produced needs no domain knowledge at all: score the
null rule too, and a column the null rule ties is not a deciding column.
styxx/discriminates.py three verdicts per column -- SEPARATES when some
candidate strictly beats the control,
NULL_TIES_BEST when candidates differ from each
other but none beats doing nothing, DEGENERATE
when everything shares one value. A column the
author declared decisive that returns anything
else is an accusation, and in strict mode an
exception, so it can be called inside a prereg
before a bar is frozen.
formula_span_census.py emits a `discrimination` block, so the retraction
is computed rather than asserted. It returns
destroys_nominal DEGENERATE and reaches
NULL_TIES_BEST -- BOTH columns, not just the one
the adversary found.
OATH_CONTRACT.md rule 10, naming the cycle that taught it.
tests 19 new; the anchor scores the real historical
numbers and requires the module to fire on them.
Changes no certificate, so no preregistration was needed -- same class as
styxx.oathready.
Two things the re-run exposed, both kept rather than papered over:
The RECON is inside the corpus its census measures, so publishing it moves
the counts. Re-running after drafting one section moved two of three
candidates, and by the fixed point S2's reach had gone 3 -> 0. The document
now quotes NO census count anywhere -- only verdicts, which are stable under
corpus growth -- and says why. Census and certificate are mutually
recursive; both were iterated to a fixed point before this commit.
The first version of the new test file pinned those same published scores
and asserted the live receipt still held them. That would have made the
suite a tripwire on corpus growth rather than on the module. The frozen
numbers stay as a historical anchor for the module's logic; what is asserted
against the live receipt is the relation -- no candidate beats the null rule
-- which is what a regression would actually have to break.
Necessary, not sufficient: a column can separate cleanly and still measure
the wrong thing, which is the defect SYNTHESIS_mention_and_use catalogues,
and which this check would have cleared in all seven earlier instances.
2482 passed, 8 skipped. corpus 365 certificates, HELD 358 FAILED 7,
receipt-drift 0, receipt-changed 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fathomlab
changed the title
`.claude/worktrees/` holds agent scratch clones of this repository. Each is
a full copy, so every certificate and every receipt in the tree has a
byte-identical phantom twin inside it -- 1,611 JSON files at the time of
writing, against 0 tracked JSON under `.claude` (its only tracked file is a
skill markdown). Neither `discover_certificates` nor the receipt search had
any exclusion, and both walked straight in.
The denominator was the larger half. The audit enumerated 365 certificates,
of which 178 were phantoms, and reported every finding twice:
before 365 certificates HELD 358 FAILED 7 verdict-drift 4
incomplete 2 receipt-changed 0
after 187 certificates HELD 182 FAILED 5 verdict-drift 2
incomplete 1 receipt-changed 1
Nothing was hidden from CI or from replicators: REPLICATIONS.md documents
`corpus_audit papers/`, and `papers/` contains no worktree, so the published
numbers were right and only a working-copy run was wrong. That is precisely
why it survived -- the defect was invisible from every angle anyone checks.
The other half was a hidden finding. CAPSTONE_universal_mind's
`mind_v0_validation.json` is present-and-CHANGED: one real file whose
content is not what was certified. Its phantom twin made two candidates out
of one, so classify_missing returned `ambiguous` ("several candidates, no
non-arbitrary choice") and the audit printed `receipt-changed 0` over a
receipt it could see had drifted. The comment in `_resolve_receipts` has
said "present and changed" about this exact file since it was written; the
classifier disagreed with it because its search population included files
that are not in the corpus.
Same defect as everything else found today: the population was defined by
what a glob matched rather than by what the thing is. Exclusion is by
directory NAME at any depth, which is deliberately blunt -- a receipt that
exists ONLY inside an excluded directory stays unresolved and reports
`absent`, never resolved out of scratch space. That direction is pinned by
test_receipt_only_in_scratch_is_absent: this fix makes the search NARROWER
and must never turn an unresolved receipt into a resolved one.
Measured before shipping: the exclusion list costs 1 tracked file
(.claude/skills/posture/SKILL.md, not a receipt) and 0 tracked files under
every other excluded name.
REPLICATIONS.md's expected line moves 185 -> 186 certificates, HELD 181 ->
182, for the one document added today. The five disclosed exceptions are
unchanged.
8 new tests. 2490 passed, 8 skipped. ruff clean over styxx/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A survey of every candidate-scoring census in this repository found seventeen. SEVEN declare one column as decisive -- "the column a design lives or dies on", "THE NUMBER THAT KILLS DESIGNS", "the second is the one that kills", "(b) is the bar". TWO of the seven score that column against any baseline. The rest do not know whether their deciding column can fail. That is a statement about evidence, not about defect, and assuming the worst would have been wrong. The mention/use census killed FOUR OF FIVE candidate designs for v0.12. It declares `of_which_nominal` decisive and carries no control. Running the one it never ran -- treat every token as a mention, the most permissive rule the design admits -- gives a control that destroys 5,159 nominal verifications, and all five candidates beat it. The column SEPARATES. That decision was sound, and it is now measured rather than assumed. The same run found a limitation in the new check, now documented and pinned by a test. On a cost/benefit design the null rule wins the BENEFIT column by construction -- a rule that fires on everything catches everything -- so `accusations_reached` comes back NULL_TIES_BEST at a control of 11. That is expected, not vacuity: it means the candidates justify themselves on cost. Declare the cost column as deciding. An instrument that flagged it would have condemned nearly every census in this repository on its first run. The survey's three counts are disclosed as a single automated pass, not a frozen instrument. Checked by hand: the deciding-column declarations are verbatim from the four files named, and the two censuses called out carry no occurrence of control/baseline/null/permissive/trivial/noop anywhere in their source. The population boundary is a judgement -- the same kind this whole document is about getting wrong -- so seventeen is an order of magnitude and seven is a floor. Also fixed, and it was still live: the census DOCSTRING still asserted "that split is the one that decides". The receipt was retracted this morning and the prose above it was not, so the file contradicted itself for several hours. Corrected, with the reason and with an explicit refusal to freeze anything against that column until the frame covers the documents it is supposed to be about. CONVERGENCE, now documented in the census: it reads stored certificates, and documents cite it as a receipt, so the two are mutually recursive. Re-running invalidates the certificates of citing documents; re-certifying those can change what the census reads. It has to be iterated to a fixed point -- two rounds each time today -- and a single run's numbers must not be quoted in a document inside the frame. Both citing documents and the census were converged before this commit. 2491 passed, 8 skipped. corpus papers: 186 certificates, HELD 182 FAILED 4, receipt-drift 0, incomplete 1, receipt-changed 1 -- matching the expected line in REPLICATIONS.md exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…llection
Three cycles died in 24 hours the same way: v0.12 froze a bar against a
marker in this lab's own corpus; v0.13's deciding column could only see the
certified sixth of this lab's documents; the corpus audit was enumerating a
population half of which was a scratch clone. RECON_v13 states the
consequence -- closing the open defect "needs a population this lab did not
write, which nobody has collected." This protocol collects it. It freezes no
clause and licenses no fix.
Committed BEFORE the first API request, so the freeze is provably prior to
the data.
The pilot's largest defect, which nobody had named: both its queries
(all_results.json, eval_results.json) are HuggingFace Trainer conventions.
"The verifier on 14 public repositories" was the verifier on ONE training
harness, and its headline -- "not one accusation is a catch" -- may be a
property of that harness's JSON schema rather than of external prose. Five
queries from other ecosystems are added to break it.
FOUR ADVERSARIAL LENSES, EIGHT VERIFIED, ONE SURVIVED REFUTATION -- and it
was a blocker in my own arithmetic:
20 repos/query x 7 queries = 140 > 120 total cap. A query's take counts
only repos new to the global seen set, so cross-query dedup never reduces
a take. Queries 1-6 fill 20 each, the total cap fires, and query 7 --
evaluation_results.json, the non-HuggingFace eval-harness arm this
protocol exists to reach -- is NEVER ISSUED A SINGLE REQUEST.
Deterministic under every overlap regime tested. Two outcome rows
quantified over "the five new queries" when only four could ever run.
And the protocol asserted, of itself, "a leg that cannot fail must not gate;
every row above is reachable from the collection rule as written." That
sentence was arithmetically false when written. A self-certifying
reachability claim, in a document about vacuous gates, that had not been
checked. Fixed by raising the total to 20x7 exactly -- NOT by reordering
queries (inspection-driven selection) or lowering the per-query cap (which
would silently re-freeze a smaller sample in six other arms) -- plus an
assert so it cannot recur.
The refuters also REFUTED an escalation worth recording: the claim that
query 7 is "the richest arm" came from reading its returns, and is
inadmissible as grounds for promoting it. It is a cap-arithmetic bug and had
to be fixed as one, blind to what the arm contains.
Other amendments, each a defect in what I had written:
control arm said "the same verifier, at the same verifier_sha256",
implemented as a SUM OF STORED COUNTS. The 186
certificates carry TEN distinct verifier shas and only
FOUR came from the current certify.py. Now re-certified
live; drift recorded, not smoothed.
replication arm promised the pilot's 14 were "collected again". They
never were -- PILOT_REPOS was only a flag. Promise
withdrawn; if none reappear, no replication is claimed.
rows 4 and 5 "behave like" is not a predicate; it could be read
either way once numbers were on screen. Now descriptive,
per-arm, with token counts and an obligation-surface
count, and an arm under 200 tokens supports nothing.
README match four exact spellings, so ReadMe.md was NO_DOC -- a
string match standing in for "has a README", the
mention/use shape again. Case-folded, and readme-like
paths recorded for every NO_DOC so NO_PAIR stays
interpretable.
blinding "adjudicators are not told" is worthless when the packet
is 100% accused tokens: membership leaks the verdict.
Packets are now salted with ABSTAINED and VERIFIED
decoys -- which buys the miss rate, the measurement the
pilot never had. On a corpus where the verifier abstains
on 19 tokens in 20, whether those abstentions are
caution or blindness is the more important question.
adjudicators named as what they are: LLM agents of the same family as
the author and the verifier. Correlated error is NOT
solved; their majority is evidence, not ground truth.
Disclosed rather than fixed: the control's accusation column is 11 events in
3 documents from one day, two of which are documents ABOUT the external
corpus whose accused tokens are external numbers quoted in internal prose --
the null arm is partly the treatment arm, quoted. The control frame also
grows underneath the measurement and this protocol's own RESULT will enter
it, so the certificate roster and per-document sha256 are recorded.
styxx-discriminates is deliberately NOT run here, and the refusal is in the
protocol: it compares candidate RULES on a shared frame, and there is one
rule and two POPULATIONS. An earlier draft of the collector called it with
the corpus as a "candidate" -- a category error that would have produced a
verdict-shaped object meaning nothing.
Collector rewritten off the pilot: API fetch instead of cloning (no timeout
silently dropping large repos), per-file provenance (repo, branch, sha,
path, bytes, sha256), receipts in TREE order not name order, and EVERY
selected repository gets a terminal status -- the pilot had four paths that
removed a repo from the denominator without counting it. Receipt rejections
now separate malformed JSON from fetch failure from over-cap, which one
counter had conflated.
29 tests, driving every failure branch. 2520 passed, 8 skipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…running The decoy sampling rule, sample sizes, seed, packet size, tie direction and the question text are all fixed here BEFORE their author saw a single collected token. Verified at commit time: no ledger file exists on disk. That ordering is the only thing that makes the blinding claim worth anything -- choosing a decoy rule after seeing the accusations would be the same post-hoc defect this lane has now catalogued nine times. WHY DECOYS. The protocol's first draft said adjudicators "are not told" what the verifier decided. That is worthless when every item in the packet is an accused token: membership alone leaks the verdict, and a panel that knows the answer measures its own agreeableness. Packets are now salted with ABSTAINED and VERIFIED tokens, shuffled together and presented identically, so inclusion carries no status. Pinned by a test that fails if any packet item ever carries `status`, `receipt_ref`, `arm` or `is_accused`. WHAT THE DECOYS BUY, which is more than blinding. The pilot could only ever measure the false-accusation rate -- of what it flagged, how much was not a claim. The abstained arm gives the MISS RATE: of what the verifier declined to check, how much was a checkable claim. On an external corpus where it abstains on roughly nineteen tokens in twenty, that is the more important number, because it is the one that separates calibrated restraint from inertness. An instrument that accuses nothing because it checks nothing is not accurate. The pilot's flattering 0.9408 abstain share has never been tested against it. The verified arm is a sanity check on the panel rather than the instrument: if adjudicators call verified tokens non-claims at a high rate, the panel and the verifier disagree about what a claim IS and every other number is suspect. TIE DIRECTION. Majority of three; no majority resolves AGAINST the instrument. An accused token nobody can call a real claim counts as a false accusation, and UNSURE never rescues one -- CLAIM/UNSURE/UNSURE is NOT_A_CLAIM. The question text says so explicitly so adjudicators use UNSURE honestly rather than defensively. Three tests pin this direction; it is the one that makes the instrument look worse. Adjudicators are named in the protocol as what they are: LLM agents of the same family as the author and the verifier. Correlated error is NOT solved. Their majority is evidence, not ground truth, unanimity is reported as the correlated-error ceiling rather than as confidence, and packets are retained so a human can re-adjudicate any of it. 14 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… survive
Collected under the frozen protocol, no arguments, 140 repositories across
seven queries. Committed before any adjudication verdict exists.
selected 140 -> CERTIFIED 82, NO_RECEIPT 50, NO_DOC 7,
RECEIPTS_UNPARSEABLE 1. Every repo has a terminal status.
external 82 docs 5179 tokens abstain 0.8183 accusations 366
control 186 docs 7803 tokens abstain 0.2621 accusations 12
The pilot reported abstain 0.9408 and 13 accusations over 507 tokens
(0.0256) and concluded the instrument was inert outside its contract. At
scale the pooled accusation rate is 0.0707 -- nearly THREE TIMES the pilot --
and the abstain share is twelve points lower. Whether those 366 are catches
or false accusations is what the blind panel is for; no such claim is made
here.
THE PILOT WAS NARROWER THAN "HUGGINGFACE TRAINER". Per arm:
all_results.json (pilot) 14 repos 511 tok abst 0.908 acc 0.0274
eval_results.json (pilot) 11 repos 2319 tok abst 0.843 acc 0.1143
metrics.json 9 repos 287 tok abst 0.986 acc 0.0139
results.json 7 repos 119 tok (n<200, no comparison)
scores.json 8 repos 30 tok (n<200, no comparison)
benchmark_results.json 18 repos 1206 tok abst 0.707 acc 0.0166
evaluation_results.json 15 repos 707 tok abst 0.799 acc 0.0750
Query 1 replicates the pilot almost exactly (511 tokens vs 507, abstain
0.908 vs 0.9408, accusation 0.0274 vs 0.0256). Query 2, also a pilot query,
gives 0.1143 -- four times that. The pilot took only 14 repositories across
both queries and query 1 fills first, so its headline was effectively ONE
FILENAME, not one harness. All 14 pilot repositories were re-reached by the
frozen queries, so this is a genuine replication rather than a claim of one.
THE ARM THE CAP BUG WOULD HAVE DELETED. `evaluation_results.json` came back
with 15 certified repos, 707 tokens, and the second-highest obligation
surface of any arm (45 lines). Had the 20x7>120 arithmetic not been caught
before collection, that entire arm would simply have been absent, and two
outcome rows would have been evaluated over four arms while claiming five.
The blocker was fixed blind to what the arm contained; this is what it
contained.
AND THE POOLED RATE IS NOT A PROPERTY OF EXTERNAL PROSE. Accusations are
severely concentrated:
ONE repository supplies 194 of 366 (53.0%); the top three supply 73.2%.
MEDIAN per-repo accusation share is 0.0 -- 43 of 73 repositories with
tokens draw ZERO accusations.
Excluding the single dominant repository: 172 / 4535 = 0.0379.
That repository alone: 194 / 644 = 0.3012.
So "the verifier accuses 7% of external tokens" is a statement about a
handful of documents, and the typical external repository is untouched. The
RESULT must report the distribution, not the pooled rate, and both are
recorded here. This is the same defect shape the lane has been cataloguing:
a pooled number standing in for a population property.
Control arm re-certified live under the pinned verifier as amended: 186
documents, 10 distinct recorded verifier shas replaced by one live run, 1
verdict changed (FINDING_behavioral_sycophancy_blackbox, already the known
drift entry).
Persistence promise held and checked: max context excerpt exactly 200 chars,
no third-party document or receipt bodies vendored, per-file sha256 recorded
so any reader can re-fetch at the pinned sha and verify byte-identity.
Packets built under the rule frozen before the ledger was read: all 366
accusations, 150 abstained decoys, 75 verified decoys, shuffled, 24 packets,
no status in any packet item. The answer key is committed separately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…esults
Post-hoc, labelled as such, and it revises no frozen number. It exists
because the frozen run produced a figure that cannot be read the way the
protocol's outcome table proposed to read it.
Row 6 says: if NO_PAIR dominates, that is "almost nobody publishes a claim
document beside machine-readable results" -- the strongest available
evidence for the contract framing. The frozen run returned NO_RECEIPT for 50
of 140 repositories, which looked like exactly that.
It is not. Of 57 repositories that yielded no receipt pair, 26 (45.6%)
demonstrably carry a results-like JSON file:
21 missed on AFFIX overall_results.json, graph_gcn_results.json
5 missed on CASE All_results.json vs all_results.json
GitHub's `filename:results.json` qualifier tokenises, so it selects
repositories whose file is `overall_results.json`. RECEIPT_NAMES is an exact,
case-sensitive basename set, so the collector then refuses that same file.
The selection rule and the inclusion rule disagree about what a results file
IS, and repositories fall into the gap.
That is RECEIPT_NAMES standing in for the class "machine-readable results
file" -- the defect SYNTHESIS_mention_and_use catalogues, occurring inside
the collector built to measure that defect's reach. Tenth instance, and it
is mine.
WHAT IS NOT DONE: RECEIPT_NAMES is not widened, the corpus is not
re-collected, and no frozen number is revised. Widening the list after seeing
which repositories fell out is selection after seeing returns, which is what
the protocol's opening paragraph forbids. The frozen numbers stand as
collected. What changes is what may be concluded from them: row 6 is
UNREADABLE from this corpus and is withdrawn rather than reported.
The honest position is that this corpus can say what the verifier does to
documents that DO pair with a receipt under a narrow filename convention. It
cannot say how common such pairing is in the world, and it never could.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… flags
The cycle's pre-committed sanity condition on the VERIFIED arm fired. Of the
verified tokens put to the blind panel, the share it agreed were claims came
back at 0.4933. The protocol said in advance that if this was low, "the panel
and the instrument disagree about what a claim IS, and every other number
here is suspect." So the false-accusation and miss rates are reported and are
NOT promoted to findings. The row is honoured, not reinterpreted.
WHAT MADE IT FAIL IS THE RESULT. The known coincidence channel -- a token
grounding against an index or a seed -- accounts for 5 items. The rest are
NOMINALLY bound: the token matched a receipt leaf whose path its own line
names, and the panel still says it is not a claim. Read them and the panel is
plainly right. Every one of these carries OATH-VERIFIED:
20 `--episodes 20 \` a command-line flag
6 `| [Blog 6](blog_post_phase5.md) |` a link label
2.5 `<img src="static/readme_examples/...">` a numeral in an HTML tag
10 `Apple M4 (10 logical cores, 16 GB RAM)` a hardware spec
256 `| Experts | 256 with K=8 |` a configuration value
0 `All scores are in **[0, 1]**` range notation
The verifier did not merely fail to check these. It SWORE AN OATH to them.
SYNTHESIS_mention_and_use catalogues eight instances of a marker being
treated as the class it co-occurs with, and every one is about ACCUSATIONS.
Nobody had looked at the other channel. The same defect is in the
VERIFICATIONS, at roughly half the nominally-bound external tokens sampled --
and a false verification is worse than a false accusation, because the
affirmative attestation is the entire product. An accusation says "check
this". A verification says "I have checked this, and it is sworn to a
receipt." The second is the one an outside reader would rely on. The pilot
could not have seen it; it measured only what the verifier accused.
THE PILOT'S HEADLINE IS WITHDRAWN, and not conditionally. It reported 13
accusations, "not one a catch" -- a false-accusation rate of 1.0 -- and that
became "OATH is a contract, not a detector" in OATH_CONTRACT.md and the PR.
At n=366 across seven query families the rate is 0.2596. The withdrawal
survives the failed gate because the panel's bias has a known DIRECTION: it
rejects tokens the verifier VERIFIED, so applied to accusations it would
produce more NOT_A_CLAIM verdicts, not fewer. 0.2596 is therefore an upper
bound and the genuine-claim share of 0.7404 a lower bound. A stricter panel
only makes 1.0 less tenable.
Why the pilot was wrong is not sample size. Its two queries were both
HuggingFace Trainer conventions and it took 14 repos across them, the first
filling before the second contributed. Arm 1 replicates it almost exactly
(511 tokens vs 507, abstain 0.908 vs 0.9408) while arm 2 gives 0.1143. The
pilot's headline was ONE FILENAME. All 14 pilot repos were re-reached, so
that first row is a real replication rather than a claim of one.
CONCENTRATION. Pooled accusation share 0.0707, but one repository supplies
194 of 366 and the top three supply 0.7322. The MEDIAN repository's share is
0.0 and 43 draw zero. The dominant document is a model card with full
retrieval tables and bootstrap CIs, and its accusations are the LEAST false
in the corpus (0.1856 inside vs 0.343 outside). The instrument fires where
claims are dense, which is correct; the pooled rate just counts how many
claim-dense documents were drawn.
TWO OUTCOME ROWS WITHDRAWN. Row 6 read NO_PAIR as "almost nobody publishes
results" -- but 26 of the probed no-pair repos demonstrably do, missed on
affix (overall_results.json) or case (All_results.json). The abstention rows
are downstream of the failed gate. Neither is concluded from.
THE PANEL IS NOT THREE READERS. 0.9814 unanimity, 11 splits, 16 UNSURE of
1773 votes. That is the correlated-error ceiling disclosed in advance, not
confidence. Packets and answer key are committed so a human can
re-adjudicate; until then this is a machine's opinion about a machine.
RESULT certifies OATH-HELD: 53 verified, 4 abstained, 0 contradicted.
2534 passed, 8 skipped. corpus papers: 187 certificates, HELD 183 FAILED 4,
receipt-drift 0. REPLICATIONS.md expected line updated 186 -> 187.
Licenses nothing: no clause, no bar, no version bump.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pilot's headline is corrected in the three places it was actually shipped, not only in the paper that withdrew it. A retraction that lives only in the RESULT is not a retraction. OATH_CONTRACT.md the "why a contract, not a detector" section README.md the section of the same name styxx/oathready.py the module docstring an adopter reads first Each now states the original finding, marks the second half withdrawn, gives the replacing number (false-accusation rate 0.2596 at n=366 across seven filename conventions), and says why the original was wrong: it replicates on its own query and nowhere else, so it was a fact about ONE FILENAME rather than about external writing. Each also carries the newer and worse finding, because an adopter needs it before adopting anything: of external tokens the verifier VERIFIED, a blind panel judged only about half to be claims at all. The rest are command-line flags, link labels and hardware specifications carrying OATH-VERIFIED because a value happened to match a receipt field. What survives and what does not, stated in all three: proof-carrying cognition requiring an author who emits receipts survives. "Nearly silent outside the contract" does not -- the instrument is noisy in BOTH directions, and the direction that should worry a reader is the one where it speaks confidently. 2534 passed, 8 skipped. ruff clean over styxx/. corpus papers: 187 certificates, HELD 183 FAILED 4, receipt-drift 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… one line
The verifier confesses composition per token; the auditor was still blind to
it. Now it folds each certificate's epistemics_summary into a corpus total,
printed on the audit's own summary line:
epistemics: 6008 verified | obligated 2505 unobligated 3503 (rate 0.5831)
| weakest 2041 (0.3397) | 0 pre-v1
That is the RESULT_unobligated_oath number (0.5811 over 192 certs) recomputed
live over the current 195 -- 0.5831 -- and "0 pre-v1" confirms every
certificate has been reissued under epistemics-summary v1. A single command
now answers, over an entire corpus: of what this swears to, how much was the
verifier ever obligated to examine.
ADDITIVE, invariant frozen and tested: the fold READS the summary and SUMS
it; it touches no verdict, no count, no drift number. test_the_fold_never
_touches_verdict_or_counts re-audits the live corpus and asserts the classic
summary line -- the one REPLICATIONS.md pins -- is byte-for-byte what it was.
Composition is labelled composition, not quality: claim-share stays the
panel's job, and the reading field says so.
Certificates predating v1 contribute nothing rather than garbage and are
counted in certificates_without_summary (0 here, since the corpus was
reissued). The fold is empty-safe.
5 new tests. 2571 passed, 11 skipped. corpus papers: 195 certificates, HELD
190 FAILED 5, classic audit line unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A verified count with no boundary is the green-checkmark half-truth this
instrument exists to reject -- and our own CLI printed exactly that:
"OATH-HELD verified=53" with no hint that most of the 53 were volunteered.
The terminal line now reads:
OATH-HELD verified=10 abstained=7 contradicted=0
of 10 verified: 3 obligated, 7 volunteered (70%) -- 3 by value match alone
An operator sees the boundary without opening the JSON. Read straight off
the certificate's own epistemics_summary; no re-derivation. Pinned by a test
asserting the volunteered percentage reaches the terminal.
(The RESULT cert this was smoke-tested against reverted -- its diff was only
the epistemics_summary block a pre-v1 cert gains on re-issue, verdict and
counts identical.)
2572 passed, 11 skipped. corpus audit line unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sentences out of 2,738
The product claim is "styxx verifies what an AI agent says it changed
against what actually changed." The most meaningful corpus available: this
branch. 54 commits authored by an AI agent, every message gated with
styxx.diffgate against its own parent diff.
commits gated 54
sentences the gate NEVER READ 2,732
claims extracted 6
verified 3
contradicted (leads) 3
Coverage of the agent's actual reporting: ~0.2%. The tests-passed figures,
panel results, invariants, retractions -- the entire substance -- sat in
the never-read band.
ALL THREE CONTRADICTIONS HAND-ADJUDICATED AND NONE SURVIVED: each is prose
ABOUT a file (the drift entry, the CAPSTONE finding, a test-fired
narrative) read as a claim the commit touched it. The catalogued
mention-vs-use defect, three for three, firing on the prose of the agent
that catalogued it. Of what the gate did read, half was misread.
Adjudicator = the messages' author; conflict disclosed, n=3, each case
one-glance checkable.
THE FINDING: diffgate has the same disease OATH had, now measured on the
same terms -- tiny template-bound obligation surface, mention-vs-use
dominating its extractions, and an unmeasured never-read band that this
run makes visible. OATH declines ~2 claims in 5; the agent-gate declines
~499 sentences in 500. This does not falsify the product thesis. It PRICES
it: the pipeline runs end to end, and claim extraction from agent prose is
the open problem, exactly as the obligation predicate is for documents.
Owed: an agent-report extractor measured the way the obligation predicate
now is (blind ground truth, held-out, null-rule control); uncovered_
sentences promoted to first-class the way epistemics_summary promoted the
abstained band; the three false accusations as regression fixtures.
RESULT certifies OATH-HELD 8/0 -- and the new CLI line confessed it in
real time: "of 8 verified: 1 obligated, 7 volunteered (88%)". The
instrument's honesty is ambient now.
2572 passed, 11 skipped. corpus papers: 196 certificates, HELD 191 FAILED
5; corpus epistemics line: 6016 verified, rate 0.5834, 0 pre-v1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verification correctness is not verification coverage -- the one-line charter, now a repo-root document where every number is bound to a committed receipt and the document itself carries an OATH certificate. The four receipts: the 0.5811 internal unobligated-oath census, the 0.72 external composition, the claimhood join (0.8472/0.7826 obligated vs 0.3654 volunteered abroad), and the agent branch attestation (54 commits, 6 claims read, 2732 sentences never read). The verifier's own terminal line on this document: "of 14 verified: 7 obligated, 7 volunteered (50%) -- 3 by value match alone". The positioning doc confesses its boundary the same way every certificate does. That is the position. Root-level doc; corpus audit scans papers/ only -- corpus line unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nned
The two smaller debts from RESULT_agent_gate_boundary_2026_08_30, paid:
1. DiffGate now carries sentences_total and uncovered_texts -- the
never-read band is auditable, not just countable, in the result object,
in --out, and on every CLI run ("never read: N of M sentences"). Same
promotion epistemics_summary gave OATH's abstained band. Observation
only: A/B at the pinned range origin/main..04f7531 reproduces all 54
gates byte-identical on stdout; the receipt differs only in the head
argument spelling ("HEAD" vs explicit sha), verified line-by-line.
2. The three mention-vs-use false accusations from the agent sweep are
pinned verbatim in test_diffgate_false_accusations.py, in the file
whose charter is exactly this. xfail(strict) for the aspiration, a
catalogue test for the live defect: a repair that flips one without
the other fails the suite. No repair is attempted here -- lexical
patches are measured dead in this corpus; the repair must wait for the
blind ground truth the next unit builds.
2575 passed, 11 skipped, 3 xfailed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The follow-up RESULT_agent_gate_boundary owed: measure the extractor against a blind ground truth instead of the author's hand adjudication. Frozen before the measurement harness exists. Red-teamed before freezing -- two adversarial passes, 22 findings, every BLOCKING one incorporated (REDTEAM_agent_claim_prereg_2026_08_30.json, kept verbatim): the flagged set is now explicitly in the packets (the draft's headline estimand had no blind ground truth at all); 1-1-1 splits adjudicate NO-MAJORITY with an instability threshold; the G-V re-run can only use the v2 instruction text pre-committed here; null-rule precision gets inverse-probability weights so the extractor is not graded on its own home turf; the mention-vs-use decoys report but never gate, because their key's author is the contested adjudicator; and the panel can overturn the author -- the 3 hand-adjudicated false accusations publish individually, retraction clause frozen. In this commit: the prereg, the red-team transcript, seat instructions v1 AND v2, the deterministic packet builder, the packets themselves (6 flagged + 294 sampled of 2,824 sentences at pin origin/main..a6994ac, 3 packets x 130 with all 30 decoys in each, dev/held-out split recorded), and the salted SHA-256 of the answer key. The key and its salt exist only outside the repository until every seat output is committed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All 130 labels per seat, all three packets, plus the probe's 130 corpus-or-synthetic calls. Committed BEFORE the answer key enters the repository, exactly as the frozen prereg orders the reveal. Disclosed deviation, recorded in the file itself: seat p2-seat2's first run died on a session usage limit before returning any labels and was re-run fresh with the identical v1 instructions and packet; no labels from the failed run were ever observed. All other seats replayed from cache, byte-identical. A second harness note for the fold to weigh: packet 3's first LAUNCH (killed before any seat completed) briefly carried two phantom duplicate lines from a transcription error; the run was stopped, relaunched from the frozen packet file, and no seat output from the phantom run survives. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
overturns the author on one of three
The reveal and the fold of the frozen prereg, in protocol order: seat
outputs committed first, answer key + salt only after (salted sha256
verified against the freeze commit, and re-verified inside the fold).
Validity: probe J 0.28 (under the 0.5 void line), nine seats at 24/24 on
the gating decoys, unanimity 0.9718, zero NO-MAJORITY. All three gates
evaluated exactly as frozen:
G-V PASS.
G1 PASS by one sentence: never-read claim density 0.0204 against the
0.02 floor -- 6 A's in 294. One fewer and the boundary claim would
have been ruled an overclaim.
G2 PASS with the mandated counts attached: E1 = 2/6 vs best null 3/12
(weighted 0.2061); no significance is claimed at these n.
THE RETRACTION CLAUSE FIRED. The blind majority read the 8cb66a4
sentence as a claim; the synthetic decoy of that exact shape drew the
same reading from eight of nine seats against the author's key; and the
prereg's own frozen tense-and-agency rule sides with the panel. The
boundary RESULT's "three for three false accusations" is withdrawn by
amendment (document re-certified, OATH-HELD 8/0): two of three stand,
the third is a claim the diff does not support -- a defensible catch.
The red-team fix that kept the mention-vs-use decoys out of the validity
gate is the only reason this could surface instead of being instructed
away.
Repair bar now set: DEV-only development, HELD-OUT labels sealed as
salted hashes (salt outside the repo until that cycle), beat N2's 0.2061
weighted precision or stay home.
RESULT certified OATH-HELD 26/0. corpus papers: 197 certificates, HELD
192 FAILED 5. 2575 passed, 11 skipped, 3 xfailed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ists
The repair the baseline RESULT licensed, under its binding rules: DEV-only
development, HELD-OUT sealed, and the bar is N2's weighted precision
0.2061 -- the verb-stem null -- not the current templates.
Four frozen conjuncts (action head, concrete object, stative block, no
other actor) plus ONE exception cited before implementation: negative-
scope statives ("the rung ladder is untouched") DO flag, because the
blind panel adjudicated exactly that sentence A. The exception is
learned from a DEV label and declared here, not discovered later.
The verb list is deliberately the WEAK conjunct -- N2 measures that list
alone, so the candidate earns nothing unless structure adds precision
over it. If it does not, the RESULT carries verbatim "the structural
detector adds no precision over the verb-stem null at this sample size"
and the lexical-death finding extends to shallow structure.
Stage 2 is the failable gate: a fresh blind panel over the detector's own
flags, 60 flagged + 60 unflagged controls + the same 30 decoys, with a
discrimination gate as well as a precision gate. Stage 1's HELD-OUT
report is descriptive by frozen rule -- the sealed split holds 4 A
sentences, under the 5-A floor the baseline prereg set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage 1 of the frozen prereg. styxx/claimdetect.py implements the four conjuncts exactly as specified before it existed -- action head, concrete object, stative block with the declared negative-scope exception, no other actor -- plus the separate RESULT rule for evidence outside any diff. DEV telemetry (NOT a result; the gate is Stage 2's fresh blind panel): STRUCT-1 precision 1.0 (2/2) recall 0.5 N1 path precision 0.2222 (2/9) recall 0.5 N2 verbs precision 0.1053 (2/19) recall 0.5 STRUCT-1 is strictly narrower than the verb list it is built on, at equal recall -- structure rejects what the list accepts. Pinned by a test. THE CENSUS, and the reason this cycle exists: over the pinned 2,824 sentences the templates parsed 7 claims and never read 2,818 sentences. STRUCT-1 says 40 of those are structurally checkable claims the templates could not parse -- 5.71x what the gate read. diffgate now carries them as `unparsed_claims`, and the CLI confesses the number on every run. OBSERVER INVARIANT, tested by breaking it on purpose: claimdetect is imported lazily inside a try/except, so a broken observer changes no verdict, no claim, no count. G-AB re-ran all 54 gates of the pinned attestation range byte-identical. TWO MEASURED MISSES, pinned as tests rather than patched away, because conjuncts 1 and 2 are frozen: "9 new tests." (no action verb) and "certify: collapse the ladder's third rung into spec-or-historical" -- decoy #2, a known-A all nine seats labelled A, missed for want of a concrete object. Recall bounds get published, not fixed quietly. One licensed DEV fix recorded in the source: the stem suffix set lacked `ied`/`ies`, so "Modified" -- the corpus's commonest action verb -- never matched. Morphology, not vocabulary; the N2 verb LIST is unchanged. 2600 passed, 11 skipped, 3 xfailed. LEDGER rebuilt for the new prereg. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…seat ran The frozen Stage 2 needs 60 STRUCT-1-flagged sentences with a floor of 45. Only 38 exist on the pinned corpus (39 at branch head). The sample cannot be drawn and the floor cannot be met on any version of this corpus. Same defect class that killed the v13 ladder prereg: a population unsatisfiable the moment it was written. Caught before collection -- the best time -- and published rather than quietly shrunk. Changed, and only this: the flagged arm becomes a CENSUS of all 38 (removing every degree of selection freedom a 60-of-38 draw never had), the control arm matches at 38 on the unchanged seed, and the floor drops 45 -> 30 because a floor above the population size makes the gate unfailable-by-construction -- which is the same defect as a gate that cannot fail, wearing the opposite coat. NOT changed: G-S2P still requires A-share > 0.2061 with the same verbatim failure sentence; G-S2LIFT still requires flagged > control; same decoys, same instruction texts, same topology, same majority and NO-MAJORITY rules. STRUCT-1 is frozen as shipped (struct-1/2026-08-30, 5534b70) -- no conjunct or regex moves for the rest of this cycle, whatever happens. At n=38 no significance is claimed and none may be quoted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Built per the amendment, before any seat runs. Flagged arm is exhaustive (no selection freedom), controls drawn on the unchanged seed 20260831 from the same available pool, all 30 frozen decoys in every packet, 3 packets of 56/55/55 under opaque ids that distinguish nothing. Key sealed outside the repository; only its salted sha256 is committed here, verified at fold exactly as the baseline cycle did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All nine fresh seats returned on the first run: no re-runs, no v2 instruction text used, no seat discarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
and the control arm holds zero claims.
Stage 2's failable gate, run on a census of every STRUCT-1 flag that
exists. Nine fresh blind seats, three packets, the same thirty frozen
decoys, the same instruction text. No re-runs, no seat discarded, zero
NO-MAJORITY, unanimity 0.9578, every seat cleared validity.
STRUCT-1 flagged (all 38) 16 A A-share 0.4211
matched control (all 38) 0 A A-share 0.0
frozen bar (N2 verb-stem null) 0.2061
G-S2P PASS 0.4211 > 0.2061
G-S2LIFT PASS 0.4211 > 0.0
After a whole family of lexical candidates died measured, one survived --
on a census, so there was no draw to make favourably. The sentences the
detector declined contained no claims at all in this sample.
THE HONEST HALF: 22 of 38 flags are not claims, every one listed, in
three nameable classes -- the action verb sitting inside a quoted code
span, behavioural present ("firewall.py reads git history") that the
frozen tense rule cannot separate from commit past, and splitter
fragments. Two have obvious structural fixes. NEITHER IS APPLIED: the
amendment froze STRUCT-1 for this cycle whatever the outcome, and a
repair built after seeing the failures it must fix is the discipline this
week was spent enforcing.
AND THE DOCUMENT IS OATH-FAILED. One token: the `9` inside the verbatim
quotation "9 new tests.", which this RESULT quotes in order to report
that STRUCT-1 misses it. Mention-versus-use, eleventh catalogued
instance, firing on a paper about a detector built to tell mention from
use. Published FAILED rather than reworded -- quotations do not get
edited to appease an instrument. Two independent instruments named the
same root defect on the same day.
Worth: over the pinned corpus the templates parsed 7 claims and never
read 2,818 sentences; STRUCT-1 flags 40 checkable claims inside that
blind spot, 5.71x what the gate read.
corpus papers: 198 certificates, HELD 192 FAILED 6. 2600 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…, or die measured Answers this repository's own open issue #39, the largest known hole in the flagship instrument: 0.5227 of full-precision decimals sit on lines without trigger vocabulary and are never checked. Every lexical repair to that predicate is measured dead. The 2026-08-28 structural census scored the reporting-verb rule at recall 0.0235 / precision 0.40 -- indistinguishable from obligating everything -- while "decimal with 2+ fractional digits AND outside any code span" scored recall 0.3765 / precision 0.80. But those are IN-SAMPLE, and that RECON says in its own status line that it licenses no clause and no repair. This prereg exists to earn that licence on held-out data or refuse it. The candidate is taken VERBATIM from the RECON's best rule -- nothing added, tuned, or reweighted -- and composes by UNION with the existing predicate, never replacement, under first-writer semantics. THE COST INVERSION, stated before any number: obligating a token that does not bind produces an ACCUSATION, so this predicate's precision IS one minus the false-accusation rate. Precision gates first and recall is secondary -- the opposite weighting from yesterday's claim-detector cycle. A rule that closes the gap by manufacturing accusations has closed nothing. Four gates, each failable. G-O1P precision >= 0.70 held-out (0.80 was in-sample; 0.70 is the regression allowance fixed NOW so it cannot be chosen later). G-O1NULL must beat obligate-everything on the same adjudications. G-O1R recall >= 0.20. And G-O1REG, the ship gate: a corpus-wide A/B where every new accusation is hand-adjudicated and published individually, and ONE false accusation blocks the release regardless of the precision result. Ground truth is a fresh blind panel on abstentions excluded from the RECON's 225 and from both claim-detector cycles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…decoys 198 documents re-certified live; 1,968 abstained tokens survive the exclusions (the RECON's adjudicated triples and both claim-detector packet sets); the frozen predicate splits them 482 positive / 1,486 negative -- the 52% gap, counted on our own corpus. Sample built on seed 20260901, key sealed outside the repo, salted sha256 committed before any seat runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
held-out, and the negative publishes at full prominence. The frozen verbatim sentence: "the structural obligation clause does not survive held-out adjudication." Nine fresh seats, every one 16/16 on decoys, unanimity 0.9762, zero NO-MAJORITY. The census's best structural rule -- 0.80 precision in-sample -- adjudicated 26/58 (0.4483) on fresh blind tokens against a ship bar of 0.70. G-O1NULL and G-O1R both PASSED (it beats obligate-everything 0.4483 > 0.3391 and would catch ~2 in 5 unchecked claims), but obligation manufactures accusations, precision gates first, and a rule that would false-accuse 32 of every 58 tokens it obligates is an accusation generator, not a predicate. G-O1REG never ran. THE FAILURE HAS A NAME: BARS. All 32 would-be false accusations listed; read together they are thresholds -- prereg'd bars, gate criteria, floors, config values -- quantities SET, not REPORTED. The seats separated bar-from-result almost perfectly on one line of context; the predicate sees only the token's shape. The next candidate (precision shape AND outside code AND not bar-adjacent) is named and NOT built -- it goes to its own freeze and its own fresh panel. Also the cleanest demonstration this lane owns of why in-sample licenses nothing: a third of the census rule's apparent precision evaporated on fresh tokens. Had 0.80 shipped, the corpus would carry a wave of false accusations wearing its badge. AND: the RESULT is itself OATH-FAILED, on the denominators inside the counts statement its own prereg mandates verbatim -- range-sanity cannot tell a rate from the denominator that produced it. Second RESULT in two days failed by its own verifier for protocol-mandated text; the pattern is named (the verifier taxes the disclosures the preregs order) and the tax is paid in public, not reworded away. corpus papers: 199 certificates, HELD 192 FAILED 7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ublished test_the_fold_never_touches_verdict_or_counts hardcoded failed == 5. Two RESULTs were then published OATH-FAILED on purpose (the quotation specimen and the mandated-counts specimen), FAILED moved 5 -> 7, and CI correctly went red -- the pin did its job on the wrong kind of drift. The number now lives in ONE place: the test reads the expected audit line out of REPLICATIONS.md and asserts the live corpus matches it. A deliberate change updates one file; an accidental drift still fails both. The held >= 188 floor stays as the denominator guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ZENODO_nothing_crosses_unseen_2026_08_31.md synthesizes the week's arc for deposit: the boundary measured on the agent gate, the blind-panel methodology that can overrule its builder, STRUCT-1's positive, OBLIGATE-1's negative, and the verifier taxing its own mandated disclosures. Certified OATH-HELD 14/0 against seven receipts, one of which is a pre-deposit corpus-audit snapshot -- pinned to the moment BEFORE this document's own certificate joined the count it reports, with the recursion named in the text. The certificate is the corpus's 200th. Audit line now: 200 certificates, HELD 193, FAILED 7. REPLICATIONS bumped; ledger rebuilt. Also closes a protocol gap the deposit assembly caught: the stage2 and obligate1 answer keys were hash-committed at build time but their plaintext never landed after the seat outputs did. Both revealed now, hashes re-verified against the sealed salt on the way in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OBLIGATE-1 died held-out at 0.4483 with its failure class named in one word: bars. OBLIGATE-2 is the same census base plus four frozen bar-markers (comparator adjacency, bar vocabulary, interval position, gate-table criterion cells). Provenance disclosed first: the markers were written after reading the 32 blind false positives, which is what a dev set is for -- and why those 115 adjudications are spent forever. Every gate runs on fresh tokens. New this cycle, learned from the last two: a FEASIBILITY PRECONDITION (>= 30 fresh positives must exist before any packet is built, or the cycle aborts pre-panel), and G-O2BAR, a class-specific kill-shot -- the bar band the conjunct throws away must itself adjudicate < 0.30 CLAIM, or the conjunct is amputating the wrong limb. The corpse is the bar: G-O2NULL requires beating OBLIGATE-1's 0.4483, not just the null -- beating the null but not the corpse means the conjunct subtracted nothing. G-O2REG inherited verbatim: one false accusation in the corpus A/B blocks the ship. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ealed DEV telemetry on the spent adjudications (may not be quoted as a result): the bar-markers keep 13 of OBLIGATE-1's 58 dev positives and 11 of those were CLAIMs. Fresh sample: 60 positive + 30 bar-band + 30 other-negative, seed 20260902, 3 packets of 56, decoy reuse per the prereg's disclosed decision, key sealed with salted sha256 committed before any seat runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
family's first, and it still does not ship
Nine fresh seats, all 16/16 on decoys, unanimity 0.9881, zero
NO-MAJORITY. Two gates passed, two failed, and a frozen line does not
move for one adjudication:
G-O2P PASS 44/59 -> 0.7458 (bar 0.70) -- first held-out precision
survival in the structural obligation family
G-O2NULL PASS beats the null 0.5678 AND the corpse 0.4483
G-O2R FAIL weighted recall 0.0916 vs 0.10 -- mandated sentence:
"the clause is a rounding error on the gap"
G-O2BAR FAIL bar-band CLAIM share 0.3333 vs 0.30 max -- mandated:
"the conjunct is amputating the wrong limb"
G-O2REG never ran. The clause does not ship.
What the failures teach, decomposed in the RESULT: the recall ceiling is
INHERITED from the 2dp-outside-code base (the general remainder
contributes ~624 weighted missed claims vs the bar-band's ~111) -- no
bar-marker tuning can touch it; and one in three discarded bar-adjacent
tokens is a real claim, so the marker needs comparison-ROLE sensitivity,
not wider windows. Both priors go to the successor's freeze; nothing is
built here.
AND the RESULT is OATH-FAILED on 44/59's own digits -- the denominator-
blindness specimen, third consecutive paper. Three specimens is a queue:
the defect graduates to a repair candidate with its own future freeze.
corpus papers: 201 certificates, HELD 193 FAILED 8.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
repair prereg targets what the ledger actually names Three specimen notes blamed range-sanity. The stored ledgers say obligation_source=vocabulary with empty hits: the true mechanism is the v0.3 COUNT-BINDING filter, whose path-vocabulary requirement strips correct receipted denominators (fields named valid/claims never appear in prose). Corrections appended to all three RESULTs at full prominence -- the accusations were wrong for the reason stated, but the clause blamed was not the one firing, and a lab that publishes its verifier's defects publishes its own misdiagnoses the same way. FROZEN: V11_FRACTION_COHERENCE. A/B forming an explicit fraction whose same-line ratio r satisfies round(A/B, decimals(r)) == r binds JOINTLY iff both operands match leaves under ONE common receipt parent -- the shared-subtree requirement replaces path-vocabulary as the anti-coincidence guard. Rescue-only: the clause can never accuse. Out of scope, stated now: bare count pairs and quoted fragments stay accused as the clause's honest boundary. Four gates: fixtures (the ratio-coherent specimens flip, the others must NOT), a five-mutant battery per fixture where one verifying mutant kills the clause, an ABSOLUTE corpus A/B (zero movement other than UNGROUNDED->VERIFIED via derived-fraction, zero HELD->FAILED, one wrong movement blocks the ship), and full-suite invariants. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t leaf Declared before the formal gate run. 115 is the pooled sum 58+57, computed inside the mandated counts statement and stored nowhere; the clause refusing it is the anti-coincidence guard working. 26 was never accused (it already binds through the ordinary ladder). Fixture list corrected; thresholds untouched. Also recorded: G-F3 caught the first implementation firing unconditionally, and caught a patch script that silently failed to apply the strict guard -- comment said strict, code said everything, the absolute gate said no. V11 clause + strict guard + 26-test battery + A/B harness in this commit; formal gate run and ship decision follow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
disclosures -- all four gates PASS, and the gates caught two lies on the way
The first verdict-changing verifier repair since the epistemics freeze.
Integer operands of an explicit A/B whose same-line ratio r satisfies
round(A/B, decimals(r)) == r bind jointly iff both values sit under ONE
common receipt parent. Strictly rescue-only by construction.
G-F1 corrected fixtures PASS (3 specimens flip; the pooled 115 is
REFUSED both times -- its refusal is
part of the gate)
G-F2 five-mutant battery PASS (zero mutants verify)
G-F3 absolute corpus A/B PASS (3 tokens moved, all licensed, zero
wrong, zero HELD->FAILED)
G-F4 invariants PASS (2608 green after the re-issue flow)
THE GATES EARNED THEIR KEEP TWICE: the first implementation fired
unconditionally -- G-F3 caught it re-attributing healthy VERIFIED tokens
and elevating an ABSTAIN through the degenerate 0/38 case; then a patch
script silently failed to install the strict guard -- comment said
rescue-only, code said everything, and the A/B caught the code
contradicting its own comment. Both catches are in the amendment.
Outcome, honestly bounded: the OBLIGATE-2 RESULT re-certifies OATH-HELD;
OBLIGATE-1 and STRUCT-1 stay FAILED on the pooled 115, the bare count
pair, and the quoted fragment -- the clause's frozen boundary, with the
residual classes named as successors. Stored history untouched; verdict
drift tracked by the audit.
corpus papers: 202 certificates, HELD 195, FAILED 7.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First external bug report in this project's history, from its longest-running production MCP user (months of systemwide Claude Code deployment): every observe/verify call on Claude returned classification="adversarial" / gate="fail", because Anthropic's API exposes no logprobs and the vitals fallback classified the ABSENCE of trajectory data as adversarial behaviour. Every tool chain ended in a standing false accusation, and the user assumed it was their fault. It was ours. The instrument could not tell "no data" from "bad data" -- the exact defect class this week's research measured in three other instruments (mention-vs-use, bar-vs-result, denominator blindness), found in production by an outside user before the lab found it itself. The repaired contract: no logprobs -> classification "unmeasured", gate "unmeasured", measured=false, with the reason naming the missing channel and pointing at the text-only tools that need no logprobs. The same third answer diffgate's `measured` flag gives -- the gate did not run, and says so instead of picking the accusing half. Pinned by tests/test_unmeasured_not_adversarial.py (the not-stacc fixture), including the guard that real trajectories still classify. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mped The branch now carries everything a release needs the moment it merges: version 7.47.0 and a CHANGELOG entry covering the epistemics_summary, V11 fraction-coherence (four gates), the never-read band and STRUCT-1 observer on diffgate, and the not-stacc fix (unmeasured, not adversarial, when a provider ships no logprobs). Post-merge main is release-ready: build, twine, tag -- one command each. Full suite green (2614 passed after the editable reinstall refreshed the stale-install detector's metadata -- the detector was correct). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One self-contained HTML file carrying a document's exact bytes, every receipt's exact bytes, the certificate verbatim, and two layers of verification the READER runs: WebCrypto tamper-evidence in any browser (offline, zero external requests) and full re-verification at the real instrument via 'python -m styxx.capsule verify'. Creation refuses to lie: no capsule mints unless every hash matches and the certificate reproduces at the live verifier. The embedded document bytes are the newline-canonical bytes the certificate hashed, making capsules byte-faithful across CRLF checkouts by construction. Tamper battery (11 tests): round-trip, CRLF mint, flipped document byte, doctored receipt, hand-edited ledger, hand-edited verdict, garbage parse, and three creation refusals. First real capsule minted over the OBLIGATE-2 split-verdict RESULT (OATH-HELD 22/19/0) and included; in-browser check confirms integrity badges, band painting, and the TAMPERED banner on a forged copy — where the forged number renders bandless. What no layer proves — receipts back to reality — is printed in the capsule's own footer. Spec frozen first: SPEC_oath_capsule_v01_2026_08_31.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #41. Review that one first — this branch is based on it, so the diff here is the 12 commits on top.
Twenty-one commits, and most of them are corrections rather than additions. The short version: the verifier was pointed outside this lab for the first time, it failed, and everything examined afterwards turned out to be failing the same way — including the claim this lane believed was its strongest.
UPDATE 2026-08-27 — the pilot's headline is withdrawn, and the verified channel is worse
A frozen protocol, red-teamed before collection, then 140 repositories across seven filename
conventions instead of fourteen across two.
(
PROTOCOL_oath_external_corpus_2026_08_27.md→RESULT_oath_external_corpus_2026_08_27.md,OATH-HELD, 53 verified / 0 contradicted.)
The pre-committed sanity gate FAILED. Of tokens the verifier had verified, a blind
three-seat panel agreed only
0.4933were claims at all. The protocol said in advance that ifthis was low, every other number in the cycle is suspect — so the headline numbers are reported and
not promoted to findings. The row is honoured, not reinterpreted.
What made it fail is the result. The known coincidence channel explains 5 items. The rest are
nominally bound — matched to a receipt leaf whose path their own line names — and the panel still
says they are not claims. It is right. Tokens carrying OATH-VERIFIED include a command-line
flag (
--episodes 20), a link label ([Blog 6](blog_post_phase5.md)), a numeral insidean
<img>tag, a hardware spec (Apple M4 (10 logical cores, 16 GB RAM)) and rangenotation (
All scores are in [0, 1]).The verifier did not fail to check these — it swore an oath to them. All eight previously
catalogued instances of the mention/use defect concern accusations. This is the same defect in
the verifications, and a false verification is worse: an accusation says check this, a
verification says I have checked this. That is the one an outside reader relies on. It was found
as a side effect of salting the blind panel with decoys to fix a blinding leak — nobody set out to
look at that channel.
"Every accusation it made was false" is withdrawn — in
OATH_CONTRACT.md,README.mdandstyxx/oathready.py, not only in the paper. At n=366 thefalse-accusation rate is
0.2596. The withdrawal survives the failed gate because the panel'sbias has a known direction: it rejects tokens the verifier verified, so applied to accusations it
yields more NOT_A_CLAIM verdicts, not fewer.
0.2596is an upper bound, and the pilot's1.0is not tenable.Why the pilot was wrong is not sample size. Both its queries were HuggingFace
Trainerconventions, and the first filled before the second contributed. Arm 1 replicates it almost exactly
(511 tokens against 507, abstain
0.908against0.9408); arm 2 gives0.1143.The headline was one filename. All 14 pilot repositories were re-reached, so that arm is a real
replication rather than a claim of one.
And the rate is not a property of external prose. One repository supplies 194 of the 366
accusations; the median repository's share is
0.0and 43 draw zero. The dominant documentis a model card with full benchmark tables, and its accusations are the least false in the corpus
(
0.1856inside against0.343outside).Two outcome rows withdrawn.
NO_PAIRlooked like "nobody publishes results" — but 26probed repositories demonstrably do, missed on affix (
overall_results.json) or case(
All_results.json). That isRECEIPT_NAMESstanding in for the class, inside thecollector built to measure that defect's reach. The list is not widened: doing so after seeing
which repositories fell out is selection after seeing returns.
The panel is not three readers.
0.9814unanimity is the correlated-error ceilingdisclosed in advance, not confidence. Packets and answer key are committed so a human can
re-adjudicate any of it.
The red team earned its keep before a byte moved. Of 8 verified findings, 1 survived refutation:
20 × 7 = 140 against a 120 total cap, so query 7 — the non-HuggingFace eval-harness arm — would
never have been issued a single request. Fixed blind to what the arm contained; it came back with
707 tokens and the second-highest obligation surface of any arm. The protocol had also asserted of
itself that "every row above is reachable from the collection rule as written", which was
arithmetically false when frozen.
The measurement that started it
All 178 committed OATH certificates were on documents this lab wrote. Pointed at 12 public repositories selected mechanically by GitHub code search — their own README against their own committed metrics — the shipped verifier abstains on 0.9408 of 507 tokens, verifies 17, and accuses 13. Not one accusation is a catch. Every one traces to trigger vocabulary read out of context:
rateon "learning rate" in a limitations paragraph,meanon the left margin of a pasteddescribe()table, anddeltaon\Deltainside a LaTeX formula, accusing the literal1in\left(1 \pm ...\right)— a mathematical constant, accused of being an unmet claim.Stated fairly: this tests OATH outside its own stated contract.
certify.pyscopes itself to "the doc's own cited result JSONs"; external READMEs cite none. It is not a broken promise — it is a measurement of whether the instrument transfers, which had never been made.What it clarifies is worth more than what it costs. Proof-carrying code does not verify arbitrary binaries; it requires a compiler that emits the proof. OATH is a contract, not a detector.
So the contract shipped
OATH_CONTRACT.md— ten rules an outside author can adopt without adopting anything else here, each learned by getting it wrong, each naming the cycle that taught it. With an explicit what this does not buy you: a document can keep the contract perfectly and be completely wrong.styxx/oathready.py+styxx-oathready— the check that tells an author whether they kept it. Four kinds partitioning the ledger exactly: bound, coincident (grounds at an index or seed — reported separately, because a coincidence looks like evidence), accused (always naming the word that caused it), abstained (never a failure, never a non-zero exit).Four instruments, one defect
SYNTHESIS_mention_and_use_2026_08_26.md. The OATH obligation predicate, OATH on quotations, the ledger's refusal classifier, and diffgate's claim extractor all infer claimhood from co-occurrence — they read what sits near a token and conclude what it asserts. It breaks hardest on writing that is about claims rather than making them, which is the genre this lab produces most of.Corrections in this branch
papers/LEDGER.mdrefusal listSHIPPED,PRODUCT,DO,BUILTas machinery refusals; cycle 156 (which built the ledger) counted as a loss because its verdict quotes the ledger's own negatives count. 16 → 9REPLICATIONS.mdcorpus-audit targetThe prior-art survey, and what it retired
A protocol frozen before searching, with a pre-committed outcome table naming what each finding would cost, and built so it structurally could not return "novel". 18 sources, all 18 surviving independent re-fetch by an agent that did not record them, 0 dropped.
It fired
DISCIPLINE_PRIOR_ART.honest-signalimplements preregistration-as-runtime-precondition, and better — it gates the merge rather than the scoring and tests the kill criterion for vacuity. The claim is retired in the RESULT and the north star's "the first time trust can be a measurement instead of a gamble" is withdrawn in the text, with a related-work section in the body.Its vacuity check is now adopted (
Experiment.vacuous_gates). Census: 1 of 40 gated preregs trips it, and that one is disclosed — the real finding is a schema gap, not a lie.What survives, at the strength that survives
Q2 is settled and it is ours: arXiv 2606.27934 does not hash the verifier, so a third party cannot detect that the checker changed.
verifier_sha256in every certificate is what makes drift detectable, and it caught a real one here.Beyond that: the negative — every system found assumes the contract is kept, and nobody pointed such machinery at prose never written to carry receipts — and the scale of self-application. Neither is a mechanism. Both are checkable.
v0.12 and v0.13: two clauses proposed, two clauses dead
PREREG_oath_v12_formula_constant_2026_08_26.mdwas frozen, scored, and killed by its owngate (
V12_UNDERREACH). It froze a bar against a LINE-level marker and specified a SPAN-levelclause, so the bar was measuring something the clause would never read. It ships OFF.
A successor census then scored five span definitions and looked like a design ready to write —
every candidate destroying nothing on the column the receipt named "the column that decides".
Three design lenses and three adversaries returned 11 blockers, unanimous against freezing,
and the sharpest was aimed at the census rather than the designs:
Verified by scoring the rule that does nothing at all — no span test, every bare numeral on a line
carrying a backslash command. It scores identically to the best candidate. The column could not
separate the best rule from the worst, so nothing downstream of it was evidence. It was blind
because it can only see the certified sixth of the corpus, and every real measurement those rules
would silence lives in the uncertified rest.
That is a vacuous gate — catalogued as VP-D in
RECON_vacuous_pass_2026_08_27.md, committed thesame day, by that RECON's author, and found by an adversary. v0.13 was not frozen. The full
record, including the other verified blockers (both designs silence genuine claims, including one
token the shipped verifier itself obligates; the entire affirmative case is one formula this lab
wrote three times in 24 hours; shipping would have required flipping the author's own certificates
FAILED → HELD) is in
RECON_v13_not_frozen_2026_08_27.md.What was built instead of a clause
protocol.require_nonvacuous_gatescloses the structural half of "a leg that cannot fail mustnot gate". Its own comment discloses the half it cannot reach — "an unfailable BAR needs domain
knowledge this parser does not have and is not attempted". That residual is exactly what killed
the census: an outcome row did name the column, so the structural check passed.
styxx/discriminates.py+styxx-discriminatescloses it, and needs no domain knowledge,because the technique is empirical: score the null rule too. Three verdicts per column —
SEPARATESwhen some candidate strictly beats the control,NULL_TIES_BESTwhen candidatesdiffer from each other but none beats doing nothing,
DEGENERATEwhen everything shares onevalue. A column declared decisive that returns anything else is an accusation, and in strict mode
an exception, so it can be called inside a preregistration before a bar is frozen.
Run against the span census it returns
destroys_nominal DEGENERATEandreaches NULL_TIES_BEST— both columns, not just the one the adversary found. The receipt carries thatverdict, so the retraction is computed rather than asserted. Changes no certificate, so no
preregistration was needed. Necessary, not sufficient: a column can separate cleanly and still
measure the wrong thing.
Two things the re-run exposed, kept rather than papered over
The RECON is inside the corpus its own census measures, so publishing it moves the counts —
re-running after drafting one section moved two of three candidates, and by the fixed point one
candidate's reach had gone 3 → 0. The document now quotes no census count anywhere, only verdicts,
which are stable under corpus growth. Census and certificate are mutually recursive; both were
iterated to a fixed point before commit.
The first version of the new test file pinned those same published scores against the live
receipt. That would have made the suite a tripwire on corpus growth rather than on the module.
The frozen numbers stay as a historical anchor; what is asserted live is the relation — no
candidate beats the null rule.
The audited corpus was 49% scratch clone
.claude/worktrees/holds agent scratch clones of this repository. Each is a full copy, so everycertificate and receipt in the tree has a byte-identical phantom twin inside it. Neither
discover_certificatesnor the receipt search had any exclusion, and both walked straight in.178 of 365 enumerated certificates were phantoms, and every finding was reported twice.
Nothing was hidden from CI or from replicators —
REPLICATIONS.mddocumentscorpus_audit papers/, andpapers/contains no worktree — which is exactly why it survived: the publishednumbers were right and only a working-copy run was wrong.
The other half was a hidden finding.
CAPSTONE_universal_mind'smind_v0_validation.jsonispresent-and-CHANGED. Its phantom twin made two candidates out of one, so the classifier returned
ambiguousand the audit printedreceipt-changed 0over a receipt it could see had drifted.The comment in
_resolve_receiptshad said "present and changed" about that exact file since itwas written; the classifier disagreed because its search population included files that are not in
the corpus. Same defect as everything else here.
The exclusion makes the search narrower, never looser — a receipt existing only inside scratch
reports
absent, and that direction is pinned by a test.Applied retrospectively: the decision that mattered survives
Seventeen candidate-scoring censuses in this repository. Seven declare one column decisive
("the column a design lives or dies on", "THE NUMBER THAT KILLS DESIGNS", "(b) is the bar").
Two of the seven score it against any baseline.
That is a statement about evidence, not defect — and assuming the worst would have been wrong.
The mention/use census killed four of five candidate designs for v0.12, declares
of_which_nominaldecisive, and carries no control. Running the one it never ran gives a controldestroying 5,159 nominal verifications, and all five candidates beat it. The column
SEPARATES. That decision was sound, and is now measured rather than assumed.The same run found a limitation in the new check, documented and pinned by a test: on a
cost/benefit design the null rule wins the benefit column by construction, so
NULL_TIES_BESTthere is expected rather than vacuous. An instrument that flagged it would have condemned nearly
every census in this repository on its first run.
Also still live until today: the span census's docstring still asserted "that split is the one
that decides" hours after the receipt retracted it. The file contradicted itself. Corrected.
Also in the branch
Three certificates in this branch are OATH-FAILED on purpose, each accused on the example it
cites. They are published failing rather than reworded until they pass.
Suite green (2,491 passed, 8 skipped).
ruffclean overstyxx/. Corpus audit matches the expected line in REPLICATIONS.md exactly: 186 certificates, HELD 182 / FAILED 4, receipt-drift 0, incomplete 1, receipt-changed 1.ruffclean overstyxx/.mainuntouched.🤖 Generated with Claude Code