Mutation testing: the harness, nineteen vectors, the triage record - #64
Open
EvanWinget wants to merge 7 commits into
Open
Mutation testing: the harness, nineteen vectors, the triage record#64EvanWinget wants to merge 7 commits into
EvanWinget wants to merge 7 commits into
Conversation
tools/mutate.py breaks the reference in one small way at a time and asks whether the vector corpus notices. It generates every mutant of every module in python/bitlisp from a fixed edit set (comparison operators flipped to their neighbor or negation, arithmetic and bitwise operators swapped, and/or exchanged, integer constants moved by one, booleans flipped, if tests negated, not removed, raise deleted, break/continue exchanged), runs the corpus against each mutant in a private mirror of the tree, and reports the survivors with their diffs. With --tests, survivors also run the pytest suite, separating what nothing catches from what the tests catch and the corpus does not. Mirrors copy python/ and tools/ and link vectors/ and puzzles/, so mutants never touch the checkout and parallel workers never see each other's edits. The unmutated tree must pass every oracle the mutants face before any mutant runs, otherwise a broken mirror would report every mutant killed (which is how the first pass's --tests column went wrong before puzzles/ was linked). Timeouts count as kills. The unit test checks that every module yields parseable, distinct mutants, that a constant mutant changes exactly its constant, that the mirror shares data and copies code, and that a broken error-code table dies in the corpus. CLAUDE.md gains the command.
Nineteen cases, each a behavior the spec states that no vector exercised, found by tools/mutate.py as mutants the corpus let survive. Every case passes the reference. The two path cases were cross-checked against the consensus oracle (chia-rs, flags 0), the three seal cases against the vendored Bitcoin Core framework. VM.md section 3.1: an interior zero byte in a path atom is not a leading zero byte and is not costed (path_interior_zero_byte, path_leading_and_interior_zero_bytes). The surviving mutant counted every zero byte. VM.md section 2 (D5): the one-byte atom 0x7f in the long form is non-minimal (nonminimal_one_byte_atom_7f), a lone 0xfc prefix byte is bad_encoding (lone_prefix_fc), and a length of 8,191 in the three-byte form is non-minimal (nonminimal_length_e0_at_8191). The corpus pinned the 0x40 boundary but not the 0x2000 one. CONDITIONS.md time asserts: ASSERT_SEQUENCE_HEIGHT accepts 0 and rejects -1 (seqheight_zero, seqheight_negative). The other three time asserts had both cases, this one had neither. CONDITIONS.md section 1, VALIDATION.md rule 6: a reserved declared cost of exactly -1 is bad_condition_arg, not reserved_cost_too_low (reserved_cost_minus_one). The corpus had -500. CONDITIONS.md self asserts: ASSERT_MY_SCRIPTPUBKEY and ASSERT_MY_AMOUNT take exactly one operand (four arity cases). Only the outpoint and taproot asserts had arity cases. CONDITIONS.md message family: a pair in a non-amount specifier field is bad_condition_arg (assure_script_specifier_pair), an empty ASSERT_ANNOUNCEMENT is bad_condition_arity (assert_announcement_arity_zero), and a specifier composing amount with tapleaf parses its fields in operand order (assure_amount_tapleaf_specifier_parses). No composed case had carried an amount before an identity field. VALIDATION.md rule 3 (C9): an amount specifier over a zero-amount prevout balances (amount_specifier_over_zero_amount_input_balances). The amount domain is 0 to MAX_MONEY and the corpus started at 1. CONDITIONS.md seals: the compact-size boundary at 253 in the txid and outputs hash, a 252-byte and a 253-byte output script under SEAL_OUTPUTS and 253 outputs under SEAL. The one-byte form was the only one any seal vector exercised.
docs/mutation-triage.md records what tools/mutate.py found on its first pass and how each survivor was judged: the method, the six survivor classes (equivalent, unreachable guard, same code, model precondition, beyond reach, gap) with the reason each of the first five is accepted without a vector, the per-module numbers on main at PR 60 (1,733 mutants, 1,551 killed, 179 corpus survivors of which the pytest suite kills 55, 3 timeouts), the nineteen gaps with the mutant that exposed each and the vector that closes it, and the representative sites of every accepted survivor. Line numbers are left out on purpose: the classes are what a later pass compares against, and the re-running section says how. docs/README.md gains the entry.
EvanWinget
force-pushed
the
mutation-testing
branch
from
August 24, 2026 04:22
00ac043 to
c3fa730
Compare
A mutant that makes the reference raise outside its error taxonomy stops the corpus runner on an escaping exception before any vector reaches a verdict. That is detection, but by Python rather than by the corpus, and counting it as a kill overstates what the corpus guards. The corpus driver now exits with its own code on a vector's verdict, so any other nonzero exit is classified crashed, and the pytest pass treats only exit 1 (failing tests) as a kill. The summary gains the column and the JSON report carries the verdict. The unit test pins the mapping.
Six findings from the multi-agent review of PR 64, each reproduced before fixing. The runner's malformed-case catch turned ValueError and KeyError escaping a mutated reference into vector verdicts, inflating kill counts. run_vectors gains MalformedCase, a VectorError subclass raised on that path, and the corpus driver classifies it as a crash. The driver stopped at the first failing file in alphabetical order, so a mutant's killed-versus-crashed verdict was an artifact of file ordering and every VM kill first paid the condition and validation files. It now continues past crashing files, lets a vector verdict win over a crash, and visits files cheapest first, in an order the baseline run measures. Under --tests the pytest oracle collected the mirror's own copy of the harness test, which mutated the already-mutated package, built nested mirrors, and manufactured false kills. That test and the corpus-runner test (a repeat of the run every survivor just passed) are excluded, and a generated conftest disables Hypothesis deadlines so parallel suites on a loaded machine cannot fail a latency check and report a false kill. The report keeps each run's output tail so a test kill names its failing test. Negating an if test whose condition is a bare equality-class comparison or a "not" duplicated the comparison swap or the "not" removal, one semantic mutant under two ids. Those negation sites are skipped, and AugAssign, IfExp, and While now get the operator swaps and test negations the docstring already claimed, closing the untouched augmented-assignment and expression-test sites the review counted, the message ledger's "+= 1" among them. Worker mirrors are removed at exit instead of accumulating in the temporary directory. Crashed and timed-out mutants are listed by site like survivors. __init__ is excluded from the inventory: its mutants would measure the export list, not the reference.
The two 253-byte seal cases are removed. The review ran the mutation both ways: the pre-existing seal boundary case and the seal unit suite already pin the 253 side of the compact-size boundary, and with both cases deleted every _compact_size mutant is still killed, so only the 252 case, which is kept, closes a gap. The largest case in the corpus pinned nothing. The two path-cost cases are renamed from interior to trailing zero bytes: programs 0100 and 000100 place the zero byte at the tail, and the old names claimed a position the bytes do not have. Case content is unchanged.
The full re-run after the review fold-ins counts 1,651 mutants: 1,241 killed by the corpus, 225 crashed (detected by Python before any verdict, no kill credit), 180 survived, 5 timed out. The pass table gains the crashed column, and the numbers paragraph is rewritten from the new run: the pytest suite kills 52 of the 180 survivors and 128 survive both oracles. Three former test kills were artifacts of the mirror's own harness test mutating the already-mutated package and now survive genuinely: the error-code guard in errors.py, a frozen flag in conditions.py, and sha256tree's argument index in operators.py. Each already sits in an accepted class. The one new survivor, from the newly mutated expression tests, is the taproot-versus-scriptpubkey name choice in validation.py's unsatisfied-scriptpubkey message. It only names a condition in a message and joins the same-code class beside the name table index.
Owner
Author
Review pass (/code-review 64 max, Fable finders, adversarial verify)Ten confirmed findings. Six are folded into 19a725e, two are vector corrections in dac86ba, and two were dropped at the finding cap but resolved anyway, outcomes below.
Final pass after the fold-ins: 1,651 mutants, 1,241 killed by the corpus, 225 crashed, 180 survived (52 killed by the pytest suite, 128 survive both, all 180 in the accepted classes), 5 timeouts. Corpus at 1,149 cases with this PR's 17 vectors. Lint clean, CI green. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Three commits, in review order:
tools: the mutation harness.tools/mutate.pygenerates small semantic mutants of every module inpython/bitlisp/(flipped comparisons, swapped operators, off-by-one constants, negatediftests, deletedraise, exchangedbreak/continue) and runs the vector corpus against each in a private mirror of the tree. Survivors are reported with diffs.--testsruns the pytest suite over corpus survivors.python/tests/test_mutate.pycovers the generator, the mirror, and one end-to-end kill.CLAUDE.mdgains the command,ci/lint/codespell-ignore-words.txtgainsNotIn(theastclass).vectors: the gaps the first mutation pass found. Nineteen cases across eight files, each a behavior the spec states that no vector exercised. The commit message cites the spec section for each.vectors/README.mdgains a mutation-coverage section.docs: the mutation triage record.docs/mutation-triage.mdwith the method, the survivor classes and why each is accepted, the pass numbers, the gap table, and the representative accepted survivors.docs/README.mdentry.No spec change. No implementation change in
python/bitlisp/: every gap was in the corpus, not the reference.Numbers
On
mainat PR 60 with the new vectors: 1,733 mutants, 1,551 killed (89.5%), 3 timeouts (kills), 179 corpus survivors. The pytest suite kills 55 of those, 124 survive both. Before the nineteen vectors the pass had 195 survivors. Every remaining survivor is in one of five accepted classes: equivalent mutants, guards no input reaches, checks that share an error code with the one that then fires, the transaction model's own preconditions intx.py, and boundaries no vector can carry (the 2^20 and 2^27 length forms, a 12 GB atom, a signature scalar above the group order, a 65,536-byte script).What the gaps were
The ones worth a reviewer's attention:
vm/paths.json). Cross-checked against chia-rs.0x7fin the long form, a lone0xfcprefix, and a length of 8,191 in the three-byte form were all unpinned non-minimal encodings (vm/serialize.json). The corpus pinned the 0x40 boundary and not the 0x2000 one.ASSERT_SEQUENCE_HEIGHThad no zero and no negative case where its three siblings had both.reserved_cost_too_low.ASSERT_MY_SCRIPTPUBKEYandASSERT_MY_AMOUNThad no arity cases.validation/seals.json, cross-checked against the vendored Core framework).Review guide
docs/mutation-triage.mdfirst: it is the argument for what was and was not turned into a vector..venv/bin/python tools/mutate.py --only tx:0 --listprints a diff for any id from the report. Ids are ordinal within a module and shift when the module changes, which is why the record names sites rather than ids..venv/bin/python tools/run_vectors.py(1,151 cases),.venv/bin/pytest python/tests,ci/lint/lint.sh. The full mutation pass is.venv/bin/python tools/mutate.py --tests, about two hours on 8 cores.--module costsruns one module in a couple of minutes.A note on the harness itself
The first
--testscolumn reported every survivor killed. The mirror lackedpuzzles/, which two test modules read at collection, so pytest failed before running a test. The harness now linkspuzzles/and refuses to start unless the unmutated mirror passes the same oracles the mutants face.