feat(scripts): semantic dedup replay with a coverage gate in front of it (#1451)

[ozand]

Refs #1451. Measurement issue: the deliverable is a number and a recommendation. The replay is here; the number is not, and this PR makes the reason reproducible rather than a claim in a comment.

What the script does

scripts/replay_semantic_dedup.py, four separable stages, all read-only:

stage	what it does
extract	rebuilds the historical candidate set from the ledger alone
audit	decides whether the comparison set the issue names can still be built
embed	batched gateway embedding calls, texts and requests counted separately
score	the decisive number, at a threshold fixed before scoring

Stage 2 is a gate, not a formality: score is unreachable without it, so a partially adjudicable population cannot be scored and presented as the population the baseline describes.

It reproduces #1328 exactly, then stops

$ replay_semantic_dedup.py extract --ledger-dir <copy of state/ledger> \
    --until 2026-09-06T02:15:00+00:00
{"ledger_rows": 28388, "cycles": 1771, "window_hours": 24,
 "name_key_blocks": 122, "later_success": 51, "later_partial": 27}

122 / 51 / 27 is #1328's published "prior failed outcome, same (demand id, path)" row, to the row. The extraction is therefore the same one, not a second invention.

The 109 / 44 row the issue names as the comparison set is the stricter one: prior real failure per the bridge's own criterion, which is bridge._is_real_result applied to the prior's result artifact. Those are pruned:

$ replay_semantic_dedup.py audit --ledger-dir ... --results-index ...
{"name_key_blocks": 122, "adjudicable": 45, "missing_result_artifact": 77,
 "blocked_priors_among_adjudicable": 12,
 "bridge_criterion_set_lower_bound": 33, "bridge_criterion_set_upper_bound": 110,
 "missing_ts_range": ["2026-07-21T01:51:34Z", "2026-08-17T22:45:08Z"],
 "complete": false}
exit status 2

The set size is bounded, not known, because an unaudited prior might have been a blocked stub. Detail and the missing field list are in the issue comment.

The threshold, fixed in code before any scoring

NULL_PERCENTILE = 99.0. The rule: block iff cosine ≥ the 99th percentile of the corpus's own unrelated-pair null distribution, where a pair is unrelated when it shares neither demand_id nor target_path.

A rule rather than a constant, for two reasons stated in the module header. Asymmetric cost: #1328's standing rule is that a falsely suppressed live task costs more than a paid duplicate, and a percentile states the false-block rate against unrelated work directly. Model independence: a hardcoded cosine silently becomes a different rule when the gateway's embedding model changes, and proposal titles from one repository share vocabulary and paths, so their similarity floor is high and unknown in advance.

#1451 rules out importing 0.95 from the ShinkaEvolve ablation. A test asserts that value appears nowhere in the executable code — the module header names it only to record the refusal, and the test strips comments before checking.

Embedding cost

0 texts, 0 requests, no gateway call. The budgeted ~218 would have scored a population that cannot be shown to be the one the 40.4% baseline describes. embed_texts counts texts and requests separately (batching reduces requests, never texts) and refuses a short gateway response rather than padding it; both are covered by tests with a fake client.

Tests

tests/test_replay_semantic_dedup.py, 31 tests, all passing:

• candidate reconstruction: same demand and path cools, different path does not, outside the window does not, a prior success does not, skipped-* is the dedup stack working and not a failure, gz archives and the live file are both read, a torn last line is skipped not fatal, --until reproduces a historical window;
• the ported is_real_result agrees with nanobot.runtime.bridge._is_real_result case by case (ported so the script runs against a field dump without the runtime on the path, so the agreement is asserted rather than assumed);
• the coverage gate: full coverage yields an exact set, a missing prior yields a range, the audit names the dates of what is missing, the subcommand exits non-zero;
• the threshold rule: percentile of the null, unrelated pairs only, refuses when every pair is related, no 0.95 in code;
• scoring reports both directions, and the cost counter separates texts from requests.

Script and test only. No runtime wiring, no proposer change, nothing written on the host.

🤖 Generated with Claude Code