This repository asks a narrow empirical question: when does a published benchmark score contain performance that fails to transfer to a closely matched replacement? “Benchmark maxing” is the claim being audited; the directly measured construct is renewal-specific score non-transfer, not developer intent or an identified optimization mechanism.
The current paper is an executed, reproducible secondary analysis, not a proposal. It combines a controlled sensitivity demonstration with three public score panels:
| Component | Scale | Result |
|---|---|---|
| ConStat known-exposure controls | 70 committed; 61 selected by an outcome-based notebook rule | 58/70 pass four benchmark-specific screens; 57/61 pass four secondary screens |
| ConStat real-model synthetic renewals | 47 models × 4 benchmarks | Bonferroni core: 2/188 (also Holm); +1 BH-only; 0 BY |
| ConStat surface-rephrase comparator | 47 models × 4 benchmarks | 0/188 global BH |
| GSM1k public–fresh triangulation | 63 models | reporting set: 3/63 Holm∩BY; 11 two-sided BH; 14 one-sided sensitivity |
| MMLU-CF descriptive split comparator | 44 models × 2 protocols | all 88 displayed gaps are within ±2 percentage points |
The conservative ConStat core requires the estimate and source one-sided lower bound to exceed 2 percentage points plus Bonferroni evidence against zero across all 188 cells; those two cells necessarily also pass Holm. This guarantee is conditional on the fixed selected family and valid source p-values, is not selection-adjusted, and does not control a 2-point null. GSM1k's three-cell Holm∩BY intersection is a conservative reporting convention, not a new combined error guarantee, and has no lower-bound test against 2 points. In the purposive 13-model overlap, 6 signs agree and Spearman rho is 0.04; no model-population interval or p-value is claimed.
- Paper — complete manuscript and primary claims.
- Executed results — concise numerical report.
- Analysis contract — estimands, decision rule, correction families, controls, and prohibited claims.
- Executed plan — end-to-end research decisions.
- Source manifest — exact upstream versions, URLs, hashes, licenses, and extraction methods.
- Source inclusion protocol — case-study search and inclusion boundary.
- Corpus dispositions — a decision, reason, locator, date, screener, and adjudication status for every item in the bounded 60-source corpus.
- Renewal evidence map — detailed author-assessment roles or exclusions for all 21 candidates.
- Figure descriptions — accessible long descriptions for the manuscript figures.
- Reviewer record — original critiques and conference-style review rounds.
- Research portfolio — prospective extensions kept separate from executed evidence.
Python 3.13 and the complete dependency graph in uv.lock are used below. The check command validates all 13 tracked source artifacts plus both hash-locked evidence maps and compares 31 regenerated files without modifying committed results.
uv sync --frozen
uv run --frozen python scripts/benchmark_score_audit.py --check
uv run --frozen python -m unittest discover -s tests -vTo re-extract the tracked inputs from local upstream checkouts, pass their paths explicitly:
git clone https://github.com/eth-sri/ConStat.git /tmp/constat
git -C /tmp/constat checkout e691cdd6a9f8cb6c72ad0eee291cd3ee6619b65d
curl -L https://arxiv.org/src/2405.00332v4 -o /tmp/gsm1k-source.tar
curl -L https://raw.githubusercontent.com/microsoft/MMLU-CF/fd89eefd0815a5aed7d99e74eb6a086eabbe448e/README.md \
-o /tmp/mmlu-cf-README.md
uv run --frozen python scripts/prepare_score_audit_data.py \
--constat-tables /tmp/constat/tables \
--gsm1k-source-tar /tmp/gsm1k-source.tar \
--mmlu-cf-readme /tmp/mmlu-cf-README.md \
--output-dir data/score_auditThe importer refuses any upstream artifact whose SHA-256, archive member, row count, or model uniqueness differs from the frozen contract. Its end-to-end fixture and adversarial failure paths are covered by the test suite.
The exact reviewed release is v1.5.0-score-audit. CHECKSUMS.sha256 binds the paper, long descriptions, evidence maps, tracked score inputs, and generated results; CI constructs and verifies its exact path set before review. GitHub's release asset digests provide an independent transport check.
Build the manuscript with Tectonic:
cd paper
tectonic benchmark_maxing_paper.texanalysis/ Frozen retrospective analysis contract
data/ Tracked aggregate score tables and provenance
scripts/ Deterministic import and analysis pipelines
tests/ Rule, hash, panel-size, and frozen-result tests
results/ Generated report, tables, and figures
paper/ LaTeX source, bibliography, and reviewed PDF
research/ Executed plan plus prospective research portfolio
docs/reviews/ Original Korean and English critiques
archive/pilot/ Superseded six-configuration pilot, retained for auditability
The results support renewal-specific public-form score non-transfer in a small Bonferroni core, conditional on the fixed selected family, valid imported p-values, and renewal validity. “Benchmark maxing” remains the audited claim label, not an identified optimization process. The results do not prove intentional targeting, verbatim memorization, a unique exposure channel, population-wide prevalence, cross-renewal stability, or a corrected leaderboard. Aggregate source tables cannot replace item-level re-evaluation, and this retrospective reviewer-informed contract is not a preregistration.
The repository does not currently declare a repository-wide license because the rights holder has not been specified. Upstream material retains its original Apache-2.0, MIT, or CC-BY-4.0 terms; see THIRD_PARTY_NOTICES.md and the source manifest. Public visibility does not itself grant reuse rights.