Apply Bloom's statement-selection verdicts; add the bloom68 config - #35
Merged
Conversation
Bloom's verdicts (2026-08-25/26) on the Erdős benchmark selection: - Problem 208: erdos_208.parts.i (his explicit pick, overriding the earlier "stronger part" choice of parts.ii; 812 stays parts.i). - Problem 508 (Hadwiger-Nelson): the value-typed HadwigerNelsonProblem stays an excluded row, but three derived prove-or-disprove samples (χ(ℝ²) = 5/6/7) stand in for it. The derivation is a pure-text special case in scripts/erdos_isolation.py -- the vendored source stays pristine upstream FC -- recomputed byte-for-byte by tests/test_erdos.py; compile and disproof certification run in the container suite like every other spec. The bloom_selection subset is now 50 statements over 48 problems. - Autoformalized problems: new bloom_selection subset (now apn_erdos_autoformalized's default) keeping 18 of the 20 statements -- 1206 represented by part i alone, 1207 dropped entirely, 713 kept split into its two parts. The selection doc is cut down to the selection rules; the per-problem table lives in subsets/bloom_selection.json.
Each derived 508 spec now declares HadwigerNelsonProblem.eq5/eq6/eq7 instead of three same-named statements, restoring the sample-id-is-the- declaration-name convention and dropping the manifest decl_name override. The isolation structural gate resolves the source name (HN_DECL) and maps the cut prediction through the rename.
tadamcz
marked this pull request as ready for review
August 28, 2026 13:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Applies Thomas Bloom's verdicts (emails of 2026-08-25/26) to the two Erdős datasets, and adds the canonical run config. With both
bloom_selectionsubsets in effect, the benchmark totals 68 samples over 65 distinct problems: 50 inerdos(508 contributes three samples) + 18 inerdos_autoformalized(713 contributes two).erdos (FC-formalized, 48 problems)
erdos_208.parts.i: Bloom's explicit pick, overriding the earlier "stronger part" choice ofparts.ii. (812 staysparts.i, already his pick.)HadwigerNelsonProblemstays an excluded manifest row, and three derived prove-or-disprove samples — χ(ℝ²) = 5, = 6, = 7 — stand in for it (Greg's proposal, adopted by Bloom). The derivation is a pure-text special case inscripts/erdos_isolation.py: the vendoredSources/508.leanstays byte-identical to the FC pin (the statement certificate is untouched), the target theorem is cut out, itsanswer(sorry)placeholder replaced by the literal, and the theorem renamed toHadwigerNelsonProblem.eqN— so each sample's id is its declaration name, per the usual convention. The fast suite re-derives the committed specs byte-for-byte; the container suite compiles them and certifies their.disproofdeclarations like every other spec (it skips only the answer-form certificate, which cannot apply to a value-typed source, and asserts the derived statement's type is sorryAx-free instead).bloom_selectionis now 50 statements over 48 problems.ERDOS_PROBLEM_STATEMENT_SELECTION.mdcut down to the selection rules plus Bloom's full list of 70 problem numbers; the per-problem table is redundant withsubsets/bloom_selection.json.erdos_autoformalized (18 problems, 20 statements)
subsets/bloom_selection.json, nowapn_erdos_autoformalized's default subset: 18 statements — dropsErdos1206.erdos_1206.parts.ii(part i alone represents 1206) andErdos1207.erdos_1207(dropped from the benchmark entirely); 713 stays split into two samples. The manifest and vendored files are untouched.configs/bloom68.yaml
The canonical eval-set config for the selection (the first git-tracked file in
configs/): both tasks with theirbloom_selectionsubsets, run with all affordances —agent_type: deep(subagents) andliterature: true(offline arXiv snapshot) — per the 2026-08-28 agreement. gpt-5.6-sol + claude-fable-5, $200/sample cap (agreed; $27.2k hard ceiling), aligned with the recently launched configs (benchmarks@main,inspect-ai==0.3.259,retry_attempts: 5). Points atLeanOpenProblems@main, so launch after merge.Notes for review
χ(ℝ²) = 5needsOfNat ℕ∞ 5, which Mathlib has, so this should be routine.