Skip to content

Apply Bloom's statement-selection verdicts; add the bloom68 config - #35

Merged
tadamcz merged 9 commits into
mainfrom
bloom-verdicts
Aug 28, 2026
Merged

Apply Bloom's statement-selection verdicts; add the bloom68 config#35
tadamcz merged 9 commits into
mainfrom
bloom-verdicts

Conversation

@tadamcz

@tadamcz tadamcz commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Applies Thomas Bloom's verdicts (emails of 2026-08-25/26) to the two Erdős datasets, and adds the canonical run config. With both bloom_selection subsets in effect, the benchmark totals 68 samples over 65 distinct problems: 50 in erdos (508 contributes three samples) + 18 in erdos_autoformalized (713 contributes two).

erdos (FC-formalized, 48 problems)

  • 208 → erdos_208.parts.i: Bloom's explicit pick, overriding the earlier "stronger part" choice of parts.ii. (812 stays parts.i, already his pick.)
  • 508 (Hadwiger–Nelson) resurrected as three samples: the value-typed HadwigerNelsonProblem stays an excluded manifest row, and three derived prove-or-disprove samples — χ(ℝ²) = 5, = 6, = 7 — stand in for it (Greg's proposal, adopted by Bloom). The derivation is a pure-text special case in scripts/erdos_isolation.py: the vendored Sources/508.lean stays byte-identical to the FC pin (the statement certificate is untouched), the target theorem is cut out, its answer(sorry) placeholder replaced by the literal, and the theorem renamed to HadwigerNelsonProblem.eqN — so each sample's id is its declaration name, per the usual convention. The fast suite re-derives the committed specs byte-for-byte; the container suite compiles them and certifies their .disproof declarations like every other spec (it skips only the answer-form certificate, which cannot apply to a value-typed source, and asserts the derived statement's type is sorryAx-free instead).
  • bloom_selection is now 50 statements over 48 problems.
  • ERDOS_PROBLEM_STATEMENT_SELECTION.md cut down to the selection rules plus Bloom's full list of 70 problem numbers; the per-problem table is redundant with subsets/bloom_selection.json.

erdos_autoformalized (18 problems, 20 statements)

  • New subsets/bloom_selection.json, now apn_erdos_autoformalized's default subset: 18 statements — drops Erdos1206.erdos_1206.parts.ii (part i alone represents 1206) and Erdos1207.erdos_1207 (dropped from the benchmark entirely); 713 stays split into two samples. The manifest and vendored files are untouched.

configs/bloom68.yaml

The canonical eval-set config for the selection (the first git-tracked file in configs/): both tasks with their bloom_selection subsets, run with all affordances — agent_type: deep (subagents) and literature: true (offline arXiv snapshot) — per the 2026-08-28 agreement. gpt-5.6-sol + claude-fable-5, $200/sample cap (agreed; $27.2k hard ceiling), aligned with the recently launched configs (benchmarks@main, inspect-ai==0.3.259, retry_attempts: 5). Points at LeanOpenProblems@main, so launch after merge.

Notes for review

  • Exactly one of χ(ℝ²)=5/6/7 is true, so two of the three 508 samples are disprovable-in-principle statements — worth keeping in mind when counting solve rates.
  • The three derived specs have not been compiled locally (no local Lean toolchain); CI's isolation suite is the authoritative gate. χ(ℝ²) = 5 needs OfNat ℕ∞ 5, which Mathlib has, so this should be routine.
  • Fast suites and mypy pass locally.

Bloom's verdicts (2026-08-25/26) on the Erdős benchmark selection:

- Problem 208: erdos_208.parts.i (his explicit pick, overriding the
  earlier "stronger part" choice of parts.ii; 812 stays parts.i).
- Problem 508 (Hadwiger-Nelson): the value-typed HadwigerNelsonProblem
  stays an excluded row, but three derived prove-or-disprove samples
  (χ(ℝ²) = 5/6/7) stand in for it. The derivation is a pure-text
  special case in scripts/erdos_isolation.py -- the vendored source
  stays pristine upstream FC -- recomputed byte-for-byte by
  tests/test_erdos.py; compile and disproof certification run in the
  container suite like every other spec. The bloom_selection subset is
  now 50 statements over 48 problems.
- Autoformalized problems: new bloom_selection subset (now
  apn_erdos_autoformalized's default) keeping 18 of the 20 statements --
  1206 represented by part i alone, 1207 dropped entirely, 713 kept
  split into its two parts.

The selection doc is cut down to the selection rules; the per-problem
table lives in subsets/bloom_selection.json.
Each derived 508 spec now declares HadwigerNelsonProblem.eq5/eq6/eq7
instead of three same-named statements, restoring the sample-id-is-the-
declaration-name convention and dropping the manifest decl_name
override. The isolation structural gate resolves the source name
(HN_DECL) and maps the cut prediction through the rename.
@tadamcz tadamcz changed the title Apply Thomas Bloom's final statement-selection verdicts Apply Bloom's statement-selection verdicts; add the bloom68 config Aug 28, 2026
@tadamcz
tadamcz marked this pull request as ready for review August 28, 2026 13:12
@tadamcz
tadamcz merged commit 8d201a4 into main Aug 28, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant