Conversation
build_eee_record slugified the model directory independently of model_info.id, so every dotted id was published under a directory no reader holding the id would look in. The validator rejects those records and the nightly hal ingestion has been red since 2026-08-31. Also add a claude haiku 4.5 override. Anthropic spells its ids with dashes and the sibling claude haiku 4.1 entry already does, so the auto-derived anthropic/claude-haiku-4.5 was wrong on its own terms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightly Adapter ingestion workflow has failed on
halsince at least2026-08-31. The validator rejects records whose
model_info.idaddresses adirectory the file is not in, and
halproduced three of them —google/gemini-2.0-flashsitting in.../gemini-2-0-flash/,openai/gpt-4.1in
.../gpt-4-1/, andanthropic/claude-haiku-4.5in.../claude-haiku-4-5/.The path defect
build_eee_recordreturnedslugify(model_slug_clean)as the model directory.slugifymaps every run of non-alphanumerics to a dash, so it turned each dotin an id into a dash while the id kept it. That published one model under two
spellings, one of which nothing addresses.
Dots win here rather than dashes, because dotted directories are what the
datastore already holds.
data/lexam/openai/hasgpt-4.1,gpt-4.1-miniandgpt-4.1-nano,data/lexam/google/hasgemini-2.5-pro, anddata/wild/openai/hasgpt-4.1.lexam'sregistry_snapshot.jsonalso listsopenai/gpt-4.1andgoogle/gemini-2.5-proas registry-canonical, so dashingthe ids instead would have put
halin conflict with it.halis the outlierand the fix is to stop slugifying.
The Anthropic spelling
claude haiku 4.5had noMODEL_ID_OVERRIDESentry, so it auto-derived to thedotted
anthropic/claude-haiku-4.5. Anthropic's own ids use dashes and thesibling
claude haiku 4.1entry already spells itanthropic/claude-haiku-4-1,so this one was wrong independently of the path bug. Adding the override leaves
the published
claude-haiku-4-5directory correct and unchanged.Verification
A live run over all 12 leaderboards produces 246 records and 0 failures, and
every one now sits at the path its
model_info.idnames — checked byrecomputing
<developer>/<model>from each record's id and comparing it to thefile's parents. Four of the 18 distinct ids move directory:
openai/gpt-4.1(18 records),
google/gemini-2.0-flash(16),google/gemini-2.5-pro-preview(2) and
deepseek/deepseek-v3.1(1). The other 14 are byte-identical inplacement.
Two tests are added. Both fail on the tree without this change.
uv run pytest tests -qgives 1226 passed, 46 skipped, anduv run --only-group dev ruff check every_eval_ever tests scriptsis clean.Datastore follow-up, not in this PR
The cron store is additive and does not prune, so those four moves leave 20
orphaned directories holding 37 stale records in
evaleval/EEE_raw, one perbenchmark collection the model appears in —
data/hal-gaia/openai/gpt-4-1,data/hal-usaco/google/gemini-2-0-flashand so on. That deletion is anoutward-facing change to the published repo and wants its own pass.
The other three nightly failures are separate and unaddressed here.
terminal_bench_2raisesno ranked leaderboard rows found in source HTML,which is upstream layout drift and a real code fix.
global_mmlu_liteandhlehit HF 503s writing to
evaleval/EEE_raw, andexgentichits a 429 readingExgentic/results— both infrastructure rather than code.