Skip to content

Address the hal datastore path by the model id it publishes - #289

Open
borgr wants to merge 1 commit into
mainfrom
fix/hal-datastore-path
Open

borgr wants to merge 1 commit into
mainfrom
fix/hal-datastore-path

Conversation

@borgr

@borgr borgr commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

The nightly Adapter ingestion workflow has failed on hal since at least
2026-08-31. The validator rejects records whose model_info.id addresses a
directory the file is not in, and hal produced three of them —
google/gemini-2.0-flash sitting in .../gemini-2-0-flash/, openai/gpt-4.1
in .../gpt-4-1/, and anthropic/claude-haiku-4.5 in .../claude-haiku-4-5/.

The path defect

build_eee_record returned slugify(model_slug_clean) as the model directory.
slugify maps every run of non-alphanumerics to a dash, so it turned each dot
in an id into a dash while the id kept it. That published one model under two
spellings, one of which nothing addresses.

Dots win here rather than dashes, because dotted directories are what the
datastore already holds. data/lexam/openai/ has gpt-4.1, gpt-4.1-mini and
gpt-4.1-nano, data/lexam/google/ has gemini-2.5-pro, and
data/wild/openai/ has gpt-4.1. lexam's registry_snapshot.json also lists
openai/gpt-4.1 and google/gemini-2.5-pro as registry-canonical, so dashing
the ids instead would have put hal in conflict with it. hal is the outlier
and the fix is to stop slugifying.

The Anthropic spelling

claude haiku 4.5 had no MODEL_ID_OVERRIDES entry, so it auto-derived to the
dotted anthropic/claude-haiku-4.5. Anthropic's own ids use dashes and the
sibling claude haiku 4.1 entry already spells it anthropic/claude-haiku-4-1,
so this one was wrong independently of the path bug. Adding the override leaves
the published claude-haiku-4-5 directory correct and unchanged.

Verification

A live run over all 12 leaderboards produces 246 records and 0 failures, and
every one now sits at the path its model_info.id names — checked by
recomputing <developer>/<model> from each record's id and comparing it to the
file's parents. Four of the 18 distinct ids move directory: openai/gpt-4.1
(18 records), google/gemini-2.0-flash (16), google/gemini-2.5-pro-preview
(2) and deepseek/deepseek-v3.1 (1). The other 14 are byte-identical in
placement.

Two tests are added. Both fail on the tree without this change.

uv run pytest tests -q gives 1226 passed, 46 skipped, and
uv run --only-group dev ruff check every_eval_ever tests scripts is clean.

Datastore follow-up, not in this PR

The cron store is additive and does not prune, so those four moves leave 20
orphaned directories holding 37 stale records in evaleval/EEE_raw, one per
benchmark collection the model appears in — data/hal-gaia/openai/gpt-4-1,
data/hal-usaco/google/gemini-2-0-flash and so on. That deletion is an
outward-facing change to the published repo and wants its own pass.

The other three nightly failures are separate and unaddressed here.
terminal_bench_2 raises no ranked leaderboard rows found in source HTML,
which is upstream layout drift and a real code fix. global_mmlu_lite and hle
hit HF 503s writing to evaleval/EEE_raw, and exgentic hits a 429 reading
Exgentic/results — both infrastructure rather than code.

build_eee_record slugified the model directory independently of
model_info.id, so every dotted id was published under a directory no
reader holding the id would look in. The validator rejects those records
and the nightly hal ingestion has been red since 2026-08-31.

Also add a claude haiku 4.5 override. Anthropic spells its ids with
dashes and the sibling claude haiku 4.1 entry already does, so the
auto-derived anthropic/claude-haiku-4.5 was wrong on its own terms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant