Add the LM-Harmony adapter, keeping its two evaluation protocols unjoinable

[borgr]

Converts the results matrix behind Train-before-Test Harmonizes Language Model Rankings (arXiv:2507.05195) into data/lm-harmony/.

The source is one file — notebooks/all_results.json in socialfoundations/lm-harmony — shaped {block: {task: {hf_model_id: score}}} over four blocks, 27 tasks and 61 models, with a published standard error for every cell. Model ids are already HF developer/model strings, so no developer inference is involved.

Coverage: 3,087 source cells → 61 records carrying 2,928 results, 0 dropped, 3 tasks excluded. All 61 pass the CLI validator with semantic checks on and no warnings.

Decision log

Two protocols, one benchmark, and they must not join. direct_eval is an ordinary zero-shot lm-eval number. train_before_test fine-tunes the model on the task's own training split first, which is the paper's contribution and a different quantity. Both are published, so both are converted. They share an evaluation_name because they are the same benchmark, and only direct_eval takes a canonical metric_id — the fine-tuned protocol gets lm_harmony.train_before_test.<metric>. A consumer joining on accuracy therefore cannot pool a zero-shot score with a task-trained one. Alternative rejected: one metric_id with the protocol only in additional_details, which leaves the wrong join one careless GROUP BY away. Confidence high.

Task metadata read from the task: field, never from filenames. Every dataset, config and split comes from the vendored lm_eval/tasks/**.yaml at the pinned commit, matched on the yaml's own task: value. Matching by filename would have converted BasqueGLUE's qnli and BigBench's social_iqa — right name, wrong dataset, and the record would have validated. Confidence high.

cola is bounded [-1, 1]. It is scored by Matthews correlation, and nine of the 61 models score below zero. Under the [0, 1] the other metrics use, those nine are a hard validator error. Confidence high.

acc_norm → normalized-accuracy. This one I got wrong first. The hosted resolver returns no_match for every spelling of acc_norm, so the first version emitted the bare harness key as the metric_id. The registry does carry it, as normalized-accuracy with acc_norm already an alias — the live Space simply lags the seed data. Nine tasks including mathqa and sciq now carry a joinable canonical id instead of a fragmenting one. Worth generalising: an API no_match is not evidence a registry entity is absent. Confidence high after correction.

Legacy dataset paths resolved, and the source's own spelling kept. Eleven of sixteen dataset_path values are legacy Hub names that redirect (sciq → allenai/sciq, glue → nyu-mll/glue, math_qa → allenai/math_qa). hf_repo carries the resolved repo; additional_details.lm_eval_dataset_path keeps what the harness actually asked for. Confidence high.

Three tasks deliberately not published. wiki_2025, arxiv_2025 and stackexchange_2025 are the paper's post-cutoff perplexity corpora, scored as bits_per_byte for 53 of the 61 models. The repository commits no task definition for them, so the dataset the scores cover cannot be named, and bits_per_byte has no registry metric. They are SourceRecordExclusions in the report rather than records under an invented dataset. Confidence medium — a maintainer may prefer them published with source_type: other. This is the one place I would welcome a different call.

An unrecognised task is a failure, not a skip. A task appearing upstream and absent from the adapter's table exits non-zero rather than being dropped, because its dataset, split and metric would otherwise be guesses. Confidence high.

Two provenance caveats travel with every result rather than being tidied away. lm-eval scores what it calls a task's test docs, which for a benchmark withholding test labels is its validation split. And the published runs cap the scored split with --dataset_param.max_num_test under a seeded permutation, so a split larger than the cap was scored on a random subsample. The source does not state the resulting n per task, so num_samples is left unset rather than derived from the stderr. Confidence high — deriving n would have been inference dressed as data.

Automation

Registered in catalog.py as a daily unit. --emit-source-version resolves and prints the upstream commit without downloading the results file, so an unchanged repository is skipped. The revision is resolved to a sha before anything is fetched, and --allow-unpinned-source is required to proceed without one, so no record cites a moving reference. --input-json replays a saved payload and requires an explicit --revision, so replayed bytes cannot be attributed to whatever main is at conversion time.

Verification

• uv run python -m every_eval_ever validate '<out>/data/lm-harmony/*/*/*.json' — 61 passed, 0 errors, 0 warnings, at the final datastore path with semantic checks on.
• uv run python -m every_eval_ever.check_duplicate_entries — clean.
• uv run pytest tests/test_lm_harmony_adapter.py — 21 passed. Offline and fixture-based, including a merge-gate assertion with run_semantic_checks=True, the [-1, 1] bound, the protocol-separation invariant, exact score and stderr round-trips, and the CLI guards.
• uv run pytest tests — 1,029 passed, 45 skipped.
• uv run ruff check — clean.

Related

• Registry ids this depends on: Add the benchmarks, aliases and LAMBADA split a harness-derived conversion needs eval-card-registry#64
• Data: a submission to evaleval/EEE_datastore follows