Conversation
lm-eval v0.4 writes a metric key as `acc,none` and its companion as
`acc_stderr,none`. v0.3 and earlier write bare `acc` and `acc_stderr`. The
converter recognised the companion only in the comma form, so converting a
pre-v0.4 log produced two defects at once: every standard error fell through
as a metric of its own, with the spread published as `score_details.score`,
and the sibling score lost its uncertainty, because the lookup built a
`{metric}_stderr,{filter}` key that cannot exist in that format. One sciq
task came out as four results -- accuracy 0.891, "0.0099", acc_norm 0.816,
"0.0123" -- where two of the four are dispersion wearing a score's clothes,
and a consumer averaging that task's scores would mix them in.
Both call sites now test for either spelling, and the companion key is
derived in the same format as the score key it belongs to. The same sciq task
now yields two results, each carrying its own standard error.
`acc_norm` also gains its canonical id. It is length-normalized accuracy, a
different computation from `acc` on the same items, and the registry carries
it as `normalized-accuracy` with `acc_norm` already among its aliases -- the
hosted resolver reports no match only because the live Space lags the seed
data, which is why the id map had it namespaced as an unregistered metric.
The pre-v0.4 format had no fixture, which is why this survived. It has one
now, taken from a real published log, plus a ConverterCase asserting four
results across two tasks, both metric ids present, and an uncertainty on
every score. The fixture sits in its own directory so the existing
directory-scan test still describes what it scans.
No published datastore record carries the bad ids, so this corrects what the
converter would produce rather than data already submitted.
Full suite: 1,003 passed, 45 skipped.
The same format gap as `acc_norm`, one metric further on. lm-eval v0.3 reports perplexity under `ppl` and v0.4 renamed it `perplexity`, but both the canonical id map and the bounds table knew only the new spelling. So a pre-v0.4 log's perplexity came out as an unregistered `lm-evaluation-harness.ppl` with no bounds, which is two records of the same quantity under two ids depending on which harness version produced the log. Found converting lambada_openai, where v0.3 reports `ppl` and `acc` side by side: the accuracy resolved and carried its bounds, the perplexity did not. Full suite: 1,003 passed, 45 skipped.
…arness JSON Stability AI committed the lm-evaluation-harness outputs behind StableLM under `evals/`, and the in-tree lm_eval converter already reads that format. So this adds no conversion. It adds the three things such a repository needs before the converter can be pointed at it, and it is deliberately small enough to copy for the next one. Pin the source: the revision resolves to a commit sha before any file is fetched, so a record cannot cite bytes that moved. Repair the model identity: the converter takes model_info.id from config.model_args's pretrained= value and refuses an id with no publishing namespace, which is right, because a placeholder developer routes unrelated models into one directory. 28 of 29 files name a full repo id. The one run from a local checkout has its org supplied through ORGLESS_MODEL_ORG, where the decision stays visible, and an unlisted org-less id fails its row instead of getting a guess. Pin one collection: left alone the converter files each task into its own bare collection -- data/sciq/, data/piqa/, data/lambada_openai/ -- which mixes these numbers with every other source's records for the same benchmark and loses the provenance. collection_override keeps the source together. 29 open-weights models over 11 benchmarks: Pythia, GPT-J-6B, GPT-NeoX-20B, BLOOM, OPT, LLaMA-1, Llama-2, MPT-7B, Falcon-7B, RedPajama-INCITE, Qwen-7B, Cerebras BTLM, OpenLLaMA, Mistral-7B, phi-1.5, Baichuan2 and five StableLM releases. Two of those benchmarks are why the source is worth having: SciQ for 29 models and LAMBADA-OpenAI for 29, both of which the datastore held almost nothing for. evals/open_llm_leaderboard/ holds per-task files for two models rather than one file per model. That shape is excluded by prefix and reported on every run, so the 8 files are visible rather than silently absent. Coverage: 29 source files -> 293 records carrying 530 results, 0 dropped, 8 skipped. All 293 pass the CLI validator with semantic checks on and no warnings, every result carries bounds, and no duplicates. Includes the pre-v0.4 converter fixes this depends on, so the branch produces correct records on its own; they are also open separately for review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stability AI committed the lm-evaluation-harness outputs behind StableLM under
evals/, and the in-treelm_evalconverter already reads that format. So this adds no conversion. It adds the three things a public repository of harness JSON needs before the converter can be pointed at it, and it is deliberately small enough to copy for the next one.Coverage: 29 source files → 293 records carrying 530 results, 0 dropped, 8 skipped. All 293 pass the CLI validator with semantic checks on and no warnings, every result carries bounds, no duplicates.
What a wrapper has to add
Pin the source. The revision resolves to a commit sha before any file is fetched, so no record can cite bytes that moved.
--allow-unpinned-sourceis required to proceed without one, and a truncated git-tree listing stops the run rather than publishing a silent subset.Repair the model identity. The converter takes
model_info.idfromconfig.model_args'spretrained=value and refuses an id with no publishing namespace — correctly, since a placeholder developer routes unrelated models into one directory. 28 of the 29 files name a full repo id.evals/stablelm-3b-4e1t.jsonwas run from a local checkout, so its org is supplied throughORGLESS_MODEL_ORGwhere the decision stays visible; an org-less id that is not listed there fails its row instead of getting a guess.Pin one collection. Left alone the converter files each task into its own bare collection —
data/sciq/,data/piqa/,data/lambada_openai/— which mixes these numbers with every other source's records for the same benchmark and loses the provenance.publish_evaluation_logs(collection_override=...)keeps the source together. This is the one behaviour worth knowing about if you copy the file.Why this source
29 open-weights models over 11 benchmarks, and unusually it reaches the pre-2024 generation in machine-readable form: Pythia (2.8b-deduped, 6.9b, 12b), GPT-J-6B, GPT-NeoX-20B, BLOOM (3b, 7b1), OPT (2.7b, 6.7b), LLaMA-1, Llama-2 (7b, 13b), MPT-7B, Falcon-7B, RedPajama-INCITE-7B-Base, Qwen-7B and Qwen-7B-Chat, Cerebras BTLM-3B-8K, OpenLLaMA ×3, Mistral-7B, phi-1.5, Baichuan2-7B-Base, and five StableLM releases.
Two of the benchmarks are the reason it is worth having. SciQ for 29 models and LAMBADA-OpenAI for 29, both of which the datastore held almost nothing for. MPT-7B's SciQ number, for instance, exists only because Stability measured it —
mosaicml/llm-foundryhas no SciQ anywhere.Decision log
A wrapper rather than a new adapter. Confidence high. Re-implementing conversion for a format the repo already reads would be two code paths to keep in step.
Not parameterised into a generic harness-JSON adapter. There is a second repository of the same shape, so the pattern recurs — but a parameterised adapter is a cross-package design change, and a small file others can copy costs less than a shared abstraction that has to fit every future source. Confidence medium; I would take the argument the other way.
Mirrors published as the source ran them.
huggyllama/llama-7bfor LLaMA-1 andkittn/mistral-7B-v0.1-hffor Mistral-7B are community mirrors, not first-party repos. Aliasing a mirror onto its upstream is the eval-card-registry's job. Confidence high.RedPajama-INCITE-7B-Base2filename vs...-Baseinmodel_args. The identity comes frommodel_args, which is what the harness was actually given. Confidence high.truthfulqa_mc'smc1/mc2stay namespaced. They are task-specific constructs rather than global metrics, solm-evaluation-harness.mc1is right and the registry carries no canonical for them. Confidence high.Dependency
The runs predate lm-eval v0.4, and the converter's handling of that format was wrong: standard errors were published as scores and
acc_norm/pplreached no canonical id. This branch includes those fixes (#285) so it produces correct records standing alone; they are open separately for review because they change what every lm_eval conversion produces.Verification
uv run python -m every_eval_ever validate '<out>/data/stablelm-evals/*/*/*.json'— 293 passed, 0 errors, 0 warnings, at the final datastore path with semantic checks on.uv run python -m every_eval_ever.check_duplicate_entries— clean.uv run pytest tests/test_stablelm_evals_adapter.py— 16 passed, offline, no network.uv run pytest tests— 1,023 passed, 45 skipped.uv run ruff check— clean.sciqaccuracy stays 0.891 and itslambada_openaiaccuracy 0.5173685231903745, each with its published standard error attached.Related
evaleval/EEE_datastorefollows