Auto-BenchmarkCards

Auto-BenchmarkCards produces structured documentation for AI benchmarks from evaluation records, papers, dataset repositories, and linked web sources. It records source provenance, composes a structured BenchmarkCard, and checks eligible claims against the collected evidence.

The main branch contains the current development package and a reusable card composer. The research code and evaluation artifacts for Auto-BenchmarkCards: Automated Documentation for AI Benchmarks are frozen at v0.2.0. The Python package and command retain the singular name auto_benchmarkcard and benchmarkcard for compatibility.

Research-release pipeline. A benchmark enters as a name and an evaluation record and leaves as a published card. Red marks a field recorded as "Not specified"; the triangle marks a field that shipped with a warning.

Paper release

The paper evaluates the 40-field cards and configuration frozen in v0.2.0. Its results do not evaluate subsequent changes on main.

Resource	Role
GitHub release v0.2.0	Versioned research code and evaluation release
Hugging Face revision 0a86cea5…	Frozen evaluated corpus of 530 JSON cards
Auto-BenchmarkCards-code-and-data.zip attached to the GitHub release and the arXiv submission	Frozen paper artifact with the code, all 530 cards, provenance, and replay inputs
Evaluation Cards	Live community deployment; dynamic and not used to compute the paper's results

The frozen corpus is a research snapshot with known defects, not a human-verified reference database. Subsequent corrections to the living corpus do not change the inputs analyzed in the paper.

Important interpretation limits

• In the stratified sample, an estimated 47.80% of cards had at least one screen-detected and verifier-confirmed material finding. This is not an estimate of overall defect prevalence because the screen's recall is unknown.
• In the overlapping filled-field universe, post-composition warnings had 3.03% weighted precision and 1.86% recall for source-unsupported content. They are review cues, not correctness certificates.
• Each card contains a possible_risks list produced by a separate risk identifier. In the sampled cards, the automated source judge classified 547 of 761 candidate risk assertions as relevant and grounded in the collected evidence and 214 as not. These judgements were not human-validated and are excluded from the headline field-support results. Treat every item as a prompt for human review, not as a verified property of a benchmark.

No headline result in the paper depends on the candidate risk assignments.

Repository contents

• src/auto_benchmarkcard/: package and command-line application
• src/auto_benchmarkcard/tools/composer/: reusable extraction, composition, and schema validation library
• scripts/: batch generation, corpus assembly, and evaluation programs
• tests/: offline and artifact-backed regression tests
• eval/: frozen evaluation sample, results, public human-label projections, manifests, and checksums
• docs/: architecture notes and figure sources
• spaces/benchmarkcard-webhook/: optional Hugging Face Space integration

The public evaluation snapshot is based on 531 attempted entries and 530 published cards. See eval/README.md for result scopes, artifact locations, sanitization policy, and offline reproduction commands. The complete corpus and per-field provenance ship in the release archive.

Composer library

The composer accepts a CardSchema describing the card's sections, fields, and allowed values. It uses that schema for evidence extraction, Stage B (formatting the selected evidence into card fields), and validation and repair. The benchmark schema is the default; other applications can supply their own.

The library in this integration comes from composer-library commit 94c7713. Modelcards uses it with a separate model-card schema and records its exact composer revision in composer-pin.json. That pin remains valid independently of subsequent work on main.

Install

Auto-BenchmarkCards requires Python 3.11 or newer.

git clone --branch main https://github.com/evaleval/auto-benchmarkcard.git
cd auto-benchmarkcard
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Copy .env.example to .env and add only the credentials required by your chosen backends. .env is ignored by Git. The entailment stage also requires a separately built Merlin solver binary. Set MERLIN_BIN to its executable path; the default is external/merlin/bin/merlin in the checkout. Merlin source and build instructions are available in the Merlin repository.

pyproject.toml is the dependency specification. It pins a compatible FactReasoner fork; the current adapter supplies Mellea-backed extraction and entailment components. requirements-lock.txt is a compatibility shim for older install commands and contains -e .; it does not lock transitive dependencies.

Run

Generate from an Every Eval Ever export:

benchmarkcard generate ./external/eee_samples \
  -b "MMLU,TruthfulQA" \
  -o ./output

Generate from Unitxt:

benchmarkcard generate-unitxt glue -o ./output

Inspect the available commands:

benchmarkcard --help
benchmarkcard validate

Each run writes the final card, collected source artifacts, provenance, and factuality outputs to a timestamped directory.

Reproduce the reported statistics

The GitHub release attaches the same code-and-data archive distributed with the arXiv version. Its REPRODUCIBILITY.md gives seven offline commands for recomputing the reported summaries from frozen model outputs and annotations. Those commands require no credentials or new hosted-model calls.

Use the release archive or a separate checkout of v0.2.0 for those commands. Some runtime helpers on main have changed since the evaluated version, so the historical instrument check correctly rejects them. The frozen evaluation files and their checksum baseline are retained unchanged.

Fresh generation is not byte-reproducible because it depends on changing web sources and hosted models. The archive records the evaluated configuration, software environment, frozen outputs, and checksums so the reported numerical analysis can be replayed independently of those services.

Development

Development checks are run manually. For example, after installing the dev extra, check the composer's schema interface with:

python -m pytest tests/test_card_schema.py

Additional setup and frozen-artifact rules are documented in DEVELOPMENT.md. This code follows URLs found in input records and has not been hardened as a public network service. Run it in an isolated environment when processing untrusted inputs; see SECURITY.md.

Citation

Please cite the accompanying paper. The arXiv identifier will be added to CITATION.cff after arXiv assigns it.

Aris Hofmann, Inge Vejsbjerg, Jan Batzner, Leshem Choshen, Jenny Chim, Avijit Ghosh, and Elizabeth M. Daly. “Auto-BenchmarkCards: Automated Documentation for AI Benchmarks.” 2026.

Licenses

Material	License
Source code and repository documentation	MIT
Generated 530-card corpus	CDLA-Permissive-2.0
Author-created evaluation instruments, reports, aggregates, and derived outputs	CC BY 4.0
Sanitized participant label projections	Published as consented research records; no separate copyright license is asserted over participant-authored expression
Third-party sources, excerpts, dependencies, and benchmark content	Their original terms; see third-party notices

DATA-LICENSE.md defines the file-level boundary. The licenses apply only to rights held by the relevant providers and do not grant rights over the benchmarks described by the cards.