Skip to content

Add aiXamine evaluator org, family, and 9 service composites - #63

Closed
fatihdeniz wants to merge 2 commits into
evaleval:mainfrom
fatihdeniz:seed/aixamine-evaluator
Closed

fatihdeniz wants to merge 2 commits into
evaleval:mainfrom
fatihdeniz:seed/aixamine-evaluator

Conversation

@fatihdeniz

Copy link
Copy Markdown
Contributor

Registers aiXamine (aixamine.qcri.org, paper) as an evaluator so its EEE records resolve to real entities.

  • seed/orgs.yamlaixamine org (tags: ["academic","evaluator"])
  • seed/families.yamlaixamine family grouping the 9 service leaderboards
  • seed/composites.yaml — 9 aixamine-<service> composites mapping each source_config

Seed-only (+91 lines). seed --local ingests cleanly; gate suite passes (the one failure is an unrelated models.dev drift check).

@borgr

borgr commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Verified locally against current main and I recommend merging.

Ran the same steps the dry-run job runs, on main + this branch:

rm -f fixtures/*.parquet && LOCAL_MODE=true uv run eval-card-registry seed --local
LOCAL_MODE=true uv run pytest

Result: 1 failed, 868 passed. The single failure is test_refresh_modelsdev_dedup.py::test_noncatalog_regen_is_confluent_with_committed, which is a pre-existing break on main — tests/fixtures/modelsdev_api.snapshot.json has not moved since 923986e while seed/models/sources/models_dev.generated.yaml has been refreshed six times since, most recently d608d6b. Unmodified main gives the identical 1 failed, 868 passed, so this branch adds no failure. Fixing it means re-pinning the snapshot, which is separate work.

The content reads clean to me. One evaluator org, one family, nine service composites each with a single config and a category/tags pair that matches what the service measures. Additive only, no existing entry touched.

Worth landing this together with the adapter in evaleval/every_eval_ever#283, since that adapter's records resolve against these entities.

@j-chim

j-chim commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Thank you for the PR. We've integrated this via #67 with commits and attribution intact. Two fixes were applied on top: the bare QCRI alias was dropped from the org, because QCRI is a separate Hugging Face uploader and any of its records would have folded into the evaluator, and one composite's category was aligned with its sibling.

@j-chim j-chim closed this Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants