Add aiXamine adapter

[fatihdeniz]

Adds an adapter for aiXamine (aixamine.qcri.org, paper), converting its public-API reports into EEE records — one aggregate log per (model, service) across 9 services.

• adapters/aixamine/ — the adapter + tests
• catalog.py — scheduled entry (9 collections, weekly)

API models are date-stamped with their access date; HF models keep their repo id. Tests pass. Output validates clean.

[borgr]

Reviewed this on a local rebase onto current main, since the fork CI gate has never released and the branch has therefore never been tested. Good news first — the rebase is clean, ruff check passes, and the full suite is 1345 passed, 1 skipped. The shape of the adapter is right, the catalog entry looks correct for nine collections on a weekly cadence, and emitting one aggregate log per (model, service) is the natural grain for this source.

Four things I'd want changed before this lands, two of them verified by running the code.

---

1. [high] A model with no developer disappears without a trace

_outputs_for (adapters/aixamine/adapter.py:279) returns early when _resolve_developer yields None. No failure, no exclusion, nothing in the report. I ran it against a report whose model carries no developer and a flat name:

total_records   : 1
converted       : 0
failures        : 0
exclusions      : 0
raise_if_incomplete -> returned normally (run would exit 0)

Under --all on the weekly cron, a change to the search payload's developer field could drop every report in the sweep and the job would still go green. raise_for_failed_records (helpers/io.py:392) returns early on an empty failure list, which is why the exit status cannot see this.

The hotfix commit was right that the run should not crash. The record just needs to say so:

def _outputs_for(report, model, catalog, out_root, retrieved_ts, outputs, failures):
    """Append this (report, model)'s service logs to outputs / failures."""
    if _resolve_developer(model) is None:
        failures.append(SourceRecordFailure(
            source_ref=f"{SRC} report {report.get('_id')}",
            reason="model has no developer and its name carries no namespace",
            source_record={"model": model.get("name")}))
        return

A test asserting the failure lands in the report would pin it.

2. [high] Two examinations of the same open-weights model share one evaluation_id

evaluation_id=f"{collection}/{developer}_{name.replace('/', '_')}" (adapter.py:200) has no run segment. For API models _dated_name appends the access date so successive examinations differ, but for accessType == "huggingface" it returns the bare name. Two reports with different scores collapse:

report r1 (Jan): aixamine_hallucination/meta-llama_meta-llama_Llama-3.1-8B-Instruct
report r2 (Jul): aixamine_hallucination/meta-llama_meta-llama_Llama-3.1-8B-Instruct

The schema asks for eval_name/model_id/retrieved_timestamp (eval_types.py:558). The report id is the stable choice here, since it stays constant across reruns of the same examination while retrieved_timestamp would change on every cron pass:

            evaluation_id=f"{collection}/{model_id.replace('/', '_')}/{report.get('_id') or retrieved_ts}",

That also drops the doubled developer visible above, where meta-llama appears twice because _dated_name already returns the namespaced name.

3. [medium] Every test in every service publishes the same metric_id

_metric_config sets metric_id=f"{SRC}.rate" for all 46 static and 5 dynamic tests. A consumer joining on metric_id pools a SimpleQA factuality score with a jailbreak refusal rate and a BBQ bias score as one quantity. Sibling adapters give each distinct quantity its own id — cocoabench.overall.accuracy_percent, bfcl.overall.rank. Suggested change, which keeps the unregistered namespace convention:

        metric_id=f"{SRC}.{test_value}",

metric_kind="accuracy" is worth a second look on the same grounds. A refusal rate on anthropic-redteam is not accuracy.

4. [medium] max_score=100.0 is asserted for all 51 tests

The docstring says aiXamine scores are a 0-100 rate, and that is plausible for the safety and refusal tests. Declaring it for every test means a consumer normalizing by max_score silently rescales any test that turns out to report a 0-1 proportion or an unbounded count. Does the API document the range per test, or is 0-100 an observation across the reports you captured? If the latter, leaving min_score/max_score unset for tests where nothing states the bound is the safer record.

---

Three smaller ones, none blocking.

• EvalLibrary(name=SRC, version="unknown") (adapter.py:196) writes a placeholder the repo rejects elsewhere — require_identity in helpers/io.py:203 refuses the literal "unknown". Omitting the version says the same thing without asserting it.
• source_type="documentation" with evaluator_relationship=first_party. These numbers are aiXamine's own evaluation runs read from its API rather than values transcribed from a page, so evaluation_run looks like the closer fit. Say if you read that enum the other way.
• _bundle_from_fixture uses bare open() with no encoding and no context manager. json.loads((d / "report.json").read_text(encoding="utf-8")) matches the rest of the repo and closes the handle.

Happy to push the rebase I tested to your branch if that helps, and the CI gate needs a maintainer click either way — it is queued along with the others.

[fatihdeniz]

Thanks for the thorough pass, really helpful. Good call testing it on a rebase. Please go ahead and push the rebase you tested. I'll build the fixes on top of it. I have started working through them. Only for max_score: aiXamine scores are always normalized to a 0–100 range, so it is a real per-test bound rather than just an observation, so we'll keep it.

[borgr]

Two things, and thanks for the quick turnaround.

max_score is settled — if aiXamine normalizes every test to 0–100 then the bound is real and asserting it is correct. Dropping that finding, keep it as it is.

The rebase. I rebased your three commits onto current main and verified it, but my environment blocks me from pushing to a fork branch, so I can't land it for you. The rebase is clean, with no conflicts, and on it the full suite is 1428 passed, 1 skipped with ruff check clean. Note main moved again since my review — #291 merged, which is where the extra 83 tests come from.

Two ways to get it:

Rebase it yourself, which is three commands and gives you the identical tree:

git remote add upstream https://github.com/evaleval/every_eval_ever.git   # if needed
git fetch upstream main
git rebase --onto upstream/main $(git merge-base HEAD upstream/main) feature/aixamine-adapter

That replays aiXamine EEE adapter initial version, Hotfix: unknown developer and API model date info added onto 7136b1e41. Then force-push to your branch and build the fixes on top.

Or @borgr can push my verified rebase, since maintainerCanModify is on for this PR. Either is fine — say which you'd prefer if you'd rather not do it by hand.

Whichever way, the CI approval click is still needed afterwards for the run to leave action_required.

[borgr]

Rebased your branch onto current main (7136b1e41) and force-pushed it, since #291 had moved the base under you — heads up that feature/aixamine-adapter is now 1f0ffb8dd, so please git fetch and reset before your next commit rather than pulling.

Nothing of yours changed. Same three commits, same 503 insertions across the four files you touched, no content edits from me:

1f0ffb8dd  API model date info added
af391a7c2  Hotfix: unknown developer
a7dd38617  aiXamine EEE adapter initial version

Green locally on the rebase — 1345 passed, 1 skipped, ruff clean. I will release the fork CI run on the new head.

The two findings from my review above are still open, and both are yours to judge rather than blockers I want to impose. The silent drop when _resolve_developer returns None (_outputs_for, adapter.py:276-279) and the evaluation_id collision for two HF models whose names differ only by the org prefix (line 202). Happy to draft either patch if you would rather I did.

[borgr]

Verdict — one word from mergeable. Everything I raised on the previous head is fixed and covered by tests. The only change left is the paper to paper_url key rename below. Push that and #283 is good to merge from my side. Detail follows.

---

All four commits do what they say, and the tests you added pin the right things — test_evaluation_id_stable_per_report_no_collision in particular asserts both halves of the property (same report survives a second retrieval, different reports diverge, and the old meta-llama_meta-llama shape is gone). Verified at c8b8439ee, 1431 passed, 1 skipped, ruff clean. Collected count goes 1419 on main to 1432 here, so +13 tests.

Correcting a number from my previous comment. I quoted 1345 passed for the rebase, which came from a stale run against the pre-#291 base. The rebase figure was 1428.

Two notes on your evaluation_id choice, because I think you improved on the documented convention and it is worth saying why. eval_types.py:558 documents eval_name/model_id/retrieved_timestamp, and you used the report id instead. That is the better key — evaluation_id is consumed as COUNT(DISTINCT evaluation_id) AS n_evaluation_runs (helpers/eee_stats.py:195), so keying on the run counts runs, whereas keying on the retrieval would count a re-fetch of the same run as a second run. Keep it.

One thing I would change before merge, and it is one word.

source_metadata.additional_details uses paper (adapter.py:213). The repo's key is paper_url, in nine places, all in the same object — benchpress:575, mmlu_pro:264/326, mt_bench:307/456, wild:300, openeval:906, paperswithcode:289, llm_stats:127, and arc_agi's ('paperUrl', 'paper_url') mapping. Only hal:835 says paper, and that one also points at a PDF, so it is the outlier twice over.

-                    "paper": PAPER,
+                    "paper_url": PAPER,

Worth doing now rather than later. A published record is immutable under its UUID in the flat view, so renaming the key after these records exist means re-emitting every aiXamine record with fresh UUIDs. We are working through exactly that on another collection this week. Your PAPER constant is already the abs page, so only the key changes.

One optional nit, take it or leave it. report.get('_id') or retrieved_ts falls back to the timestamp when the report has no id, which quietly restores the collision the commit fixes — the same run retrieved twice would get two ids and count as two runs. Since _outputs_for now has a good pattern for a missing required field, a SourceRecordFailure would be more consistent than a fallback:

-            evaluation_id=f"{collection}/{model_id.replace('/', '_')}/{report.get('_id') or retrieved_ts}",
+            evaluation_id=f"{collection}/{model_id.replace('/', '_')}/{report['_id']}",

with the _id check alongside the developer check in _outputs_for. If aiXamine always sends _id then the fallback is unreachable and this is cosmetic, so your call.

Also confirming a non-issue so you do not spend time on it. _iso_from returning None is right — static tests carry a duration rather than a wall-clock time, and the dynamic path already passes generatedAt through.

Push the key rename and #283 is ready to merge. The fork CI run needs a maintainer to release it, which we will do on the new head.

[fatihdeniz]

I appreciate the detailed comments. I followed your recommendations. Now, (1) no-developer models record a failure, (2) evaluation_id keys on the report id, (3) we have per-test metric_id with metric_kind reviewed. Plus the minors: source_type=evaluation_run, utf-8 fixture reads.
Kept version="unknown". Also renamed the paper key to paper_url, and used report _id for evaluation_id with an _id check in _outputs_for.

Suite seems to be green ;) Thanks again for all the help!