feat: add sayf-eval results-record converter

[Aymen311]

What

Adds a converter for sayf-eval — a model-agnostic cybersecurity LLM-evaluation framework — so its results can be ingested into the Every Eval Ever schema:

every_eval_ever convert sayf_eval --log_path <sayf-eval output dir> --output_dir data

sayf-eval writes a canonical results record per run that embeds the full pipeline configuration (decoding params, <think> handling, denominator policy, judge model) and per-task dataset provenance alongside the scores. The converter maps that record onto the schema.

Design

• One EvaluationLog per task, mirroring the lm-eval converter. Each task contributes accuracy, plus a second evaluation_results entry where applicable — CVSS MAD for vuln-scoring tasks (lower_is_better: true, [0,10]) and micro-F1 for ATT&CK-extraction tasks.
• The LLM judge is recorded under metric_config.llm_scoring; decoding/pipeline settings under generation_config.generation_args (+ additional_details for the rest); provenance as the matching source_data variant (HF / URL / other).
• Reads the record as JSON — does not import sayf-eval, so the converter has no new runtime dependency (core deps only).
• Aggregate-only by design. sayf-eval is a cybersecurity benchmark whose per-sample item text is dual-use and kept private, so the converter emits only aggregate score files and never instance-level _samples.jsonl (detailed_evaluation_results is optional in the schema). The results record contains no prompt/gold/response text.

Changes

• every_eval_ever/converters/sayf_eval/ — adapter.py (SayfEvalAdapter), __main__.py, __init__.py.
• every_eval_ever/cli.py — register sayf_eval as a convert source + dispatch.
• tests/test_sayf_eval_adapter.py + tests/data/sayf_eval/results_fixture.json.
• every_eval_ever/converters/README.md — sayf-eval section.

Testing

• 10 converter tests pass; output passes the repo's validate_file checks.
• Verified end-to-end via every_eval_ever convert sayf_eval → every_eval_ever validate: one aggregate JSON per task, zero _samples.jsonl, correct lower_is_better per metric, judge + pipeline config present, no item text.
• No schema change (no type regeneration); packaging auto-discovers the new subpackage; ruff clean against the repo config.

[nelaturuharsha]

assigned to @whymath

[mrshu]

⚒️ review-anvil report

Review decision: COMMENT — The converter has four confirmed data-quality and partial-conversion risks.
Result: Four focused changes would make repeated and partial conversions safer while preserving the PR's aggregate-only design.
Scope: This PR adds a sayf-eval aggregate results-record converter, CLI route, documentation, fixtures, and offline tests.
Checks: 8 single-reviewer concerns checked; 0 confirmed, 0 ruled out, 8 set aside because exact-head verification did not complete.
Second check: targeted, 2 reviewers; 3 kept, 1 clarified, 0 set aside, 0 removed.

Earlier review comments

No earlier review comments were present.

What I noticed

• RAV-RUN1-R1-F001 [high] source-provenance every_eval_ever/converters/sayf_eval/adapter.py:184 — The adapter puts a task slug in source_data.dataset_name. This replaces upstream names such as CTI-Bench MCQ, while the empty default prefix routes data into bare task collections. (inline)
• RAV-RUN1-R1-F003 [high] record-identity every_eval_ever/converters/sayf_eval/adapter.py:364 — evaluation_id contains the current retrieval time. Re-converting the same source run creates a new logical evaluation each time. (inline)
• RAV-RUN1-R1-F005 [high] partial-conversion every_eval_ever/converters/sayf_eval/adapter.py:418-422 — One invalid task raises after earlier task logs were built. The file-level failure path then discards those valid sibling logs. (inline)
• RAV-RUN1-R1-F007 [medium] judge-metadata every_eval_ever/converters/sayf_eval/adapter.py:164 — LlmScoring.input_prompt receives scoring prose or an invented fallback sentence. The record therefore presents a description as the actual judge prompt. (inline)

Non-blocking low findings (1 item)

• RAV-RUN1-R1-F013 [low] tests — The publish test checks Pydantic validation but does not run the repository semantic validation gate on final paths.

Things to try (4 items)

• [high] source-provenance — Upstream dataset names can stay in source_data, while an explicit sayf-eval collection override handles routing. (RAV-RUN1-R1-P001; covers RAV-RUN1-R1-F001)
• [high] record-identity — A stable source-run token can occupy the timestamp segment of evaluation_id; retrieval time can remain in retrieved_timestamp. (RAV-RUN1-R1-P002; covers RAV-RUN1-R1-F003)
• [high] partial-conversion — A task-level error boundary can keep valid task logs and record each rejected task in failure accounting. (RAV-RUN1-R1-P003; covers RAV-RUN1-R1-F005)
• [medium] judge-metadata — llm_scoring can be present only when the source supplies the real prompt template. Otherwise, supported score data can remain without invented prompt text. (RAV-RUN1-R1-P004; covers RAV-RUN1-R1-F007)

Set aside / Outside this change (8 items)

• RAV-RUN1-R1-F002 [high] score-validation — We set this aside because the verification run could not inspect the exact PR head.
• RAV-RUN1-R1-F004 [high] empty-record-accounting — We set this aside because the zero-output CLI path was not reproduced.
• RAV-RUN1-R1-F006 [medium] model-metadata — We set this aside because the model classification path was not reproduced at the exact PR head.
• RAV-RUN1-R1-F008 [medium] source-provenance — We set this aside because malformed provenance behavior was not reproduced.
• RAV-RUN1-R1-F009 [medium] timestamp-handling — We set this aside because malformed timestamp behavior was not reproduced.
• RAV-RUN1-R1-F010 [medium] source-type — We set this aside because the source artifact classification was not confirmed against the exact PR head.
• RAV-RUN1-R1-F011 [medium] failure-accounting — We set this aside because the source-record unit used by total_records was not confirmed.
• RAV-RUN1-R1-F012 [medium] failure-reporting — We set this aside because publication failure behavior was not reproduced.

Run details

• Target: PR #220 (feat/sayf-eval-converter, 7 files, +928/-1)
• Run ordinal: 1
• Rounds: 1/1 completed; adaptive off; material findings remain
• Mix: 3 codex-exec
• Focus: correctness, maintainability, simplicity, production blast-radius, and constructive positive review language
• Earlier review comments: none
• Finding counts: 0 critical, 3 high, 1 medium, 1 low, 0 nit
• Checks: concerns=8; confirmed=0/ruled-out=0/set-aside=8/lowered=0
• Second check: targeted; reviewers=2; kept=3/clarified=1/set-aside=0/removed=0; approval changed no
• Set aside: 8 items; reason=verification could not access the exact PR head

---

Reviewed with review-anvil.

[whymath]

Hey @Aymen311 it looks like the sayf-eval repo is a closed repo, and I wasn't able to find any published eval results generated using this harness either. Is there a plan to open-source the harness and generate a lot of open/public results using it? If that's the case then we can continue looking into this converter approach for the harness itself.

On the other hand, if the plan is to continue to keep the harness closed but only publish some eval results, then maybe an adapter might be more appropriate so that we can still get the data in into the EEE datastore without having to make all the structural changes needed for a converter.

cc @borgr @nelaturuharsha

[Aymen311]

Hello again @whymath, Thanks for the review. I've addressed all four confirmed points in d02bd8687 (provenance, stable ids, partial conversion, judge prompt), plus a follow-up in e0323ef54 that routes each task into its own datastore collection. I also refreshed the fixture (it now includes judge_prompt_templates) and added test cases for stable ids, partial conversion, and the no-template path, and the published paths pass the repo's validate_file semantic gate. Details are inline on each thread, and I'm happy to re-run the checks against the updated head.

Sayf-eval is now open source (https://github.com/qcri/sayf-eval), and the repo now ships a public leaderboard. Those are aggregate-only results records for 10 models across 24 cybersecurity sub-tasks, in exactly the schema this converter ingests. Surfacing them on HF Community-Evals through this converter is the immediate use case for the new domain.

[Aymen311]

Small follow-up in 280923871: sayf-eval removed its private CISSP task (it was never released or benchmarked), so a sayf-eval record no longer carries an 'other'/private source. I dropped cissp from the test fixture and the adapter tests accordingly (now ate, athena_vsp, mcq), and kept coverage of the adapter's 'other' → SourceDataPrivate mapping via a direct unit test rather than a fixture task.

[Aymen311]

@mrshu or @nelaturuharsha can you help here ?

[borgr]

Rebased onto current main so this can get CI. One conflict, in every_eval_ever/converters/README.md, where main had added ### Reruns and ### Options under AlpacaEval at the same spot this branch appends the ## sayf-eval section. Both kept, AlpacaEval's subsections first.

Also added one commit. tests/test_documentation_commands.py on main rejects a bare pip install anywhere in the docs, so the sayf-eval install line is now uv pip install sayf-eval. That was the only failure after the rebase.

Rebased branch locally: 1350 passed, 1 skipped, ruff clean.

[Aymen311]

Thanks @borgr for the tweaks!

In the meantime, we've released the writeup paper behind the harness: https://arxiv.org/pdf/2609.08765 (it includes results, failure modes...etc)
Please let us know if anything can be done from our part to move this forward.

[borgr]

Rechecked all four of mrshu's findings against the rebased branch. Three are fixed and I've resolved them with the line each one landed on. F001 keeps the upstream dataset_name and falls back to the task slug only when the record has no name, F005 gives each task its own error boundary so one bad task no longer discards its siblings, and F007 emits llm_scoring only when the record carries the real judge template.

F003 is the one still open, and only in its fallback. The created_at-derived run_token is right. run_token or retrieved_timestamp then mints a wall-clock id for a record with no created_at, and the bare except Exception above it does the same for an unparseable one, so those records still get a fresh identity per conversion. I left a suggested replacement on that thread, which raises instead of guessing — the F005 boundary in this same PR is what makes that safe, since the failure now costs one task rather than the file. If sayf-eval always writes created_at, say so on the thread and I'll resolve it as unreachable.

Everything else here reads well, and the local run on the rebase was 1350 passed, 1 skipped with ruff clean. Two things outside your control before this can land. The CI run needs a maintainer to release the fork gate, and I can't click that from where I'm working, so it's queued with the others. Thanks for the writeup link as well — worth citing from the converter README once this is in.

[Aymen311]

Thanks again @borgr, and appreciate you rebasing and sorting out the docs test. I pushed the F003 fallback fix you suggested in 17a698d41 (raise instead of minting a clock-based id, plus the two branch tests), so all four of mrshu's findings should be squared away now. Happy to make any further tweaks, or squash the commits down if that would help the merge.

[borgr]

Thanks @Aymen311 — verified 17a698d41 locally and all four of mrshu's findings are now closed. tests/test_sayf_eval_adapter.py is 17 passed, ruff check is clean, and I've resolved the last thread. No squash needed, the merge squashes anyway.

Congratulations on the paper. One optional thing you asked about, and it is genuinely optional — nothing blocks on it. The converter takes source_organization_name / source_organization_url / source_organization_logo_url from metadata_args, but there is nowhere for the paper itself to land, so a consumer reading these records has no route back to the methodology. The sibling adapters carry it in additional_details, for example every_eval_ever/adapters/aixamine/adapter.py uses additional_details={"paper": PAPER, "homepage": HOMEPAGE}. If you want the arXiv link on every record you publish, adding it to the SourceMetadata(...) call is the place:

            additional_details={
                'paper': 'https://arxiv.org/abs/2609.08765',
            },

Note abs rather than pdf — the rest of the repo stores the landing page.

The only thing left on this PR is the CI approval click, since fork runs sit at action_required until a maintainer releases them. That is with @borgr along with the other fork PRs. Once it goes green this is ready to merge from my side.

[Aymen311]

Added it in 03a3bf015 — every record now carries source_metadata.additional_details.paper = https://arxiv.org/abs/2609.08765 (the abs landing page as you noted, not the PDF), with a test asserting it. Thanks for the pointer, and for getting this over the line. Whenever you release the fork CI it should be good to merge from our side.

[Aymen311]

Also linked it from the converter README in 203cd13a1, next to the source link in the sayf-eval section, so the docs point to the methodology too.

[borgr]

Thanks — both commits look right, and the abs-vs-PDF distinction landed correctly.

One correction to my own suggestion, though. I wrote additional_details={'paper': ...} and the repo's established key for this is paper_url.

It appears as paper_url in nine places across the adapters, every one of them inside SourceMetadata.additional_details — benchpress (adapter.py:575), mmlu_pro (264, 326), mt_bench (307, 456), wild (300), openeval (906), paperswithcode (289), llm_stats (127), plus arc_agi's ('paperUrl', 'paper_url') field mapping. The only other 'paper' in the tree is hal/adapter.py:835, and that one points at a PDF rather than an abs page, so it is the outlier on both counts.

This is worth changing before merge rather than after. A consumer filtering source_metadata.additional_details.paper_url across collections would silently skip sayf_eval, and a published record is immutable under its UUID in the flat view, so renaming the key later means re-emitting every sayf_eval record with fresh UUIDs. We are working through exactly that situation on another collection right now.

Three lines. In every_eval_ever/converters/sayf_eval/adapter.py

-_PAPER = 'https://arxiv.org/abs/2609.08765'
+_PAPER_URL = 'https://arxiv.org/abs/2609.08765'

-            additional_details={'paper': _PAPER},
+            additional_details={'paper_url': _PAPER_URL},

and the assertion in tests/test_sayf_eval_adapter.py

-    assert (details or {})['paper'] == 'https://arxiv.org/abs/2609.08765'
+    assert (details or {})['paper_url'] == 'https://arxiv.org/abs/2609.08765'

The leading underscore matches this file's own constants (_CLOSED_PROVIDERS, _LOCAL_PREFIXES, _SECONDARY_METRICS), so only the key string needs to follow the cross-adapter convention. Your comment above it can stay as written. The README link is fine as it is.

Push that and #220 is ready to merge — nothing else outstanding from my side. The fork CI run needs a maintainer to release it, which we will do on the new head.

[Aymen311]

Done in d5f56eabf — renamed the key to paper_url (constant _PAPER_URL) to match the cross-adapter convention, and updated the test assertion. Good catch on the immutability angle. ruff clean and the records still validate. Should be ready to merge whenever you release the fork CI. Thanks again!