Update changed cron records instead of adding new files

[tommasocerruti]

The cron skips unchanged records, but when a record changes, it adds a new file and leaves the old one in the datastore. Over time, this can create multiple versions of the same evaluation and unnecessarily grow the datastore.

Could cron records use a stable identity and update the existing file when their content changes?
(Git history would still preserve the previous version)

[evijit]

We should in general find a deduplication method.

[nelaturuharsha]

(1) We agree on for immutability of records available on the datastore. We however, need a retention policy and then a cold storage where people who use it can download it, before finally deleting the same.

(2) You're right that this history of change is preserved via git - but in view of the above, I would not modify a record in place, but treat it as changed. We have no yet guaranteed full immutability due to functional reasons.

(3) It also brings in a good discussion for when the schema changes and moves forward

(4) We should be careful about what we do for de-duplication. We need a grounded method to decide what entails such a thing - and shouldn't rush through designing this, as we might miss out on data we can include otherwise, we've discussed before on slack.

[borgr]

The blocker is enforced, not conventional, and this turns out to gate every data correction rather than only datastore growth.

converters.common.publication.publish_evaluation_logs calls require_uuid4 on every filename it is handed. So a record cannot be published under a name derived from its identity — the publisher rejects it. That is why a changed cron record lands beside the old one instead of replacing it.

What that costs, measured. I sampled the datastore for a separate reason and the number is stark. Of 51,802 aggregate records on main, 26,611 (51.4%) carry a metric_id that cannot identify a metric — missing, generic, or an exact copy of evaluation_name. Four adapters produce 23,373 of them, all four on the daily cron. Method was three records per collection, which settled 98 of the 107 collections because one adapter writes every record in a collection, plus a full scan of the 9 that disagreed.

Fixing those four adapters today would publish ~20,702 corrected records beside the wrong ones. The datastore would then hold both, the wrong half would still fail the gate, and the correction would have made the count worse before better. The same applies to #287, which corrects declared metric bounds in three cron adapters.

So this is not only about unnecessary growth. It is the mechanism by which any adapter fix reaches published data, and without it every correction is a duplication.

Shape I would propose

• A cron record's filename derives from its identity — collection, developer, model, and the evaluation identity within the record — so a re-run overwrites in place and git history keeps the previous version, which is what the issue already suggests.
• A hand submission keeps uuid4. There, "multiple evaluations of the same model coexist without conflicts" is the point, and CONTRIBUTING.md § "UUID naming convention" is right about it.
• So require_uuid4 stops being unconditional and becomes a property of the submission path rather than of the publisher.

Touches the publisher, the cron's fingerprint and skip logic, CONTRIBUTING.md's UUID section, and the dataset card. That is a schema-and-publisher change, so per CONTRIBUTING.md it wants its approach agreed here before a PR — which is what this comment is for rather than a patch.

What it unblocks

Once a correction can replace a record: the metric-identity fix across four adapters, re-landing check_metric_identity (reverted in #266, precondition satisfied by #246 since 2026-08-28 and still absent from main), #287's bounds corrections, and re-running terminal_bench_2, which is already correct at main while its 2,671 published records are stale.

Found while building an agentic PR-review skill, context on #250.