You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bottom line. Thirteen changes to our contributor docs and agent skills, drawn from what
SGLang, NeMo-RL, PyTorch, vLLM, Ray, HF TRL/Transformers and LangChain put in theirs. Each
is a paragraph or a line of config, each stands alone — approve, reject or defer them
individually. The three I'd take first:
A1 — ask what a schema or validator change does to already-published records. The
one item here with a cost we've already paid: issue #242 is 84% of the datastore
failing the merge gate on two fields.
A2 — a fixed shape for review findings, each labelled must-fix / question / FYI, so
a review can be triaged by scanning rather than read in full.
A3 — check for existing work before starting it. The daily-cron feature has been
implemented three times (#184, #247, #249), the last two a day apart.
What I'd like: a yes/no/later on each. I'm happy to write whichever ones you want. Three
questions from the same survey that need a decision rather than a patch are in #251.
Terms are defined in the appendix at the bottom for anyone reading this cold.
All thirteen, for scanning
The last column is the one to weigh: three of these fix something that has already cost us
review time, and the rest are arguments. Reply with a verdict per number — A1 yes, A4 later, A7 no is a complete answer, and no is a useful one.
#
Change
Lands in
Already cost us?
A1
Say what a schema or validator change does to already-published records
Adapters are deliberately self-contained; propagate a shared-pattern fix or say why not
AGENTS.md, 2 sentences
No — already our design, just unwritten
A13
Disclose agent authorship and name a human who can answer
PR template, 1 line
No — CONTRIBUTING already assumes we know
A1. Ask what a schema or validator change does to the records already published
One-liner: On any PR that changes the schema or a validator rule, say how many
already-published records it would invalidate.
Why. Such a change doesn't only constrain new contributions — it re-judges everything
already published. That is how #242 happened: 84% of the datastore
now fails the merge gate on two missing fields. CONTRIBUTING § "Changing the schema, the
validator, or the
publisher"
covers regenerating types, the skill-pinning test and a version bump — all of it
forward-looking.
Change. One required question on those PRs: how many published records would now fail,
and what happens to them? The validator runs over the datastore, so the answer is a count
rather than an opinion — followed by one of: nothing breaks · N records break, here's the
follow-up issue · this needs a schema_version bump.
Note. A question, not a veto. Where a new requirement can land as a warning and become an
error once the backlog clears, that's usually the cheap path.
A2. Give review findings a fixed shape
One-liner: Three lines per finding — what and where, what it breaks, the ask — each
labelled must-fix / question / FYI.
Why. Long, evenly-weighted review prose is expensive to act on: the ask is buried and
nothing marks what blocks a merge, so the reader reconstructs severity themselves. Agent
reviewers make this worse by being fluent — pages of correct text in which every sentence
looks equally important.
Change. A field template in the review skill. A template rather than a style note,
because an agent follows a template exactly and interprets "be concise" however it likes:
What and where — one line, citing a symbol and file. Not a line number, which is wrong
after the next push and which agent reviewers produce by default.
What it breaks — one line, concrete: these inputs give that wrong output. A finding
that can't state this is a question, not a finding.
The ask — must-fix · question ("confirm this is intentional") · FYI (no
action).
One comment carries a handful of findings and says how many it held back, rather than posting
a wall.
Note. PyTorch resolves severity ambiguity the other way, declaring every comment a
must-fix and banning nits. That fits code review and not data review, where plenty of real
findings are "confirm this was deliberate" — neither a blocker nor a nit. Worth knowing too
that Hugging Face dataset discussions have no inline comments and no resolvable threads, so
on the datastore side every finding lands in one flat comment: the structure is the only
thing separating twelve findings from twelve paragraphs.
A3. Check whether the work already exists before starting it
One-liner: Four searches before starting, and if one hits, comment on the existing work
instead of opening a second PR.
Why. The daily-cron feature has been implemented three times — #184 (opened 11 Jun, closed 7 Aug), #247 (opened 11 Aug, since closed) and #249 (12 Aug, the one still open), all against #159, the last two a day apart.
Nothing in AGENTS.md, CONTRIBUTING.md, the skills or the templates says to look first, and now
that starting is nearly free the cost lands as duplicate PRs in the review queue.
Change. Four searches, written as commands so there's nothing to remember:
open PRs and issues matching the source or feature name;
the adapter list in every_eval_ever/adapters/README.md;
the collection folders already in the datastore, since a
source can be converted with no adapter PR open.
Then the rule that makes searching matter, which Ray, vLLM and Transformers all state
explicitly: on a hit, don't open the PR — say what's missing on the existing one.
Otherwise an agent opens it anyway and mentions the duplicate in the description.
Note. Searching the published output rather than only the PR list is the half that matters
for a data repo, and no surveyed repo does it.
A4. Two lines in the PR template
One-liner: What this PR deliberately leaves alone, and what changes for someone
consuming the data.
Why. A description that says what a PR does but not what it leaves alone invites a
reviewer to ask for the adjacent fix, or a later contributor to bundle one in — SGLang's
env-var guide carries an explicit "Out of scope" section for exactly that reason. And our
"What / source" field invites a paragraph where a reviewer needs a sentence: which records
appear, change or disappear.
Change. Add an Out of scope: slot and a one-sentence lead on consumer-visible effect.
A5. One home per fact
One-liner: CONTRIBUTING.md restates field rules that reference/fields.md and the
schema own — replace them with links.
Why.CONTRIBUTING § "Filling in the
schema"
specifies the evaluation_id format, the source_type values, the inference_platform vs inference_engine split and the timestamp fields — all of it owned by reference/fields.md
and eval.schema.json. tests/test_skill_conversion.py
fails when the schema moves out from under the skill; nothing does that for CONTRIBUTING, so
that copy goes stale silently. AGENTS.md already states the rule: don't restate the rule
anywhere else — prose that duplicates the check is what went stale last time.
Change. Replace the duplicated specs with links, and add the reviewer's question for any
PR changing a field or an allowed-value list: what documents this, and did it move too? One
home per rule; examples stay wherever they help.
A6. A scratch directory that can't reach a diff
One-liner: One git-ignored folder, named in AGENTS.md, for anything not meant for the PR.
A conversion run produces staged records, adapter_reports/ output, temporary clones and
notes — every one a candidate stray file, and we already needed #231 to ignore .claude/settings.local.json.
A7. Any list of suppressed problems may only shrink
One-liner: Commit the count of suppressed warnings and expected failures, and fail the
build when it grows.
Why, and the honest version of it. We don't have this problem yet: the suite carries one xfail (test_inspect_edge_cases.py)
and one pytest.mark.skip
(test_helm_adapter.py),
and no coverage-omit or filterwarnings config at all. So this is the one item here proposed
against a cost we haven't paid — include it only if a cheap ratchet now looks better than the
argument later. The pattern it forecloses is the familiar one: each suppression is justified
once, nobody re-reads the aggregate, and the list only grows.
Change. Commit the count and fail the build when it grows, so removals are free and
additions cost one sentence. The related habit is worth stating either way, since it's free:
prefer an expected failure to a skip — an expected-failure test tells you when the problem
is fixed, a skipped test never speaks again. This is the check-shaped version of the PR
template's "no warnings either".
A8. Say when a new test isn't worth adding
One-liner: "What future change would turn this test red?" If the only answer is
"editing the test", don't add it.
Why. The PR template requires an offline unit test per adapter, which is right, and across
dozens of adapters is also dozens of chances to add a test that restates the implementation.
Those never fail for a reason anyone cares about, need updating whenever the code moves, and
make coverage look better than it is.
Change. A short paragraph in CONTRIBUTING with that question and the reviewer's version of
it: if I deleted this test, would some realistic regression now slip through?
A9. Route by path: "before editing this file, read that one"
One-liner: A short file-pattern → read-this-first table in AGENTS.md.
Why. AGENTS.md indexes skills by description, which works when the task announces its
domain ("convert this leaderboard") and fails when the task is small but lands somewhere
load-bearing — a base adapter, the publisher, validation_core.py. The agent never learns a
skill applied.
Change. Half a dozen rows, keyed on directory patterns rather than filenames and pointing
only at guidance that already exists, so it needs touching when a skill is added or renamed
and not when a file moves. If it can't stay that short, it isn't worth having: a stale routing
table is worse than none, because it sends an agent to guidance that no longer matches the
code.
A10. Notice when a PR touches files its title doesn't mention
One-liner: A hard rule in the skills against staging paths outside the PR's lane; for
humans, an advisory comment at most.
Why. Scope creep is usually strays rather than a change of mind: an editor config, a
lockfile, a commit from a messy rebase, a shared-code fix inside a PR titled after one adapter.
Each one costs a reviewer the time to work out whether it was deliberate.
Change. Write down which paths each title
prefix
normally touches, then enforce it asymmetrically: the skills say "don't stage paths outside
this PR type's lane", which an agent follows for free; humans get a bot comment listing
out-of-lane paths, or nothing at all. As a blocking check this would be pure friction on
legitimate cross-cutting PRs, and nearly all the value is in the agent half anyway — two
sentences per skill, no CI.
A11. Verify a finding before posting it, and post only problems
One-liner: Check each finding against the code before posting it, and treat "nothing
found" as a complete review.
Why. A wrong review comment is cheap to write and expensive to disprove, and two of them
teach a contributor to skim the rest. A review padded with what's fine buries the parts that
aren't.
Change. Two lines in the review contract, applying to every agent review rather than to
one skill: verify each finding against the actual code and sources, dropping it below a
confidence floor rather than hedging; and report only problems — no praise, clean sections
omitted. #241 already does this for
the deployment-metadata question; this is the same rule generalised, since it's a property of
agent review rather than of that question. The verification pass roughly doubles review
tokens, so scope it to findings that ask the contributor to do work.
A12. Write down that adapters are deliberately self-contained
One-liner: Duplication across adapters is intended, so fixing a shared pattern in one
adapter and not its siblings is a bug — say so, and make it a reviewer question.
Why. This is already our design; it just isn't written anywhere, so each instance looks
like a local judgement call. HF TRL states the same choice explicitly — consistency over
correctness, self-contained files, a pattern fixed in one trainer but not the others counts
as a bug. Unstated, the design decays in two directions at once: shared code grows if source == "…" branches while the adapters drift apart, and neither half is visible in a
single diff.
Change. Two sentences in AGENTS.md — duplication across adapters is expected, and changing
a shared pattern carries a duty to propagate it or to say why not — plus the reviewer question
that makes it checkable: does this pattern exist in sibling adapters, and did they change
too?
A13. Say when a contribution was agent-authored, and who can answer for it
One-liner: Disclose agent involvement in the PR description and name a human who can
answer review questions.
Why. CONTRIBUTING already routes "a large agent-authored change" to the human lane, which
only works if we know it was one. The failure this prevents is a PR whose author can't answer a
review question — Ray, vLLM and Transformers now refuse agent-authored PRs outright for that
reason. Refusing them would be incoherent for us, since we ship skills to invite exactly this,
so the alternative is knowing what we're looking at.
Change. One line in the PR template. Not a warning label: the point is lane routing and a
name to ask, not a lower standard for those PRs.
Appendix — terms used above
AGENTS.md — the
instruction file an AI coding agent reads when working in this repo; indexes the skills and
states the repo's principles.
Skill — a folder of instructions for an agent doing one job, entry point SKILL.md. We
ship eee-dataset-conversion
(turn a leaderboard or paper into EEE records) and eee-datastore-pr-review
(review a data PR on the datastore). A skill loads a short SKILL.md and pulls in a reference/ file only when relevant.
Decision log — the block in the PR
template
recording each non-obvious choice, its confidence, and whether it would recur elsewhere
(General? yes/no).
Skill-gap issue — this
template,
for reporting where a skill was silent or wrong; you don't need to know the fix.
Skill-pinning test — tests/test_skill_conversion.py,
which re-validates the conversion skill's templates and one frozen record through the real
validator, so a schema change fails there rather than in someone's PR.
Merge gate — the semantic checks uv run python -m every_eval_ever validate enforces
beyond the JSON schema, listed in reference/datastore-gate.md.
Adapter — the script under every_eval_ever/adapters/<name>/ converting one source. Datastore — the Hugging Face
dataset the records live in.
Bottom line. Thirteen changes to our contributor docs and agent skills, drawn from what
SGLang, NeMo-RL, PyTorch, vLLM, Ray, HF TRL/Transformers and LangChain put in theirs. Each
is a paragraph or a line of config, each stands alone — approve, reject or defer them
individually. The three I'd take first:
one item here with a cost we've already paid: issue
#242 is 84% of the datastore
failing the merge gate on two fields.
a review can be triaged by scanning rather than read in full.
implemented three times (#184,
#247,
#249), the last two a day apart.
What I'd like: a yes/no/later on each. I'm happy to write whichever ones you want. Three
questions from the same survey that need a decision rather than a patch are in
#251.
Terms are defined in the appendix at the bottom for anyone reading this cold.
All thirteen, for scanning
The last column is the one to weigh: three of these fix something that has already cost us
review time, and the rest are arguments. Reply with a verdict per number —
A1 yes, A4 later, A7 nois a complete answer, and no is a useful one.Out of scope:slot, and one sentence on consumer-visible effectreference/fields.md.gitignore+ AGENTS.mdxfail, 1skip, no omit configA1. Ask what a schema or validator change does to the records already published
Why. Such a change doesn't only constrain new contributions — it re-judges everything
already published. That is how
#242 happened: 84% of the datastore
now fails the merge gate on two missing fields. CONTRIBUTING § "Changing the schema, the
validator, or the
publisher"
covers regenerating types, the skill-pinning test and a version bump — all of it
forward-looking.
Change. One required question on those PRs: how many published records would now fail,
and what happens to them? The validator runs over the datastore, so the answer is a count
rather than an opinion — followed by one of: nothing breaks · N records break, here's the
follow-up issue · this needs a
schema_versionbump.Note. A question, not a veto. Where a new requirement can land as a warning and become an
error once the backlog clears, that's usually the cheap path.
A2. Give review findings a fixed shape
Why. Long, evenly-weighted review prose is expensive to act on: the ask is buried and
nothing marks what blocks a merge, so the reader reconstructs severity themselves. Agent
reviewers make this worse by being fluent — pages of correct text in which every sentence
looks equally important.
Change. A field template in the review skill. A template rather than a style note,
because an agent follows a template exactly and interprets "be concise" however it likes:
after the next push and which agent reviewers produce by default.
that can't state this is a question, not a finding.
action).
One comment carries a handful of findings and says how many it held back, rather than posting
a wall.
Note. PyTorch resolves severity ambiguity the other way, declaring every comment a
must-fix and banning nits. That fits code review and not data review, where plenty of real
findings are "confirm this was deliberate" — neither a blocker nor a nit. Worth knowing too
that Hugging Face dataset discussions have no inline comments and no resolvable threads, so
on the datastore side every finding lands in one flat comment: the structure is the only
thing separating twelve findings from twelve paragraphs.
A3. Check whether the work already exists before starting it
Why. The daily-cron feature has been implemented three times —
#184 (opened 11 Jun, closed 7 Aug),
#247 (opened 11 Aug, since closed) and
#249 (12 Aug, the one still open), all against
#159, the last two a day apart.
Nothing in AGENTS.md, CONTRIBUTING.md, the skills or the templates says to look first, and now
that starting is nearly free the cost lands as duplicate PRs in the review queue.
Change. Four searches, written as commands so there's nothing to remember:
every_eval_ever/adapters/README.md;datastore, since a
source can be converted with no adapter PR open.
Then the rule that makes searching matter, which Ray, vLLM and Transformers all state
explicitly: on a hit, don't open the PR — say what's missing on the existing one.
Otherwise an agent opens it anyway and mentions the duplicate in the description.
Note. Searching the published output rather than only the PR list is the half that matters
for a data repo, and no surveyed repo does it.
A4. Two lines in the PR template
Why. A description that says what a PR does but not what it leaves alone invites a
reviewer to ask for the adjacent fix, or a later contributor to bundle one in — SGLang's
env-var guide carries an explicit "Out of scope" section for exactly that reason. And our
"What / source" field invites a paragraph where a reviewer needs a sentence: which records
appear, change or disappear.
Change. Add an
Out of scope:slot and a one-sentence lead on consumer-visible effect.A5. One home per fact
Why. CONTRIBUTING § "Filling in the
schema"
specifies the
evaluation_idformat, thesource_typevalues, theinference_platformvsinference_enginesplit and the timestamp fields — all of it owned byreference/fields.mdand
eval.schema.json.tests/test_skill_conversion.pyfails when the schema moves out from under the skill; nothing does that for CONTRIBUTING, so
that copy goes stale silently. AGENTS.md already states the rule: don't restate the rule
anywhere else — prose that duplicates the check is what went stale last time.
Change. Replace the duplicated specs with links, and add the reviewer's question for any
PR changing a field or an allowed-value list: what documents this, and did it move too? One
home per rule; examples stay wherever they help.
A6. A scratch directory that can't reach a diff
A conversion run produces staged records,
adapter_reports/output, temporary clones andnotes — every one a candidate stray file, and we already needed
#231 to ignore
.claude/settings.local.json.A7. Any list of suppressed problems may only shrink
Why, and the honest version of it. We don't have this problem yet: the suite carries one
xfail(test_inspect_edge_cases.py)and one
pytest.mark.skip(
test_helm_adapter.py),and no coverage-omit or
filterwarningsconfig at all. So this is the one item here proposedagainst a cost we haven't paid — include it only if a cheap ratchet now looks better than the
argument later. The pattern it forecloses is the familiar one: each suppression is justified
once, nobody re-reads the aggregate, and the list only grows.
Change. Commit the count and fail the build when it grows, so removals are free and
additions cost one sentence. The related habit is worth stating either way, since it's free:
prefer an expected failure to a skip — an expected-failure test tells you when the problem
is fixed, a skipped test never speaks again. This is the check-shaped version of the PR
template's "no warnings either".
A8. Say when a new test isn't worth adding
Why. The PR template requires an offline unit test per adapter, which is right, and across
dozens of adapters is also dozens of chances to add a test that restates the implementation.
Those never fail for a reason anyone cares about, need updating whenever the code moves, and
make coverage look better than it is.
Change. A short paragraph in CONTRIBUTING with that question and the reviewer's version of
it: if I deleted this test, would some realistic regression now slip through?
A9. Route by path: "before editing this file, read that one"
Why. AGENTS.md indexes skills by description, which works when the task announces its
domain ("convert this leaderboard") and fails when the task is small but lands somewhere
load-bearing — a base adapter, the publisher,
validation_core.py. The agent never learns askill applied.
Change. Half a dozen rows, keyed on directory patterns rather than filenames and pointing
only at guidance that already exists, so it needs touching when a skill is added or renamed
and not when a file moves. If it can't stay that short, it isn't worth having: a stale routing
table is worse than none, because it sends an agent to guidance that no longer matches the
code.
A10. Notice when a PR touches files its title doesn't mention
Why. Scope creep is usually strays rather than a change of mind: an editor config, a
lockfile, a commit from a messy rebase, a shared-code fix inside a PR titled after one adapter.
Each one costs a reviewer the time to work out whether it was deliberate.
Change. Write down which paths each title
prefix
normally touches, then enforce it asymmetrically: the skills say "don't stage paths outside
this PR type's lane", which an agent follows for free; humans get a bot comment listing
out-of-lane paths, or nothing at all. As a blocking check this would be pure friction on
legitimate cross-cutting PRs, and nearly all the value is in the agent half anyway — two
sentences per skill, no CI.
A11. Verify a finding before posting it, and post only problems
Why. A wrong review comment is cheap to write and expensive to disprove, and two of them
teach a contributor to skim the rest. A review padded with what's fine buries the parts that
aren't.
Change. Two lines in the review contract, applying to every agent review rather than to
one skill: verify each finding against the actual code and sources, dropping it below a
confidence floor rather than hedging; and report only problems — no praise, clean sections
omitted. #241 already does this for
the deployment-metadata question; this is the same rule generalised, since it's a property of
agent review rather than of that question. The verification pass roughly doubles review
tokens, so scope it to findings that ask the contributor to do work.
A12. Write down that adapters are deliberately self-contained
Why. This is already our design; it just isn't written anywhere, so each instance looks
like a local judgement call. HF TRL states the same choice explicitly — consistency over
correctness, self-contained files, a pattern fixed in one trainer but not the others counts
as a bug. Unstated, the design decays in two directions at once: shared code grows
if source == "…"branches while the adapters drift apart, and neither half is visible in asingle diff.
Change. Two sentences in AGENTS.md — duplication across adapters is expected, and changing
a shared pattern carries a duty to propagate it or to say why not — plus the reviewer question
that makes it checkable: does this pattern exist in sibling adapters, and did they change
too?
A13. Say when a contribution was agent-authored, and who can answer for it
Why. CONTRIBUTING already routes "a large agent-authored change" to the human lane, which
only works if we know it was one. The failure this prevents is a PR whose author can't answer a
review question — Ray, vLLM and Transformers now refuse agent-authored PRs outright for that
reason. Refusing them would be incoherent for us, since we ship skills to invite exactly this,
so the alternative is knowing what we're looking at.
Change. One line in the PR template. Not a warning label: the point is lane routing and a
name to ask, not a lower standard for those PRs.
Appendix — terms used above
AGENTS.md— theinstruction file an AI coding agent reads when working in this repo; indexes the skills and
states the repo's principles.
SKILL.md. Weship
eee-dataset-conversion(turn a leaderboard or paper into EEE records) and
eee-datastore-pr-review(review a data PR on the datastore). A skill loads a short
SKILL.mdand pulls in areference/file only when relevant.reviewed":
Fast vs Needs-a-human, decided by the change and not by who wrote it.
template
recording each non-obvious choice, its confidence, and whether it would recur elsewhere
(
General? yes/no).template,
for reporting where a skill was silent or wrong; you don't need to know the fix.
tests/test_skill_conversion.py,which re-validates the conversion skill's templates and one frozen record through the real
validator, so a schema change fails there rather than in someone's PR.
uv run python -m every_eval_ever validateenforcesbeyond the JSON schema, listed in
reference/datastore-gate.md.every_eval_ever/adapters/<name>/converting one source.Datastore — the Hugging Face
dataset the records live in.
cc @nelaturuharsha