Skip to content

Thirteen small contribution-workflow changes, each independently approvable #250

Description

@borgr

Bottom line. Thirteen changes to our contributor docs and agent skills, drawn from what
SGLang, NeMo-RL, PyTorch, vLLM, Ray, HF TRL/Transformers and LangChain put in theirs. Each
is a paragraph or a line of config, each stands alone — approve, reject or defer them
individually. The three I'd take first:

  • A1 — ask what a schema or validator change does to already-published records. The
    one item here with a cost we've already paid: issue
    #242 is 84% of the datastore
    failing the merge gate on two fields.
  • A2 — a fixed shape for review findings, each labelled must-fix / question / FYI, so
    a review can be triaged by scanning rather than read in full.
  • A3 — check for existing work before starting it. The daily-cron feature has been
    implemented three times (#184,
    #247,
    #249), the last two a day apart.

What I'd like: a yes/no/later on each. I'm happy to write whichever ones you want. Three
questions from the same survey that need a decision rather than a patch are in
#251.

Terms are defined in the appendix at the bottom for anyone reading this cold.

All thirteen, for scanning

The last column is the one to weigh: three of these fix something that has already cost us
review time, and the rest are arguments. Reply with a verdict per number — A1 yes, A4 later, A7 no is a complete answer, and no is a useful one.

# Change Lands in Already cost us?
A1 Say what a schema or validator change does to already-published records one required PR question Yes#242, 84% of the datastore failing the gate
A2 Fixed three-line shape for review findings, labelled must-fix / question / FYI template in review skill No — argument from review load
A3 Four searches before starting; on a hit, comment instead of opening a PR CONTRIBUTING + skills Yes — cron built 3× (#184, #247, #249)
A4 Out of scope: slot, and one sentence on consumer-visible effect PR template, 2 lines No — SGLang precedent
A5 Replace duplicated field specs with links to reference/fields.md CONTRIBUTING Yes — duplication is live in both files today
A6 One git-ignored scratch directory, named in AGENTS.md .gitignore + AGENTS.md Partly — #231
A7 Commit the count of suppressed warnings and expected failures; fail on growth CI check No — we have 1 xfail, 1 skip, no omit config
A8 "What change would turn this test red?" — if only editing it, don't add it CONTRIBUTING, 1 para No — no instance found in the suite
A9 File-pattern → read-this-first routing table AGENTS.md, ~6 rows No — argument from how skills load
A10 Don't stage paths outside the PR type's lane 2 sentences per skill, no CI No — no instance cited
A11 Verify each finding before posting; report only problems review contract, 2 lines Partly — #241 does it for one question
A12 Adapters are deliberately self-contained; propagate a shared-pattern fix or say why not AGENTS.md, 2 sentences No — already our design, just unwritten
A13 Disclose agent authorship and name a human who can answer PR template, 1 line No — CONTRIBUTING already assumes we know

A1. Ask what a schema or validator change does to the records already published

One-liner: On any PR that changes the schema or a validator rule, say how many
already-published records it would invalidate.

Why. Such a change doesn't only constrain new contributions — it re-judges everything
already published. That is how
#242 happened: 84% of the datastore
now fails the merge gate on two missing fields. CONTRIBUTING § "Changing the schema, the
validator, or the
publisher"

covers regenerating types, the skill-pinning test and a version bump — all of it
forward-looking.

Change. One required question on those PRs: how many published records would now fail,
and what happens to them?
The validator runs over the datastore, so the answer is a count
rather than an opinion — followed by one of: nothing breaks · N records break, here's the
follow-up issue · this needs a schema_version bump.

Note. A question, not a veto. Where a new requirement can land as a warning and become an
error once the backlog clears, that's usually the cheap path.


A2. Give review findings a fixed shape

One-liner: Three lines per finding — what and where, what it breaks, the ask — each
labelled must-fix / question / FYI.

Why. Long, evenly-weighted review prose is expensive to act on: the ask is buried and
nothing marks what blocks a merge, so the reader reconstructs severity themselves. Agent
reviewers make this worse by being fluent — pages of correct text in which every sentence
looks equally important.

Change. A field template in the review skill. A template rather than a style note,
because an agent follows a template exactly and interprets "be concise" however it likes:

  1. What and where — one line, citing a symbol and file. Not a line number, which is wrong
    after the next push and which agent reviewers produce by default.
  2. What it breaks — one line, concrete: these inputs give that wrong output. A finding
    that can't state this is a question, not a finding.
  3. The askmust-fix · question ("confirm this is intentional") · FYI (no
    action).

One comment carries a handful of findings and says how many it held back, rather than posting
a wall.

Note. PyTorch resolves severity ambiguity the other way, declaring every comment a
must-fix and banning nits. That fits code review and not data review, where plenty of real
findings are "confirm this was deliberate" — neither a blocker nor a nit. Worth knowing too
that Hugging Face dataset discussions have no inline comments and no resolvable threads, so
on the datastore side every finding lands in one flat comment: the structure is the only
thing separating twelve findings from twelve paragraphs.


A3. Check whether the work already exists before starting it

One-liner: Four searches before starting, and if one hits, comment on the existing work
instead of opening a second PR.

Why. The daily-cron feature has been implemented three times —
#184 (opened 11 Jun, closed 7 Aug),
#247 (opened 11 Aug, since closed) and
#249 (12 Aug, the one still open), all against
#159, the last two a day apart.
Nothing in AGENTS.md, CONTRIBUTING.md, the skills or the templates says to look first, and now
that starting is nearly free the cost lands as duplicate PRs in the review queue.

Change. Four searches, written as commands so there's nothing to remember:

  • open PRs and issues matching the source or feature name;
  • closed PRs — Add scheduled adapter ingestion workflow #184 was closed, which is exactly why the next two missed it;
  • the adapter list in every_eval_ever/adapters/README.md;
  • the collection folders already in the
    datastore, since a
    source can be converted with no adapter PR open.

Then the rule that makes searching matter, which Ray, vLLM and Transformers all state
explicitly: on a hit, don't open the PR — say what's missing on the existing one.
Otherwise an agent opens it anyway and mentions the duplicate in the description.

Note. Searching the published output rather than only the PR list is the half that matters
for a data repo, and no surveyed repo does it.


A4. Two lines in the PR template

One-liner: What this PR deliberately leaves alone, and what changes for someone
consuming the data.

Why. A description that says what a PR does but not what it leaves alone invites a
reviewer to ask for the adjacent fix, or a later contributor to bundle one in — SGLang's
env-var guide carries an explicit "Out of scope" section for exactly that reason. And our
"What / source" field invites a paragraph where a reviewer needs a sentence: which records
appear, change or disappear.

Change. Add an Out of scope: slot and a one-sentence lead on consumer-visible effect.


A5. One home per fact

One-liner: CONTRIBUTING.md restates field rules that reference/fields.md and the
schema own — replace them with links.

Why. CONTRIBUTING § "Filling in the
schema"

specifies the evaluation_id format, the source_type values, the inference_platform vs
inference_engine split and the timestamp fields — all of it owned by
reference/fields.md
and eval.schema.json.
tests/test_skill_conversion.py
fails when the schema moves out from under the skill; nothing does that for CONTRIBUTING, so
that copy goes stale silently. AGENTS.md already states the rule: don't restate the rule
anywhere else — prose that duplicates the check is what went stale last time.

Change. Replace the duplicated specs with links, and add the reviewer's question for any
PR changing a field or an allowed-value list: what documents this, and did it move too? One
home per rule; examples stay wherever they help.


A6. A scratch directory that can't reach a diff

One-liner: One git-ignored folder, named in AGENTS.md, for anything not meant for the PR.

A conversion run produces staged records, adapter_reports/ output, temporary clones and
notes — every one a candidate stray file, and we already needed
#231 to ignore
.claude/settings.local.json.


A7. Any list of suppressed problems may only shrink

One-liner: Commit the count of suppressed warnings and expected failures, and fail the
build when it grows.

Why, and the honest version of it. We don't have this problem yet: the suite carries one
xfail (test_inspect_edge_cases.py)
and one pytest.mark.skip
(test_helm_adapter.py),
and no coverage-omit or filterwarnings config at all. So this is the one item here proposed
against a cost we haven't paid — include it only if a cheap ratchet now looks better than the
argument later. The pattern it forecloses is the familiar one: each suppression is justified
once, nobody re-reads the aggregate, and the list only grows.

Change. Commit the count and fail the build when it grows, so removals are free and
additions cost one sentence. The related habit is worth stating either way, since it's free:
prefer an expected failure to a skip — an expected-failure test tells you when the problem
is fixed, a skipped test never speaks again. This is the check-shaped version of the PR
template's "no warnings either".


A8. Say when a new test isn't worth adding

One-liner: "What future change would turn this test red?" If the only answer is
"editing the test", don't add it.

Why. The PR template requires an offline unit test per adapter, which is right, and across
dozens of adapters is also dozens of chances to add a test that restates the implementation.
Those never fail for a reason anyone cares about, need updating whenever the code moves, and
make coverage look better than it is.

Change. A short paragraph in CONTRIBUTING with that question and the reviewer's version of
it: if I deleted this test, would some realistic regression now slip through?


A9. Route by path: "before editing this file, read that one"

One-liner: A short file-pattern → read-this-first table in AGENTS.md.

Why. AGENTS.md indexes skills by description, which works when the task announces its
domain ("convert this leaderboard") and fails when the task is small but lands somewhere
load-bearing — a base adapter, the publisher, validation_core.py. The agent never learns a
skill applied.

Change. Half a dozen rows, keyed on directory patterns rather than filenames and pointing
only at guidance that already exists, so it needs touching when a skill is added or renamed
and not when a file moves. If it can't stay that short, it isn't worth having: a stale routing
table is worse than none, because it sends an agent to guidance that no longer matches the
code.


A10. Notice when a PR touches files its title doesn't mention

One-liner: A hard rule in the skills against staging paths outside the PR's lane; for
humans, an advisory comment at most.

Why. Scope creep is usually strays rather than a change of mind: an editor config, a
lockfile, a commit from a messy rebase, a shared-code fix inside a PR titled after one adapter.
Each one costs a reviewer the time to work out whether it was deliberate.

Change. Write down which paths each title
prefix

normally touches, then enforce it asymmetrically: the skills say "don't stage paths outside
this PR type's lane", which an agent follows for free; humans get a bot comment listing
out-of-lane paths, or nothing at all. As a blocking check this would be pure friction on
legitimate cross-cutting PRs, and nearly all the value is in the agent half anyway — two
sentences per skill, no CI.


A11. Verify a finding before posting it, and post only problems

One-liner: Check each finding against the code before posting it, and treat "nothing
found" as a complete review.

Why. A wrong review comment is cheap to write and expensive to disprove, and two of them
teach a contributor to skim the rest. A review padded with what's fine buries the parts that
aren't.

Change. Two lines in the review contract, applying to every agent review rather than to
one skill: verify each finding against the actual code and sources, dropping it below a
confidence floor rather than hedging; and report only problems — no praise, clean sections
omitted. #241 already does this for
the deployment-metadata question; this is the same rule generalised, since it's a property of
agent review rather than of that question. The verification pass roughly doubles review
tokens, so scope it to findings that ask the contributor to do work.


A12. Write down that adapters are deliberately self-contained

One-liner: Duplication across adapters is intended, so fixing a shared pattern in one
adapter and not its siblings is a bug — say so, and make it a reviewer question.

Why. This is already our design; it just isn't written anywhere, so each instance looks
like a local judgement call. HF TRL states the same choice explicitly — consistency over
correctness
, self-contained files, a pattern fixed in one trainer but not the others counts
as a bug. Unstated, the design decays in two directions at once: shared code grows
if source == "…" branches while the adapters drift apart, and neither half is visible in a
single diff.

Change. Two sentences in AGENTS.md — duplication across adapters is expected, and changing
a shared pattern carries a duty to propagate it or to say why not — plus the reviewer question
that makes it checkable: does this pattern exist in sibling adapters, and did they change
too?


A13. Say when a contribution was agent-authored, and who can answer for it

One-liner: Disclose agent involvement in the PR description and name a human who can
answer review questions.

Why. CONTRIBUTING already routes "a large agent-authored change" to the human lane, which
only works if we know it was one. The failure this prevents is a PR whose author can't answer a
review question — Ray, vLLM and Transformers now refuse agent-authored PRs outright for that
reason. Refusing them would be incoherent for us, since we ship skills to invite exactly this,
so the alternative is knowing what we're looking at.

Change. One line in the PR template. Not a warning label: the point is lane routing and a
name to ask, not a lower standard for those PRs.

Appendix — terms used above
  • AGENTS.md — the
    instruction file an AI coding agent reads when working in this repo; indexes the skills and
    states the repo's principles.
  • Skill — a folder of instructions for an agent doing one job, entry point SKILL.md. We
    ship
    eee-dataset-conversion
    (turn a leaderboard or paper into EEE records) and
    eee-datastore-pr-review
    (review a data PR on the datastore). A skill loads a short SKILL.md and pulls in a
    reference/ file only when relevant.
  • Review lanesCONTRIBUTING § "How your PR gets
    reviewed"
    :
    Fast vs Needs-a-human, decided by the change and not by who wrote it.
  • Decision log — the block in the PR
    template

    recording each non-obvious choice, its confidence, and whether it would recur elsewhere
    (General? yes/no).
  • Skill-gap issuethis
    template
    ,
    for reporting where a skill was silent or wrong; you don't need to know the fix.
  • Skill-pinning test
    tests/test_skill_conversion.py,
    which re-validates the conversion skill's templates and one frozen record through the real
    validator, so a schema change fails there rather than in someone's PR.
  • Merge gate — the semantic checks uv run python -m every_eval_ever validate enforces
    beyond the JSON schema, listed in
    reference/datastore-gate.md.
  • Adapter — the script under every_eval_ever/adapters/<name>/ converting one source.
    Datastore — the Hugging Face
    dataset
    the records live in.

cc @nelaturuharsha

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions