From d7704e436454185fd54ed3414aded8ded3fdbb80 Mon Sep 17 00:00:00 2001 From: nelaturuharsha Date: Sun, 9 Aug 2026 03:42:42 +0200 Subject: [PATCH 1/5] docs: checkpoint datastore PR reviews --- .../skills/eee-datastore-pr-review/SKILL.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/.agents/skills/eee-datastore-pr-review/SKILL.md b/.agents/skills/eee-datastore-pr-review/SKILL.md index 827e6b51e..e727a5063 100644 --- a/.agents/skills/eee-datastore-pr-review/SKILL.md +++ b/.agents/skills/eee-datastore-pr-review/SKILL.md @@ -55,6 +55,33 @@ Re-read the allowed deployment values from `every_eval_ever/validator/validation_core.py` and the live schema. Existing records and old bot comments may use obsolete vocabularies. +## Progress checkpoints + +Emit an incremental checkpoint to the caller at every boundary below. Checkpoints are +run receipts, not Hugging Face discussion comments: do not post them to the PR unless +the operator explicitly asks. Each checkpoint must include the phase, current PR head +SHA, facts established since the prior checkpoint, affected file/model counts, command +exit statuses or evidence URLs when applicable, blockers, and the next action. + +Required checkpoints: + +1. **Snapshot:** after selecting the PR head and matching bot run. +2. **Diagnosis:** after reproducing the gate and grouping its findings. +3. **Evidence and patch plan:** after resolving model-specific evidence and before the + first live PR mutation. Include every proposed field change and any unresolved axis. +4. **Local repair:** after the repaired diff passes local validation and content review. +5. **Remote receipt:** immediately after each uploaded commit or validator-trigger + comment, including the returned commit SHA or discussion event id. +6. **Bot result:** after each completed bot run, tied to its head/fingerprint; repeat + diagnosis and repair checkpoints for another iteration. + +For a phase lasting more than 60 seconds, emit a heartbeat at least once per minute +with the current evidence surface or bounded poll, completed/remaining counts, and +whether local or remote state changed. Use bounded polling calls of at most 45 seconds +so progress messages can be delivered. A checkpoint is not an approval gate: continue +unless the operating contract requires an operator decision or the operator asked to +pause at that phase. + ## Workflow ### 1. Establish the exact PR state From 1dbac8d2fa45d6c55fab3178e6fc18883988d9cf Mon Sep 17 00:00:00 2001 From: nelaturuharsha Date: Sun, 9 Aug 2026 03:45:28 +0200 Subject: [PATCH 2/5] docs: keep review runs active after checkpoints --- .agents/skills/eee-datastore-pr-review/SKILL.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.agents/skills/eee-datastore-pr-review/SKILL.md b/.agents/skills/eee-datastore-pr-review/SKILL.md index e727a5063..790746461 100644 --- a/.agents/skills/eee-datastore-pr-review/SKILL.md +++ b/.agents/skills/eee-datastore-pr-review/SKILL.md @@ -62,6 +62,8 @@ run receipts, not Hugging Face discussion comments: do not post them to the PR u the operator explicitly asks. Each checkpoint must include the phase, current PR head SHA, facts established since the prior checkpoint, affected file/model counts, command exit statuses or evidence URLs when applicable, blockers, and the next action. +When a progress or parent-message channel is available, send the checkpoint through it +and continue in the same run. Do not end a turn merely to deliver a checkpoint. Required checkpoints: From 3d09249b7060e206ec3fc267d6a39b8e1132a3c8 Mon Sep 17 00:00:00 2001 From: nelaturuharsha Date: Sun, 9 Aug 2026 04:06:26 +0200 Subject: [PATCH 3/5] docs: require approval for deployment research --- .../skills/eee-datastore-pr-review/SKILL.md | 50 +++++++++++++++---- .../assets/deployment-metadata-proposal.md | 24 +++++++++ 2 files changed, 64 insertions(+), 10 deletions(-) create mode 100644 .agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md diff --git a/.agents/skills/eee-datastore-pr-review/SKILL.md b/.agents/skills/eee-datastore-pr-review/SKILL.md index 790746461..3af35256d 100644 --- a/.agents/skills/eee-datastore-pr-review/SKILL.md +++ b/.agents/skills/eee-datastore-pr-review/SKILL.md @@ -5,8 +5,8 @@ description: >- dataset. Use when given an EEE_datastore discussion or PR URL, asked to run or reproduce `/eee validate changed`, resolve EEE validator errors or warnings, research model deployment_type or model_availability, edit the - changed datastore records, rerun the bot, or prepare canonical-registry - follow-ups. + changed datastore records, prepare human-approved deployment metadata + proposals, rerun the bot, or prepare canonical-registry follow-ups. --- # Review and repair an EEE datastore PR @@ -31,6 +31,9 @@ not sufficient. - If asked only to review, prepare a patch and findings without uploading or commenting. If asked to fix, update the supplied PR, trigger its validator, and iterate on that same ref. +- Before changing `deployment_type` or `model_availability` on the live PR, obtain + explicit human approval of a proposal bound to the current PR head. Authorization + to fix the PR is not approval of research-derived field values. - Ask the operator before a policy decision: minting a new canonical id, changing a schema/validator rule, dropping non-trivial data, choosing an ambiguous metric or bound, or making another structural change. Do not hide such a choice in a data @@ -69,8 +72,9 @@ Required checkpoints: 1. **Snapshot:** after selecting the PR head and matching bot run. 2. **Diagnosis:** after reproducing the gate and grouping its findings. -3. **Evidence and patch plan:** after resolving model-specific evidence and before the - first live PR mutation. Include every proposed field change and any unresolved axis. +3. **Research proposal:** after resolving model-specific evidence. Render + `assets/deployment-metadata-proposal.md`, report its SHA-256, and pause for explicit + human approval before editing deployment fields or mutating the live PR. 4. **Local repair:** after the repaired diff passes local validation and content review. 5. **Remote receipt:** immediately after each uploaded commit or validator-trigger comment, including the returned commit SHA or discussion event id. @@ -80,9 +84,9 @@ Required checkpoints: For a phase lasting more than 60 seconds, emit a heartbeat at least once per minute with the current evidence surface or bounded poll, completed/remaining counts, and whether local or remote state changed. Use bounded polling calls of at most 45 seconds -so progress messages can be delivered. A checkpoint is not an approval gate: continue -unless the operating contract requires an operator decision or the operator asked to -pause at that phase. +so progress messages can be delivered. Continue after ordinary checkpoints. The +research proposal is an approval gate; do not continue past it without an explicit +approval matching both its digest and PR head. ## Workflow @@ -154,12 +158,33 @@ appendix, source repository, and official API/release documentation. Use current research where facts may have changed, but pin the evidence revision or date relevant to the submitted evaluation. -Batch models only after proving that they share the same evidence. Keep an evidence -table with raw model label, canonical model id, both decisions, source URL/revision, -and confidence. +Batch models only after proving that they share the same evidence. + +Before editing either deployment axis: + +1. Copy `assets/deployment-metadata-proposal.md` to run notes outside the datastore + repository and fill one row per exact submitted `model_info.id`. Do not substitute + a folder slug or an unapproved canonical id. +2. Use only the five table columns in the template. Reference sources as `S1`, `S2`, + and so on; list each full URL and pinned revision/date once below the table. +3. Include every model whose deployment fields would change, including mechanical + vocabulary migrations. State unresolved axes as `unknown` rather than omitting a + row. +4. Compute the completed file's SHA-256. Return the rendered Markdown (or a clickable + path in a shared workspace), digest, PR head, model count, and affected-file count. +5. Stop and request explicit human approval of that digest at that PR head. Do not edit + the records, upload a commit, or comment on the Hugging Face discussion while + approval is pending. + +Approval covers only the exact table and head named by the human. Before applying it, +re-fetch `refs/pr/`. If the head changed, evidence changed, a proposed value +changed, or a new model entered scope, regenerate the artifact and obtain approval +again. Record the approver and approval time in the decision log. ### 5. Make the repair +- For deployment metadata, begin only after the research proposal is explicitly + approved and the remote head still matches it. Apply only its approved rows. - Edit only files implicated by a finding. Avoid mass reformatting unrelated data. - Preserve UUID filenames and stable evaluation identities unless identity itself is the defect. @@ -190,6 +215,9 @@ When the task authorizes a fix, upload exact add/delete operations to the existi as `parent_commit` so concurrent updates fail instead of being overwritten. Never set `create_pr=True` for a repair round. +If the commit changes either deployment axis, require the approved proposal digest and +head in the decision log before uploading. General authorization to fix is insufficient. + After the commit lands: 1. Comment `/eee validate changed` on the same discussion with @@ -230,6 +258,8 @@ Return: - local validation and duplicate-check results; - content spot-checks performed; - deployment/availability evidence table, including researched `unknown` values; +- research proposal path or rendered table, SHA-256, approved head, approver, and + approval time; - registry and adapter follow-ups with cross-links or candidate tables; - decision log and any unresolved blocker. diff --git a/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md b/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md new file mode 100644 index 000000000..2d8a40a9a --- /dev/null +++ b/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md @@ -0,0 +1,24 @@ +# EEE deployment metadata proposal + +- PR: `` +- PR head: `<40-character commit SHA>` +- Schema version: `` +- Scope: ` models / files` +- Status: **PENDING HUMAN APPROVAL** + +| model_id | proposed deployment_type | proposed model_availability | confidence | source(s) | +|---|---|---|---|---| +| `` | `` | `` | `` | `` | + +## Sources + +- **S1** — [``](), `` — `` + +## Unresolved questions + +- None. + +## Approval + +Approve only the completed proposal's reported SHA-256 at the PR head above. Any +change to the head, table, or evidence invalidates approval. From 68eed5c55b4bd2400b086d9c784ec5f7b6371c0b Mon Sep 17 00:00:00 2001 From: nelaturuharsha Date: Sun, 9 Aug 2026 04:15:44 +0200 Subject: [PATCH 4/5] docs: define metadata missingness evidence standard --- .../skills/eee-datastore-pr-review/SKILL.md | 42 ++++++-- .../assets/deployment-metadata-proposal.md | 17 +++- .../reference/metadata-missingness.md | 96 +++++++++++++++++++ .../reference/model-deployment.md | 44 ++++++++- 4 files changed, 185 insertions(+), 14 deletions(-) create mode 100644 .agents/skills/eee-datastore-pr-review/reference/metadata-missingness.md diff --git a/.agents/skills/eee-datastore-pr-review/SKILL.md b/.agents/skills/eee-datastore-pr-review/SKILL.md index 3af35256d..914de7c9b 100644 --- a/.agents/skills/eee-datastore-pr-review/SKILL.md +++ b/.agents/skills/eee-datastore-pr-review/SKILL.md @@ -6,7 +6,8 @@ description: >- or reproduce `/eee validate changed`, resolve EEE validator errors or warnings, research model deployment_type or model_availability, edit the changed datastore records, prepare human-approved deployment metadata - proposals, rerun the bot, or prepare canonical-registry follow-ups. + proposals, distinguish record omissions from genuinely unavailable metadata, + rerun the bot, or prepare canonical-registry follow-ups. --- # Review and repair an EEE datastore PR @@ -23,9 +24,11 @@ not sufficient. declared bounds, or change a value merely to silence the validator. - Make `unknown` a researched conclusion, not a default. Record which relevant surfaces were checked before retaining it. -- Distinguish absent record metadata from unavailable source evidence. A missing or - null `model_info.additional_details` object means the record needs investigation; - it does not establish either axis as `unknown`. +- Distinguish absent record metadata from unavailable source evidence for every field. + Describe an unchecked absence as "not surfaced in the submitted record," not as + missing from the underlying evaluation. A missing or null + `model_info.additional_details` object means the record needs investigation; it does + not establish either axis as `unknown`. - Keep work on the supplied `refs/pr/` ref. Do not open a replacement PR for another repair round. - If asked only to review, prepare a patch and findings without uploading or @@ -48,6 +51,10 @@ Before editing, read these sibling references: - `../eee-dataset-conversion/reference/datastore-submission.md` - `../eee-dataset-conversion/reference/verification.md` +Read `reference/metadata-missingness.md` whenever a field is absent, null, defaulted, +or claimed to be unavailable. Apply it to deployment metadata and reproducibility +fields such as temperature and maximum output tokens. + Read `reference/model-deployment.md` whenever either model deployment axis is missing, stale, invalid, or suspicious. Read `../eee-dataset-conversion/reference/registry.md` when an id is unresolved or a @@ -136,6 +143,12 @@ Group findings by root cause rather than by file. For each group, record: - the source evidence needed for a correct fix; - proposed change and confidence. +Classify every apparent omission with `reference/metadata-missingness.md`. Do not call +an absent field genuinely missing while its status is `record_absent` or +`research_incomplete`. If a README, eval card, methodology page, leaderboard, paper, +repository, or API exposes it, classify it as `available_not_surfaced` and identify +the adapter/submission gap. + Inspect content even when the validator omits it. At minimum check suspicious zeroes, score scale and bounds, metric identity, `source_data`, duplicate overall/subtask aggregates, stable `evaluation_id`, model identity, answer leakage, and companion @@ -148,9 +161,10 @@ actually supplied `additional_details`, supplied only one axis, or supplied neit ### 4. Research ambiguous metadata -For deployment warnings, apply `reference/model-deployment.md` to each exact model -variant and evaluation run. Determine the two axes independently. Do not infer one -from the other, from the developer folder, or from a provider-wide rule. +Apply `reference/metadata-missingness.md` before deciding any absent field is truly +unavailable. For deployment warnings, then apply `reference/model-deployment.md` to +each exact model variant and evaluation run. Determine the two axes independently. Do +not infer one from the other, from the developer folder, or from a provider-wide rule. Search all relevant primary surfaces before choosing `unknown`: record payload and run config, generating adapter, pinned model card, evaluator methodology, paper and @@ -166,10 +180,13 @@ Before editing either deployment axis: repository and fill one row per exact submitted `model_info.id`. Do not substitute a folder slug or an unapproved canonical id. 2. Use only the five table columns in the template. Reference sources as `S1`, `S2`, - and so on; list each full URL and pinned revision/date once below the table. + and so on; list each full URL and pinned revision/date once below the table. Record + deployment (`D`) and availability (`A`) confidence and sources separately inside + their shared cells. 3. Include every model whose deployment fields would change, including mechanical - vocabulary migrations. State unresolved axes as `unknown` rather than omitting a - row. + vocabulary migrations. Propose `unknown` only for `conflicting_sources` or + `unavailable_after_search`, and complete the template's unknown-rationale table. + If any axis is `research_incomplete`, do not finalize the proposal. 4. Compute the completed file's SHA-256. Return the rendered Markdown (or a clickable path in a shared workspace), digest, PR head, model count, and affected-file count. 5. Stop and request explicit human approval of that digest at that PR head. Do not edit @@ -196,6 +213,9 @@ again. Record the approver and approval time in the decision log. - If generated records are wrong, fix or prepare the generating adapter in the code repo as well; otherwise the next refresh will restore the defect. Keep adapter code out of the datastore PR and cross-link its separate PR. +- Treat `available_not_surfaced` as an extraction defect: backfill the approved value + in the data repair and prepare an adapter/submission follow-up so regeneration does + not erase it. - Review the resulting diff for accidental deletion, unrelated churn, and a mechanical replacement applied to semantically different models. @@ -257,6 +277,8 @@ Return: - files changed, grouped by root cause; - local validation and duplicate-check results; - content spot-checks performed; +- missingness classification for each absent field, including source-available + extraction gaps and the surfaces checked before any unavailable claim; - deployment/availability evidence table, including researched `unknown` values; - research proposal path or rendered table, SHA-256, approved head, approver, and approval time; diff --git a/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md b/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md index 2d8a40a9a..23492fe16 100644 --- a/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md +++ b/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md @@ -8,13 +8,26 @@ | model_id | proposed deployment_type | proposed model_availability | confidence | source(s) | |---|---|---|---|---| -| `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | ## Sources - **S1** — [``](), `` — `` -## Unresolved questions +## Unknown rationale + +Complete one row for every `unknown` cell. Do not finalize this proposal while any +axis is `research_incomplete`. + +Coverage codes: `R` record/run artifacts, `A` adapter/config, `E` eval card/README/ +methodology/leaderboard, `M` exact model card/release, `P` official provider/API, +`W` paper/repository/archive. + +| model_id | axis | status/reason | checked | confusion or evidence gap | evidence needed to resolve | +|---|---|---|---|---|---| +| `` | `` | `: ` | `` | `` | `` | + +## Other unresolved questions - None. diff --git a/.agents/skills/eee-datastore-pr-review/reference/metadata-missingness.md b/.agents/skills/eee-datastore-pr-review/reference/metadata-missingness.md new file mode 100644 index 000000000..867ef0af2 --- /dev/null +++ b/.agents/skills/eee-datastore-pr-review/reference/metadata-missingness.md @@ -0,0 +1,96 @@ +# Distinguish record omissions from unavailable metadata + +Use this protocol before claiming that any reproducibility or model field is missing. +It applies to deployment metadata, temperature, token limits, and other fields that an +adapter or submitter may have failed to surface. + +## Classify the observation + +| Status | Meaning | Required action | +|---|---|---| +| `record_absent` | The submitted record omits the field; source research has not finished | Say "not surfaced in the submitted record" and investigate | +| `research_incomplete` | One or more relevant primary surfaces or exact identities remain unchecked | Continue research; do not propose `unknown` or claim source missingness | +| `available_not_surfaced` | A reliable source contains the value but the record does not | Backfill with approval and fix/follow up on the adapter or submission path | +| `conflicting_sources` | Relevant primary sources disagree and run-level evidence does not resolve them | Document the conflict; use the field's unknown/absent representation only with approval | +| `unavailable_after_search` | The stop rule is complete and searched first-party sources do not establish the value | Document the bounded claim and use the field's unknown/absent representation with approval | + +These are research statuses, not schema values. For deployment axes, the schema value +corresponding to the last two unresolved outcomes is `unknown`. + +## Search primary surfaces + +Search the exact evaluation, model variant, aliases, and relevant historical date. +Inspect each surface when it exists; mark it not applicable with a reason rather than +silently skipping it: + +1. Raw submitted record, instance companion, run logs, and generation configuration. +2. Pinned adapter, scraper, submission payload, and evaluator client configuration. +3. First-party eval card, README, methodology page, leaderboard detail page, and data + card. +4. Evaluation paper, appendix, supplementary material, source repository, and pinned + configuration files. +5. Exact model card, release, weight-file listing, license/access page, and aliases. +6. Official provider API/SDK documentation, model catalog, release/deprecation notice, + and an archived page when current documentation may differ from the evaluated run. + +Follow every concrete primary-source lead found on those surfaces. Do not count a +search-result snippet, model family, provider name, or current library default as the +submitted run's value. A harness default is evidence only when the pinned version and +run configuration establish that the default governed this run. + +## Keep a search ledger + +For each field and exact identity, record the surface, URL or repository revision, +retrieval/release date, query or path inspected, result, and any contradiction. Reuse a +source across models only when it explicitly covers every grouped variant. + +Use concise reason codes for unresolved results: + +- `no_run_provenance` +- `identity_unresolved` +- `conflicting_primary_sources` +- `historical_evidence_unavailable` +- `source_access_blocked` +- `no_primary_evidence_after_search` + +State what specific artifact or statement would resolve the uncertainty. +`source_access_blocked` normally remains `research_incomplete`: an inaccessible known +primary source is still an unexamined lead, not evidence that the value is unavailable. + +## Apply the stop rule + +Classify a field as `unavailable_after_search` only when: + +- every relevant surface above is checked or marked not applicable with a reason; +- exact aliases, versions, checkpoints, and the evaluation date are searched; +- mutable pages are pinned or an archive/repository history is attempted; +- the ledger contains no unexamined primary-source lead; and +- the claim is scoped to the searched sources and retrieval date. + +If any condition is false, use `research_incomplete`. Never use a time limit or number +of search queries alone as proof of unavailable metadata. + +## Use a confidence threshold + +- **High:** a direct first-party run artifact or explicit exact-source statement. +- **Medium:** at least two aligned first-party sources for the exact identity, with no + contrary run-level evidence. +- **Low:** family/provider inference, alias uncertainty, a single indirect source, + secondary reporting, an assumed default, or unresolved primary-source conflict. + +Only high- or medium-confidence categorical values may enter a repair proposal. A +low-confidence candidate becomes the field's unknown/absent representation only after +the stop rule is complete; otherwise it remains `research_incomplete`. + +## Phrase findings conservatively + +- Before research: "The submitted EEE record does not surface this field." +- When found elsewhere: "The field is source-available but was not surfaced by this + adapter/submission." +- After the stop rule: "The value was not established in the searched first-party + sources as of ``." + +Do not generalize bounded search results into a claim that the underlying evaluation +never documented the field. For papers and eval cards, distinguish source completeness +from the platform's extraction coverage and emphasize that living first-party and +community submissions can improve that coverage over time. diff --git a/.agents/skills/eee-datastore-pr-review/reference/model-deployment.md b/.agents/skills/eee-datastore-pr-review/reference/model-deployment.md index cf2713585..f22d44bf6 100644 --- a/.agents/skills/eee-datastore-pr-review/reference/model-deployment.md +++ b/.agents/skills/eee-datastore-pr-review/reference/model-deployment.md @@ -4,6 +4,18 @@ Use this reference to decide the two model axes for the exact evaluated model an The live schema and validator define the allowed strings; this reference defines the evidence standard. +Read `metadata-missingness.md` first. `unknown` is a schema value; +`unavailable_after_search` is the research status that can justify proposing it. + +## Contents + +- Classify what is missing +- Keep the axes independent +- Evidence priority and confidence threshold +- Decision signals +- Research procedure +- Batch-edit guardrails + ## Classify what is missing Inspect the raw JSON, not only a parsed `EvaluationLog`. The library can materialize @@ -19,6 +31,8 @@ compatibility placeholders and make an omitted key look like an explicit `unknow - If the relevant primary sources genuinely omit the fact, keep `unknown` and list the sources checked. Missing source evidence is a valid result; missing record metadata is not evidence. +- If the required search is unfinished, classify the axis as `research_incomplete`. + Do not convert incomplete research into `unknown`. ## Keep the axes independent @@ -48,6 +62,20 @@ A model card usually establishes availability, not deployment. A leaderboard row developer folder usually establishes neither. When sources conflict, prefer the run-level source, preserve the conflict in the decision log, and lower confidence. +## Confidence threshold + +Rate the best candidate for each axis before writing the proposal: + +| Confidence | Evidence standard | Proposal action | +|---|---|---| +| High | Direct first-party evidence for the exact run or exact model artifact | Propose the supported categorical value | +| Medium | Two aligned first-party sources for the exact variant, with no contrary run-level evidence | Propose the supported categorical value | +| Low | Family/provider inference, alias uncertainty, one indirect source, secondary reporting, or unresolved conflict | Never propose the categorical guess | + +For a low-confidence candidate, continue research. If the missingness stop rule is +complete, propose `unknown` and document the reason; otherwise keep +`research_incomplete` and do not finalize the approval artifact. + ## Decision signals Use these as evidence tests, not name-based mappings: @@ -63,6 +91,16 @@ Use these as evidence tests, not name-based mappings: Do not mechanically translate stale vocabulary until checking the old validator's meaning and the submitted run. Lexical similarity is not source evidence. +For `deployment_type`, require run/evaluator evidence: raw generation metadata, a +client/backend in the pinned adapter or config, or an explicit methodology statement. +A model card cannot establish who operated inference for this evaluation. + +For `model_availability`, inspect the exact artifact, not only a family README. A +downloadable gated or restrictively licensed checkpoint is `open_weights` under the +current schema. API access alone is `closed_weights`. Treat delta-only releases, +missing base weights, removed repositories, renamed checkpoints, and availability that +changed after the evaluation date as ambiguity requiring dated evidence. + ## Research procedure 1. Extract `model_info.id`, `name`, `developer`, `inference_platform`, @@ -81,8 +119,10 @@ meaning and the submitted run. Lexical similarity is not source evidence. |---|---|---|---|---|---| | ... | ... | ... | ... | ... | high/medium/low | -6. Use `unknown` only after listing the relevant surfaces checked and why they did not - establish the axis. Do not turn lack of quick evidence into a provider-wide guess. +6. Complete the search ledger and stop rule in `metadata-missingness.md`. Use + `unknown` only for `conflicting_sources` or `unavailable_after_search`, with a reason + code and the evidence needed to resolve it. Do not turn lack of quick evidence into + a provider-wide guess. ## Batch-edit guardrails From 5f861a59bc33690e0640fcd538fe031b3c6afcf5 Mon Sep 17 00:00:00 2001 From: nelaturuharsha Date: Mon, 10 Aug 2026 04:07:53 +0200 Subject: [PATCH 5/5] docs: preserve datastore review decisions --- .../skills/eee-datastore-pr-review/SKILL.md | 42 +++++++++++++++++-- .../assets/collection-review-decision.md | 18 ++++++++ .../assets/deployment-metadata-proposal.md | 12 ++++++ .../reference/model-deployment.md | 31 +++++++++++++- 4 files changed, 99 insertions(+), 4 deletions(-) create mode 100644 .agents/skills/eee-datastore-pr-review/assets/collection-review-decision.md diff --git a/.agents/skills/eee-datastore-pr-review/SKILL.md b/.agents/skills/eee-datastore-pr-review/SKILL.md index 914de7c9b..d97779002 100644 --- a/.agents/skills/eee-datastore-pr-review/SKILL.md +++ b/.agents/skills/eee-datastore-pr-review/SKILL.md @@ -37,6 +37,10 @@ not sufficient. - Before changing `deployment_type` or `model_availability` on the live PR, obtain explicit human approval of a proposal bound to the current PR head. Authorization to fix the PR is not approval of research-derived field values. +- Preserve approved decisions with the data. After approval, copy the byte-identical + proposal into each affected collection root and append its receipt to that + collection's `REVIEW_DECISIONS.md`; do not leave provenance only in temporary run + notes or discussion history. - Ask the operator before a policy decision: minting a new canonical id, changing a schema/validator rule, dropping non-trivial data, choosing an ambiguous metric or bound, or making another structural change. Do not hide such a choice in a data @@ -153,7 +157,11 @@ Inspect content even when the validator omits it. At minimum check suspicious ze score scale and bounds, metric identity, `source_data`, duplicate overall/subtask aggregates, stable `evaluation_id`, model identity, answer leakage, and companion pairing. An out-of-range score requires finding the source scale or source value; do -not cap, clamp, or round it into validity. +not cap, clamp, round it into validity, or widen the bounds around it. When a source +publishes a normalized value, convert it to the metric's declared scale and preserve +the raw value plus explicit conversion in `score_details.details` or the decision log. +Serialize unbounded limits as the schema-supported JSON strings `"Infinity"` and +`"-Infinity"`; replacing a bare non-finite token must not change its meaning. Inspect the raw JSON before constructing an `EvaluationLog`. The model layer may auto-fill absent deployment keys with `unknown`, hiding whether the contributor @@ -174,11 +182,23 @@ to the submitted evaluation. Batch models only after proving that they share the same evidence. +For an aggregate or leaderboard source, determine whether it ran inference or merely +collected cited results. An aggregator's adapter and repository establish aggregation +provenance, not the deployment used by every cited evaluation. Follow per-result +citations to run evidence. Multiple citations, harness labels, or generation settings +do not by themselves prove different deployments. Conversely, the same model id does +not make separate runs deployment-equivalent. Use one proposal row only when every +affected file in that row shares the evidence; otherwise use scoped repeated rows as +defined in the proposal template. Ask before splitting records when one log contains +proven conflicting deployments because that changes the contribution's structure. + Before editing either deployment axis: 1. Copy `assets/deployment-metadata-proposal.md` to run notes outside the datastore - repository and fill one row per exact submitted `model_info.id`. Do not substitute - a folder slug or an unapproved canonical id. + repository. Fill one row per exact submitted `model_info.id` when its affected files + are evidence-equivalent. If the same id has distinct run evidence, repeat the exact + id with scope tags and define those tags below the table. Do not substitute a folder + slug or an unapproved canonical id. 2. Use only the five table columns in the template. Reference sources as `S1`, `S2`, and so on; list each full URL and pinned revision/date once below the table. Record deployment (`D`) and availability (`A`) confidence and sources separately inside @@ -216,6 +236,17 @@ again. Record the approver and approval time in the decision log. - Treat `available_not_surfaced` as an extraction defect: backfill the approved value in the data repair and prepare an adapter/submission follow-up so regeneration does not erase it. +- After approval, copy the proposal's exact bytes to every affected collection as + `data//deployment-metadata-proposal-.md`. Never alter + that copy: its full digest must still equal the approved digest. +- Create or append `data//REVIEW_DECISIONS.md` using + `assets/collection-review-decision.md`. Record the PR and pre-edit head, full proposal + digest and relative path, approver/time, affected records, source-backed decisions, + validation receipt, and unresolved adapter/registry work. Append an entry for other + substantive content corrections such as score-scale, metric-identity, model-identity, + or data-drop decisions even when no deployment proposal exists. Do not require an + entry for formatting-only or byte-serialization-only repairs. Preserve existing + entries and never store private research notes, credentials, or inaccessible URLs. - Review the resulting diff for accidental deletion, unrelated churn, and a mechanical replacement applied to semantically different models. @@ -227,6 +258,9 @@ or bot says “Ready to Merge.” Compare the final changed-path inventory with the initial inventory. Explain every new path, deletion, identity change, or source-value change in the decision log. +Collection-root proposal and `REVIEW_DECISIONS.md` files are intentional new paths; +verify their links, proposal digest, and append-only history even though the JSON +validator ignores Markdown. ### 7. Update and monitor the existing PR @@ -282,6 +316,8 @@ Return: - deployment/availability evidence table, including researched `unknown` values; - research proposal path or rendered table, SHA-256, approved head, approver, and approval time; +- collection-root proposal copies and `REVIEW_DECISIONS.md` entries, including a digest + check proving each proposal copy is byte-identical to the approved artifact; - registry and adapter follow-ups with cross-links or candidate tables; - decision log and any unresolved blocker. diff --git a/.agents/skills/eee-datastore-pr-review/assets/collection-review-decision.md b/.agents/skills/eee-datastore-pr-review/assets/collection-review-decision.md new file mode 100644 index 000000000..28bbae2a5 --- /dev/null +++ b/.agents/skills/eee-datastore-pr-review/assets/collection-review-decision.md @@ -0,0 +1,18 @@ +## `` — `` + +- PR: `` +- Pre-edit head: `<40-character SHA>` +- Scope: `` +- Approved proposal: `` +- Proposal SHA-256: `` +- Approval: `` +- Local verification: `` +- Matching bot receipt: `` + +### Decisions + +- ``: `` (``) + +### Follow-ups + +- `` diff --git a/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md b/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md index 23492fe16..0b48d3192 100644 --- a/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md +++ b/.agents/skills/eee-datastore-pr-review/assets/deployment-metadata-proposal.md @@ -10,6 +10,14 @@ |---|---|---|---|---| | `` | `` | `` | `` | `` | +Repeat the exact model id with a scope tag such as `[G1]` only when its submitted files +have distinct run evidence. Multiple citations or generation settings alone do not +create a group. + +## Scope manifest + +- **G1** — `` — `` + ## Sources - **S1** — [``](), `` — `` @@ -35,3 +43,7 @@ methodology/leaderboard, `M` exact model card/release, `P` official provider/API Approve only the completed proposal's reported SHA-256 at the PR head above. Any change to the head, table, or evidence invalidates approval. + +After approval, preserve this file byte-for-byte at +`data//deployment-metadata-proposal-.md` and record the +approval receipt in that collection's `REVIEW_DECISIONS.md`. diff --git a/.agents/skills/eee-datastore-pr-review/reference/model-deployment.md b/.agents/skills/eee-datastore-pr-review/reference/model-deployment.md index f22d44bf6..6c152c606 100644 --- a/.agents/skills/eee-datastore-pr-review/reference/model-deployment.md +++ b/.agents/skills/eee-datastore-pr-review/reference/model-deployment.md @@ -11,6 +11,7 @@ Read `metadata-missingness.md` first. `unknown` is a schema value; - Classify what is missing - Keep the axes independent +- Handle aggregators and repeated model ids - Evidence priority and confidence threshold - Decision signals - Research procedure @@ -48,6 +49,32 @@ Common valid combinations include hosted open-weight models and self-deployed open-weight models. Weight availability alone never proves how this evaluation served the model. +## Handle aggregators and repeated model ids + +First decide whether the submitted source ran inference or aggregates scores reported +elsewhere. A score matrix, leaderboard, or adapter marked as an aggregator does not +inherit a deployment type: its repository can establish where citations came from but +not who hosted each cited evaluation. Follow each result's `reference_url`, source +notes, or equivalent citation to the evaluator's run artifacts and methods. + +Keep these tests separate: + +- Same model id and different generation parameters do not imply different deployment. +- Different citations, benchmarks, harness labels, or evaluator relationships do not + alone prove different deployment. +- The same model id can still be self-deployed in one run and accessed through a hosted + API in another. Require run evidence before grouping them. +- `model_availability` remains an exact-model property and can be shared across runs + when the identity and dated evidence match; do not derive deployment from it. + +Before assigning the log-level `deployment_type`, inventory the run evidence for every +result in that log. Assign one categorical value only when all results are +evidence-equivalent. If results have proven conflicting serving methods, do not flatten +them to a model-wide value: ask the operator whether to split the log by provenance or, +after the missingness stop rule, retain an approval-gated `unknown`. Splitting records +is structural and requires agreement. A multi-reference log with no contradiction is +not itself a reason to split. + ## Evidence priority Prefer evidence closest to the submitted run: @@ -113,7 +140,9 @@ changed after the evaluation date as ambiguity requiring dated evidence. quantization. Do not substitute a nearby family member. 4. Pin web evidence to a revision, release date, or retrieval date. Prefer official model cards, repositories, papers, and provider docs over aggregators. -5. Record one row per evidence-equivalent model group: +5. Record one row per evidence-equivalent model/run group. Repeat a model id when its + submitted files have distinct run evidence; keep a scope manifest so approved values + map deterministically back to files: | Raw label | Canonical id | Deployment | Availability | Evidence and revision | Confidence | |---|---|---|---|---|---|