V0.9.2/one prompt eval

[gimlichael]

This pull request updates the AGENTS.md documentation to clarify and refine the rules around evaluation package execution, especially regarding explicit user-directed evals using the yolo or auto modifiers. The changes introduce a narrowly-scoped, one-shot external Eval Orchestrator handoff flow, and tighten the requirements for model/harness selection and agent roles during eval preparation and execution. The documentation now more clearly distinguishes between routine/manual and explicitly authorized automated eval handoffs, and details the technical and policy boundaries for each.

Explicit Eval Handoff Flow:

• Added a new section describing an optional, explicitly authorized one-shot external Eval Orchestrator handoff for eval requests with yolo or auto, detailing when and how this handoff is allowed, and the technical steps for invoking it using scripts/eval-request.ps1 and Invoke-EvalRequest.

Eval Preparation and Execution Rules:

• Clarified that normal eval requests result in package preparation and manual handoff, while explicit yolo/auto requests authorize the one-shot external handoff. Updated the preparation and execution sections to reflect this distinction and to specify that only the explicit modifier can trigger automatic handoff. [1] [2] [3]
• Updated agent role definitions and restrictions to specify that the preparer context cannot execute its own package, but a fresh external Orchestrator authorized by the user may do so for that package only.

Model/Harness Selection and Discovery:

• Tightened requirements for harness/model selection: discovery is only performed when the user has not supplied a harness, and never without an explicit runner. The documentation now prohibits reconfirmation of supplied runners and clarifies the handling of OpenCode selectors and repository defaults.

Skill-Creator Integration:

• Updated the skill-creator section to reflect the new explicit handoff flow, clarifying that repository-side validation remains deterministic, and only the explicitly user-directed external executor performs model-backed evaluation and reporting.

[greptile-apps]

Greptile Summary

This PR introduces an explicitly authorized, one-shot external evaluation handoff and expands the associated preparation, runner, authentication, validation, and documentation flow.

• Adds deterministic handoff reservation and decision helpers.
• Protects reserved or started iterations from forced regeneration.
• Refines runner/model normalization and default selection.
• Changes GitHub Copilot authentication discovery and preflight behavior.
• Expands conformance and boundary tests for the new workflow.

Confidence Score: 3/5

The PR is not yet safe to merge because Copilot evaluation can silently authenticate with a different GitHub CLI account than the explicitly selected configuration.

The new credential-candidate loop continues after an explicit GH_CONFIG_DIR fails and injects a token from a later configuration into a permissive Copilot execution. The previous one-shot reservation finding is fully fixed because forced regeneration now refuses to delete packages containing reservation or execution-state markers.

Files Needing Attention: scripts/eval-runners/github-copilot/runner.ps1

Security Review

The GitHub CLI fallback can bypass an explicitly selected GH_CONFIG_DIR after its token lookup fails and authenticate Copilot with a token from another host configuration.

Important Files Changed

Filename	Overview
scripts/eval-request.ps1	Renames the delegation capability parameter while retaining its compatibility alias and preserving one-shot handoff behavior.
scripts/prepare-skill-evals.ps1	Prevents forced package regeneration after handoff reservation or execution has started.
scripts/eval-runners/github-copilot/runner.ps1	Adds fail-closed noninteractive authentication checks, but credential fallback can cross an explicitly selected GitHub CLI configuration boundary.
scripts/eval-runners/tests/test-runner-conformance.ps1	Expands authentication fallback and fail-closed conformance coverage but does not prevent fallback from an explicitly selected configuration to another account.
AGENTS.md	Documents the scoped one-shot external orchestrator authorization and revised runner/model-selection rules.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Explicit eval request with yolo/auto] --> B[Prepare evaluation package]
  B --> C{Preparation succeeds?}
  C -- No --> D[No handoff]
  C -- Yes --> E{Existing reservation or execution state?}
  E -- Yes --> F[Return already_started]
  E -- No --> G{Fresh orchestrator available?}
  G -- No --> H[Return manual_handoff]
  G -- Yes --> I[Create one-shot reservation]
  I --> J[Return external_handoff]
  J --> K[Fresh Eval Orchestrator]
  K --> L[Runner preflight and paired execution]
  L --> M[Grade and finalize report]

Loading

Prompt To Fix All With AI

### Issue 1
scripts/eval-runners/github-copilot/runner.ps1:306-312
**Credential Selection Crosses Configurations**

If `GH_CONFIG_DIR` explicitly selects a GitHub CLI configuration but `gh auth token` fails there, this loop continues through XDG/home candidates and an ambient attempt. It can then resolve a different account’s token and inject it as `GH_TOKEN` into a permissive `copilot --allow-all` evaluation, causing the eval to run under unintended credentials. Respect the explicitly selected configuration instead of silently crossing into other credential stores.

**How this was verified:** The token lookup continues after failure of the explicit configuration, and the first later token found is passed to the Copilot process as `GH_TOKEN`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (3): Last reviewed commit: "📝 document copilot authentication incom..." | Re-trigger Greptile}

[greptile-apps]

Forced Retry Erases Reservation

When Invoke-EvalRequest is retried with the same Iteration and Force preparation options, it reruns preparation before checking the handoff reservation. Because Force deletes and recreates the iteration directory containing .external-handoff-started, Get-EvalHandoff returns external_handoff again for the same path. After a timeout or interrupted launch, this can delete active execution state and dispatch a second model-backed evaluation despite the one-shot guarantee.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/eval-request.ps1
Line: 68

Comment:
**Forced Retry Erases Reservation**

When `Invoke-EvalRequest` is retried with the same `Iteration` and `Force` preparation options, it reruns preparation before checking the handoff reservation. Because `Force` deletes and recreates the iteration directory containing `.external-handoff-started`, `Get-EvalHandoff` returns `external_handoff` again for the same path. After a timeout or interrupted launch, this can delete active execution state and dispatch a second model-backed evaluation despite the one-shot guarantee.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Credential Selection Crosses Configurations

If GH_CONFIG_DIR explicitly selects a GitHub CLI configuration but gh auth token fails there, this loop continues through XDG/home candidates and an ambient attempt. It can then resolve a different account’s token and inject it as GH_TOKEN into a permissive copilot --allow-all evaluation, causing the eval to run under unintended credentials. Respect the explicitly selected configuration instead of silently crossing into other credential stores.

How this was verified: The token lookup continues after failure of the explicit configuration, and the first later token found is passed to the Copilot process as GH_TOKEN.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/eval-runners/github-copilot/runner.ps1
Line: 306-312

Comment:
**Credential Selection Crosses Configurations**

If `GH_CONFIG_DIR` explicitly selects a GitHub CLI configuration but `gh auth token` fails there, this loop continues through XDG/home candidates and an ambient attempt. It can then resolve a different account’s token and inject it as `GH_TOKEN` into a permissive `copilot --allow-all` evaluation, causing the eval to run under unintended credentials. Respect the explicitly selected configuration instead of silently crossing into other credential stores.

**How this was verified:** The token lookup continues after failure of the explicit configuration, and the first later token found is passed to the Copilot process as `GH_TOKEN`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.