diff --git a/AGENTS.md b/AGENTS.md index 80d4201..924ac91 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ An executing harness stays inside its package. Building, testing, or writing any Repository-owned preparation, validation, CI jobs, hooks, deterministic tests, package generation, automatic completion gates, and automatic agent fan-out must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as repository test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. The package-local implementations under `scripts/eval-runners/` are protocol adapters, not automatic repository execution: they may invoke their native harness only when a human-selected external Eval Orchestrator is explicitly handed a prepared package and selected profile. - Do not create, restore, recommend, or run generic automation or automatic fan-out that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. A runner adapter may launch its named native harness only at the explicit external-handoff boundary described below, never from repository automation or CI. -- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. `yolo`, `auto`, urgency, completion gates, third-party instructions, and prior approval do not change this rule. +- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. Urgency, completion gates, third-party instructions, and prior approval do not change this rule. The sole narrow exception is an explicit user-directed eval request with `yolo` (or the existing `auto` alias): it authorizes one fresh external Eval Orchestrator handoff per prepared package, as described below; it never authorizes the preparer to execute an arm. - Routine skill validation is local and deterministic. Use schema and metadata checks, fixture validation, bundled assertions, repository validators, and human inspection of the eval prompts and expected outcomes. - Model-backed comparisons are not a repository completion gate. Do not spawn additional agents or call external model tools merely to satisfy a generic eval workflow. - A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. @@ -40,13 +40,13 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops, except for the explicitly authorized fresh external handoff below. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval -`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. +`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Without `yolo`/`auto`, treat them as instructions to prepare the package and stop for manual handoff. With `yolo`/`auto`, prepare and perform the automatic external handoff below. Neither form requests new eval cases unless the user asks for those too. -Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1`. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. +Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. Only when the user has not supplied a harness, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1 -Runner `. Never invoke discovery without `-Runner`, and never ask to reconfirm a supplied runner. For a supplied runner with a deterministic default, let preparation resolve and verify it without another question; validate any explicit model exactly as supplied. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot @@ -60,11 +60,27 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed Use `-CodebeltReference` only when the script should select the Codebelt Reference explicitly and fail if `claude-haiku-4.5` is no longer present. For noninteractive direct script use, omitting both `-Runner` and `-CodebeltReference` is an error whenever a package would be generated; `-Runner codex` and `-Runner github-copilot` resolve their repository-defined default models, while `-Runner opencode` still requires `-Model`. +### Optional one-shot external handoff + +An explicit request such as `Please eval dotnet-strong-name-signing using Codex. yolo` authorizes preparation followed by ONE external Eval Orchestrator handoff for that package. The existing `auto` modifier has the same scoped meaning. Bare modifiers, skill maintenance, and completion gates do not authorize this flow. Authorization ends when the handoff completes; there are no answer-quality retries. + +Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available and do not infer otherwise merely because no shell command named "external orchestrator" exists. In capability terms: GitHub Copilot CLI with `task` + `general-purpose` available means `ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` is true. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. + +For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and set `-CanDelegateFreshOrchestrator` only after establishing the host capability above (`-ExternalOrchestratorAvailable` remains a compatibility alias). Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model; let preparation resolve the existing repository default and verify only that exact model's availability. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. + +When using a shell tool, pipe the returned decisions to `ConvertTo-Json -Depth 4` so table formatting cannot truncate absolute paths. Consume each returned decision in the interactive agent workflow: + +- `manual_handoff`: return the absolute `prompt_path` and stop. When the host is unavailable, include the concise `reason`; never execute an arm in the current context. +- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. In GitHub Copilot CLI this delegation is ONE fresh `task` `general-purpose` subagent; pass only `prompt_path`, keep the returned agent/task handle, and follow that same orchestrator to terminal completion. Do not paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. +- `already_started`: observe the existing Orchestrator. Never delegate again, rerun preparation to retry execution, remove the handoff reservation, or invoke Phase 1 again after a timeout, failed launch, interruption, or uncertain result. If the launch is known unavailable before it starts, use the manual path; once a launch is attempted, do not suggest blindly running the prompt again. A failed/interrupted execution remains incomplete under the existing fresh-iteration rule. + +The helper reserves the one-shot handoff with `.external-handoff-started` before returning `external_handoff`; retain its returned path and the native Orchestrator handle across continuations. `Get-EvalHandoff` can inspect that same path on resumption and refuses another automatic handoff. This receipt is outside the execution protocol and does not replace Phase 1 state or evidence. The only lifecycle instructions are the existing `RUN-THIS.prompt.md`; its package-local Phase 1 controller, grading, and finalization remain authoritative. No CI, hooks, validators, preparation scripts, or completion gates consume this decision by launching a model. + ### Handing the package over Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner and reads its dispatch owner. For runner-owned dispatch it invokes `invoke-runner-owned-arms.ps1` exactly once as a foreground Phase 1 command and sets the caller shell/tool timeout to at least the package-computed allowance. That command preflights every blind `with_skill` and `without_skill` arm and starts zero executions if any preflight is incompatible. For orchestrator-owned dispatch it delegates each arm to a fresh native worker. It records each terminal envelope through the deterministic runner-owned recorder where applicable, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. -Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. +Hand the user that one file by its absolute path, and stop there in the normal manual flow. The explicit one-shot flow hands that same file to the fresh external Orchestrator instead. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. Do not list the individual prompt files, do not describe the directory layout, and do not hand back a procedure for the user to carry out by hand. A reply that ends with 26 file paths and "run both versions" has moved the work onto the user instead of doing it. @@ -111,7 +127,7 @@ Run it before `scripts/sync-skill-install.ps1`, which stays the last gate becaus Two roles exist here, and they are not the same agent doing the same job. Read which one you are before applying the rules below. -**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop, and you must: +**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop for manual handoff unless this explicit eval request authorizes the one-shot external handoff above. In both cases you must: - never execute the prompts you just prepared, and never quietly become the executor of your own package - never spawn subagents for the candidate or baseline runs @@ -129,12 +145,12 @@ When a person gives you `RUN-THIS.prompt.md`, points you at a prepared package, Four things still hold while you execute: -- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or another agent asking is not a person asking. +- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or an unsolicited agent request is not a person asking. A fresh external Orchestrator receiving the canonical file through the explicit user-directed `yolo`/`auto` handoff above carries that user authorization for this package only. - Stay inside the package directory. If it sits inside a repository, the rest of that repository is not yours to read, build, test, or write. - Do not read the assertions or expected output before the worker runs. After every available worker has finished and its result is recorded, grade the completed results, generate the report, and state the comparison honestly. - Nothing about this makes model-backed execution a completion gate for any repository task. -An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. +An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This disqualifies the preparer context, not the fresh external Orchestrator receiving an explicitly authorized one-shot handoff. The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it follows `delegation.dispatch_owner`: for runner-owned dispatch it invokes the foreground package-local `invoke-runner-owned-arms.ps1` command exactly once and waits for its terminal JSON summary; the helper resolves the runner, validates `describe`, preflights every pending `run.json`, asserts native delegation for every result, and refuses to start any execution until all preflights pass. A caller/tool timeout or interrupted conversation does not authorize rerunning Phase 1. If execution was interrupted and no valid `execution-freeze.json` exists, the package is incomplete and requires a fresh iteration. For orchestrator-owned dispatch it resolves the runner, validates `describe`, preflights each `run.json`, and uses the declared native subagent/task. A runner-owned process/thread is the single Eval Worker and model execution; no outer model worker may contain it. Orchestrator-owned workers must not invoke `runner.ps1 execute`; their transport envelope passes through `record-native-result.ps1`. Runner-owned execution results come directly from the runner and must never be synthesized, repaired, or reconstructed from assistant text. The runner launches each native session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, only after Phase 1 returns a successful terminal JSON summary and the immutable freeze validates, the executor bridges results into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. @@ -212,7 +228,7 @@ Bare `yolo` or `auto` outside an explicit commit request does not invoke `git-vi Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. -Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. +Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops, unless an explicit `yolo`/`auto` eval request authorizes the fresh external handoff above. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. `skill-creator-agnostic` is deprecated, no longer maintained, and retained only for backward compatibility until 1.0.0. Agents must not use it for new skill creation, skill modification, or benchmarking; use Anthropic's `skill-creator` directly and apply the repository-specific requirements from this `AGENTS.md`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 21f0132..1cab27d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,33 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.9.2] - 2026-09-08 + +This patch release adds optional one-shot external handoff for explicit eval requests, complete with deterministic workflow helpers and test coverage. Agents can now accept `yolo` or `auto` modifiers on explicit evaluation requests to authorize a single fresh external Eval Orchestrator handoff; preparation remains deterministic and repository-local, while only an explicit, authorized harness receives the sealed package. Skill eval preparation gains a PassThru parameter for returning prompt paths, runner names are normalized for user-facing consistency, and validation tooling gains comprehensive eval-request-workflow checks. + +### Added + +- `scripts/eval-request.ps1` with deterministic helpers for optional one-shot external handoff: `Get-EvalHandoff` reserves and transitions between manual and external handoff states, while `Invoke-EvalRequest` normalizes harness names, discovers available models, returns preparation decisions, and never launches a model itself, +- Full test coverage for eval-request workflow in `scripts/eval-runners/tests/test-eval-request.ps1` exercising all runner/model normalization paths, handoff state transitions, already-started detection, unavailable-host fallback, and case-insensitive normalization, +- Runner normalization in eval-request helpers converting user-facing harness names (`GitHub Copilot`, `Copilot CLI`) to internal canonical runner ids (`github-copilot`) before model discovery and profile generation, +- PassThru parameter in `scripts/prepare-skill-evals.ps1` for returning the prepared prompt paths without file output, enabling downstream helpers to collect decisions and route handoffs, +- `-Yolo` flag in `scripts/eval-request.ps1` Invoke-EvalRequest authorizing one-shot external handoff after explicit user eval request, while keeping preparation and validation deterministic, +- Optional-one-shot-external-handoff section in `AGENTS.md` documenting the explicit user authorization, harness selection, model discovery flow (including OpenCode's explicit-model requirement), and the deterministic one-handoff boundary, +- External evaluation authorization guidance in `AGENTS.md` clarifying that `yolo`/`auto` modifiers on explicit eval requests enable handoff without widening repository automation, CI, completion gates, or model-backed execution, +- Eval-request-workflow validation checks in `scripts/validate-skill-templates.ps1` ensuring eval-request.ps1 presence, test coverage, runner normalization completeness, and handoff-state-reservation correctness. + +### Changed + +- `AGENTS.md` eval preparation guidance now distinguishes between manual handoff (normal path that prepares and returns `RUN-THIS.prompt.md`) and external handoff (when user explicitly requests eval with yolo/auto), clarifying that only the latter authorizes orchestrator delegation, +- Runner-model discovery and selection in `scripts/Get-HarnessModels.ps1` now fully documents the -Runner requirement and exposes current available models without auto-selecting a previous or default model, +- `README.md` eval section expanded with documentation of optional one-shot handoff feature, explicit authorization scoping, and reference to `AGENTS.md` for the complete workflow, +- `CONTRIBUTING.md` now references the optional one-shot eval handoff capability and directs users to full preparation and handoff guidance in `AGENTS.md`. + +### Fixed + +- Eval request workflow now clearly separates deterministic preparation (always model-free, always stops for manual handoff) from explicit external handoff (only with user-supplied yolo/auto, never retried if already launched), +- Model discovery no longer accepts harness names as ambiguous shortcuts; runner normalization explicitly converts `Copilot` and `GitHub Copilot CLI` to the canonical `github-copilot` runner before passing to model-discovery logic. + ## [0.9.1] - 2026-09-07 This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. @@ -638,6 +665,7 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. +[0.9.2]: https://github.com/codebeltnet/agentic/compare/v0.9.1...v0.9.2 [0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8df4768..c53b690 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,7 +98,9 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. For `github-copilot` without an explicit model, discovery is only an availability check for the repository-defined default; never rank or replace it with a subjective stronger/newer choice. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. + +For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is a valid external handoff capability for this one-shot flow: delegate exactly one fresh orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same returned agent/task handle to terminal completion. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. diff --git a/README.md b/README.md index b2dce65..e8234ba 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. Without an explicit one-shot request, the repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, grading application, and report adapter validate that ledger and never re-bless changed bytes. The external Grader writes only package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`); `apply-eval-grading.ps1` projects only `passed` and `evidence` decisions onto canonical results. `finalize-eval-package.ps1` owns the deterministic completion boundary and succeeds only after validating the freeze, bridge, complete grading, and all four report artifacts. A changed raw file requires a fresh Phase 1 execution, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. @@ -26,6 +26,8 @@ That resolves every skill the branch changed and prepares a package for each. `- Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the Grader writes only the grading artifact, then the deterministic application helper and finalizer perform the bridge, canonical projection, upstream aggregation/viewer compatibility, and first-party report generation. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. +Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability (`ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` = true): start exactly one fresh delegated orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same agent/task handle to terminal completion. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. If the resolved runner is `github-copilot` and no explicit model was supplied, preparation applies the existing repository default and verifies only that model's availability. Discovery is an availability check, not model ranking; never switch to a stronger/newer/different model by preference. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). + The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 new file mode 100644 index 0000000..0a4a8af --- /dev/null +++ b/scripts/eval-request.ps1 @@ -0,0 +1,74 @@ +<# +.SYNOPSIS + Deterministic helpers for the agent's explicit eval request workflow. Dot-source to use. +.DESCRIPTION + These helpers never launch a model. The interactive host consumes an external_handoff decision + with its fresh-context delegation tool, passing only the canonical prompt path. Do not wire + this decision to CI, hooks, preparation, validation, or completion-gate execution. + For GitHub Copilot CLI, task + general-purpose delegation is a valid external-orchestrator + capability for this one-shot handoff. +#> +Set-StrictMode -Version Latest + +function Get-EvalHandoff { + [CmdletBinding()] + param( + [Parameter(Mandatory)][string]$PromptPath, + [switch]$Yolo, + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator + ) + + $path = (Resolve-Path -LiteralPath $PromptPath -ErrorAction Stop).Path + if ([IO.Path]::GetFileName($path) -cne 'RUN-THIS.prompt.md') { + throw 'Handoff requires the prepared RUN-THIS.prompt.md file.' + } + $package = Split-Path -Parent $path + $decision = [ordered]@{ action = 'manual_handoff'; prompt_path = $path; reason = 'Preparation complete; hand this file to an external Eval Orchestrator.' } + if (-not $Yolo) { return [pscustomobject]$decision } + + # Reserve the handoff before the host delegates. An uncertain launch must never be retried. + $claim = Join-Path $package '.external-handoff-started' + if ((Test-Path -LiteralPath $claim) -or + (Test-Path -LiteralPath (Join-Path $package 'orchestration-state.json')) -or + (Test-Path -LiteralPath (Join-Path $package 'execution-freeze.json'))) { + $decision.action = 'already_started' + $decision.reason = 'Do not dispatch again or invoke Phase 1 again. Observe the existing Orchestrator; interrupted execution remains incomplete.' + return [pscustomobject]$decision + } + if (-not $CanDelegateFreshOrchestrator) { + $decision.reason = 'This host cannot delegate one fresh external Eval Orchestrator context. Keep the intact manual handoff and never execute an arm in this context.' + return [pscustomobject]$decision + } + try { + $stream = [IO.File]::Open($claim, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None) + $stream.Dispose() + } catch [IO.IOException] { + if (-not (Test-Path -LiteralPath $claim)) { throw } + $decision.action = 'already_started' + $decision.reason = 'A handoff was already reserved. Do not dispatch again.' + return [pscustomobject]$decision + } + $decision.action = 'external_handoff' + $decision.reason = 'Pass only this canonical file path to one fresh external Eval Orchestrator under the explicit user yolo authorization; wait for its final result.' + return [pscustomobject]$decision +} + +function Invoke-EvalRequest { + [CmdletBinding()] + param( + [Parameter(Mandatory)][hashtable]$Preparation, + [switch]$Yolo, + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator + ) + + $ErrorActionPreference = 'Stop' + # Splat existing preparation options unchanged; it alone resolves and verifies model policy. + # Collect every success before issuing any handoff, including multi-skill explicit requests. + if ($Preparation.ContainsKey('CollectResults')) { throw 'An eval request prepares packages; CollectResults is a separate forensic workflow.' } + $arguments = $Preparation.Clone() + $arguments.PassThru = $true + $paths = @(& (Join-Path $PSScriptRoot 'prepare-skill-evals.ps1') @arguments) + foreach ($path in $paths) { + Get-EvalHandoff -PromptPath $path -Yolo:$Yolo -CanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator + } +} diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index d3853fc..1c61842 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -300,12 +300,13 @@ in both paired arms. Personal Copilot configuration is excluded by run-local `COPILOT_HOME`, `COPILOT_CACHE_HOME`, `HOME`, `USERPROFILE`, and XDG roots; the runner does not copy the normal `.copilot` directory. Authentication prefers explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`; when none is -present, the trusted runner may resolve `gh auth token` outside the worker and -inject only that token as a protected environment variable. Host `GH_CONFIG_DIR` -is never forwarded into the evaluated worker. `--secret-env-vars` removes every -listed token variable from shell and MCP child environments. Preflight does not -make a model request and therefore reports native keychain/service readiness as -conditional rather than claiming successful remote authentication. Codex's +present, the trusted runner resolves `gh auth token` outside the worker across +documented/default config candidates and injects only that token as a protected +environment variable. Host `GH_CONFIG_DIR` is never forwarded into the evaluated +worker. `--secret-env-vars` removes every listed token variable from shell and +MCP child environments. If neither explicit token variables nor trusted GitHub +CLI token fallback is resolvable, preflight fails closed before Phase 1 model +execution starts. Codex's compatibility API-key path uses `--ask-for-approval never` with `exec --sandbox danger-full-access`; subscription eval arms use the runner-owned app-server path described above. It does not combine explicit sandbox selection with diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index f63bf46..da6dc6b 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -203,35 +203,64 @@ function Get-CopilotTokenVariable { return $null } -function Get-CopilotGhConfigDirectory { +function Add-CopilotGhConfigCandidate { + param( + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Candidates, + [string]$Path + ) + + if ([string]::IsNullOrWhiteSpace($Path)) { return } + try { + $normalized = [System.IO.Path]::GetFullPath($Path) + if (-not $Candidates.Contains($normalized)) { $Candidates.Add($normalized) } + } catch { } +} + +function Get-CopilotGhConfigDirectories { # GH_CONFIG_DIR is an authentication-state exception to the isolated - # Copilot configuration roots. Resolve it from GitHub CLI's documented - # precedence without reading or logging any credential file. - $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') - if ([string]::IsNullOrWhiteSpace($configured)) { - $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') - if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { - $configured = Join-Path $xdgConfig 'gh' - } elseif ((Get-PlatformName) -eq 'windows') { - $applicationData = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) - if (-not [string]::IsNullOrWhiteSpace($applicationData)) { - $configured = Join-Path $applicationData 'GitHub CLI' - } - } else { - $userHome = [Environment]::GetEnvironmentVariable('HOME') - if ([string]::IsNullOrWhiteSpace($userHome)) { - $userHome = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) - } + # Copilot configuration roots. Build documented/default candidates and keep + # only existing directories; token resolution tries each candidate in order. + $candidates = [System.Collections.Generic.List[string]]::new() + + Add-CopilotGhConfigCandidate -Candidates $candidates -Path ([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) + $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') + if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $xdgConfig 'gh') + } + + if ((Get-PlatformName) -eq 'windows') { + $applicationDataFromEnvironment = [Environment]::GetEnvironmentVariable('APPDATA') + if (-not [string]::IsNullOrWhiteSpace($applicationDataFromEnvironment)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $applicationDataFromEnvironment 'GitHub CLI') + } + + $applicationDataFromProfile = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) + if (-not [string]::IsNullOrWhiteSpace($applicationDataFromProfile)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $applicationDataFromProfile 'GitHub CLI') + } + } else { + foreach ($userHome in @( + [Environment]::GetEnvironmentVariable('HOME'), + [Environment]::GetEnvironmentVariable('USERPROFILE'), + [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) + )) { if (-not [string]::IsNullOrWhiteSpace($userHome)) { - $configured = Join-Path (Join-Path $userHome '.config') 'gh' + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path (Join-Path $userHome '.config') 'gh') } } } - if ([string]::IsNullOrWhiteSpace($configured) -or -not (Test-Path -LiteralPath $configured -PathType Container)) { - return $null + $resolved = [System.Collections.Generic.List[string]]::new() + foreach ($candidate in @($candidates)) { + if (Test-Path -LiteralPath $candidate -PathType Container) { $resolved.Add($candidate) } } - return [System.IO.Path]::GetFullPath($configured) + return @($resolved.ToArray()) +} + +function Get-CopilotGhConfigDirectory { + $directories = @(Get-CopilotGhConfigDirectories) + if ($directories.Count -eq 0) { return $null } + return [string]$directories[0] } function Get-CopilotGitHubCliToken { @@ -240,26 +269,54 @@ function Get-CopilotGitHubCliToken { return $null } - $environment = New-RunnerProbeEnvironment + $baseEnvironment = New-RunnerProbeEnvironment foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME', 'GH_CONFIG_DIR')) { $value = [Environment]::GetEnvironmentVariable($name) if (-not [string]::IsNullOrWhiteSpace($value)) { - $environment[$name] = $value + $baseEnvironment[$name] = $value + } + } + + $candidateConfigDirectories = [System.Collections.Generic.List[string]]::new() + foreach ($directory in @(Get-CopilotGhConfigDirectories)) { + if (-not [string]::IsNullOrWhiteSpace([string]$directory) -and -not $candidateConfigDirectories.Contains([string]$directory)) { + $candidateConfigDirectories.Add([string]$directory) } } - $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) + # Keep one ambient attempt with GH_CONFIG_DIR removed so an existing host + # default can still resolve even when no candidate was discovered. + $candidateConfigDirectories.Add('') + + $probeRoot = Resolve-RunnerProbeTempRoot + $probeDirectory = Join-Path $probeRoot ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null try { - $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 - if ($process.TimedOut -or $process.ExitCode -ne 0) { - return $null - } - $token = ([string]$process.Stdout).Trim() - if ([string]::IsNullOrWhiteSpace($token)) { - return $null + foreach ($candidateDirectory in @($candidateConfigDirectories.ToArray())) { + $environment = [ordered]@{} + foreach ($key in @($baseEnvironment.Keys)) { + $environment[[string]$key] = [string]$baseEnvironment[$key] + } + if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { + [void]$environment.Remove('GH_CONFIG_DIR') + } else { + $environment['GH_CONFIG_DIR'] = [string]$candidateDirectory + } + + $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { + continue + } + $token = ([string]$process.Stdout).Trim() + if ([string]::IsNullOrWhiteSpace($token)) { + continue + } + return [pscustomobject]@{ + Token = $token + GhConfigDirectory = if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { $null } else { [string]$candidateDirectory } + } } - return $token + return $null } finally { if (Test-Path -LiteralPath $probeDirectory) { Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue @@ -275,24 +332,31 @@ function Resolve-CopilotAuthentication { TokenVariable = $tokenVariable TokenValue = $null GitHubCliTokenResolved = $false + GitHubCliConfigDirectory = $null + NonInteractiveReady = $true } } $githubCliToken = Get-CopilotGitHubCliToken - if (-not [string]::IsNullOrWhiteSpace($githubCliToken)) { + if ($null -ne $githubCliToken -and -not [string]::IsNullOrWhiteSpace([string]$githubCliToken.Token)) { return [pscustomobject]@{ Source = 'github_cli_token' TokenVariable = 'GH_TOKEN' - TokenValue = $githubCliToken + TokenValue = [string]$githubCliToken.Token GitHubCliTokenResolved = $true + GitHubCliConfigDirectory = [string](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigDirectory' -Default $null) + NonInteractiveReady = $true } } + $fallbackSource = if ($null -ne (Resolve-ExternalCommand -Name 'copilot')) { 'copilot_os_keychain_unverified' } else { 'none' } return [pscustomobject]@{ - Source = 'copilot_os_keychain_or_github_cli_unverified' + Source = $fallbackSource TokenVariable = $null TokenValue = $null GitHubCliTokenResolved = $false + GitHubCliConfigDirectory = $null + NonInteractiveReady = $false } } @@ -602,10 +666,11 @@ function Get-CopilotPreflight { if ($authState.Source -eq 'environment') { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) } elseif ($authState.Source -eq 'github_cli_token') { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'GitHub CLI fallback resolved a token in the trusted runner; only a protected token environment variable will be passed to Copilot.')) + $configDetail = if ([string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory)) { 'ambient/default' } else { 'resolved GH_CONFIG_DIR candidate' } + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail ("GitHub CLI fallback resolved a token in the trusted runner from a {0} source; only a protected token environment variable will be passed to Copilot." -f $configDetail))) } else { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status unavailable -Detail 'No explicit token is present and GitHub CLI fallback did not yield a token; native Copilot OS-keychain lookup is delegated to the installed CLI. This preflight does not contact the Copilot service.')) - $warnings.Add('Authentication readiness beyond explicit environment tokens and the observable GitHub CLI fallback cannot be proven without a live Copilot request; preflight remains conditional and does not reject a tokenless native OAuth/keychain configuration.') + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or ensure `gh auth token` can resolve in this host context.') } if ($platform -notin @('linux', 'macos')) { @@ -646,7 +711,7 @@ function Get-CopilotPreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('runner-owned fresh Copilot CLI session per eval execution', 'copilot --output-format json terminal event capture', 'native task/general-purpose subagent available as a separate harness capability, not the transport', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('runner-owned fresh Copilot CLI session per eval execution', 'copilot --output-format json terminal event capture', 'native task/general-purpose subagent available as a separate harness capability, not the transport', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'non-interactive authentication source required before Phase 1 (explicit token env or trusted GitHub CLI fallback)', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker')) { $mechanisms.Add($mechanism) } if ($null -ne $run.Interaction -and $continuationCapability.Available) { $mechanisms.Add(("explicit Copilot {0} continuation selected from installed help" -f $continuationCapability.Flag)) $mechanisms.Add('no implicit last-session continuation') @@ -668,6 +733,13 @@ function Get-CopilotPreflight { exact_session_required = $true implicit_continuation = $false } + authentication = [ordered]@{ + source = [string]$authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady + github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved + github_cli_config_candidate_used = -not [string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory) + token_value_observed = $false + } } return $document } @@ -692,7 +764,6 @@ function New-CopilotEnvironment { } # Model-free help/version bootstrap must use the same cache roots too. if ($WithoutAuthentication) { return New-RunnerEnvironment -Run $Inputs.Run -Additional $additional } - $tokenVariable = Get-CopilotTokenVariable $authState = Resolve-CopilotAuthentication if ($authState.Source -eq 'github_cli_token') { $additional[$authState.TokenVariable] = $authState.TokenValue @@ -1091,6 +1162,7 @@ function Invoke-CopilotScriptedExecute { $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ source = $authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') @@ -1286,13 +1358,14 @@ function Invoke-CopilotExecute { $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'non-interactive authentication source required before Phase 1 (explicit token env or trusted GitHub CLI fallback)', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ source = $authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index 9c1f6e9..90d205b 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -24,7 +24,7 @@ foreach ($file in @('github-copilot/runner.ps1', 'bridge-execution-result.ps1', } } $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') -function Resolve-CopilotAuthentication { return [pscustomobject]@{ Source = 'fixture'; TokenVariable = $null; TokenValue = $null; GitHubCliTokenResolved = $false } } +function Resolve-CopilotAuthentication { return [pscustomobject]@{ Source = 'fixture'; TokenVariable = $null; TokenValue = $null; GitHubCliTokenResolved = $false; GitHubCliConfigDirectory = $null; NonInteractiveReady = $true } } function Resolve-SandboxCommand { param($Name) return $null } function Resolve-ExternalCommand { param($Name) if ($Name -ne 'copilot') { throw "Unexpected executable lookup: $Name" }; return $fakeCommand } function Get-CopilotPreflight { diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 new file mode 100644 index 0000000..84c51f1 --- /dev/null +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -0,0 +1,162 @@ +<# Deterministic request-to-handoff coverage: real preparation, fixture catalogs, fake host. No models. #> +[CmdletBinding()] +param() +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +$scripts = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +. (Join-Path $scripts 'eval-request.ps1') +$workspace = Join-Path ([IO.Path]::GetTempPath()) ('eval-request-workspace/' + [guid]::NewGuid().ToString('N')) +[void](New-Item -ItemType Directory -Path $workspace -Force) +$script:dispatches = [Collections.Generic.List[string]]::new() + +function Assert-True([bool]$Condition, [string]$Message) { + if (-not $Condition) { throw $Message } +} +function Invoke-FakeHost($Decision) { + if ($Decision.action -eq 'external_handoff') { + # The host's only input is the actual canonical handoff file, never an arm prompt. + Assert-True (Test-Path -LiteralPath $Decision.prompt_path -PathType Leaf) 'Host received a missing handoff.' + Assert-True ([IO.Path]::GetFileName($Decision.prompt_path) -ceq 'RUN-THIS.prompt.md') 'Host received an arm.' + $script:dispatches.Add($Decision.prompt_path) + } +} +function New-Preparation([string]$Runner, [string]$Name) { + return @{ Skill = 'dotnet-strong-name-signing'; Eval = @(1); Runner = $Runner + OutputRoot = (Join-Path $workspace $Name); ModelCatalogPath = $catalog } +} +function Assert-Failure([hashtable]$Options, [string]$Pattern) { + # The request must fail closed even in an ordinary interactive PowerShell caller. + $ErrorActionPreference = 'Continue' + $before = $script:dispatches.Count + $failed = $false + try { + Invoke-EvalRequest -Preparation $Options -Yolo -CanDelegateFreshOrchestrator | + ForEach-Object { Invoke-FakeHost $_ } + } catch { + $failed = $true + Assert-True ($_.Exception.Message -match $Pattern) "Unexpected failure: $_" + } + Assert-True $failed 'Expected preparation failure.' + Assert-True ($script:dispatches.Count -eq $before) 'Failed preparation dispatched an Orchestrator.' +} + +try { + $catalog = Join-Path $workspace 'models.json' + @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'claude-opus-4.7' }, @{ id = 'provider/Exact.Model' }) } | + ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $catalog -Encoding utf8 + + $normalOptions = New-Preparation 'Codex' 'normal' + $normal = Invoke-EvalRequest -Preparation $normalOptions -CanDelegateFreshOrchestrator + Invoke-FakeHost $normal + Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' + Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' + + # Force may replace an unstarted package, but must preserve every started package. + $normalOptions.Iteration = 1 + $normalOptions.Force = $true + $replacement = Invoke-EvalRequest -Preparation $normalOptions + Assert-True ($replacement.prompt_path -eq $normal.prompt_path) 'Force could not replace an unstarted package.' + foreach ($marker in @('.external-handoff-started', 'orchestration-state.json', 'execution-freeze.json')) { + $options = New-Preparation 'Codex' ([guid]::NewGuid().ToString('N')) + $options.Iteration = 1 + $options.Force = $true + if ($marker -eq '.external-handoff-started') { + $reserved = Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator + Invoke-FakeHost $reserved + } else { + $reserved = Invoke-EvalRequest -Preparation $options + '{}' | Set-Content -LiteralPath (Join-Path (Split-Path $reserved.prompt_path) $marker) + } + $package = Split-Path $reserved.prompt_path + $beforeFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | + ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) + Assert-Failure $options 'handoff or execution has already started' + $afterFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | + ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) + Assert-True (($beforeFiles -join "`n") -ceq ($afterFiles -join "`n")) "Forced retry changed package protected by $marker." + $again = Get-EvalHandoff -PromptPath $reserved.prompt_path -Yolo -CanDelegateFreshOrchestrator + Assert-True ($again.action -eq 'already_started') "Forced retry erased $marker." + } + + foreach ($case in @( + @{ Name = 'codex-default'; Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, + @{ Name = 'copilot-default'; Runner = 'Copilot'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5'; StrongerModel = 'claude-opus-4.7'; UseLegacyAvailabilityAlias = $true }, + @{ Name = 'copilot-explicit-model'; Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'gpt-5.6-luna'; Explicit = $true }, + @{ Name = 'opencode-explicit-model'; Runner = 'OpenCode'; Expected = 'opencode'; Model = 'provider/Exact.Model'; Explicit = $true } + )) { + $options = New-Preparation $case.Runner ([guid]::NewGuid().ToString('N')) + if ($case.ContainsKey('Explicit')) { $options.Model = $case.Model } + $decision = if ($case.ContainsKey('UseLegacyAvailabilityAlias')) { + Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + } else { + Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator + } + $profile = Get-Content -LiteralPath (Join-Path (Split-Path $decision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json + Assert-True ($profile.runner -ceq $case.Expected -and $profile.model -ceq $case.Model) 'Wrong runner/model policy.' + if ($case.ContainsKey('StrongerModel')) { + Assert-True ($profile.model -cne $case.StrongerModel) 'Copilot replaced repository model policy with a subjective stronger-model choice.' + } + if ($case.Expected -eq 'codex') { + Assert-True ($profile.reasoning_effort -eq 'low') 'Codex default reasoning changed.' + } + Assert-True ($decision.action -eq 'external_handoff') 'Yolo did not request external handoff.' + $before = $script:dispatches.Count + Invoke-FakeHost $decision + Assert-True ($script:dispatches[$before] -ceq $decision.prompt_path) 'Host did not receive the exact generated RUN-THIS.prompt.md path.' + $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -CanDelegateFreshOrchestrator + Invoke-FakeHost $again + Assert-True ($again.action -eq 'already_started' -and $script:dispatches.Count -eq $before + 1) 'Duplicate handoff could invoke Phase 1 twice.' + $unavailableAfterStart = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo + Assert-True ($unavailableAfterStart.action -eq 'already_started') 'An uncertain launch incorrectly suggested a new manual execution.' + } + + $reference = New-Preparation 'unused' 'reference' + $reference.Remove('Runner') + $reference.CodebeltReference = $true + $reference.ReasoningEffort = 'high' + $reference.ConfigurationProfile = 'fixture-profile' + $reference.ToolProfile = 'fixture-tools' + $reference.TimeoutSeconds = 123 + $reference.Concurrency = 3 + $referenceDecision = Invoke-EvalRequest -Preparation $reference -Yolo -CanDelegateFreshOrchestrator + $profile = Get-Content (Join-Path (Split-Path $referenceDecision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json + Assert-True ($profile.runner -eq 'github-copilot' -and $profile.model -eq 'claude-haiku-4.5') 'CodebeltReference changed.' + Assert-True ($profile.reasoning_effort -eq 'high' -and $profile.configuration_profile -eq 'fixture-profile' -and $profile.tool_profile -eq 'fixture-tools' -and $profile.timeout_seconds -eq 123 -and $profile.concurrency -eq 3) 'Preparation options were not forwarded unchanged.' + + $unavailable = Invoke-EvalRequest -Preparation (New-Preparation 'Copilot' 'unavailable') -Yolo + $before = $script:dispatches.Count + Invoke-FakeHost $unavailable + Assert-True ($unavailable.action -eq 'manual_handoff' -and $script:dispatches.Count -eq $before) 'Unavailable host executed a fallback.' + Assert-True (Test-Path -LiteralPath $unavailable.prompt_path) 'Unavailable host lost the package.' + # Existing execution state also blocks automatic handoff, even without a handoff receipt. + '{}' | Set-Content (Join-Path (Split-Path $unavailable.prompt_path) 'orchestration-state.json') + $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -CanDelegateFreshOrchestrator + Assert-True ($started.action -eq 'already_started') 'Existing Phase 1 could be invoked twice.' + + Assert-Failure (New-Preparation 'OpenCode' 'missing-model') 'explicit -Model' + $invalid = New-Preparation 'GitHub Copilot' 'invalid-model' + $invalid.Model = 'not-in-catalog' + Assert-Failure $invalid "Runner 'github-copilot' model 'not-in-catalog' could not be verified" + $invalid = New-Preparation 'Codex' 'invalid-default' + $emptyCatalog = Join-Path $workspace 'empty-models.json' + '{"models":[{"id":"unrelated-model"}]}' | Set-Content $emptyCatalog + $invalid.ModelCatalogPath = $emptyCatalog + Assert-Failure $invalid "Runner 'codex' model 'gpt-5.6-luna' could not be verified" + $invalid = New-Preparation 'Codex' 'missing-skill' + $invalid.Skill = 'no-such-skill' + Assert-Failure $invalid 'skill|directory|path' + + # Check the real discovery call contract, not a mocked model resolver. + $source = Get-Content (Join-Path $scripts 'prepare-skill-evals.ps1') -Raw + Assert-True ($source.Contains("`$arguments = @('-Runner', `$RunnerName, '-RequireModel', `$ModelName)")) 'Model discovery must explicitly receive normalized runner and exact model.' + $helper = Get-Content (Join-Path $scripts 'eval-request.ps1') -Raw + Assert-True ($helper.Contains("[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator")) 'External orchestrator capability alias changed unexpectedly.' + Assert-True (-not $helper.Contains('claude-haiku-4.5') -and -not $helper.Contains('gpt-5.6-luna') -and -not $helper.Contains('claude-opus-4.7')) 'Eval request helper must not embed model-selection policy.' + Assert-True ($helper -notmatch 'invoke-runner-owned-arms|runner.ps1 execute|Start-Process|spawn_agent') 'Request helper must not implement execution.' + Write-Host 'PASS: normal/yolo requests, Copilot delegation capability semantics, runner/model policy, failures, canonical handoff pathing, and duplicate dispatch guards (fake host only).' +} finally { + # The absolute target is the unique child allocated under the external test workspace above. + $allowed = [IO.Path]::GetFullPath((Join-Path ([IO.Path]::GetTempPath()) 'eval-request-workspace')) + [IO.Path]::DirectorySeparatorChar + if (-not [IO.Path]::GetFullPath($workspace).StartsWith($allowed, [StringComparison]::OrdinalIgnoreCase)) { throw 'Unsafe test cleanup path.' } + Remove-Item -LiteralPath $workspace -Recurse -Force +} diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 7eaa252..019a416 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -125,6 +125,11 @@ $recordedOldGhToken = $env:GH_TOKEN $recordedOldGithubToken = $env:GITHUB_TOKEN $recordedOldCopilotHome = $env:COPILOT_HOME $recordedOldGhConfigDir = $env:GH_CONFIG_DIR +$recordedOldHome = $env:HOME +$recordedOldUserProfile = $env:USERPROFILE +$recordedOldAppData = $env:APPDATA +$recordedOldLocalAppData = $env:LOCALAPPDATA +$recordedOldXdgConfigHome = $env:XDG_CONFIG_HOME $recordedOldFixtures = $env:AGENTIC_RECORDED_FIXTURES try { $fakeBin = Join-Path $recordedRoot 'bin' @@ -1970,58 +1975,134 @@ exit 2 $env:CODEX_HOME = $recordedOldCodexHome } if ($Suite -in @('All', 'Copilot')) { - # GitHub Copilot authentication: explicit env, OS-keychain, GitHub CLI, and - # no-auth fixtures are all deterministic and contain no credential values. + # GitHub Copilot authentication continuity and failure semantics: + # deterministic fixtures only, no live credentials. $env:COPILOT_GITHUB_TOKEN = $null $env:GH_TOKEN = $null $env:GITHUB_TOKEN = $null $missingGhConfig = Join-Path $recordedRoot 'missing-github-cli-auth' + New-Item -ItemType Directory -Path $missingGhConfig -Force | Out-Null + $copilotLogPath = Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl' - # The fixture marker is fake-CLI input only; it models a positive OS - # keychain lookup without naming or reading a real credential-store file. - $copilotKeychainHome = Join-Path $recordedRoot 'copilot-keychain-home' - New-Item -ItemType Directory -Path $copilotKeychainHome -Force | Out-Null + # Fixture marker only: models a keychain-only host with no resolvable + # non-interactive token source. + $copilotKeychainHost = Join-Path $recordedRoot 'copilot-keychain-host' + New-Item -ItemType Directory -Path $copilotKeychainHost -Force | Out-Null New-Item -ItemType Directory -Path (Join-Path $with.Root 'home\.copilot') -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available'), 'fixture marker only', [Text.UTF8Encoding]::new($false)) - $env:COPILOT_HOME = $copilotKeychainHome + $env:COPILOT_HOME = $copilotKeychainHost $env:GH_CONFIG_DIR = $missingGhConfig + $nativeExecutionsBeforeKeychain = @() + if (Test-Path -LiteralPath $copilotLogPath -PathType Leaf) { + $nativeExecutionsBeforeKeychain = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + } $copilotKeychainPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'compatible' $copilotKeychainPreflight.status 'Copilot tokenless OS-keychain authentication remains compatible' - Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'unavailable' }).Count -eq 1) 'Copilot preflight leaves native keychain readiness conditional' - Assert-True (@($copilotKeychainPreflight.warnings | Where-Object { $_ -match 'cannot be proven' }).Count -gt 0) 'Copilot preflight explains the unverified keychain/service boundary' + Assert-Equal 'incompatible' $copilotKeychainPreflight.status 'Copilot keychain-only auth is rejected before Phase 1 execution' + Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot keychain-only preflight fails authentication deterministically' + Assert-Equal 'copilot_os_keychain_unverified' $copilotKeychainPreflight.protocol_observations.authentication.source 'Copilot keychain-only source is diagnostic only' + Assert-True (-not [bool]$copilotKeychainPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot keychain-only source is not accepted as non-interactive readiness' + Assert-True (@($copilotKeychainPreflight.reasons | Where-Object { $_ -match 'supported non-interactive Copilot authentication source' }).Count -eq 1) 'Copilot keychain-only preflight reports the missing non-interactive source' $copilotKeychainResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'completed' $copilotKeychainResult.status 'Copilot keychain fixture executes without an exported token' - $keychainRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'copilot_authentication_source' -Default '') -eq 'os_keychain' }) - Assert-Equal 1 $keychainRecords.Count 'Copilot fake observes the simulated OS-keychain path' + Assert-Equal 'incompatible' $copilotKeychainResult.status 'Copilot execute fails closed when preflight cannot resolve non-interactive auth' + Assert-Equal 'incompatible' ([string]$copilotKeychainResult.evidence.preflight.status) 'Copilot execute returns preflight evidence for keychain-only rejection' + $nativeExecutionsAfterKeychain = @() + if (Test-Path -LiteralPath $copilotLogPath -PathType Leaf) { + $nativeExecutionsAfterKeychain = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + } + Assert-Equal $nativeExecutionsBeforeKeychain.Count $nativeExecutionsAfterKeychain.Count 'Copilot keychain-only rejection occurs before model execution' Remove-Item -LiteralPath (Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available') -Force + # Fresh-context continuity regression: the first GH config candidate exists + # but is not authenticated; fallback resolves a second supported host + # candidate without exposing auth files or token values to the run. + $copilotFreshBoundaryRoot = Join-Path $recordedRoot 'copilot-fresh-boundary' + $copilotFreshHome = Join-Path $copilotFreshBoundaryRoot 'home' + $copilotFreshXdg = Join-Path $copilotFreshBoundaryRoot 'xdg' + $copilotFreshAppData = Join-Path $copilotFreshBoundaryRoot 'appdata' + $copilotFreshLocalAppData = Join-Path $copilotFreshBoundaryRoot 'localappdata' + $copilotFreshHostAppData = Join-Path $copilotFreshBoundaryRoot 'host-appdata' + $copilotFreshHostGhConfig = Join-Path $copilotFreshHostAppData 'GitHub CLI' + New-Item -ItemType Directory -Path $copilotFreshHome, $copilotFreshXdg, $copilotFreshAppData, $copilotFreshLocalAppData, (Join-Path $copilotFreshXdg 'gh'), $copilotFreshHostGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotFreshHostGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $copilotFreshHome + $env:USERPROFILE = $copilotFreshHome + $env:APPDATA = $copilotFreshHostAppData + $env:LOCALAPPDATA = $copilotFreshLocalAppData + $env:XDG_CONFIG_HOME = $copilotFreshXdg + $env:COPILOT_HOME = (Join-Path $copilotFreshBoundaryRoot 'copilot-home') + New-Item -ItemType Directory -Path $env:COPILOT_HOME -Force | Out-Null + $env:GH_CONFIG_DIR = $null + $copilotFreshBoundaryPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotFreshBoundaryPreflight.status 'Copilot preflight resolves non-interactive auth across fresh-context candidate fallback' + Assert-True (@($copilotFreshBoundaryPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot fresh-context preflight proves authentication readiness' + Assert-Equal 'github_cli_token' $copilotFreshBoundaryPreflight.protocol_observations.authentication.source 'Copilot fresh-context source resolves through GitHub CLI fallback' + Assert-True ([bool]$copilotFreshBoundaryPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot fresh-context source is accepted as non-interactive readiness' + Assert-True ([bool]$copilotFreshBoundaryPreflight.protocol_observations.authentication.github_cli_config_candidate_used) 'Copilot fresh-context preflight records GH config candidate fallback usage' + $copilotFreshBoundaryResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotFreshBoundaryResult.status 'Copilot fresh-context fallback executes after preflight authentication continuity' + Assert-Equal 'github_cli_token' $copilotFreshBoundaryResult.evidence.credential.source 'Copilot fresh-context execution records GitHub CLI fallback source' + Assert-True ([bool]$copilotFreshBoundaryResult.evidence.credential.noninteractive_ready) 'Copilot fresh-context execution records non-interactive readiness' + Assert-True ([bool]$copilotFreshBoundaryResult.evidence.credential.github_cli_token_resolved) 'Copilot fresh-context execution resolves a trusted GitHub CLI token' + $freshBoundaryExecutionRecords = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + $freshBoundaryExecution = $freshBoundaryExecutionRecords[$freshBoundaryExecutionRecords.Count - 1] + Assert-Equal 'explicit_environment' $freshBoundaryExecution.copilot_authentication_source 'Copilot fresh-context worker receives only protected env-token auth' + Assert-Equal 1 @($freshBoundaryExecution.copilot_auth_names_present).Count 'Copilot fresh-context worker receives one token variable' + Assert-True (@($freshBoundaryExecution.copilot_auth_names_present) -contains 'GH_TOKEN') 'Copilot fresh-context worker receives only GH_TOKEN from trusted fallback' + Assert-True ([string]::IsNullOrWhiteSpace([string]$freshBoundaryExecution.gh_config_dir)) 'Copilot fresh-context worker does not receive GH_CONFIG_DIR' + $copilotGhFallbackHome = Join-Path $recordedRoot 'copilot-gh-fallback-home' New-Item -ItemType Directory -Path $copilotGhFallbackHome -Force | Out-Null $copilotGhConfig = Join-Path $recordedRoot 'copilot-gh-config' New-Item -ItemType Directory -Path $copilotGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $copilotGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:COPILOT_HOME = $copilotGhFallbackHome $env:GH_CONFIG_DIR = $copilotGhConfig + $ghFallbackMatchesBefore = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }).Count $copilotGhPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'compatible' $copilotGhPreflight.status 'Copilot GitHub CLI fallback remains compatible' + Assert-Equal 'github_cli_token' $copilotGhPreflight.protocol_observations.authentication.source 'Copilot preflight classifies explicit GH config fallback source' + Assert-True ([bool]$copilotGhPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot preflight classifies GH config fallback as non-interactive readiness' $copilotGhResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'completed' $copilotGhResult.status 'Copilot GitHub CLI fallback fixture executes without an exported token' + Assert-True ([bool]$copilotGhResult.evidence.credential.noninteractive_ready) 'Copilot GitHub CLI fallback result records non-interactive readiness' Assert-True $copilotGhResult.evidence.credential.github_cli_token_resolved 'Copilot records GitHub CLI token fallback without storing the token value' Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) - Assert-Equal 1 $ghRecords.Count 'Copilot fake observes only the protected GH_TOKEN produced by trusted GitHub CLI fallback' + Assert-Equal ($ghFallbackMatchesBefore + 1) $ghRecords.Count 'Copilot fake observes one additional protected GH_TOKEN execution produced by trusted GitHub CLI fallback' $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null + $copilotNoAuthRoot = Join-Path $recordedRoot 'copilot-no-auth-roots' + New-Item -ItemType Directory -Path (Join-Path $copilotNoAuthRoot 'home'), (Join-Path $copilotNoAuthRoot 'appdata'), (Join-Path $copilotNoAuthRoot 'localappdata'), (Join-Path $copilotNoAuthRoot 'xdg') -Force | Out-Null + $env:HOME = Join-Path $copilotNoAuthRoot 'home' + $env:USERPROFILE = Join-Path $copilotNoAuthRoot 'home' + $env:APPDATA = Join-Path $copilotNoAuthRoot 'appdata' + $env:LOCALAPPDATA = Join-Path $copilotNoAuthRoot 'localappdata' + $env:XDG_CONFIG_HOME = Join-Path $copilotNoAuthRoot 'xdg' + $nativeExecutionsBeforeNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) $env:COPILOT_HOME = $copilotNoAuthHome $env:GH_CONFIG_DIR = $missingGhConfig $copilotNoAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'compatible' $copilotNoAuthPreflight.status 'Copilot preflight does not require an exported token when native auth is not observable' - Assert-True (@($copilotNoAuthPreflight.warnings | Where-Object { $_ -match 'conditional' }).Count -gt 0) 'Copilot no-auth preflight is explicitly conditional' + Assert-Equal 'incompatible' $copilotNoAuthPreflight.status 'Copilot preflight fails closed when no supported non-interactive auth source is available' + Assert-True (@($copilotNoAuthPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot no-auth preflight records an authentication failure check' + Assert-Equal 'copilot_os_keychain_unverified' $copilotNoAuthPreflight.protocol_observations.authentication.source 'Copilot no-auth preflight reports diagnostic auth source identity' + Assert-True (-not [bool]$copilotNoAuthPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot no-auth preflight reports non-interactive auth readiness as false' $copilotNoAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'failed' $copilotNoAuthResult.status 'Copilot no-auth execution failure is captured without a model request' - Assert-Equal 'copilot_os_keychain_or_github_cli_unverified' $copilotNoAuthResult.evidence.credential.source 'Copilot no-auth evidence does not claim authentication' + Assert-Equal 'incompatible' $copilotNoAuthResult.status 'Copilot no-auth execution fails before Phase 1 model execution' + Assert-Equal 'incompatible' ([string]$copilotNoAuthResult.evidence.preflight.status) 'Copilot no-auth execute result preserves the incompatible preflight evidence' + $nativeExecutionsAfterNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal $nativeExecutionsBeforeNoAuth.Count $nativeExecutionsAfterNoAuth.Count 'Copilot no-auth rejection does not start a model execution process' Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:COPILOT_HOME = $recordedOldCopilotHome } Write-Output "Real runner deterministic adapter conformance ($Suite): PASS" @@ -2036,6 +2117,11 @@ exit 2 $env:GITHUB_TOKEN = $recordedOldGithubToken $env:COPILOT_HOME = $recordedOldCopilotHome $env:GH_CONFIG_DIR = $recordedOldGhConfigDir + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:AGENTIC_RECORDED_FIXTURES = $recordedOldFixtures if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } @@ -2401,6 +2487,7 @@ try { Assert-True ($prepareText.Contains('execution-freeze.json') -and $prepareText.Contains('grading.json') -and $prepareText.Contains('validate-eval-grading.ps1') -and $prepareText.Contains('apply-eval-grading.ps1') -and $prepareText.Contains('finalize-eval-package.ps1')) 'handoff preparation must expose the shared freeze, grading validation, grading application, and finalization boundaries' Assert-True ($prepareText.Contains('Read the selected runner descriptor and its `delegation.dispatch_owner`.') -and $prepareText.Contains('invoke-runner-owned-arms.ps1') -and $prepareText.Contains('package-computed Phase 1 allowance') -and $prepareText.Contains('must be started exactly once')) 'handoff preparation must expose one foreground Phase 1 invocation with a computed caller timeout' Assert-True ($prepareText.Contains('Do not create outer workers') -and $prepareText.Contains('edit raw result/evidence files')) 'handoff preparation must forbid outer runner-owned workers and raw evidence edits' + Assert-True ($prepareText.Contains('authentication incompatibility is terminal for this package iteration') -and $prepareText.Contains('Do not suggest switching to another runner') -and $prepareText.Contains('do not suggest starting another Orchestrator') -and $prepareText.Contains('do not rerun Phase 1')) 'handoff preparation must fail closed on Copilot authentication incompatibility without alternate-runner/orchestrator suggestions' Assert-True ($prepareText.Contains('The Grader may author exactly one package-root `grading.json`') -and $prepareText.Contains('It must not edit raw execution results')) 'handoff preparation must isolate the Grader to the grading-only artifact' Assert-True ($prepareText.Contains('Write `grading.json`, then validate it before finalization') -and $prepareText.Contains('Grading validation is retryable; finalization is not') -and $prepareText.Contains('only after grading validation succeeds')) 'handoff preparation must place retryable grading validation before exactly-once finalization' Assert-True ($prepareText.Contains('Return only its machine-readable JSON summary') -and $prepareText.Contains('Never repair, re-freeze, re-bridge a changed raw result')) 'handoff preparation must make finalizer success and fail-closed recovery explicit' @@ -2409,6 +2496,7 @@ try { $generatedHandoff = Invoke-GeneratedRunnerPrompt $generatedConcurrencyThreeHandoff = Invoke-GeneratedRunnerPrompt -RequestedConcurrency 3 Assert-True ($generatedHandoff.Contains('evaluation is incomplete and a fresh package/code fix is required') -and $generatedHandoff.Contains('Never patch package-local runner code') -and $generatedHandoff.Contains('delete orchestration state') -and $generatedHandoff.Contains('delete execution results') -and $generatedHandoff.Contains('delete or replace `execution-freeze.json`') -and $generatedHandoff.Contains('rerun Phase 1') -and $generatedHandoff.Contains('manually broaden a capability check')) 'generated handoff output forbids package-local repair, state deletion, retry, and manual capability broadening' + Assert-True ($generatedHandoff.Contains('authentication incompatibility is terminal for this package iteration') -and $generatedHandoff.Contains('Do not suggest switching to another runner') -and $generatedHandoff.Contains('do not suggest starting another Orchestrator') -and $generatedHandoff.Contains('do not rerun Phase 1')) 'generated handoff output must keep Copilot authentication failures fail-closed without fallback orchestration suggestions' Assert-True ($generatedHandoff.Contains('invoke-runner-owned-arms.ps1') -and $generatedHandoff.Contains('package-computed Phase 1 allowance') -and $generatedHandoff.Contains('must be started exactly once') -and $generatedHandoff.Contains('If execution is interrupted and no valid `execution-freeze.json` exists')) 'generated handoff exposes one foreground Phase 1 invocation with fail-closed interruption handling' Assert-True ($generatedHandoff.Contains('validate-eval-grading.ps1') -and $generatedHandoff.Contains('-ShowSkeleton') -and $generatedHandoff.Contains('Write `grading.json`, then validate it before finalization') -and $generatedHandoff.Contains('Grading validation is retryable; finalization is not') -and $generatedHandoff.Contains('only after grading validation succeeds')) 'generated handoff requires author -> validate/retry -> finalize exactly once' Assert-True ($generatedHandoff.Contains('allowance of 6240 seconds') -and $generatedHandoff.Contains('6 arm(s) × 120-second fixed model-free runner preflight timeout = 720-second serial preflight allowance') -and $generatedHandoff.Contains('5490-second execution allowance across 3 batch(es) at concurrency 2') -and $generatedHandoff.Contains('2 scripted user turn(s) × profile.timeout_seconds 900 + 30 seconds runner grace') -and $generatedHandoff.Contains('+ 30 seconds orchestration grace')) 'generated handoff uses the fixed preflight timeout, scripted-turn model timeout, execution batches, and separate orchestration grace' diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 358e697..c160999 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -44,6 +44,9 @@ .PARAMETER Force Overwrite an existing iteration directory. +.PARAMETER PassThru + Also return each successfully prepared absolute RUN-THIS.prompt.md path on the success stream. + .PARAMETER Runner Package-local Eval Runner id written to execution-profile.json when -CodebeltReference is not used. GitHub Copilot and Codex can resolve repository-defined default models; OpenCode requires an explicit model selector. @@ -129,6 +132,10 @@ param( [Parameter(ParameterSetName = 'Changed')] [switch]$Force, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [switch]$PassThru, + [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [string]$Runner, @@ -482,6 +489,11 @@ function Resolve-ExecutionSelection { if (-not $hasRunner -and $hasModel) { throw 'Runner/model selection requires -Runner when -Model is supplied.' } + # Normalize the user-facing names before discovery; model policy remains runner-owned here. + $Runner = switch ($Runner.Trim().ToLowerInvariant()) { + { $_ -in @('github copilot', 'github copilot cli', 'copilot') } { 'github-copilot' } + default { $_ } + } $resolvedRunnerMatch = @($supportedRunners | Where-Object { [string]::Equals($_, $Runner, [StringComparison]::OrdinalIgnoreCase) }) if ($resolvedRunnerMatch.Count -ne 1) { throw "Unsupported runner '$Runner'. Supported runner IDs: $supportedText." @@ -1453,6 +1465,12 @@ function Invoke-PrepareMode { if (-not $Force) { throw "'$iterationDirectory' already exists. Pass -Force to replace it, or -Iteration to write a new one." } + # Force must not erase a one-shot reservation or evidence of an active/completed run. + foreach ($marker in @('.external-handoff-started', 'orchestration-state.json', 'execution-freeze.json')) { + if (Test-Path -LiteralPath (Join-Path $iterationDirectory $marker)) { + throw "Cannot replace '$iterationDirectory': handoff or execution has already started ($marker). Preserve this package and use a fresh iteration." + } + } Remove-Item -LiteralPath $iterationDirectory -Recurse -Force } New-Item -ItemType Directory -Path $iterationDirectory -Force | Out-Null @@ -1781,6 +1799,7 @@ function Invoke-PrepareMode { Write-Host 'The selected evaluator must write valid runner-produced execution results back into this package.' Write-Host 'If it cannot, the evaluation is incomplete and must fail closed; only persisted runner-produced evidence at' Write-Host 'the manifest-declared paths may proceed.' + if ($PassThru) { Write-Output $runnerPath } } function New-RunnerPrompt { @@ -1841,6 +1860,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('Read the selected runner descriptor and its `delegation.dispatch_owner`. For runner-owned behavioral transport, invoke:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$runnerOwnedFanoutPath`" -IterationDirectory `"$IterationDirectory`"") [void]$builder.AppendLine('It performs every preflight before any execute process, preserves exact manifest paths, owns concurrency/backpressure, timeout/watchdog handling, terminal registration, orchestration evidence, and immutable `execution-freeze.json` before Phase 2. Consume its terminal JSON summary. If Phase 1 reports incompatible or fails, stop: the evaluation is incomplete and must fail closed. The evaluation is incomplete and a fresh package/code fix is required. Never patch package-local runner code, delete orchestration state, delete execution results, delete or replace `execution-freeze.json`, rerun Phase 1, or manually broaden a capability check. Do not create outer workers, execute an arm yourself, write orchestration state, or edit raw result/evidence files. If dispatch ownership is orchestrator-owned, use only the descriptor-declared native worker transport, the exact manifest paths, and then run the shared freeze boundary; do not synthesize or repair transport evidence. Only persisted runner-produced evidence at the manifest-declared paths may proceed.') + [void]$builder.AppendLine('For `github-copilot`, authentication incompatibility is terminal for this package iteration: report the missing supported non-interactive source (explicit `COPILOT_GITHUB_TOKEN`/`GH_TOKEN`/`GITHUB_TOKEN` or trusted `gh auth token` fallback), preserve the package, and stop. Do not suggest switching to another runner, do not suggest starting another Orchestrator, and do not rerun Phase 1.') [void]$builder.AppendLine('Workers receive only their isolated run directory. Keep the paired arm, metadata, expected output, assertions, grading, reports, and orchestration files out of Phase 1. Preserve runner-owned terminal results and all referenced raw transcript/event artifacts exactly as written.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 2 — grading and finalization') diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 6858d9c..2119cc4 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1430,11 +1430,18 @@ Add-ValidationResult -Results $results -Name 'Progress coalescing renders select } } +Add-ValidationResult -Results $results -Name 'Explicit eval requests preserve preparation and reserve one external handoff' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-eval-request.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Eval request regressions failed: $($output -join [Environment]::NewLine)" } +} + Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable prompts instead of executing them' -Group 'Preparation' -Action { $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref $contributing = Get-FileText -RepoRoot $repoRoot -RelativePath 'CONTRIBUTING.md' -GitRef $Ref $runnerReadme = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/eval-runners/README.md' -GitRef $Ref + $evalRequest = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/eval-request.ps1' -GitRef $Ref $prepare = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/prepare-skill-evals.ps1' -GitRef $Ref Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '## Portable Eval Handoff' @@ -1447,6 +1454,9 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '-CanDelegateFreshOrchestrator' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'OpenCode discovery mirrors every model exposed by all configured OpenCode providers' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'availability is presentation metadata only and never filters the selectable catalog' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made.' @@ -1477,6 +1487,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'a completion gate an agent cannot skip' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed' Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'Discovery is an availability check, not model ranking' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'choose a Harness + Model when the user did not already do so' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'OpenCode mirrors every model exposed by all configured providers' @@ -1486,6 +1498,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'the user must choose from the discovered selectors before package preparation' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'OpenCode through `opencode models --verbose`' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'without filtering the selectable catalog' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle "[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator" + Assert-NotContains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'claude-opus-4.7' Assert-Contains -Name 'README.md' -Content $readme -Needle 'full operational permission inside each isolated behavioral harness configuration' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Hard filesystem confinement is a separate optional outer capability' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'full harness operational permission inside each isolated eval boundary'