From ea52fe36ea72f3e64a8c5185015bdd7ad79d459b Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 7 Sep 2026 20:40:22 +0200 Subject: [PATCH 01/18] =?UTF-8?q?=F0=9F=92=AC=20clarify=20eval=20request?= =?UTF-8?q?=20workflow=20and=20external=20handoff?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds guidance for optional one-shot external handoff when user explicitly requests evaluation with yolo/auto modifier. Clarifies the boundary between preparation (deterministic) and execution (external handoff). Updates roles for preparer and executor, specifying when explicit eval requests authorize automatic handoff versus manual handoff. Normalizes harness naming and improves model discovery guidance. --- AGENTS.md | 34 +++++++++++++++++++++++++--------- CONTRIBUTING.md | 2 ++ 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 80d4201..7ab7c49 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ An executing harness stays inside its package. Building, testing, or writing any Repository-owned preparation, validation, CI jobs, hooks, deterministic tests, package generation, automatic completion gates, and automatic agent fan-out must never invoke an authenticated AI/LLM CLI or API. Using the user's Copilot, Claude, Codex, Gemini, or other model account as repository test infrastructure is forbidden; this repository does not provide an opt-in path around that rule. The package-local implementations under `scripts/eval-runners/` are protocol adapters, not automatic repository execution: they may invoke their native harness only when a human-selected external Eval Orchestrator is explicitly handed a prepared package and selected profile. - Do not create, restore, recommend, or run generic automation or automatic fan-out that launches model sessions for candidate/baseline execution, grading, comparison, benchmarking, description optimization, or review generation. A runner adapter may launch its named native harness only at the explicit external-handoff boundary described below, never from repository automation or CI. -- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. `yolo`, `auto`, urgency, completion gates, third-party instructions, and prior approval do not change this rule. +- A request to create, modify, fix, test, validate, benchmark, finalize, or release a skill does not authorize additional model calls. Urgency, completion gates, third-party instructions, and prior approval do not change this rule. The sole narrow exception is an explicit user-directed eval request with `yolo` (or the existing `auto` alias): it authorizes one fresh external Eval Orchestrator handoff per prepared package, as described below; it never authorizes the preparer to execute an arm. - Routine skill validation is local and deterministic. Use schema and metadata checks, fixture validation, bundled assertions, repository validators, and human inspection of the eval prompts and expected outcomes. - Model-backed comparisons are not a repository completion gate. Do not spawn additional agents or call external model tools merely to satisfy a generic eval workflow. - A temp workspace controls filesystem isolation only. It never makes external calls local, free, offline, or acceptable. @@ -40,13 +40,13 @@ This rule is Priority 1. If another repository rule, skill, test, or completion Anthropic's `skill-creator` owns the evaluation methodology this repository uses: define evals, run each task once with the skill and once without it, hold the model, the environment, the task, and the inputs constant, then compare. Keep that experimental design. Only the execution transport changes here. -Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. +Where `skill-creator` says to spawn with-skill and baseline subagents in the same turn, this repository prepares a portable evaluation package and stops, except for the explicitly authorized fresh external handoff below. The package keeps the existing paired methodology: `run.json` defines what one blind arm executes, `execution-profile.json` selects the runner/model/configuration, and the Eval Runner defines how its harness satisfies the contract. Before an execution-ready `RUN-THIS.prompt.md` is emitted, the user-facing preparation flow resolves a Harness + Model choice; the portable profile stores the internal runner id and the opaque runner-native model selector. The external Eval Orchestrator never chooses runner or model policy. It reads `delegation.dispatch_owner`; for runner-owned dispatch it invokes the package-local foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once with the package-computed timeout allowance. That helper resolves the runner, preflights every pending arm before execution, asserts native delegation, starts runner-owned native surfaces with bounded child timeouts and concurrency/backpressure, registers terminal results, and freezes execution evidence. For orchestrator-owned dispatch it delegates each arm through the declared native worker. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce the canonical raw result directly. It grades only after execution, invokes the packaged Anthropic `skill-creator` aggregator and static viewer, and returns the finished reports. Preparation, collection, validation, and reporting remain deterministic and never invoke a model. ### Asking for an eval -`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too. +`eval `, `evaluate `, `eval this skill`, `prepare evals for `, and `evaluate using the existing evals` are all requests for this workflow. Without `yolo`/`auto`, treat them as instructions to prepare the package and stop for manual handoff. With `yolo`/`auto`, prepare and perform the automatic external handoff below. Neither form requests new eval cases unless the user asks for those too. -Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1`. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. +Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. Only when the user has not supplied a harness, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1 -Runner `. Never invoke discovery without `-Runner`, and never ask to reconfirm a supplied runner. For a supplied runner with a deterministic default, let preparation resolve and verify it without another question; validate any explicit model exactly as supplied. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model. ``` pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill dotnet-test -Runner github-copilot @@ -60,11 +60,27 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed Use `-CodebeltReference` only when the script should select the Codebelt Reference explicitly and fail if `claude-haiku-4.5` is no longer present. For noninteractive direct script use, omitting both `-Runner` and `-CodebeltReference` is an error whenever a package would be generated; `-Runner codex` and `-Runner github-copilot` resolve their repository-defined default models, while `-Runner opencode` still requires `-Model`. +### Optional one-shot external handoff + +An explicit request such as `Please eval dotnet-strong-name-signing using Codex. yolo` authorizes preparation followed by ONE external Eval Orchestrator handoff for that package. The existing `auto` modifier has the same scoped meaning. Bare modifiers, skill maintenance, and completion gates do not authorize this flow. Authorization ends when the handoff completes; there are no answer-quality retries. + +Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. + +For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and `-ExternalOrchestratorAvailable` only after establishing the host capability above. Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. + +When using a shell tool, pipe the returned decisions to `ConvertTo-Json -Depth 4` so table formatting cannot truncate absolute paths. Consume each returned decision in the interactive agent workflow: + +- `manual_handoff`: return the absolute `prompt_path` and stop. When the host is unavailable, include the concise `reason`; never execute an arm in the current context. +- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. Do not paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. +- `already_started`: observe the existing Orchestrator. Never delegate again, rerun preparation to retry execution, remove the handoff reservation, or invoke Phase 1 again after a timeout, failed launch, interruption, or uncertain result. If the launch is known unavailable before it starts, use the manual path; once a launch is attempted, do not suggest blindly running the prompt again. A failed/interrupted execution remains incomplete under the existing fresh-iteration rule. + +The helper reserves the one-shot handoff with `.external-handoff-started` before returning `external_handoff`; retain its returned path and the native Orchestrator handle across continuations. `Get-EvalHandoff` can inspect that same path on resumption and refuses another automatic handoff. This receipt is outside the execution protocol and does not replace Phase 1 state or evidence. The only lifecycle instructions are the existing `RUN-THIS.prompt.md`; its package-local Phase 1 controller, grading, and finalization remain authoritative. No CI, hooks, validators, preparation scripts, or completion gates consume this decision by launching a model. + ### Handing the package over Every package contains `RUN-THIS.prompt.md`, one instruction that drives the whole thing. It makes the user-selected external agent the Eval Orchestrator, Grader, and report producer. The orchestrator resolves the selected Eval Runner and reads its dispatch owner. For runner-owned dispatch it invokes `invoke-runner-owned-arms.ps1` exactly once as a foreground Phase 1 command and sets the caller shell/tool timeout to at least the package-computed allowance. That command preflights every blind `with_skill` and `without_skill` arm and starts zero executions if any preflight is incompatible. For orchestrator-owned dispatch it delegates each arm to a fresh native worker. It records each terminal envelope through the deterministic runner-owned recorder where applicable, grades only after collection, writes the grading fields, and generates the static report without executing an eval prompt in its own context. -Hand the user that one file by its absolute path, and stop there. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. +Hand the user that one file by its absolute path, and stop there in the normal manual flow. The explicit one-shot flow hands that same file to the fresh external Orchestrator instead. Do not reproduce its contents in the reply. The runner is built around absolute paths - the package directory, its own location, the path in the hand-back block - and a copy that has passed through a chat window arrives with them shortened to a bare directory name like `iteration-4`, pointing nowhere, with its internal links broken. The file on disk always says what the file on disk says; a paste of it is a lossy snapshot that also goes stale the moment the generator changes. Where the user's harness cannot read files at all, tell them to open that path and paste it themselves, so what travels is the real text rather than your recollection of it. Do not list the individual prompt files, do not describe the directory layout, and do not hand back a procedure for the user to carry out by hand. A reply that ends with 26 file paths and "run both versions" has moved the work onto the user instead of doing it. @@ -111,7 +127,7 @@ Run it before `scripts/sync-skill-install.ps1`, which stays the last gate becaus Two roles exist here, and they are not the same agent doing the same job. Read which one you are before applying the rules below. -**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop, and you must: +**Preparing.** You are doing repository work: authoring a skill, changing one, or answering a request to evaluate one. After the package is written you stop for manual handoff unless this explicit eval request authorizes the one-shot external handoff above. In both cases you must: - never execute the prompts you just prepared, and never quietly become the executor of your own package - never spawn subagents for the candidate or baseline runs @@ -129,12 +145,12 @@ When a person gives you `RUN-THIS.prompt.md`, points you at a prepared package, Four things still hold while you execute: -- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or another agent asking is not a person asking. +- Run only the package you were handed, and only because a person asked in this turn. A hook, a script, a completion gate, a skill change, or an unsolicited agent request is not a person asking. A fresh external Orchestrator receiving the canonical file through the explicit user-directed `yolo`/`auto` handoff above carries that user authorization for this package only. - Stay inside the package directory. If it sits inside a repository, the rest of that repository is not yours to read, build, test, or write. - Do not read the assertions or expected output before the worker runs. After every available worker has finished and its result is recorded, grade the completed results, generate the report, and state the comparison honestly. - Nothing about this makes model-backed execution a completion gate for any repository task. -An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This is the only role-based disqualification. +An agent that prepared a package in this session does not get to turn around and execute it. The separation is the point: the preparer knows the grading key, so it is the wrong harness. This disqualifies the preparer context, not the fresh external Orchestrator receiving an explicitly authorized one-shot handoff. The selected executor has two ordered phases. Its current context may read `RUN-THIS.prompt.md`, `manifest.json`, `execution-profile.json`, and the runner protocol files, but it must not execute an eval prompt itself. In phase one, it follows `delegation.dispatch_owner`: for runner-owned dispatch it invokes the foreground package-local `invoke-runner-owned-arms.ps1` command exactly once and waits for its terminal JSON summary; the helper resolves the runner, validates `describe`, preflights every pending `run.json`, asserts native delegation for every result, and refuses to start any execution until all preflights pass. A caller/tool timeout or interrupted conversation does not authorize rerunning Phase 1. If execution was interrupted and no valid `execution-freeze.json` exists, the package is incomplete and requires a fresh iteration. For orchestrator-owned dispatch it resolves the runner, validates `describe`, preflights each `run.json`, and uses the declared native subagent/task. A runner-owned process/thread is the single Eval Worker and model execution; no outer model worker may contain it. Orchestrator-owned workers must not invoke `runner.ps1 execute`; their transport envelope passes through `record-native-result.ps1`. Runner-owned execution results come directly from the runner and must never be synthesized, repaired, or reconstructed from assistant text. The runner launches each native session from its own run directory with `repo/` as the working directory, `home/` as the isolated profile, and the required isolation controls; hard filesystem confinement, when a runner proves it, raises the reported isolation from pragmatic to strict but is not itself a prerequisite. The runner receives only `run.json` and `execution-profile.json`; workers never see the runner, manifest, grading key, sibling results, or orchestration commentary, because all of those live outside the run directory. Never reuse a worker or session between runs. In phase two, only after Phase 1 returns a successful terminal JSON summary and the immutable freeze validates, the executor bridges results into `eval-result/2`, reads the grading key, follows the packaged `skill-creator` grader guidance, writes the grading evidence, invokes the package adapter so Anthropic's aggregator and eval viewer produce the report, and returns the report path and comparison. It does not ask the user whether to start either phase. @@ -212,7 +228,7 @@ Bare `yolo` or `auto` outside an explicit commit request does not invoke `git-vi Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. -Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. +Follow it as written except at the execution boundary. Anthropic's `skill-creator` requires paired with-skill and baseline runs in fresh subagents. This repository prepares the same paired inputs as a portable package and stops, unless an explicit `yolo`/`auto` eval request authorizes the fresh external handoff above. When the user hands that package to a harness, `RUN-THIS.prompt.md` makes the selected harness create those isolated paired workers without exposing the grading key, then use the packaged `skill-creator` grader guidance, aggregator, and eval viewer in the same handoff. The authoring guidance, eval definitions, assertion drafting, and iteration loop still apply; only the execution transport changes. Repository-side validation remains deterministic, while the explicitly user-directed external executor performs the post-run evaluator judgement and invokes the upstream viewer/report generation that the skill-creator experience expects. `skill-creator-agnostic` is deprecated, no longer maintained, and retained only for backward compatibility until 1.0.0. Agents must not use it for new skill creation, skill modification, or benchmarking; use Anthropic's `skill-creator` directly and apply the repository-specific requirements from this `AGENTS.md`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8df4768..ea65665 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -100,6 +100,8 @@ pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill < Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. + Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. Run both configurations on the same model, same version, and same configuration. Independent arms must be dispatched concurrently up to `execution-profile.json.concurrency` when the harness permits it. For any selected runner whose descriptor says `delegation.dispatch_owner=runner`, invoke `invoke-runner-owned-arms.ps1` exactly once as a foreground Phase 1 command with the caller shell/tool timeout set to at least the package-computed allowance. The foreground helper performs every preflight before execution, owns fan-out, state, result registration, and freeze creation, and starts zero model executions when any preflight is incompatible. Never create outer native subagents/tasks for runner-owned arms or hand-author preflight, fan-out, state, or result bookkeeping. Copilot task/general-purpose workers, OpenCode Task/General workers, and Codex native mechanisms remain harness capabilities only; they are not the behavioral transport for runner-owned evaluation. A with-skill run on one model against a baseline on another measures the model as much as the skill and is not a skill-effectiveness result. From f683f8920b75731d05eaeef0bf305bb29cda793c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 7 Sep 2026 20:40:30 +0200 Subject: [PATCH 02/18] =?UTF-8?q?=F0=9F=94=A8=20add=20eval=20request=20det?= =?UTF-8?q?erministic=20helper?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces eval-request.ps1 helper script that provides deterministic, model-free eval request workflow for agents. Handles decision logic between manual handoff (preparation-only) and external handoff (with explicit yolo/auto authorization). Includes comprehensive test coverage exercising all runner/model normalization paths and handoff state transitions. Adds PassThru parameter to prepare-skill-evals.ps1 for returning prepared prompt paths. Adds runner normalization to handle user-facing names (GitHub Copilot, Copilot CLI). Integrates validator coverage for eval request workflow in validate-skill-templates.ps1. --- scripts/eval-request.ps1 | 72 ++++++++++ .../eval-runners/tests/test-eval-request.ps1 | 125 ++++++++++++++++++ scripts/prepare-skill-evals.ps1 | 13 ++ scripts/validate-skill-templates.ps1 | 6 + 4 files changed, 216 insertions(+) create mode 100644 scripts/eval-request.ps1 create mode 100644 scripts/eval-runners/tests/test-eval-request.ps1 diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 new file mode 100644 index 0000000..f53bbdf --- /dev/null +++ b/scripts/eval-request.ps1 @@ -0,0 +1,72 @@ +<# +.SYNOPSIS + Deterministic helpers for the agent's explicit eval request workflow. Dot-source to use. +.DESCRIPTION + These helpers never launch a model. The interactive host consumes an external_handoff decision + with its fresh-context delegation tool, passing only the canonical prompt path. Do not wire + this decision to CI, hooks, preparation, validation, or completion-gate execution. +#> +Set-StrictMode -Version Latest + +function Get-EvalHandoff { + [CmdletBinding()] + param( + [Parameter(Mandatory)][string]$PromptPath, + [switch]$Yolo, + [switch]$ExternalOrchestratorAvailable + ) + + $path = (Resolve-Path -LiteralPath $PromptPath -ErrorAction Stop).Path + if ([IO.Path]::GetFileName($path) -cne 'RUN-THIS.prompt.md') { + throw 'Handoff requires the prepared RUN-THIS.prompt.md file.' + } + $package = Split-Path -Parent $path + $decision = [ordered]@{ action = 'manual_handoff'; prompt_path = $path; reason = 'Preparation complete; hand this file to an external Eval Orchestrator.' } + if (-not $Yolo) { return [pscustomobject]$decision } + + # Reserve the handoff before the host delegates. An uncertain launch must never be retried. + $claim = Join-Path $package '.external-handoff-started' + if ((Test-Path -LiteralPath $claim) -or + (Test-Path -LiteralPath (Join-Path $package 'orchestration-state.json')) -or + (Test-Path -LiteralPath (Join-Path $package 'execution-freeze.json'))) { + $decision.action = 'already_started' + $decision.reason = 'Do not dispatch again or invoke Phase 1 again. Observe the existing Orchestrator; interrupted execution remains incomplete.' + return [pscustomobject]$decision + } + if (-not $ExternalOrchestratorAvailable) { + $decision.reason = 'This host cannot hand off to a fresh external Eval Orchestrator. Use the intact manual handoff; never execute an arm in this context.' + return [pscustomobject]$decision + } + try { + $stream = [IO.File]::Open($claim, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None) + $stream.Dispose() + } catch [IO.IOException] { + if (-not (Test-Path -LiteralPath $claim)) { throw } + $decision.action = 'already_started' + $decision.reason = 'A handoff was already reserved. Do not dispatch again.' + return [pscustomobject]$decision + } + $decision.action = 'external_handoff' + $decision.reason = 'Pass only this canonical file path to one fresh external Eval Orchestrator under the explicit user yolo authorization; wait for its final result.' + return [pscustomobject]$decision +} + +function Invoke-EvalRequest { + [CmdletBinding()] + param( + [Parameter(Mandatory)][hashtable]$Preparation, + [switch]$Yolo, + [switch]$ExternalOrchestratorAvailable + ) + + $ErrorActionPreference = 'Stop' + # Splat existing preparation options unchanged; it alone resolves and verifies model policy. + # Collect every success before issuing any handoff, including multi-skill explicit requests. + if ($Preparation.ContainsKey('CollectResults')) { throw 'An eval request prepares packages; CollectResults is a separate forensic workflow.' } + $arguments = $Preparation.Clone() + $arguments.PassThru = $true + $paths = @(& (Join-Path $PSScriptRoot 'prepare-skill-evals.ps1') @arguments) + foreach ($path in $paths) { + Get-EvalHandoff -PromptPath $path -Yolo:$Yolo -ExternalOrchestratorAvailable:$ExternalOrchestratorAvailable + } +} diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 new file mode 100644 index 0000000..52ecd86 --- /dev/null +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -0,0 +1,125 @@ +<# Deterministic request-to-handoff coverage: real preparation, fixture catalogs, fake host. No models. #> +[CmdletBinding()] +param() +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +$scripts = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +. (Join-Path $scripts 'eval-request.ps1') +$workspace = Join-Path ([IO.Path]::GetTempPath()) ('eval-request-workspace/' + [guid]::NewGuid().ToString('N')) +[void](New-Item -ItemType Directory -Path $workspace -Force) +$script:dispatches = [Collections.Generic.List[string]]::new() + +function Assert-True([bool]$Condition, [string]$Message) { + if (-not $Condition) { throw $Message } +} +function Invoke-FakeHost($Decision) { + if ($Decision.action -eq 'external_handoff') { + # The host's only input is the actual canonical handoff file, never an arm prompt. + Assert-True (Test-Path -LiteralPath $Decision.prompt_path -PathType Leaf) 'Host received a missing handoff.' + Assert-True ([IO.Path]::GetFileName($Decision.prompt_path) -ceq 'RUN-THIS.prompt.md') 'Host received an arm.' + $script:dispatches.Add($Decision.prompt_path) + } +} +function New-Preparation([string]$Runner, [string]$Name) { + return @{ Skill = 'dotnet-strong-name-signing'; Eval = @(1); Runner = $Runner + OutputRoot = (Join-Path $workspace $Name); ModelCatalogPath = $catalog } +} +function Assert-Failure([hashtable]$Options, [string]$Pattern) { + # The request must fail closed even in an ordinary interactive PowerShell caller. + $ErrorActionPreference = 'Continue' + $before = $script:dispatches.Count + $failed = $false + try { + Invoke-EvalRequest -Preparation $Options -Yolo -ExternalOrchestratorAvailable | + ForEach-Object { Invoke-FakeHost $_ } + } catch { + $failed = $true + Assert-True ($_.Exception.Message -match $Pattern) "Unexpected failure: $_" + } + Assert-True $failed 'Expected preparation failure.' + Assert-True ($script:dispatches.Count -eq $before) 'Failed preparation dispatched an Orchestrator.' +} + +try { + $catalog = Join-Path $workspace 'models.json' + @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'provider/Exact.Model' }) } | + ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $catalog -Encoding utf8 + + $normalOptions = New-Preparation 'Codex' 'normal' + $normal = Invoke-EvalRequest -Preparation $normalOptions -ExternalOrchestratorAvailable + Invoke-FakeHost $normal + Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' + Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' + + foreach ($case in @( + @{ Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, + @{ Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5' }, + @{ Runner = 'Copilot'; Expected = 'github-copilot'; Model = 'gpt-5.6-luna'; Explicit = $true }, + @{ Runner = 'OpenCode'; Expected = 'opencode'; Model = 'provider/Exact.Model'; Explicit = $true } + )) { + $options = New-Preparation $case.Runner ([guid]::NewGuid().ToString('N')) + if ($case.ContainsKey('Explicit')) { $options.Model = $case.Model } + $decision = Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + $profile = Get-Content -LiteralPath (Join-Path (Split-Path $decision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json + Assert-True ($profile.runner -ceq $case.Expected -and $profile.model -ceq $case.Model) 'Wrong runner/model policy.' + if ($case.Expected -eq 'codex') { + Assert-True ($profile.reasoning_effort -eq 'low') 'Codex default reasoning changed.' + } + Assert-True ($decision.action -eq 'external_handoff') 'Yolo did not request external handoff.' + $before = $script:dispatches.Count + Invoke-FakeHost $decision + $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -ExternalOrchestratorAvailable + Invoke-FakeHost $again + Assert-True ($again.action -eq 'already_started' -and $script:dispatches.Count -eq $before + 1) 'Duplicate handoff could invoke Phase 1 twice.' + $unavailableAfterStart = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo + Assert-True ($unavailableAfterStart.action -eq 'already_started') 'An uncertain launch incorrectly suggested a new manual execution.' + } + + $reference = New-Preparation 'unused' 'reference' + $reference.Remove('Runner') + $reference.CodebeltReference = $true + $reference.ReasoningEffort = 'high' + $reference.ConfigurationProfile = 'fixture-profile' + $reference.ToolProfile = 'fixture-tools' + $reference.TimeoutSeconds = 123 + $reference.Concurrency = 3 + $referenceDecision = Invoke-EvalRequest -Preparation $reference -Yolo -ExternalOrchestratorAvailable + $profile = Get-Content (Join-Path (Split-Path $referenceDecision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json + Assert-True ($profile.runner -eq 'github-copilot' -and $profile.model -eq 'claude-haiku-4.5') 'CodebeltReference changed.' + Assert-True ($profile.reasoning_effort -eq 'high' -and $profile.configuration_profile -eq 'fixture-profile' -and $profile.tool_profile -eq 'fixture-tools' -and $profile.timeout_seconds -eq 123 -and $profile.concurrency -eq 3) 'Preparation options were not forwarded unchanged.' + + $unavailable = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'unavailable') -Yolo + $before = $script:dispatches.Count + Invoke-FakeHost $unavailable + Assert-True ($unavailable.action -eq 'manual_handoff' -and $script:dispatches.Count -eq $before) 'Unavailable host executed a fallback.' + Assert-True (Test-Path -LiteralPath $unavailable.prompt_path) 'Unavailable host lost the package.' + # Existing execution state also blocks automatic handoff, even without a handoff receipt. + '{}' | Set-Content (Join-Path (Split-Path $unavailable.prompt_path) 'orchestration-state.json') + $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -ExternalOrchestratorAvailable + Assert-True ($started.action -eq 'already_started') 'Existing Phase 1 could be invoked twice.' + + Assert-Failure (New-Preparation 'OpenCode' 'missing-model') 'explicit -Model' + $invalid = New-Preparation 'GitHub Copilot' 'invalid-model' + $invalid.Model = 'not-in-catalog' + Assert-Failure $invalid "Runner 'github-copilot' model 'not-in-catalog' could not be verified" + $invalid = New-Preparation 'Codex' 'invalid-default' + $emptyCatalog = Join-Path $workspace 'empty-models.json' + '{"models":[{"id":"unrelated-model"}]}' | Set-Content $emptyCatalog + $invalid.ModelCatalogPath = $emptyCatalog + Assert-Failure $invalid "Runner 'codex' model 'gpt-5.6-luna' could not be verified" + $invalid = New-Preparation 'Codex' 'missing-skill' + $invalid.Skill = 'no-such-skill' + Assert-Failure $invalid 'skill|directory|path' + + # Check the real discovery call contract, not a mocked model resolver. + $source = Get-Content (Join-Path $scripts 'prepare-skill-evals.ps1') -Raw + Assert-True ($source.Contains("`$arguments = @('-Runner', `$RunnerName, '-RequireModel', `$ModelName)")) 'Model discovery must explicitly receive normalized runner and exact model.' + $helper = Get-Content (Join-Path $scripts 'eval-request.ps1') -Raw + Assert-True ($helper -notmatch 'invoke-runner-owned-arms|runner.ps1 execute|Start-Process|spawn_agent') 'Request helper must not implement execution.' + Write-Host 'PASS: normal/yolo requests, runner/model policy, failures, unavailable host, canonical handoff, and duplicate dispatch guards (fake host only).' +} finally { + # The absolute target is the unique child allocated under the external test workspace above. + $allowed = [IO.Path]::GetFullPath((Join-Path ([IO.Path]::GetTempPath()) 'eval-request-workspace')) + [IO.Path]::DirectorySeparatorChar + if (-not [IO.Path]::GetFullPath($workspace).StartsWith($allowed, [StringComparison]::OrdinalIgnoreCase)) { throw 'Unsafe test cleanup path.' } + Remove-Item -LiteralPath $workspace -Recurse -Force +} diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index 358e697..af8a648 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -44,6 +44,9 @@ .PARAMETER Force Overwrite an existing iteration directory. +.PARAMETER PassThru + Also return each successfully prepared absolute RUN-THIS.prompt.md path on the success stream. + .PARAMETER Runner Package-local Eval Runner id written to execution-profile.json when -CodebeltReference is not used. GitHub Copilot and Codex can resolve repository-defined default models; OpenCode requires an explicit model selector. @@ -129,6 +132,10 @@ param( [Parameter(ParameterSetName = 'Changed')] [switch]$Force, + [Parameter(ParameterSetName = 'Prepare')] + [Parameter(ParameterSetName = 'Changed')] + [switch]$PassThru, + [Parameter(ParameterSetName = 'Prepare')] [Parameter(ParameterSetName = 'Changed')] [string]$Runner, @@ -482,6 +489,11 @@ function Resolve-ExecutionSelection { if (-not $hasRunner -and $hasModel) { throw 'Runner/model selection requires -Runner when -Model is supplied.' } + # Normalize the user-facing names before discovery; model policy remains runner-owned here. + $Runner = switch ($Runner.Trim().ToLowerInvariant()) { + { $_ -in @('github copilot', 'github copilot cli', 'copilot') } { 'github-copilot' } + default { $_ } + } $resolvedRunnerMatch = @($supportedRunners | Where-Object { [string]::Equals($_, $Runner, [StringComparison]::OrdinalIgnoreCase) }) if ($resolvedRunnerMatch.Count -ne 1) { throw "Unsupported runner '$Runner'. Supported runner IDs: $supportedText." @@ -1781,6 +1793,7 @@ function Invoke-PrepareMode { Write-Host 'The selected evaluator must write valid runner-produced execution results back into this package.' Write-Host 'If it cannot, the evaluation is incomplete and must fail closed; only persisted runner-produced evidence at' Write-Host 'the manifest-declared paths may proceed.' + if ($PassThru) { Write-Output $runnerPath } } function New-RunnerPrompt { diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 6858d9c..46241d8 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1430,6 +1430,12 @@ Add-ValidationResult -Results $results -Name 'Progress coalescing renders select } } +Add-ValidationResult -Results $results -Name 'Explicit eval requests preserve preparation and reserve one external handoff' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-eval-request.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Eval request regressions failed: $($output -join [Environment]::NewLine)" } +} + Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable prompts instead of executing them' -Group 'Preparation' -Action { $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref From 4a22669f7ae23682afc8ea5f4b22cbbd16f93fa4 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Mon, 7 Sep 2026 20:40:37 +0200 Subject: [PATCH 03/18] =?UTF-8?q?=F0=9F=93=9D=20update=20readme=20to=20doc?= =?UTF-8?q?ument=20eval=20request=20capability?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updates repository README to document the eval request workflow and optional one-shot external handoff feature. Clarifies when and how explicit yolo/auto modifiers authorize automatic delegation to external Eval Orchestrators. Reflects the deterministic preparation model and handoff boundaries for agents preparing skill evaluation packages. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index b2dce65..65e6c74 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Another repo rule is intentionally strict: every repo-managed skill ships with i Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. -Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. The repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: +Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. Without an explicit one-shot request, the repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: Phase 1 closes by writing an immutable `execution-freeze.json` ledger with the exact manifest result paths and hashes of every runner-produced execution result and referenced raw transcript/event artifact. The bridge, grading application, and report adapter validate that ledger and never re-bless changed bytes. The external Grader writes only package-root `grading.json` (`codebeltnet/agentic/eval-grading/1`); `apply-eval-grading.ps1` projects only `passed` and `evidence` decisions onto canonical results. `finalize-eval-package.ps1` owns the deterministic completion boundary and succeeds only after validating the freeze, bridge, complete grading, and all four report artifacts. A changed raw file requires a fresh Phase 1 execution, and prose cannot substitute for finalizer success. Optional scripted `interaction.json` sidecars provide deterministic same-session user turns only when the selected runner advertises and preflights that capability; ordinary single-turn runs remain unchanged. @@ -26,6 +26,8 @@ That resolves every skill the branch changed and prepares a package for each. `- Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the Grader writes only the grading artifact, then the deterministic application helper and finalizer perform the bridge, canonical projection, upstream aggregation/viewer compatibility, and first-party report generation. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. +Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). + The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. One more consistency rule matters for form-driven skills: native input fields are treated as a host feature, not something a model can rely on. Skills in this repo must stay usable with or without UI widgets, and must fall back to the same deterministic one-field-at-a-time flow when the host only supports plain chat. From 0bc863e041add08799d14dc44b06b7077f3d26fa Mon Sep 17 00:00:00 2001 From: gimlichael Date: Mon, 7 Sep 2026 23:26:43 +0200 Subject: [PATCH 04/18] =?UTF-8?q?=F0=9F=93=9D=20document=20v0.9.2=20releas?= =?UTF-8?q?e=20notes=20for=20one-shot=20eval=20handoff?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 21f0132..1cab27d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,33 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.9.2] - 2026-09-08 + +This patch release adds optional one-shot external handoff for explicit eval requests, complete with deterministic workflow helpers and test coverage. Agents can now accept `yolo` or `auto` modifiers on explicit evaluation requests to authorize a single fresh external Eval Orchestrator handoff; preparation remains deterministic and repository-local, while only an explicit, authorized harness receives the sealed package. Skill eval preparation gains a PassThru parameter for returning prompt paths, runner names are normalized for user-facing consistency, and validation tooling gains comprehensive eval-request-workflow checks. + +### Added + +- `scripts/eval-request.ps1` with deterministic helpers for optional one-shot external handoff: `Get-EvalHandoff` reserves and transitions between manual and external handoff states, while `Invoke-EvalRequest` normalizes harness names, discovers available models, returns preparation decisions, and never launches a model itself, +- Full test coverage for eval-request workflow in `scripts/eval-runners/tests/test-eval-request.ps1` exercising all runner/model normalization paths, handoff state transitions, already-started detection, unavailable-host fallback, and case-insensitive normalization, +- Runner normalization in eval-request helpers converting user-facing harness names (`GitHub Copilot`, `Copilot CLI`) to internal canonical runner ids (`github-copilot`) before model discovery and profile generation, +- PassThru parameter in `scripts/prepare-skill-evals.ps1` for returning the prepared prompt paths without file output, enabling downstream helpers to collect decisions and route handoffs, +- `-Yolo` flag in `scripts/eval-request.ps1` Invoke-EvalRequest authorizing one-shot external handoff after explicit user eval request, while keeping preparation and validation deterministic, +- Optional-one-shot-external-handoff section in `AGENTS.md` documenting the explicit user authorization, harness selection, model discovery flow (including OpenCode's explicit-model requirement), and the deterministic one-handoff boundary, +- External evaluation authorization guidance in `AGENTS.md` clarifying that `yolo`/`auto` modifiers on explicit eval requests enable handoff without widening repository automation, CI, completion gates, or model-backed execution, +- Eval-request-workflow validation checks in `scripts/validate-skill-templates.ps1` ensuring eval-request.ps1 presence, test coverage, runner normalization completeness, and handoff-state-reservation correctness. + +### Changed + +- `AGENTS.md` eval preparation guidance now distinguishes between manual handoff (normal path that prepares and returns `RUN-THIS.prompt.md`) and external handoff (when user explicitly requests eval with yolo/auto), clarifying that only the latter authorizes orchestrator delegation, +- Runner-model discovery and selection in `scripts/Get-HarnessModels.ps1` now fully documents the -Runner requirement and exposes current available models without auto-selecting a previous or default model, +- `README.md` eval section expanded with documentation of optional one-shot handoff feature, explicit authorization scoping, and reference to `AGENTS.md` for the complete workflow, +- `CONTRIBUTING.md` now references the optional one-shot eval handoff capability and directs users to full preparation and handoff guidance in `AGENTS.md`. + +### Fixed + +- Eval request workflow now clearly separates deterministic preparation (always model-free, always stops for manual handoff) from explicit external handoff (only with user-supplied yolo/auto, never retried if already launched), +- Model discovery no longer accepts harness names as ambiguous shortcuts; runner normalization explicitly converts `Copilot` and `GitHub Copilot CLI` to the canonical `github-copilot` runner before passing to model-discovery logic. + ## [0.9.1] - 2026-09-07 This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed. @@ -638,6 +665,7 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. +[0.9.2]: https://github.com/codebeltnet/agentic/compare/v0.9.1...v0.9.2 [0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1 [0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0 [0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2 From f12d15dee50f4ff2cebdcc8e94f64b971263279e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 00:06:53 +0200 Subject: [PATCH 05/18] =?UTF-8?q?=F0=9F=90=9B=20guard=20started=20eval=20p?= =?UTF-8?q?ackages=20from=20-Force=20deletion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add safeguards to prevent -Force from replacing eval packages that have already started handoff or execution. This protects the one-shot eval workflow from accidental data loss. New tests validate that the guard works correctly and that packages remain unchanged when Force is rejected. --- .../eval-runners/tests/test-eval-request.ps1 | 27 +++++++++++++++++++ scripts/prepare-skill-evals.ps1 | 6 +++++ 2 files changed, 33 insertions(+) diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 52ecd86..3f3fd4c 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -51,6 +51,33 @@ try { Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' + # Force may replace an unstarted package, but must preserve every started package. + $normalOptions.Iteration = 1 + $normalOptions.Force = $true + $replacement = Invoke-EvalRequest -Preparation $normalOptions + Assert-True ($replacement.prompt_path -eq $normal.prompt_path) 'Force could not replace an unstarted package.' + foreach ($marker in @('.external-handoff-started', 'orchestration-state.json', 'execution-freeze.json')) { + $options = New-Preparation 'Codex' ([guid]::NewGuid().ToString('N')) + $options.Iteration = 1 + $options.Force = $true + if ($marker -eq '.external-handoff-started') { + $reserved = Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + Invoke-FakeHost $reserved + } else { + $reserved = Invoke-EvalRequest -Preparation $options + '{}' | Set-Content -LiteralPath (Join-Path (Split-Path $reserved.prompt_path) $marker) + } + $package = Split-Path $reserved.prompt_path + $beforeFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | + ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) + Assert-Failure $options 'handoff or execution has already started' + $afterFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | + ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) + Assert-True (($beforeFiles -join "`n") -ceq ($afterFiles -join "`n")) "Forced retry changed package protected by $marker." + $again = Get-EvalHandoff -PromptPath $reserved.prompt_path -Yolo -ExternalOrchestratorAvailable + Assert-True ($again.action -eq 'already_started') "Forced retry erased $marker." + } + foreach ($case in @( @{ Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, @{ Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5' }, diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index af8a648..d122b36 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1465,6 +1465,12 @@ function Invoke-PrepareMode { if (-not $Force) { throw "'$iterationDirectory' already exists. Pass -Force to replace it, or -Iteration to write a new one." } + # Force must not erase a one-shot reservation or evidence of an active/completed run. + foreach ($marker in @('.external-handoff-started', 'orchestration-state.json', 'execution-freeze.json')) { + if (Test-Path -LiteralPath (Join-Path $iterationDirectory $marker)) { + throw "Cannot replace '$iterationDirectory': handoff or execution has already started ($marker). Preserve this package and use a fresh iteration." + } + } Remove-Item -LiteralPath $iterationDirectory -Recurse -Force } New-Item -ItemType Directory -Path $iterationDirectory -Force | Out-Null From 512b5035eddd6299324d7f45745d7e4d27a99491 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 00:45:03 +0200 Subject: [PATCH 06/18] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20clarify=20eval=20orc?= =?UTF-8?q?hestrator=20capability=20and=20guard=20model=20policy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename ExternalOrchestratorAvailable to CanDelegateFreshOrchestrator for semantic clarity about what capability is being tested. Add compatibility alias to preserve existing call sites. Formally document GitHub Copilot CLI task + general-purpose agent delegation as a valid external-orchestrator capability. Add guard to prevent Copilot from replacing the repository-defined default model with a subjective stronger choice. Update all tests and validators to match the new naming and expectations. --- AGENTS.md | 6 +-- CONTRIBUTING.md | 4 +- README.md | 2 +- scripts/eval-request.ps1 | 12 +++--- .../eval-runners/tests/test-eval-request.ps1 | 40 ++++++++++++------- scripts/validate-skill-templates.ps1 | 8 ++++ 6 files changed, 46 insertions(+), 26 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7ab7c49..924ac91 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -64,14 +64,14 @@ Use `-CodebeltReference` only when the script should select the Codebelt Referen An explicit request such as `Please eval dotnet-strong-name-signing using Codex. yolo` authorizes preparation followed by ONE external Eval Orchestrator handoff for that package. The existing `auto` modifier has the same scoped meaning. Bare modifiers, skill maintenance, and completion gates do not authorize this flow. Authorization ends when the handoff completes; there are no answer-quality retries. -Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. +Use the host's existing native fresh-context delegation capability for the Orchestrator, without inherited conversation, candidate instructions, grading keys, or repository context. It must be able to read/write the package and follow its canonical handoff through completion. For GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available and do not infer otherwise merely because no shell command named "external orchestrator" exists. In capability terms: GitHub Copilot CLI with `task` + `general-purpose` available means `ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` is true. For example, a host exposing `spawn_agent` with `fork_turns="none"` can delegate this one Orchestrator; this does not authorize arbitrary agents or candidate/baseline fan-out by the preparer. If no such capability exists, retain the manual handoff. Do not launch an AI CLI or invent an adapter as a fallback. -For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and `-ExternalOrchestratorAvailable` only after establishing the host capability above. Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. +For explicit requests, dot-source `scripts/eval-request.ps1` and call `Invoke-EvalRequest -Preparation @{ Skill = ''; Runner = '' }`, forwarding all user-supplied preparation options unchanged. Add `-Yolo` only for the explicit eval modifier and set `-CanDelegateFreshOrchestrator` only after establishing the host capability above (`-ExternalOrchestratorAvailable` remains a compatibility alias). Omit Runner when using the existing `CodebeltReference` selection. This deterministic helper reuses `prepare-skill-evals.ps1` and its model discovery/defaults/validation; it never calls a model. If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model; let preparation resolve the existing repository default and verify only that exact model's availability. OpenCode still requires the user's explicit model before this call. Preparation or verification failure means zero handoffs, even if earlier packages in a multi-skill request were written. When using a shell tool, pipe the returned decisions to `ConvertTo-Json -Depth 4` so table formatting cannot truncate absolute paths. Consume each returned decision in the interactive agent workflow: - `manual_handoff`: return the absolute `prompt_path` and stop. When the host is unavailable, include the concise `reason`; never execute an arm in the current context. -- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. Do not paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. +- `external_handoff`: immediately delegate ONE fresh external Orchestrator, conveying only that the user explicitly authorized this eval handoff and asking it to read and execute the file at the exact absolute `prompt_path`. In GitHub Copilot CLI this delegation is ONE fresh `task` `general-purpose` subagent; pass only `prompt_path`, keep the returned agent/task handle, and follow that same orchestrator to terminal completion. Do not paste or summarize the file, copy orchestration instructions, select another runner/model/profile, or invoke Phase 1 yourself. Wait for that Orchestrator to finish both existing phases, then return its completed report path and comparison, or its honest incomplete/failure result. Do not ask for phase-two approval. - `already_started`: observe the existing Orchestrator. Never delegate again, rerun preparation to retry execution, remove the handoff reservation, or invoke Phase 1 again after a timeout, failed launch, interruption, or uncertain result. If the launch is known unavailable before it starts, use the manual path; once a launch is attempted, do not suggest blindly running the prompt again. A failed/interrupted execution remains incomplete under the existing fresh-iteration rule. The helper reserves the one-shot handoff with `.external-handoff-started` before returning `external_handoff`; retain its returned path and the native Orchestrator handle across continuations. `Get-EvalHandoff` can inspect that same path on resumption and refuses another automatic handoff. This receipt is outside the execution protocol and does not replace Phase 1 state or evidence. The only lifecycle instructions are the existing `RUN-THIS.prompt.md`; its package-local Phase 1 controller, grading, and finalization remain authoritative. No CI, hooks, validators, preparation scripts, or completion gates consume this decision by launching a model. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ea65665..c53b690 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,9 +98,9 @@ Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, wh pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ``` -Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. +Before running the script, choose a Harness + Model when the user did not already do so. Normalize explicit harness wording immediately: `Codex` maps to `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, and `Copilot` map to `github-copilot`; `OpenCode` maps to `opencode`; matching is case-insensitive. Use `scripts/Get-HarnessModels.ps1 -Runner ` to list current selectors; it fails immediately with the supported runner IDs when `-Runner` is omitted. OpenCode mirrors every model exposed by all configured providers, with availability retained as metadata only, while GitHub Copilot and Codex list all currently available models. When OpenCode is selected without an explicit model, present the discovered exact `provider/model` selectors, ask the user to choose one, and wait; never select the first, free, recommended, previous-iteration, previous-successful, or previous-failed model automatically. Preserve an explicitly supplied OpenCode selector verbatim in `execution-profile.json`; discovery failure or incomplete metadata must not substitute another model. The Codebelt Reference shortcut is GitHub Copilot CLI + `claude-haiku-4.5`, and Codex defaults to `gpt-5.6-luna` with low reasoning; package preparation validates the resolved model against current discovery before writing `execution-profile.json`. For `github-copilot` without an explicit model, discovery is only an availability check for the repository-defined default; never rank or replace it with a subjective stronger/newer choice. The script writes `.bot/-workspace/iteration-/` with one directory per eval. Each holds the grading key `eval-metadata.json` and result stubs under `results/` at the eval-case level, plus two paired run directories, `with_skill/` and `without_skill/`. A run directory is the worker's run root: `prompt.md`, a `run.json` contract, a `repo/` working tree materialized from the fixtures, an isolated `home/`, and - for `with_skill` only - a `skill//` copy of the candidate. The grading key and results sit outside both run directories. At the root it writes `manifest.json`, `execution-profile.json`, the package-local Eval Runner protocol, the package report adapter, the exact Anthropic skill-creator grader/aggregator/viewer assets, and `RUN-THIS.prompt.md`, the one prompt you hand to the external Eval Orchestrator. That orchestrator resolves and preflights the selected runner, reads `delegation.dispatch_owner`, and either dispatches the declared orchestrator-owned native worker or starts the declared runner-owned one-arm native surface directly. It stores genuine transport-produced `execution-result.json` evidence, bridges the results, grades only after execution, and runs the adapter, which invokes `aggregate_benchmark.py` and `eval-viewer/generate_review.py --static`. It never runs an eval prompt in the coordinator context, never chooses runner/model policy, and never reuses a worker. Both worker prompts carry the same task, materialized repository, and response contract; only the operating instructions and the presence of `skill/` differ, and neither prompt identifies itself as an eval. `.gitignore` covers `.bot/*`, so nothing there reaches git. The script refuses an `-OutputRoot` inside the repository but outside `.bot/`; pass an explicit temp path when the harness does not need repository-local storage. -For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. +For an explicit eval request only, `yolo`/`auto` permits the interactive agent to hand the prepared `RUN-THIS.prompt.md` to one fresh external Eval Orchestrator and wait for its completed result. Without the modifier, preparation still stops at the manual handoff. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is a valid external handoff capability for this one-shot flow: delegate exactly one fresh orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same returned agent/task handle to terminal completion. Follow [Optional one-shot external handoff](AGENTS.md#optional-one-shot-external-handoff); `scripts/eval-request.ps1` computes and reserves the handoff but never launches a model. Host unavailability returns the manual path, preparation/model failures launch nothing, and an attempted handoff must never be dispatched again. This exception does not apply to repository automation or completion gates. Repository preparation, validation, CI, hooks, deterministic tests, and automatic completion gates never run those prompts or invoke a model. That boundary is the Priority 1 rule in `AGENTS.md`, and preparing a prompt is not permission to execute one. A human-selected external Eval Orchestrator handed a specific package may invoke the selected package-local Eval Runner; this explicit handoff boundary does not weaken the repository prohibition or authorize CI/live evals. diff --git a/README.md b/README.md index 65e6c74..e8234ba 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ That resolves every skill the branch changed and prepares a package for each. `- Each eval becomes a directory holding the grading key (`eval-metadata.json` with the expected output, assertions, and fixture and skill hashes) and prefilled result stubs, plus two paired run directories. `with_skill/` is a self-contained run root: a `prompt.md` with the effective skill instructions inlined, a `run.json` contract naming only paths inside the run, a `repo/` working tree materialized from the fixtures as real files, an isolated empty `home/`, and a `skill//` copy of the exact candidate revision. `without_skill/` is the same run with a byte-identical `repo/`, no `skill/` directory, and no mention of the skill. The grading key and results sit outside both run directories, so workers are not intentionally given them. Neither prompt identifies itself as an eval or names its configuration. `RUN-THIS.prompt.md` makes the user-selected agent the Eval Orchestrator: it reads the profile and selected runner descriptor. For runner-owned dispatch, it invokes the foreground `invoke-runner-owned-arms.ps1` Phase 1 command exactly once and sets the caller shell/tool timeout to at least the package-computed allowance. That helper owns the long-running preflight/fan-out/freeze implementation: it preflights every arm before execution, starts zero model executions when any preflight is incompatible, honors runner concurrency/backpressure, applies bounded child-process timeouts, registers terminal runner-produced evidence, and writes the immutable `execution-freeze.json` before grading. A caller timeout or interrupted conversation is not permission to rerun Phase 1; without a valid freeze the iteration is incomplete and requires a fresh package. For orchestrator-owned dispatch, the orchestrator uses the declared native worker transport. It preserves transport-owned raw evidence and uses `record-native-result.ps1` only for orchestrator-owned envelopes. After a successful Phase 1 freeze, the Grader writes only the grading artifact, then the deterministic application helper and finalizer perform the bridge, canonical projection, upstream aggregation/viewer compatibility, and first-party report generation. Missing telemetry is displayed as unavailable rather than estimated. A runner that cannot satisfy the mandatory experimental controls returns `incompatible`; lack of hard filesystem confinement downgrades the result to pragmatic isolation. If the selected external process cannot write valid runner-produced execution results at the manifest-declared paths, the evaluation is incomplete and fails closed; no response-only or reconstructed result is accepted. Packages land in gitignored `.bot/` storage by default and are not committed. -Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). +Explicit eval requests also support a one-shot handoff: `Please eval dotnet-strong-name-signing using Codex. yolo` (the existing `auto` alias works too). Normal requests still prepare and return `RUN-THIS.prompt.md` for manual handoff. With the modifier, the interactive agent uses the deterministic `scripts/eval-request.ps1` helpers, then passes that exact file to one fresh external Eval Orchestrator through its native host capability and waits for the existing grading/finalization result. In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability (`ExternalOrchestratorAvailable` / `CanDelegateFreshOrchestrator` = true): start exactly one fresh delegated orchestrator context, pass only the generated absolute `RUN-THIS.prompt.md` path, and follow that same agent/task handle to terminal completion. Supplied runners are normalized before discovery; existing defaults and explicit-model validation remain unchanged, including OpenCode's explicit-model requirement. If the resolved runner is `github-copilot` and no explicit model was supplied, preparation applies the existing repository default and verifies only that model's availability. Discovery is an availability check, not model ranking; never switch to a stronger/newer/different model by preference. Failed preparation starts no Orchestrator; an unavailable host returns the intact manual handoff. A reserved/attempted handoff is never dispatched again. This narrowly scoped user authorization does not allow execution in the preparer context, arbitrary model calls, CI execution, or model-backed completion gates. See [the agent workflow](AGENTS.md#optional-one-shot-external-handoff). The four roles are intentionally separate: the Eval Runner is the harness-specific executor for one blind arm; the Eval Orchestrator coordinates the external handoff; the Grader assesses results only after execution; and the Human Reviewer remains the final evaluator. Behavioral evaluation is portable across supported runners. Native activation is harness-specific and out of scope for v0.9.1. Anthropic is the methodology and compatibility reference, not a required Claude runtime. diff --git a/scripts/eval-request.ps1 b/scripts/eval-request.ps1 index f53bbdf..0a4a8af 100644 --- a/scripts/eval-request.ps1 +++ b/scripts/eval-request.ps1 @@ -5,6 +5,8 @@ These helpers never launch a model. The interactive host consumes an external_handoff decision with its fresh-context delegation tool, passing only the canonical prompt path. Do not wire this decision to CI, hooks, preparation, validation, or completion-gate execution. + For GitHub Copilot CLI, task + general-purpose delegation is a valid external-orchestrator + capability for this one-shot handoff. #> Set-StrictMode -Version Latest @@ -13,7 +15,7 @@ function Get-EvalHandoff { param( [Parameter(Mandatory)][string]$PromptPath, [switch]$Yolo, - [switch]$ExternalOrchestratorAvailable + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator ) $path = (Resolve-Path -LiteralPath $PromptPath -ErrorAction Stop).Path @@ -33,8 +35,8 @@ function Get-EvalHandoff { $decision.reason = 'Do not dispatch again or invoke Phase 1 again. Observe the existing Orchestrator; interrupted execution remains incomplete.' return [pscustomobject]$decision } - if (-not $ExternalOrchestratorAvailable) { - $decision.reason = 'This host cannot hand off to a fresh external Eval Orchestrator. Use the intact manual handoff; never execute an arm in this context.' + if (-not $CanDelegateFreshOrchestrator) { + $decision.reason = 'This host cannot delegate one fresh external Eval Orchestrator context. Keep the intact manual handoff and never execute an arm in this context.' return [pscustomobject]$decision } try { @@ -56,7 +58,7 @@ function Invoke-EvalRequest { param( [Parameter(Mandatory)][hashtable]$Preparation, [switch]$Yolo, - [switch]$ExternalOrchestratorAvailable + [Alias('ExternalOrchestratorAvailable')][switch]$CanDelegateFreshOrchestrator ) $ErrorActionPreference = 'Stop' @@ -67,6 +69,6 @@ function Invoke-EvalRequest { $arguments.PassThru = $true $paths = @(& (Join-Path $PSScriptRoot 'prepare-skill-evals.ps1') @arguments) foreach ($path in $paths) { - Get-EvalHandoff -PromptPath $path -Yolo:$Yolo -ExternalOrchestratorAvailable:$ExternalOrchestratorAvailable + Get-EvalHandoff -PromptPath $path -Yolo:$Yolo -CanDelegateFreshOrchestrator:$CanDelegateFreshOrchestrator } } diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 3f3fd4c..84c51f1 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -30,7 +30,7 @@ function Assert-Failure([hashtable]$Options, [string]$Pattern) { $before = $script:dispatches.Count $failed = $false try { - Invoke-EvalRequest -Preparation $Options -Yolo -ExternalOrchestratorAvailable | + Invoke-EvalRequest -Preparation $Options -Yolo -CanDelegateFreshOrchestrator | ForEach-Object { Invoke-FakeHost $_ } } catch { $failed = $true @@ -42,11 +42,11 @@ function Assert-Failure([hashtable]$Options, [string]$Pattern) { try { $catalog = Join-Path $workspace 'models.json' - @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'provider/Exact.Model' }) } | + @{ models = @(@{ id = 'gpt-5.6-luna' }, @{ id = 'claude-haiku-4.5' }, @{ id = 'claude-opus-4.7' }, @{ id = 'provider/Exact.Model' }) } | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $catalog -Encoding utf8 $normalOptions = New-Preparation 'Codex' 'normal' - $normal = Invoke-EvalRequest -Preparation $normalOptions -ExternalOrchestratorAvailable + $normal = Invoke-EvalRequest -Preparation $normalOptions -CanDelegateFreshOrchestrator Invoke-FakeHost $normal Assert-True ($normal.action -eq 'manual_handoff' -and $script:dispatches.Count -eq 0) 'Normal eval must stop at manual handoff.' Assert-True (-not (Test-Path (Join-Path (Split-Path $normal.prompt_path) '.external-handoff-started'))) 'Normal preparation reserved execution.' @@ -61,7 +61,7 @@ try { $options.Iteration = 1 $options.Force = $true if ($marker -eq '.external-handoff-started') { - $reserved = Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + $reserved = Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator Invoke-FakeHost $reserved } else { $reserved = Invoke-EvalRequest -Preparation $options @@ -74,28 +74,36 @@ try { $afterFiles = @(Get-ChildItem -LiteralPath $package -Recurse -File -Force | Sort-Object FullName | ForEach-Object { $_.FullName + ':' + (Get-FileHash -LiteralPath $_.FullName).Hash }) Assert-True (($beforeFiles -join "`n") -ceq ($afterFiles -join "`n")) "Forced retry changed package protected by $marker." - $again = Get-EvalHandoff -PromptPath $reserved.prompt_path -Yolo -ExternalOrchestratorAvailable + $again = Get-EvalHandoff -PromptPath $reserved.prompt_path -Yolo -CanDelegateFreshOrchestrator Assert-True ($again.action -eq 'already_started') "Forced retry erased $marker." } foreach ($case in @( - @{ Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, - @{ Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5' }, - @{ Runner = 'Copilot'; Expected = 'github-copilot'; Model = 'gpt-5.6-luna'; Explicit = $true }, - @{ Runner = 'OpenCode'; Expected = 'opencode'; Model = 'provider/Exact.Model'; Explicit = $true } + @{ Name = 'codex-default'; Runner = 'CoDeX'; Expected = 'codex'; Model = 'gpt-5.6-luna' }, + @{ Name = 'copilot-default'; Runner = 'Copilot'; Expected = 'github-copilot'; Model = 'claude-haiku-4.5'; StrongerModel = 'claude-opus-4.7'; UseLegacyAvailabilityAlias = $true }, + @{ Name = 'copilot-explicit-model'; Runner = 'GitHub Copilot CLI'; Expected = 'github-copilot'; Model = 'gpt-5.6-luna'; Explicit = $true }, + @{ Name = 'opencode-explicit-model'; Runner = 'OpenCode'; Expected = 'opencode'; Model = 'provider/Exact.Model'; Explicit = $true } )) { $options = New-Preparation $case.Runner ([guid]::NewGuid().ToString('N')) if ($case.ContainsKey('Explicit')) { $options.Model = $case.Model } - $decision = Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + $decision = if ($case.ContainsKey('UseLegacyAvailabilityAlias')) { + Invoke-EvalRequest -Preparation $options -Yolo -ExternalOrchestratorAvailable + } else { + Invoke-EvalRequest -Preparation $options -Yolo -CanDelegateFreshOrchestrator + } $profile = Get-Content -LiteralPath (Join-Path (Split-Path $decision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json Assert-True ($profile.runner -ceq $case.Expected -and $profile.model -ceq $case.Model) 'Wrong runner/model policy.' + if ($case.ContainsKey('StrongerModel')) { + Assert-True ($profile.model -cne $case.StrongerModel) 'Copilot replaced repository model policy with a subjective stronger-model choice.' + } if ($case.Expected -eq 'codex') { Assert-True ($profile.reasoning_effort -eq 'low') 'Codex default reasoning changed.' } Assert-True ($decision.action -eq 'external_handoff') 'Yolo did not request external handoff.' $before = $script:dispatches.Count Invoke-FakeHost $decision - $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -ExternalOrchestratorAvailable + Assert-True ($script:dispatches[$before] -ceq $decision.prompt_path) 'Host did not receive the exact generated RUN-THIS.prompt.md path.' + $again = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo -CanDelegateFreshOrchestrator Invoke-FakeHost $again Assert-True ($again.action -eq 'already_started' -and $script:dispatches.Count -eq $before + 1) 'Duplicate handoff could invoke Phase 1 twice.' $unavailableAfterStart = Get-EvalHandoff -PromptPath $decision.prompt_path -Yolo @@ -110,19 +118,19 @@ try { $reference.ToolProfile = 'fixture-tools' $reference.TimeoutSeconds = 123 $reference.Concurrency = 3 - $referenceDecision = Invoke-EvalRequest -Preparation $reference -Yolo -ExternalOrchestratorAvailable + $referenceDecision = Invoke-EvalRequest -Preparation $reference -Yolo -CanDelegateFreshOrchestrator $profile = Get-Content (Join-Path (Split-Path $referenceDecision.prompt_path) 'execution-profile.json') -Raw | ConvertFrom-Json Assert-True ($profile.runner -eq 'github-copilot' -and $profile.model -eq 'claude-haiku-4.5') 'CodebeltReference changed.' Assert-True ($profile.reasoning_effort -eq 'high' -and $profile.configuration_profile -eq 'fixture-profile' -and $profile.tool_profile -eq 'fixture-tools' -and $profile.timeout_seconds -eq 123 -and $profile.concurrency -eq 3) 'Preparation options were not forwarded unchanged.' - $unavailable = Invoke-EvalRequest -Preparation (New-Preparation 'Codex' 'unavailable') -Yolo + $unavailable = Invoke-EvalRequest -Preparation (New-Preparation 'Copilot' 'unavailable') -Yolo $before = $script:dispatches.Count Invoke-FakeHost $unavailable Assert-True ($unavailable.action -eq 'manual_handoff' -and $script:dispatches.Count -eq $before) 'Unavailable host executed a fallback.' Assert-True (Test-Path -LiteralPath $unavailable.prompt_path) 'Unavailable host lost the package.' # Existing execution state also blocks automatic handoff, even without a handoff receipt. '{}' | Set-Content (Join-Path (Split-Path $unavailable.prompt_path) 'orchestration-state.json') - $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -ExternalOrchestratorAvailable + $started = Get-EvalHandoff -PromptPath $unavailable.prompt_path -Yolo -CanDelegateFreshOrchestrator Assert-True ($started.action -eq 'already_started') 'Existing Phase 1 could be invoked twice.' Assert-Failure (New-Preparation 'OpenCode' 'missing-model') 'explicit -Model' @@ -142,8 +150,10 @@ try { $source = Get-Content (Join-Path $scripts 'prepare-skill-evals.ps1') -Raw Assert-True ($source.Contains("`$arguments = @('-Runner', `$RunnerName, '-RequireModel', `$ModelName)")) 'Model discovery must explicitly receive normalized runner and exact model.' $helper = Get-Content (Join-Path $scripts 'eval-request.ps1') -Raw + Assert-True ($helper.Contains("[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator")) 'External orchestrator capability alias changed unexpectedly.' + Assert-True (-not $helper.Contains('claude-haiku-4.5') -and -not $helper.Contains('gpt-5.6-luna') -and -not $helper.Contains('claude-opus-4.7')) 'Eval request helper must not embed model-selection policy.' Assert-True ($helper -notmatch 'invoke-runner-owned-arms|runner.ps1 execute|Start-Process|spawn_agent') 'Request helper must not implement execution.' - Write-Host 'PASS: normal/yolo requests, runner/model policy, failures, unavailable host, canonical handoff, and duplicate dispatch guards (fake host only).' + Write-Host 'PASS: normal/yolo requests, Copilot delegation capability semantics, runner/model policy, failures, canonical handoff pathing, and duplicate dispatch guards (fake host only).' } finally { # The absolute target is the unique child allocated under the external test workspace above. $allowed = [IO.Path]::GetFullPath((Join-Path ([IO.Path]::GetTempPath()) 'eval-request-workspace')) + [IO.Path]::DirectorySeparatorChar diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 46241d8..2119cc4 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1441,6 +1441,7 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref $contributing = Get-FileText -RepoRoot $repoRoot -RelativePath 'CONTRIBUTING.md' -GitRef $Ref $runnerReadme = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/eval-runners/README.md' -GitRef $Ref + $evalRequest = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/eval-request.ps1' -GitRef $Ref $prepare = Get-FileText -RepoRoot $repoRoot -RelativePath 'scripts/prepare-skill-evals.ps1' -GitRef $Ref Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '## Portable Eval Handoff' @@ -1453,6 +1454,9 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'repository automation remains deterministic and never invokes a model.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Resolve the execution configuration before running the package preparation script.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'scripts/Get-HarnessModels.ps1' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is this capability; treat it as available' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '-CanDelegateFreshOrchestrator' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If the resolved runner is `github-copilot` and no explicit model was supplied, do not pick a discovered model' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'OpenCode discovery mirrors every model exposed by all configured OpenCode providers' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'availability is presentation metadata only and never filters the selectable catalog' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made.' @@ -1483,6 +1487,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'a completion gate an agent cannot skip' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed' Assert-Contains -Name 'README.md' -Content $readme -Needle 'prepares the paired candidate and baseline inputs as a portable package and stops' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'In GitHub Copilot CLI, `task` + fresh `general-purpose` agent availability is that capability' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'Discovery is an availability check, not model ranking' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill -Runner -Model ' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'choose a Harness + Model when the user did not already do so' Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'OpenCode mirrors every model exposed by all configured providers' @@ -1492,6 +1498,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'README.md' -Content $readme -Needle 'the user must choose from the discovered selectors before package preparation' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'OpenCode through `opencode models --verbose`' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'without filtering the selectable catalog' + Assert-Contains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle "[Alias('ExternalOrchestratorAvailable')][switch]`$CanDelegateFreshOrchestrator" + Assert-NotContains -Name 'scripts/eval-request.ps1' -Content $evalRequest -Needle 'claude-opus-4.7' Assert-Contains -Name 'README.md' -Content $readme -Needle 'full operational permission inside each isolated behavioral harness configuration' Assert-Contains -Name 'README.md' -Content $readme -Needle 'Hard filesystem confinement is a separate optional outer capability' Assert-Contains -Name 'scripts/eval-runners/README.md' -Content $runnerReadme -Needle 'full harness operational permission inside each isolated eval boundary' From eee4240a02f974d51cec860b2380fe50ed016118 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 10:09:51 +0200 Subject: [PATCH 07/18] =?UTF-8?q?=E2=9C=85=20implement=20fail-closed=20cop?= =?UTF-8?q?ilot=20authentication=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive authentication validation and fail-closed behavior to the GitHub Copilot runner. The runner now detects when supported non-interactive authentication sources are unavailable (explicit tokens or trusted GitHub CLI fallback) and terminates evaluation preemptively rather than attempting execution without required auth. New test scenarios cover fresh-context environments and missing auth states. Updated handoff prompt generation documents that authentication incompatibility is terminal for the package iteration, preventing suggestions for runner switching or Phase 1 retries. --- .../eval-runners/github-copilot/runner.ps1 | 157 +++++++++++++----- .../tests/test-copilot-boundaries.ps1 | 2 +- .../tests/test-runner-conformance.ps1 | 124 ++++++++++++-- scripts/prepare-skill-evals.ps1 | 1 + 4 files changed, 223 insertions(+), 61 deletions(-) diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index f63bf46..da6dc6b 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -203,35 +203,64 @@ function Get-CopilotTokenVariable { return $null } -function Get-CopilotGhConfigDirectory { +function Add-CopilotGhConfigCandidate { + param( + [Parameter(Mandatory = $true)][AllowEmptyCollection()][System.Collections.Generic.List[string]]$Candidates, + [string]$Path + ) + + if ([string]::IsNullOrWhiteSpace($Path)) { return } + try { + $normalized = [System.IO.Path]::GetFullPath($Path) + if (-not $Candidates.Contains($normalized)) { $Candidates.Add($normalized) } + } catch { } +} + +function Get-CopilotGhConfigDirectories { # GH_CONFIG_DIR is an authentication-state exception to the isolated - # Copilot configuration roots. Resolve it from GitHub CLI's documented - # precedence without reading or logging any credential file. - $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') - if ([string]::IsNullOrWhiteSpace($configured)) { - $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') - if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { - $configured = Join-Path $xdgConfig 'gh' - } elseif ((Get-PlatformName) -eq 'windows') { - $applicationData = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) - if (-not [string]::IsNullOrWhiteSpace($applicationData)) { - $configured = Join-Path $applicationData 'GitHub CLI' - } - } else { - $userHome = [Environment]::GetEnvironmentVariable('HOME') - if ([string]::IsNullOrWhiteSpace($userHome)) { - $userHome = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) - } + # Copilot configuration roots. Build documented/default candidates and keep + # only existing directories; token resolution tries each candidate in order. + $candidates = [System.Collections.Generic.List[string]]::new() + + Add-CopilotGhConfigCandidate -Candidates $candidates -Path ([Environment]::GetEnvironmentVariable('GH_CONFIG_DIR')) + $xdgConfig = [Environment]::GetEnvironmentVariable('XDG_CONFIG_HOME') + if (-not [string]::IsNullOrWhiteSpace($xdgConfig)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $xdgConfig 'gh') + } + + if ((Get-PlatformName) -eq 'windows') { + $applicationDataFromEnvironment = [Environment]::GetEnvironmentVariable('APPDATA') + if (-not [string]::IsNullOrWhiteSpace($applicationDataFromEnvironment)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $applicationDataFromEnvironment 'GitHub CLI') + } + + $applicationDataFromProfile = [Environment]::GetFolderPath([Environment+SpecialFolder]::ApplicationData) + if (-not [string]::IsNullOrWhiteSpace($applicationDataFromProfile)) { + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path $applicationDataFromProfile 'GitHub CLI') + } + } else { + foreach ($userHome in @( + [Environment]::GetEnvironmentVariable('HOME'), + [Environment]::GetEnvironmentVariable('USERPROFILE'), + [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile) + )) { if (-not [string]::IsNullOrWhiteSpace($userHome)) { - $configured = Join-Path (Join-Path $userHome '.config') 'gh' + Add-CopilotGhConfigCandidate -Candidates $candidates -Path (Join-Path (Join-Path $userHome '.config') 'gh') } } } - if ([string]::IsNullOrWhiteSpace($configured) -or -not (Test-Path -LiteralPath $configured -PathType Container)) { - return $null + $resolved = [System.Collections.Generic.List[string]]::new() + foreach ($candidate in @($candidates)) { + if (Test-Path -LiteralPath $candidate -PathType Container) { $resolved.Add($candidate) } } - return [System.IO.Path]::GetFullPath($configured) + return @($resolved.ToArray()) +} + +function Get-CopilotGhConfigDirectory { + $directories = @(Get-CopilotGhConfigDirectories) + if ($directories.Count -eq 0) { return $null } + return [string]$directories[0] } function Get-CopilotGitHubCliToken { @@ -240,26 +269,54 @@ function Get-CopilotGitHubCliToken { return $null } - $environment = New-RunnerProbeEnvironment + $baseEnvironment = New-RunnerProbeEnvironment foreach ($name in @('HOME', 'USERPROFILE', 'APPDATA', 'LOCALAPPDATA', 'XDG_CONFIG_HOME', 'GH_CONFIG_DIR')) { $value = [Environment]::GetEnvironmentVariable($name) if (-not [string]::IsNullOrWhiteSpace($value)) { - $environment[$name] = $value + $baseEnvironment[$name] = $value + } + } + + $candidateConfigDirectories = [System.Collections.Generic.List[string]]::new() + foreach ($directory in @(Get-CopilotGhConfigDirectories)) { + if (-not [string]::IsNullOrWhiteSpace([string]$directory) -and -not $candidateConfigDirectories.Contains([string]$directory)) { + $candidateConfigDirectories.Add([string]$directory) } } - $probeDirectory = Join-Path ([System.IO.Path]::GetTempPath()) ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) + # Keep one ambient attempt with GH_CONFIG_DIR removed so an existing host + # default can still resolve even when no candidate was discovered. + $candidateConfigDirectories.Add('') + + $probeRoot = Resolve-RunnerProbeTempRoot + $probeDirectory = Join-Path $probeRoot ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) New-Item -ItemType Directory -Path $probeDirectory -Force | Out-Null try { - $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 - if ($process.TimedOut -or $process.ExitCode -ne 0) { - return $null - } - $token = ([string]$process.Stdout).Trim() - if ([string]::IsNullOrWhiteSpace($token)) { - return $null + foreach ($candidateDirectory in @($candidateConfigDirectories.ToArray())) { + $environment = [ordered]@{} + foreach ($key in @($baseEnvironment.Keys)) { + $environment[[string]$key] = [string]$baseEnvironment[$key] + } + if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { + [void]$environment.Remove('GH_CONFIG_DIR') + } else { + $environment['GH_CONFIG_DIR'] = [string]$candidateDirectory + } + + $process = Invoke-RunnerProcess -FileName $gh.FileName -ArgumentList (@($gh.Prefix) + @('auth', 'token')) -WorkingDirectory $probeDirectory -Environment $environment -TimeoutSeconds 30 + if ($process.TimedOut -or $process.ExitCode -ne 0) { + continue + } + $token = ([string]$process.Stdout).Trim() + if ([string]::IsNullOrWhiteSpace($token)) { + continue + } + return [pscustomobject]@{ + Token = $token + GhConfigDirectory = if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { $null } else { [string]$candidateDirectory } + } } - return $token + return $null } finally { if (Test-Path -LiteralPath $probeDirectory) { Remove-Item -LiteralPath $probeDirectory -Recurse -Force -ErrorAction SilentlyContinue @@ -275,24 +332,31 @@ function Resolve-CopilotAuthentication { TokenVariable = $tokenVariable TokenValue = $null GitHubCliTokenResolved = $false + GitHubCliConfigDirectory = $null + NonInteractiveReady = $true } } $githubCliToken = Get-CopilotGitHubCliToken - if (-not [string]::IsNullOrWhiteSpace($githubCliToken)) { + if ($null -ne $githubCliToken -and -not [string]::IsNullOrWhiteSpace([string]$githubCliToken.Token)) { return [pscustomobject]@{ Source = 'github_cli_token' TokenVariable = 'GH_TOKEN' - TokenValue = $githubCliToken + TokenValue = [string]$githubCliToken.Token GitHubCliTokenResolved = $true + GitHubCliConfigDirectory = [string](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigDirectory' -Default $null) + NonInteractiveReady = $true } } + $fallbackSource = if ($null -ne (Resolve-ExternalCommand -Name 'copilot')) { 'copilot_os_keychain_unverified' } else { 'none' } return [pscustomobject]@{ - Source = 'copilot_os_keychain_or_github_cli_unverified' + Source = $fallbackSource TokenVariable = $null TokenValue = $null GitHubCliTokenResolved = $false + GitHubCliConfigDirectory = $null + NonInteractiveReady = $false } } @@ -602,10 +666,11 @@ function Get-CopilotPreflight { if ($authState.Source -eq 'environment') { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) } elseif ($authState.Source -eq 'github_cli_token') { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail 'GitHub CLI fallback resolved a token in the trusted runner; only a protected token environment variable will be passed to Copilot.')) + $configDetail = if ([string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory)) { 'ambient/default' } else { 'resolved GH_CONFIG_DIR candidate' } + $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail ("GitHub CLI fallback resolved a token in the trusted runner from a {0} source; only a protected token environment variable will be passed to Copilot." -f $configDetail))) } else { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status unavailable -Detail 'No explicit token is present and GitHub CLI fallback did not yield a token; native Copilot OS-keychain lookup is delegated to the installed CLI. This preflight does not contact the Copilot service.')) - $warnings.Add('Authentication readiness beyond explicit environment tokens and the observable GitHub CLI fallback cannot be proven without a live Copilot request; preflight remains conditional and does not reject a tokenless native OAuth/keychain configuration.') + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or ensure `gh auth token` can resolve in this host context.') } if ($platform -notin @('linux', 'macos')) { @@ -646,7 +711,7 @@ function Get-CopilotPreflight { foreach ($key in $descriptor.Keys) { $descriptorCopy[$key] = $descriptor[$key] } $descriptorCopy.harness = [ordered]@{ name = 'GitHub Copilot CLI'; version = $harnessVersion } $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('runner-owned fresh Copilot CLI session per eval execution', 'copilot --output-format json terminal event capture', 'native task/general-purpose subagent available as a separate harness capability, not the transport', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('runner-owned fresh Copilot CLI session per eval execution', 'copilot --output-format json terminal event capture', 'native task/general-purpose subagent available as a separate harness capability, not the transport', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'non-interactive authentication source required before Phase 1 (explicit token env or trusted GitHub CLI fallback)', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker')) { $mechanisms.Add($mechanism) } if ($null -ne $run.Interaction -and $continuationCapability.Available) { $mechanisms.Add(("explicit Copilot {0} continuation selected from installed help" -f $continuationCapability.Flag)) $mechanisms.Add('no implicit last-session continuation') @@ -668,6 +733,13 @@ function Get-CopilotPreflight { exact_session_required = $true implicit_continuation = $false } + authentication = [ordered]@{ + source = [string]$authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady + github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved + github_cli_config_candidate_used = -not [string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory) + token_value_observed = $false + } } return $document } @@ -692,7 +764,6 @@ function New-CopilotEnvironment { } # Model-free help/version bootstrap must use the same cache roots too. if ($WithoutAuthentication) { return New-RunnerEnvironment -Run $Inputs.Run -Additional $additional } - $tokenVariable = Get-CopilotTokenVariable $authState = Resolve-CopilotAuthentication if ($authState.Source -eq 'github_cli_token') { $additional[$authState.TokenVariable] = $authState.TokenValue @@ -1091,6 +1162,7 @@ function Invoke-CopilotScriptedExecute { $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ source = $authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') @@ -1286,13 +1358,14 @@ function Invoke-CopilotExecute { $capabilities = Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem $mechanisms = [System.Collections.Generic.List[string]]::new() - foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'OS-keychain authentication delegated to Copilot', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } + foreach ($mechanism in @('copilot --output-format json', 'prompt on stdin', '--allow-all full programmatic permission', '--no-ask-user', 'repository-owned custom instructions preserved', '--disable-builtin-mcps', '--secret-env-vars shell/MCP child filtering', 'isolated COPILOT_HOME and COPILOT_CACHE_HOME', 'isolated HOME/XDG roots', 'non-interactive authentication source required before Phase 1 (explicit token env or trusted GitHub CLI fallback)', 'GitHub CLI fallback token resolved by the trusted runner when needed', 'no host GH_CONFIG_DIR exposed to the worker', 'no session continuation')) { $mechanisms.Add($mechanism) } if ($hardFilesystem) { $mechanisms.Add("external $($sandboxInfo.Source) filesystem sandbox") } else { $mechanisms.Add('pragmatic process/environment isolation without hard filesystem confinement') } if (-not $hardFilesystem) { $warnings.Add('Hard filesystem confinement was unavailable; the completed arm is reported as pragmatic isolation.') } $authState = Resolve-CopilotAuthentication $credentialEvidence = [ordered]@{ source = $authState.Source + noninteractive_ready = [bool]$authState.NonInteractiveReady github_token_variable = $authState.TokenVariable secret_env_vars = @($copilotAuthVariables) secret_env_var_scope = @('shell', 'mcp') diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index 9c1f6e9..90d205b 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -24,7 +24,7 @@ foreach ($file in @('github-copilot/runner.ps1', 'bridge-execution-result.ps1', } } $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') -function Resolve-CopilotAuthentication { return [pscustomobject]@{ Source = 'fixture'; TokenVariable = $null; TokenValue = $null; GitHubCliTokenResolved = $false } } +function Resolve-CopilotAuthentication { return [pscustomobject]@{ Source = 'fixture'; TokenVariable = $null; TokenValue = $null; GitHubCliTokenResolved = $false; GitHubCliConfigDirectory = $null; NonInteractiveReady = $true } } function Resolve-SandboxCommand { param($Name) return $null } function Resolve-ExternalCommand { param($Name) if ($Name -ne 'copilot') { throw "Unexpected executable lookup: $Name" }; return $fakeCommand } function Get-CopilotPreflight { diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 7eaa252..019a416 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -125,6 +125,11 @@ $recordedOldGhToken = $env:GH_TOKEN $recordedOldGithubToken = $env:GITHUB_TOKEN $recordedOldCopilotHome = $env:COPILOT_HOME $recordedOldGhConfigDir = $env:GH_CONFIG_DIR +$recordedOldHome = $env:HOME +$recordedOldUserProfile = $env:USERPROFILE +$recordedOldAppData = $env:APPDATA +$recordedOldLocalAppData = $env:LOCALAPPDATA +$recordedOldXdgConfigHome = $env:XDG_CONFIG_HOME $recordedOldFixtures = $env:AGENTIC_RECORDED_FIXTURES try { $fakeBin = Join-Path $recordedRoot 'bin' @@ -1970,58 +1975,134 @@ exit 2 $env:CODEX_HOME = $recordedOldCodexHome } if ($Suite -in @('All', 'Copilot')) { - # GitHub Copilot authentication: explicit env, OS-keychain, GitHub CLI, and - # no-auth fixtures are all deterministic and contain no credential values. + # GitHub Copilot authentication continuity and failure semantics: + # deterministic fixtures only, no live credentials. $env:COPILOT_GITHUB_TOKEN = $null $env:GH_TOKEN = $null $env:GITHUB_TOKEN = $null $missingGhConfig = Join-Path $recordedRoot 'missing-github-cli-auth' + New-Item -ItemType Directory -Path $missingGhConfig -Force | Out-Null + $copilotLogPath = Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl' - # The fixture marker is fake-CLI input only; it models a positive OS - # keychain lookup without naming or reading a real credential-store file. - $copilotKeychainHome = Join-Path $recordedRoot 'copilot-keychain-home' - New-Item -ItemType Directory -Path $copilotKeychainHome -Force | Out-Null + # Fixture marker only: models a keychain-only host with no resolvable + # non-interactive token source. + $copilotKeychainHost = Join-Path $recordedRoot 'copilot-keychain-host' + New-Item -ItemType Directory -Path $copilotKeychainHost -Force | Out-Null New-Item -ItemType Directory -Path (Join-Path $with.Root 'home\.copilot') -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available'), 'fixture marker only', [Text.UTF8Encoding]::new($false)) - $env:COPILOT_HOME = $copilotKeychainHome + $env:COPILOT_HOME = $copilotKeychainHost $env:GH_CONFIG_DIR = $missingGhConfig + $nativeExecutionsBeforeKeychain = @() + if (Test-Path -LiteralPath $copilotLogPath -PathType Leaf) { + $nativeExecutionsBeforeKeychain = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + } $copilotKeychainPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'compatible' $copilotKeychainPreflight.status 'Copilot tokenless OS-keychain authentication remains compatible' - Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'unavailable' }).Count -eq 1) 'Copilot preflight leaves native keychain readiness conditional' - Assert-True (@($copilotKeychainPreflight.warnings | Where-Object { $_ -match 'cannot be proven' }).Count -gt 0) 'Copilot preflight explains the unverified keychain/service boundary' + Assert-Equal 'incompatible' $copilotKeychainPreflight.status 'Copilot keychain-only auth is rejected before Phase 1 execution' + Assert-True (@($copilotKeychainPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot keychain-only preflight fails authentication deterministically' + Assert-Equal 'copilot_os_keychain_unverified' $copilotKeychainPreflight.protocol_observations.authentication.source 'Copilot keychain-only source is diagnostic only' + Assert-True (-not [bool]$copilotKeychainPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot keychain-only source is not accepted as non-interactive readiness' + Assert-True (@($copilotKeychainPreflight.reasons | Where-Object { $_ -match 'supported non-interactive Copilot authentication source' }).Count -eq 1) 'Copilot keychain-only preflight reports the missing non-interactive source' $copilotKeychainResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'completed' $copilotKeychainResult.status 'Copilot keychain fixture executes without an exported token' - $keychainRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [string](Get-JsonProperty -Object $_ -Name 'copilot_authentication_source' -Default '') -eq 'os_keychain' }) - Assert-Equal 1 $keychainRecords.Count 'Copilot fake observes the simulated OS-keychain path' + Assert-Equal 'incompatible' $copilotKeychainResult.status 'Copilot execute fails closed when preflight cannot resolve non-interactive auth' + Assert-Equal 'incompatible' ([string]$copilotKeychainResult.evidence.preflight.status) 'Copilot execute returns preflight evidence for keychain-only rejection' + $nativeExecutionsAfterKeychain = @() + if (Test-Path -LiteralPath $copilotLogPath -PathType Leaf) { + $nativeExecutionsAfterKeychain = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + } + Assert-Equal $nativeExecutionsBeforeKeychain.Count $nativeExecutionsAfterKeychain.Count 'Copilot keychain-only rejection occurs before model execution' Remove-Item -LiteralPath (Join-Path $with.Root 'home\.copilot\fixture-os-keychain-available') -Force + # Fresh-context continuity regression: the first GH config candidate exists + # but is not authenticated; fallback resolves a second supported host + # candidate without exposing auth files or token values to the run. + $copilotFreshBoundaryRoot = Join-Path $recordedRoot 'copilot-fresh-boundary' + $copilotFreshHome = Join-Path $copilotFreshBoundaryRoot 'home' + $copilotFreshXdg = Join-Path $copilotFreshBoundaryRoot 'xdg' + $copilotFreshAppData = Join-Path $copilotFreshBoundaryRoot 'appdata' + $copilotFreshLocalAppData = Join-Path $copilotFreshBoundaryRoot 'localappdata' + $copilotFreshHostAppData = Join-Path $copilotFreshBoundaryRoot 'host-appdata' + $copilotFreshHostGhConfig = Join-Path $copilotFreshHostAppData 'GitHub CLI' + New-Item -ItemType Directory -Path $copilotFreshHome, $copilotFreshXdg, $copilotFreshAppData, $copilotFreshLocalAppData, (Join-Path $copilotFreshXdg 'gh'), $copilotFreshHostGhConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotFreshHostGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $copilotFreshHome + $env:USERPROFILE = $copilotFreshHome + $env:APPDATA = $copilotFreshHostAppData + $env:LOCALAPPDATA = $copilotFreshLocalAppData + $env:XDG_CONFIG_HOME = $copilotFreshXdg + $env:COPILOT_HOME = (Join-Path $copilotFreshBoundaryRoot 'copilot-home') + New-Item -ItemType Directory -Path $env:COPILOT_HOME -Force | Out-Null + $env:GH_CONFIG_DIR = $null + $copilotFreshBoundaryPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'compatible' $copilotFreshBoundaryPreflight.status 'Copilot preflight resolves non-interactive auth across fresh-context candidate fallback' + Assert-True (@($copilotFreshBoundaryPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'passed' }).Count -eq 1) 'Copilot fresh-context preflight proves authentication readiness' + Assert-Equal 'github_cli_token' $copilotFreshBoundaryPreflight.protocol_observations.authentication.source 'Copilot fresh-context source resolves through GitHub CLI fallback' + Assert-True ([bool]$copilotFreshBoundaryPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot fresh-context source is accepted as non-interactive readiness' + Assert-True ([bool]$copilotFreshBoundaryPreflight.protocol_observations.authentication.github_cli_config_candidate_used) 'Copilot fresh-context preflight records GH config candidate fallback usage' + $copilotFreshBoundaryResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'completed' $copilotFreshBoundaryResult.status 'Copilot fresh-context fallback executes after preflight authentication continuity' + Assert-Equal 'github_cli_token' $copilotFreshBoundaryResult.evidence.credential.source 'Copilot fresh-context execution records GitHub CLI fallback source' + Assert-True ([bool]$copilotFreshBoundaryResult.evidence.credential.noninteractive_ready) 'Copilot fresh-context execution records non-interactive readiness' + Assert-True ([bool]$copilotFreshBoundaryResult.evidence.credential.github_cli_token_resolved) 'Copilot fresh-context execution resolves a trusted GitHub CLI token' + $freshBoundaryExecutionRecords = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + $freshBoundaryExecution = $freshBoundaryExecutionRecords[$freshBoundaryExecutionRecords.Count - 1] + Assert-Equal 'explicit_environment' $freshBoundaryExecution.copilot_authentication_source 'Copilot fresh-context worker receives only protected env-token auth' + Assert-Equal 1 @($freshBoundaryExecution.copilot_auth_names_present).Count 'Copilot fresh-context worker receives one token variable' + Assert-True (@($freshBoundaryExecution.copilot_auth_names_present) -contains 'GH_TOKEN') 'Copilot fresh-context worker receives only GH_TOKEN from trusted fallback' + Assert-True ([string]::IsNullOrWhiteSpace([string]$freshBoundaryExecution.gh_config_dir)) 'Copilot fresh-context worker does not receive GH_CONFIG_DIR' + $copilotGhFallbackHome = Join-Path $recordedRoot 'copilot-gh-fallback-home' New-Item -ItemType Directory -Path $copilotGhFallbackHome -Force | Out-Null $copilotGhConfig = Join-Path $recordedRoot 'copilot-gh-config' New-Item -ItemType Directory -Path $copilotGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $copilotGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:COPILOT_HOME = $copilotGhFallbackHome $env:GH_CONFIG_DIR = $copilotGhConfig + $ghFallbackMatchesBefore = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }).Count $copilotGhPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'compatible' $copilotGhPreflight.status 'Copilot GitHub CLI fallback remains compatible' + Assert-Equal 'github_cli_token' $copilotGhPreflight.protocol_observations.authentication.source 'Copilot preflight classifies explicit GH config fallback source' + Assert-True ([bool]$copilotGhPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot preflight classifies GH config fallback as non-interactive readiness' $copilotGhResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] Assert-Equal 'completed' $copilotGhResult.status 'Copilot GitHub CLI fallback fixture executes without an exported token' + Assert-True ([bool]$copilotGhResult.evidence.credential.noninteractive_ready) 'Copilot GitHub CLI fallback result records non-interactive readiness' Assert-True $copilotGhResult.evidence.credential.github_cli_token_resolved 'Copilot records GitHub CLI token fallback without storing the token value' Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) - Assert-Equal 1 $ghRecords.Count 'Copilot fake observes only the protected GH_TOKEN produced by trusted GitHub CLI fallback' + Assert-Equal ($ghFallbackMatchesBefore + 1) $ghRecords.Count 'Copilot fake observes one additional protected GH_TOKEN execution produced by trusted GitHub CLI fallback' $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null + $copilotNoAuthRoot = Join-Path $recordedRoot 'copilot-no-auth-roots' + New-Item -ItemType Directory -Path (Join-Path $copilotNoAuthRoot 'home'), (Join-Path $copilotNoAuthRoot 'appdata'), (Join-Path $copilotNoAuthRoot 'localappdata'), (Join-Path $copilotNoAuthRoot 'xdg') -Force | Out-Null + $env:HOME = Join-Path $copilotNoAuthRoot 'home' + $env:USERPROFILE = Join-Path $copilotNoAuthRoot 'home' + $env:APPDATA = Join-Path $copilotNoAuthRoot 'appdata' + $env:LOCALAPPDATA = Join-Path $copilotNoAuthRoot 'localappdata' + $env:XDG_CONFIG_HOME = Join-Path $copilotNoAuthRoot 'xdg' + $nativeExecutionsBeforeNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) $env:COPILOT_HOME = $copilotNoAuthHome $env:GH_CONFIG_DIR = $missingGhConfig $copilotNoAuthPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'compatible' $copilotNoAuthPreflight.status 'Copilot preflight does not require an exported token when native auth is not observable' - Assert-True (@($copilotNoAuthPreflight.warnings | Where-Object { $_ -match 'conditional' }).Count -gt 0) 'Copilot no-auth preflight is explicitly conditional' + Assert-Equal 'incompatible' $copilotNoAuthPreflight.status 'Copilot preflight fails closed when no supported non-interactive auth source is available' + Assert-True (@($copilotNoAuthPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot no-auth preflight records an authentication failure check' + Assert-Equal 'copilot_os_keychain_unverified' $copilotNoAuthPreflight.protocol_observations.authentication.source 'Copilot no-auth preflight reports diagnostic auth source identity' + Assert-True (-not [bool]$copilotNoAuthPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot no-auth preflight reports non-interactive auth readiness as false' $copilotNoAuthResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] - Assert-Equal 'failed' $copilotNoAuthResult.status 'Copilot no-auth execution failure is captured without a model request' - Assert-Equal 'copilot_os_keychain_or_github_cli_unverified' $copilotNoAuthResult.evidence.credential.source 'Copilot no-auth evidence does not claim authentication' + Assert-Equal 'incompatible' $copilotNoAuthResult.status 'Copilot no-auth execution fails before Phase 1 model execution' + Assert-Equal 'incompatible' ([string]$copilotNoAuthResult.evidence.preflight.status) 'Copilot no-auth execute result preserves the incompatible preflight evidence' + $nativeExecutionsAfterNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal $nativeExecutionsBeforeNoAuth.Count $nativeExecutionsAfterNoAuth.Count 'Copilot no-auth rejection does not start a model execution process' Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:COPILOT_HOME = $recordedOldCopilotHome } Write-Output "Real runner deterministic adapter conformance ($Suite): PASS" @@ -2036,6 +2117,11 @@ exit 2 $env:GITHUB_TOKEN = $recordedOldGithubToken $env:COPILOT_HOME = $recordedOldCopilotHome $env:GH_CONFIG_DIR = $recordedOldGhConfigDir + $env:HOME = $recordedOldHome + $env:USERPROFILE = $recordedOldUserProfile + $env:APPDATA = $recordedOldAppData + $env:LOCALAPPDATA = $recordedOldLocalAppData + $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome $env:AGENTIC_RECORDED_FIXTURES = $recordedOldFixtures if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } @@ -2401,6 +2487,7 @@ try { Assert-True ($prepareText.Contains('execution-freeze.json') -and $prepareText.Contains('grading.json') -and $prepareText.Contains('validate-eval-grading.ps1') -and $prepareText.Contains('apply-eval-grading.ps1') -and $prepareText.Contains('finalize-eval-package.ps1')) 'handoff preparation must expose the shared freeze, grading validation, grading application, and finalization boundaries' Assert-True ($prepareText.Contains('Read the selected runner descriptor and its `delegation.dispatch_owner`.') -and $prepareText.Contains('invoke-runner-owned-arms.ps1') -and $prepareText.Contains('package-computed Phase 1 allowance') -and $prepareText.Contains('must be started exactly once')) 'handoff preparation must expose one foreground Phase 1 invocation with a computed caller timeout' Assert-True ($prepareText.Contains('Do not create outer workers') -and $prepareText.Contains('edit raw result/evidence files')) 'handoff preparation must forbid outer runner-owned workers and raw evidence edits' + Assert-True ($prepareText.Contains('authentication incompatibility is terminal for this package iteration') -and $prepareText.Contains('Do not suggest switching to another runner') -and $prepareText.Contains('do not suggest starting another Orchestrator') -and $prepareText.Contains('do not rerun Phase 1')) 'handoff preparation must fail closed on Copilot authentication incompatibility without alternate-runner/orchestrator suggestions' Assert-True ($prepareText.Contains('The Grader may author exactly one package-root `grading.json`') -and $prepareText.Contains('It must not edit raw execution results')) 'handoff preparation must isolate the Grader to the grading-only artifact' Assert-True ($prepareText.Contains('Write `grading.json`, then validate it before finalization') -and $prepareText.Contains('Grading validation is retryable; finalization is not') -and $prepareText.Contains('only after grading validation succeeds')) 'handoff preparation must place retryable grading validation before exactly-once finalization' Assert-True ($prepareText.Contains('Return only its machine-readable JSON summary') -and $prepareText.Contains('Never repair, re-freeze, re-bridge a changed raw result')) 'handoff preparation must make finalizer success and fail-closed recovery explicit' @@ -2409,6 +2496,7 @@ try { $generatedHandoff = Invoke-GeneratedRunnerPrompt $generatedConcurrencyThreeHandoff = Invoke-GeneratedRunnerPrompt -RequestedConcurrency 3 Assert-True ($generatedHandoff.Contains('evaluation is incomplete and a fresh package/code fix is required') -and $generatedHandoff.Contains('Never patch package-local runner code') -and $generatedHandoff.Contains('delete orchestration state') -and $generatedHandoff.Contains('delete execution results') -and $generatedHandoff.Contains('delete or replace `execution-freeze.json`') -and $generatedHandoff.Contains('rerun Phase 1') -and $generatedHandoff.Contains('manually broaden a capability check')) 'generated handoff output forbids package-local repair, state deletion, retry, and manual capability broadening' + Assert-True ($generatedHandoff.Contains('authentication incompatibility is terminal for this package iteration') -and $generatedHandoff.Contains('Do not suggest switching to another runner') -and $generatedHandoff.Contains('do not suggest starting another Orchestrator') -and $generatedHandoff.Contains('do not rerun Phase 1')) 'generated handoff output must keep Copilot authentication failures fail-closed without fallback orchestration suggestions' Assert-True ($generatedHandoff.Contains('invoke-runner-owned-arms.ps1') -and $generatedHandoff.Contains('package-computed Phase 1 allowance') -and $generatedHandoff.Contains('must be started exactly once') -and $generatedHandoff.Contains('If execution is interrupted and no valid `execution-freeze.json` exists')) 'generated handoff exposes one foreground Phase 1 invocation with fail-closed interruption handling' Assert-True ($generatedHandoff.Contains('validate-eval-grading.ps1') -and $generatedHandoff.Contains('-ShowSkeleton') -and $generatedHandoff.Contains('Write `grading.json`, then validate it before finalization') -and $generatedHandoff.Contains('Grading validation is retryable; finalization is not') -and $generatedHandoff.Contains('only after grading validation succeeds')) 'generated handoff requires author -> validate/retry -> finalize exactly once' Assert-True ($generatedHandoff.Contains('allowance of 6240 seconds') -and $generatedHandoff.Contains('6 arm(s) × 120-second fixed model-free runner preflight timeout = 720-second serial preflight allowance') -and $generatedHandoff.Contains('5490-second execution allowance across 3 batch(es) at concurrency 2') -and $generatedHandoff.Contains('2 scripted user turn(s) × profile.timeout_seconds 900 + 30 seconds runner grace') -and $generatedHandoff.Contains('+ 30 seconds orchestration grace')) 'generated handoff uses the fixed preflight timeout, scripted-turn model timeout, execution batches, and separate orchestration grace' diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index d122b36..c160999 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -1860,6 +1860,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('Read the selected runner descriptor and its `delegation.dispatch_owner`. For runner-owned behavioral transport, invoke:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$runnerOwnedFanoutPath`" -IterationDirectory `"$IterationDirectory`"") [void]$builder.AppendLine('It performs every preflight before any execute process, preserves exact manifest paths, owns concurrency/backpressure, timeout/watchdog handling, terminal registration, orchestration evidence, and immutable `execution-freeze.json` before Phase 2. Consume its terminal JSON summary. If Phase 1 reports incompatible or fails, stop: the evaluation is incomplete and must fail closed. The evaluation is incomplete and a fresh package/code fix is required. Never patch package-local runner code, delete orchestration state, delete execution results, delete or replace `execution-freeze.json`, rerun Phase 1, or manually broaden a capability check. Do not create outer workers, execute an arm yourself, write orchestration state, or edit raw result/evidence files. If dispatch ownership is orchestrator-owned, use only the descriptor-declared native worker transport, the exact manifest paths, and then run the shared freeze boundary; do not synthesize or repair transport evidence. Only persisted runner-produced evidence at the manifest-declared paths may proceed.') + [void]$builder.AppendLine('For `github-copilot`, authentication incompatibility is terminal for this package iteration: report the missing supported non-interactive source (explicit `COPILOT_GITHUB_TOKEN`/`GH_TOKEN`/`GITHUB_TOKEN` or trusted `gh auth token` fallback), preserve the package, and stop. Do not suggest switching to another runner, do not suggest starting another Orchestrator, and do not rerun Phase 1.') [void]$builder.AppendLine('Workers receive only their isolated run directory. Keep the paired arm, metadata, expected output, assertions, grading, reports, and orchestration files out of Phase 1. Preserve runner-owned terminal results and all referenced raw transcript/event artifacts exactly as written.') [void]$builder.AppendLine() [void]$builder.AppendLine('## Phase 2 — grading and finalization') From 3b1703d442d841ff9df38910ac70a3bb70b0b267 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 10:10:06 +0200 Subject: [PATCH 08/18] =?UTF-8?q?=F0=9F=93=9D=20document=20copilot=20authe?= =?UTF-8?q?ntication=20incompatibility=20handling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update eval-runners README to explain Copilot authentication requirements and fail-closed behavior. Documents the fresh-context authentication testing strategy, the GitHub CLI token fallback mechanism, and the fail-closed incompatibility policy that prevents evaluation execution when required non-interactive authentication sources are unavailable. --- scripts/eval-runners/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index d3853fc..1c61842 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -300,12 +300,13 @@ in both paired arms. Personal Copilot configuration is excluded by run-local `COPILOT_HOME`, `COPILOT_CACHE_HOME`, `HOME`, `USERPROFILE`, and XDG roots; the runner does not copy the normal `.copilot` directory. Authentication prefers explicit `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`; when none is -present, the trusted runner may resolve `gh auth token` outside the worker and -inject only that token as a protected environment variable. Host `GH_CONFIG_DIR` -is never forwarded into the evaluated worker. `--secret-env-vars` removes every -listed token variable from shell and MCP child environments. Preflight does not -make a model request and therefore reports native keychain/service readiness as -conditional rather than claiming successful remote authentication. Codex's +present, the trusted runner resolves `gh auth token` outside the worker across +documented/default config candidates and injects only that token as a protected +environment variable. Host `GH_CONFIG_DIR` is never forwarded into the evaluated +worker. `--secret-env-vars` removes every listed token variable from shell and +MCP child environments. If neither explicit token variables nor trusted GitHub +CLI token fallback is resolvable, preflight fails closed before Phase 1 model +execution starts. Codex's compatibility API-key path uses `--ask-for-approval never` with `exec --sandbox danger-full-access`; subscription eval arms use the runner-owned app-server path described above. It does not combine explicit sandbox selection with From 7faba4ad894c95e70e36924871113a40b68da08c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 18:59:07 +0200 Subject: [PATCH 09/18] =?UTF-8?q?=F0=9F=92=AC=20document=20eval=20infrastr?= =?UTF-8?q?ucture=20and=20project=20conventions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update repository documentation with eval infrastructure details, workspace git scenario support, Copilot eval worker projection requirements, and grading contract specifications. These changes reflect improvements to the portable eval handoff and runner integrity requirements. --- README.md | 4 ++++ scripts/eval-runners/README.md | 8 ++++++++ 2 files changed, 12 insertions(+) diff --git a/README.md b/README.md index e8234ba..fbf3803 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,10 @@ One repo-wide convention matters especially for scaffolding skills: prefer dynam Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. +Git scenarios may use `workspace.git: true` for a staged initial commit, or a declarative object with `base_branch`, `feature_branch`, and `commits` (each containing a `message` and repository-relative `files` map; string values write UTF-8 content and null deletes a file). Preparation creates local `origin/` and `origin/HEAD` refs before applying feature commits, with fixed identity and timestamps. No remote access or setup commands are required. `dotnet-change-impact` eval 9 uses this to exercise default-base resolution against real paired Git history. + +Copilot eval workers now require physical projections outside package/source ancestry, preserving staged repository instructions while excluding ambient ancestor instructions. Captured boundary contradictions invalidate execution. Phase 2 explicitly follows the packaged Anthropic grader and requires assertion-specific, source-backed PASS evidence; generic completion statements cannot finalize. See the [runner integrity contract](scripts/eval-runners/README.md) for the projection limits and checkpoint telemetry rules. + Skill validation is local and deterministic. The Priority 1 **AI/LLM Evaluation Automation Prohibition** in `AGENTS.md` forbids repository preparation, validation, CI, hooks, deterministic tests, automatic fan-out, graders, and completion gates from using an authenticated Copilot, Claude, Codex, Gemini, or other model account. There is no repository opt-in switch. A human-selected external Eval Orchestrator may invoke an explicitly selected package-local Eval Runner for a package it was handed; that boundary never authorizes live model execution in CI or automatic repository workflows. Evaluation keeps Anthropic's `skill-creator` methodology and portable paired-run conventions while replacing only the execution transport. `run.json` remains the runner-neutral one-arm contract; `execution-profile.json` selects the runner/model/configuration; and `delegation.dispatch_owner` declares whether the orchestrator dispatches a native subagent/task or starts the runner-owned native execution surface directly. Orchestrator-owned envelopes pass through `record-native-result.ps1`; runner-owned transports produce `execution-result.json` directly before the existing `eval-result/2` bridge and reports. The user-facing preparation flow normalizes explicit harness intent before emitting `RUN-THIS.prompt.md`; the portable profile stores the internal runner id and the opaque runner-native model selector, with no provider field. The conforming real runners are GitHub Copilot, Codex, and OpenCode, with a deterministic fake runner used for conformance. GitHub Copilot CLI with `claude-haiku-4.5` is the Codebelt Reference evaluation configuration — a repository convention for economical, stable comparison, not an Anthropic default — and Codex defaults to `gpt-5.6-luna` with low reasoning. Package preparation validates the resolved model against the current harness catalog before writing the profile. OpenCode discovery mirrors every model exposed by all configured providers, preserves exact `provider/model` selectors, and retains availability only as presentation metadata; it never filters paid or unknown models. When OpenCode is selected without an explicit model, the user must choose from the discovered selectors before package preparation, and no first/free/recommended/previous model may be selected automatically. GitHub Copilot and Codex discovery lists all currently available models. Without an explicit one-shot request, the repository prepares the paired candidate and baseline inputs as a portable package and stops; the external Eval Orchestrator resolves, preflights, follows the selected dispatch owner, then grades and reports. Runners grant full operational permission inside each isolated behavioral harness configuration so the evaluated agent can perform ordinary engineering work noninteractively. Hard filesystem confinement is a separate optional outer capability: it raises reported isolation confidence from pragmatic to strict when proven, but it is not a universal platform prerequisite, so Windows is a first-class pragmatic target; mandatory experimental controls remain fail-closed. Native skill activation is not evaluated in v0.9.1. Adding or modifying a skill triggers package preparation automatically, as a completion gate an agent cannot skip: diff --git a/scripts/eval-runners/README.md b/scripts/eval-runners/README.md index 1c61842..976ea0e 100644 --- a/scripts/eval-runners/README.md +++ b/scripts/eval-runners/README.md @@ -139,6 +139,14 @@ Operational permission, native skill isolation, and filesystem isolation are dis Native delegation mechanisms: +Copilot execution requires an allowlisted physical temporary projection outside the logical package and source checkout ancestry. Only the arm's staged repository, prepared home, prompt and candidate (with-skill only) are copied. Repository `AGENTS.md` and `.github/copilot-instructions.md` remain enabled and identical in both arms. Ambient ancestor instructions and filesystem links make projection preflight fail closed. Runtime repository changes are synchronized back after link validation; captures remain parent-owned in the logical arm. Execution evidence records both boundaries. On hosts without an outer sandbox this prevents package discovery through the working-directory ancestry, but does not deny arbitrary absolute host paths: projection is not OS confinement. Structured tool events are a conservative contradiction detector; forbidden package/source accesses make the arm incompatible, and the bridge independently rejects contradictory completed transcripts. Native events remain available for diagnosis. + +Copilot `session.usage_checkpoint` billing counters are cumulative: the last checkpoint wins, including across resumed turns. `promptCacheBreakState` entries are per-call cache snapshots, deduplicated by native `model_call_id`; exposed prompt/cache buckets are used only when `assistant.usage` is absent. Tool schema tokens and native snapshots remain in evidence, not generated-token buckets. Missing output tokens stay unavailable, and premium requests/nano-AI units are never converted to currency. See [GitHub's event contract](https://docs.github.com/en/copilot/how-tos/copilot-sdk/features/streaming-events). + +Phase 2 must read and follow the exact packaged `tools/skill-creator/agents/grader.md` after the execution freeze and bridge. PASS evidence requires `Source: output` (or a captured run artifact path), `Quote: `, and `Reason: ` on separate lines. The deterministic validator checks source membership, verbatim quote presence, nonempty evidence and rejects reused or generic PASS evidence. It cannot establish semantic truth; the Grader must fail uncertain/unverified expectations. Execution evidence remains immutable and finalization remains exactly once. + +Historical integrity note: `dotnet-change-impact` Copilot iteration 4 is contaminated and must not be used as effectiveness evidence. Its missing `.external-handoff-started` is consistent with manual handoff, but the package records do not establish the original user request. The current helper reserves the marker before every external handoff and contains no removal path; no reservation-policy change is justified from absence alone. Codex iteration 2 and OpenCode iteration 3 have reservations. New protocol code applies only to newly prepared iterations. + - GitHub Copilot: runner-owned behavioral transport. The runner starts one fresh Copilot CLI session per eval execution (`copilot -C --model --output-format json`, prompt on stdin) and captures that From ca663cfa005757d5c4623db01c9939667b2c12d9 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 18:59:20 +0200 Subject: [PATCH 10/18] =?UTF-8?q?=F0=9F=91=B7=20enhance=20eval=20infrastru?= =?UTF-8?q?cture=20with=20grading=20and=20isolation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement boundary violation detection for Copilot eval workers with physical projection outside package ancestry. Add isolation.ps1 support, enhance grading contract to enforce source-backed assertion evidence, update eval schema with new requirements, and fix git environment variable preservation in eval workspace initialization. Add eval-git-workspace.ps1 to support declarative git scenario setup for evaluations with staged commits and repository history. --- scripts/eval-git-workspace.ps1 | 52 +++++ .../eval-runners/bridge-execution-result.ps1 | 3 + .../contracts/execution-result.schema.json | 13 +- .../contracts/grading.schema.json | 2 +- .../eval-runners/eval-grading-contract.ps1 | 46 +++- .../eval-runners/github-copilot/isolation.ps1 | 34 +++ .../eval-runners/github-copilot/runner.ps1 | 208 +++++++++++++++++- .../tests/test-copilot-boundaries.ps1 | 85 ++++++- .../eval-runners/tests/test-eval-request.ps1 | 1 + .../tests/test-integrity-finalization.ps1 | 25 ++- .../tests/test-runner-conformance.ps1 | 4 +- scripts/prepare-skill-evals.ps1 | 13 +- scripts/validate-skill-templates.ps1 | 19 +- 13 files changed, 470 insertions(+), 35 deletions(-) create mode 100644 scripts/eval-git-workspace.ps1 create mode 100644 scripts/eval-runners/github-copilot/isolation.ps1 diff --git a/scripts/eval-git-workspace.ps1 b/scripts/eval-git-workspace.ps1 new file mode 100644 index 0000000..0394705 --- /dev/null +++ b/scripts/eval-git-workspace.ps1 @@ -0,0 +1,52 @@ +# Declarative, local-only Git scenarios. No hooks, remotes, or model calls run. +function Assert-EvalGitScenario { + param([object]$Scenario) + if ($Scenario -is [bool]) { return } + if ($Scenario -isnot [pscustomobject]) { throw 'workspace.git must be a boolean or a declarative Git scenario.' } + $allowed = @('base_branch', 'feature_branch', 'commits') + foreach ($name in $Scenario.PSObject.Properties.Name) { if ($name -notin $allowed) { throw "Unknown workspace.git field '$name'." } } + foreach ($name in $allowed) { if ($name -notin $Scenario.PSObject.Properties.Name) { throw "workspace.git requires '$name'." } } + foreach ($branch in @($Scenario.base_branch, $Scenario.feature_branch)) { + if ($branch -isnot [string] -or $branch -notmatch '^[a-zA-Z0-9][a-zA-Z0-9/_-]*$' -or $branch.Contains('//') -or $branch.EndsWith('/')) { throw 'Invalid scenario branch name.' } + } + if ($Scenario.base_branch -eq $Scenario.feature_branch -or @($Scenario.commits).Count -lt 1) { throw 'Git scenario needs distinct base/feature branches and feature commits.' } + foreach ($commit in $Scenario.commits) { + if ([string]::IsNullOrWhiteSpace([string]$commit.message) -or $commit.files -isnot [pscustomobject]) { throw 'Each Git scenario commit requires message and files.' } + foreach ($name in $commit.PSObject.Properties.Name) { if ($name -notin @('message', 'files')) { throw "Unknown commit field '$name'." } } + foreach ($file in $commit.files.PSObject.Properties) { + if ($file.Name -notmatch '^[a-zA-Z0-9_][a-zA-Z0-9_./-]*$' -or $file.Name -match '(^|/)(\.\.?|\.git|bin|obj)(/|$)' -or $file.Name.EndsWith('/')) { throw "Unsafe Git scenario file '$($file.Name)'." } + if ($null -ne $file.Value -and $file.Value -isnot [string]) { throw 'Git scenario file content must be a string or null (delete).' } + } + } +} + +function Add-EvalGitScenario { + param([string]$RepoDirectory, [object]$Scenario) + Assert-EvalGitScenario -Scenario $Scenario + if ($Scenario -is [bool]) { return } + $identity = @('-c', 'user.name=Eval Harness', '-c', 'user.email=eval-harness@localhost', '-c', 'commit.gpgsign=false', '-c', 'core.autocrlf=false', '-c', 'core.hooksPath=') + function Invoke-ScenarioGit { param([string[]]$Arguments) + & git @identity -C $RepoDirectory @Arguments 2>$null | Out-Null + if ($LASTEXITCODE -ne 0) { throw "Git scenario command failed: $($Arguments -join ' ')" } + } + $oldAuthor = $env:GIT_AUTHOR_DATE; $oldCommitter = $env:GIT_COMMITTER_DATE + try { + $env:GIT_AUTHOR_DATE = '2020-01-02T00:00:00Z'; $env:GIT_COMMITTER_DATE = $env:GIT_AUTHOR_DATE + Invoke-ScenarioGit -Arguments @('branch', '-m', [string]$Scenario.base_branch) + # Local tracking refs and symbolic HEAD exercise default resolution + # without a network remote or paths back to the package. + Invoke-ScenarioGit -Arguments @('update-ref', "refs/remotes/origin/$($Scenario.base_branch)", 'HEAD') + Invoke-ScenarioGit -Arguments @('symbolic-ref', 'refs/remotes/origin/HEAD', "refs/remotes/origin/$($Scenario.base_branch)") + Invoke-ScenarioGit -Arguments @('checkout', '-b', [string]$Scenario.feature_branch, '--quiet') + foreach ($commit in $Scenario.commits) { + foreach ($file in $commit.files.PSObject.Properties) { + $path = [IO.Path]::GetFullPath((Join-Path $RepoDirectory $file.Name)) + if (-not $path.StartsWith([IO.Path]::GetFullPath($RepoDirectory) + [IO.Path]::DirectorySeparatorChar, [StringComparison]::OrdinalIgnoreCase)) { throw 'Git scenario path escaped repo.' } + if ($null -eq $file.Value) { if (Test-Path -LiteralPath $path) { Remove-Item -LiteralPath $path -Force } } + else { [void][IO.Directory]::CreateDirectory((Split-Path -Parent $path)); [IO.File]::WriteAllText($path, $file.Value, [Text.UTF8Encoding]::new($false)) } + } + Invoke-ScenarioGit -Arguments @('add', '-A') + Invoke-ScenarioGit -Arguments @('commit', '--quiet', '-m', [string]$commit.message) + } + } finally { $env:GIT_AUTHOR_DATE = $oldAuthor; $env:GIT_COMMITTER_DATE = $oldCommitter } +} diff --git a/scripts/eval-runners/bridge-execution-result.ps1 b/scripts/eval-runners/bridge-execution-result.ps1 index d46e8de..c04cac3 100644 --- a/scripts/eval-runners/bridge-execution-result.ps1 +++ b/scripts/eval-runners/bridge-execution-result.ps1 @@ -259,6 +259,9 @@ try { $artifactPaths.Add((Get-ResultRelativeArtifactPath -EvalDirectory $evalDirectory -FullPath $full)) } + . (Join-Path $PSScriptRoot 'github-copilot/isolation.ps1') + Assert-CopilotCapturedBoundary -Raw $raw -RunData $runData + if ($RequireNativeDelegation) { if ([string]$raw.status -eq 'incompatible') { throw 'An incompatible native-worker arm is diagnostic only and cannot be bridged into a gradeable canonical result.' diff --git a/scripts/eval-runners/contracts/execution-result.schema.json b/scripts/eval-runners/contracts/execution-result.schema.json index f0bf233..0e05500 100644 --- a/scripts/eval-runners/contracts/execution-result.schema.json +++ b/scripts/eval-runners/contracts/execution-result.schema.json @@ -185,8 +185,8 @@ "prompt_fidelity": { "const": true }, "prompt_sha256": { "type": "string", "pattern": "^[0-9a-fA-F]{64}$" }, "terminal_result_capture": { "const": true }, - "paired_arm_visible": { "const": false }, - "grading_material_visible": { "const": false }, + "paired_arm_visible": { "type": "boolean" }, + "grading_material_visible": { "type": "boolean" }, "nested_model_execution": { "const": false }, "model_execution_count": { "const": 1 } } @@ -197,5 +197,12 @@ "warnings": { "type": "array", "items": { "type": "string" } }, "compatibility_deviations": { "type": "array", "items": { "type": "string" } }, "attempt_count": { "const": 1 } - } + }, + "allOf": [{ + "if": { "properties": { "status": { "const": "completed" } } }, + "then": { "properties": { "evidence": { "properties": { "delegation": { "properties": { + "paired_arm_visible": { "const": false }, + "grading_material_visible": { "const": false } + } } } } } } + }] } diff --git a/scripts/eval-runners/contracts/grading.schema.json b/scripts/eval-runners/contracts/grading.schema.json index 23b4aac..f16fc12 100644 --- a/scripts/eval-runners/contracts/grading.schema.json +++ b/scripts/eval-runners/contracts/grading.schema.json @@ -20,7 +20,7 @@ "assertion_index": { "type": "integer", "minimum": 0 }, "assertion": { "type": "string", "minLength": 1 }, "passed": { "type": "boolean" }, - "evidence": { "type": "string" } + "evidence": { "type": "string", "minLength": 1, "pattern": "\\S" } } } } diff --git a/scripts/eval-runners/eval-grading-contract.ps1 b/scripts/eval-runners/eval-grading-contract.ps1 index c1a7127..0c8cd29 100644 --- a/scripts/eval-runners/eval-grading-contract.ps1 +++ b/scripts/eval-runners/eval-grading-contract.ps1 @@ -52,8 +52,33 @@ function Assert-EvalGradingEntryShape { if ([string]$Entry.configuration -notin @('with_skill', 'without_skill')) { throw "grading.json configuration '$($Entry.configuration)' is unsupported." } if ([string]::IsNullOrWhiteSpace([string]$Entry.eval_name) -or [string]::IsNullOrWhiteSpace([string]$Entry.assertion)) { throw 'grading.json eval_name and assertion must be non-empty strings.' } if ($Entry.passed -isnot [bool]) { throw 'grading.json passed must be a boolean; incomplete grading is not finalizable.' } - if ($Entry.evidence -isnot [string]) { throw 'grading.json evidence must be a string.' } -} + if ($Entry.evidence -isnot [string]) { throw 'grading.json evidence must be a string.' } + if ([string]::IsNullOrWhiteSpace($Entry.evidence)) { throw 'grading.json evidence must be non-empty.' } + if ($Entry.passed -and $Entry.evidence -notmatch '(?s)^Source: ([^\r\n]+)\r?\nQuote: (.+?)\r?\nReason: (\S.*)$') { + throw 'PASS evidence requires Source, a verbatim Quote, and an assertion-specific Reason on separate lines.' + } +} + +function Assert-EvalPassEvidence { + param([object]$Entry, [object]$Canonical, [object]$Record) + if (-not $Entry.passed) { return } + [void]($Entry.evidence -match '(?s)^Source: ([^\r\n]+)\r?\nQuote: (.+?)\r?\nReason: (\S.*)$') + $source = $Matches[1].Trim(); $quote = $Matches[2].Trim(); $reason = $Matches[3].Trim() + if ($source -eq 'output') { $content = [string]$Canonical.output } + else { + # Only native captured artifacts are admissible, never grading keys or + # grader-created files. Freeze validation already pins their bytes. + $raw = Read-RunnerJson -Path $Record.ExecutionResultPath + $artifacts = @($raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -ceq $source }) + if ($artifacts.Count -ne 1) { throw "PASS evidence source '$source' is not a captured run artifact." } + $path = Resolve-ContainedPath -BasePath (Split-Path -Parent $Record.RunManifestPath) -RelativePath $source -FieldName 'PASS evidence source' -Kind File + $content = [IO.File]::ReadAllText($path) + } + if ([string]::IsNullOrWhiteSpace($quote) -or -not $content.Contains($quote, [StringComparison]::Ordinal)) { throw 'PASS evidence quote is absent from its frozen source.' } + if ($reason -eq $quote -or $reason -eq $Entry.assertion -or $reason -match '^(?i:eval(?:uation)? completed(?: with output)?|passed|verified|as expected|done|looks good)[.!]?$') { + throw 'PASS evidence must explain how the cited observation establishes this assertion.' + } +} function Assert-EvalGradingContract { param( @@ -124,17 +149,26 @@ function Assert-EvalGradingContract { throw "grading.json assertion cardinality $($submitted.Count) does not match the required $($expected.Count)." } - $validated = @{} + $validated = @{} + $passEvidence = @{} foreach ($entry in $submitted) { Assert-EvalGradingEntryShape -Entry $entry $key = Get-EvalGradingEntryKey -Entry $entry if (-not $expected.ContainsKey($key)) { throw "grading.json identifies an unknown eval/configuration/assertion '$key'." } if ($validated.ContainsKey($key)) { throw "grading.json contains duplicate grading entry '$key'." } $target = $expected[$key] - if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion) { + if ([string]$entry.eval_name -ne [string]$target.eval_name -or [string]$entry.assertion -ne [string]$target.assertion) { throw "grading.json assertion identity '$key' does not match eval-metadata.json exactly." - } - $validated[$key] = $entry + } + $armKey = "$($entry.eval_id)|$($entry.configuration)" + $record = @($records | Where-Object { $_.EvalId -eq $entry.eval_id -and $_.Configuration -eq $entry.configuration })[0] + Assert-EvalPassEvidence -Entry $entry -Canonical $canonicalByKey[$armKey] -Record $record + if ($entry.passed) { + $evidenceKey = $armKey + '|' + ([regex]::Replace($entry.evidence.Trim(), '\s+', ' ')).ToLowerInvariant() + if ($passEvidence.ContainsKey($evidenceKey)) { throw 'Repeated PASS evidence across assertions is not assertion-specific.' } + $passEvidence[$evidenceKey] = $true + } + $validated[$key] = $entry } foreach ($key in $expected.Keys) { if (-not $validated.ContainsKey($key)) { throw "grading.json is missing required grading entry '$key'." } diff --git a/scripts/eval-runners/github-copilot/isolation.ps1 b/scripts/eval-runners/github-copilot/isolation.ps1 new file mode 100644 index 0000000..a847514 --- /dev/null +++ b/scripts/eval-runners/github-copilot/isolation.ps1 @@ -0,0 +1,34 @@ +function Find-CopilotBoundaryContradictions { + param([object]$Data, [object]$Projection) + # Structured tool arguments/results can contradict projection isolation; + # their absence never proves OS confinement. + $text = ($Data | ConvertTo-Json -Depth 100 -Compress).Replace('\\', '/').Replace('\', '/') + foreach ($root in @($Projection.PackageRoot, $Projection.SourceRepositoryRoot) | Where-Object { $_ }) { + if ($text.IndexOf(([string]$root).Replace('\', '/'), [StringComparison]::OrdinalIgnoreCase) -ge 0) { 'Tool event references forbidden package/source path.' } + } + if ($text -match '(?i)(eval-metadata\.json|(?:^|[/"\s])(?:with_skill|without_skill)(?:[/"\s]|$)|(?:\.\./)+(?:results|tools|progress)(?:/|"|\s)|(?:execution-freeze|orchestration-state|grading|benchmark)\.(?:json|md)|(?:skill-creator-)?report\.html|RUN-THIS\.prompt\.md|\.external-handoff-started)') { + 'Tool event references forbidden grading, paired-arm, or orchestration material.' + } +} + +function Assert-CopilotCapturedBoundary { + param([object]$Raw, [object]$RunData) + if ($Raw.runner.name -ne 'github-copilot' -or $Raw.status -ne 'completed') { return } + $paths = Get-JsonProperty -Object $Raw.evidence -Name execution_paths -Default $null + if (-not [bool](Get-JsonProperty -Object $paths -Name projection_proven -Default $false)) { throw 'Completed Copilot execution lacks a proven physical projection.' } + $package = Split-Path -Parent (Split-Path -Parent $RunData.RunRoot) + $source = [string](Get-JsonProperty -Object $paths -Name source_repository_root -Default '') + $physical = [string](Get-JsonProperty -Object $paths -Name physical_run_root -Default '') + if ([string]::IsNullOrWhiteSpace($physical) -or (Test-PathInside -BasePath $package -CandidatePath $physical) -or ($source -and (Test-PathInside -BasePath $source -CandidatePath $physical))) { throw 'Invalid Copilot physical projection boundary.' } + $proof = [pscustomobject]@{ PackageRoot = $package; SourceRepositoryRoot = $source } + $transcript = @($Raw.artifacts | Where-Object { $_.scope -eq 'run' -and $_.path -eq 'evidence/copilot-events.jsonl' }) + if ($transcript.Count -ne 1) { throw 'Copilot native transcript is missing.' } + $path = Resolve-ContainedPath -BasePath $RunData.RunRoot -RelativePath $transcript[0].path -FieldName 'Copilot transcript' -Kind File + $parsed = ConvertFrom-JsonLines -Text ([IO.File]::ReadAllText($path)) + if (@($parsed.Errors).Count) { throw 'Copilot transcript contains unparseable events; boundary inspection is incomplete.' } + foreach ($event in $parsed.Events) { + if ([string](Get-JsonProperty -Object $event -Name type -Default '') -match '^(tool\.|command\.)') { + if (@(Find-CopilotBoundaryContradictions -Data (Get-JsonProperty -Object $event -Name data -Default $null) -Projection $proof).Count) { throw 'Copilot transcript contradicts claimed isolation; grading is forbidden.' } + } + } +} diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index da6dc6b..49d61ee 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -27,12 +27,16 @@ param( $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest . (Join-Path $PSScriptRoot '..\runner-common.ps1') +. (Join-Path $PSScriptRoot 'isolation.ps1') # GitHub Copilot checks these token variables before its OS credential store and # GitHub CLI fallback. The values are forwarded only to the Copilot process; # --secret-env-vars removes them from shell and MCP child environments. $copilotAuthVariables = @('COPILOT_GITHUB_TOKEN', 'GH_TOKEN', 'GITHUB_TOKEN') $script:copilotHomeCleanupSafe = $true +$script:copilotLogicalInputs = $null +$script:copilotProjection = $null +$script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() function Invoke-CopilotProcess { param( @@ -566,6 +570,13 @@ function Get-CopilotPreflight { $warnings = [System.Collections.Generic.List[string]]::new() $profile = $Inputs.Profile $run = $Inputs.Run + try { + if ($null -eq $script:copilotProjection) { [void](Get-CopilotProjectionPlan -Inputs $Inputs) } + $checks.Add((New-PreflightCheck -Name 'physical_projection' -Status passed -Detail 'Allowlisted physical workspace outside package/source ancestry is required.')) + } catch { + $reasons.Add($_.Exception.Message) + $checks.Add((New-PreflightCheck -Name 'physical_projection' -Status failed -Detail $_.Exception.Message)) + } $commandInfo = Resolve-ExternalCommand -Name 'copilot' $platform = Get-PlatformName $sandboxInfo = if ($platform -eq 'linux') { Resolve-SandboxCommand -Name 'bwrap' } elseif ($platform -eq 'macos') { Resolve-SandboxCommand -Name 'sandbox-exec' } else { $null } @@ -808,6 +819,7 @@ function Write-CopilotCapture { [Parameter(Mandatory = $true)][AllowEmptyString()][string]$Text ) + if ($null -ne $script:copilotLogicalInputs) { $RunData = $script:copilotLogicalInputs } $path = Join-Path $RunData.Run.RunRoot ($RelativePath -replace '/', [System.IO.Path]::DirectorySeparatorChar) New-Item -ItemType Directory -Path (Split-Path -Parent $path) -Force | Out-Null [System.IO.File]::WriteAllText($path, $Text, [System.Text.UTF8Encoding]::new($false)) @@ -839,6 +851,8 @@ function Read-CopilotEvents { $usageNumToolCalls = 0 $usageSeen = $false $toolStarts = 0 + $lastCheckpoint = $null + $checkpointCalls = [ordered]@{} $sessionError = $null $eventCounts = @{} $sessionIds = [System.Collections.Generic.List[string]]::new() @@ -855,6 +869,10 @@ function Read-CopilotEvents { } if ($eventCounts.ContainsKey($eventType)) { $eventCounts[$eventType]++ } else { $eventCounts[$eventType] = 1 } $data = Get-JsonProperty -Object $event -Name 'data' -Default $null + if ($eventType -match '^(tool\.|command\.)' -and $null -ne $script:copilotProjection) { + $violations = @(Find-CopilotBoundaryContradictions -Data $data -Projection $script:copilotProjection) + foreach ($violation in $violations) { $script:copilotBoundaryViolations.Add($violation) } + } foreach ($eventSessionId in @(Get-CopilotEventSessionIds -Event $event)) { if ($sessionIds -notcontains $eventSessionId) { $sessionIds.Add($eventSessionId) } } @@ -865,6 +883,19 @@ function Read-CopilotEvents { if (-not [string]::IsNullOrWhiteSpace([string]$timestamp) -and $eventTimestamps -notcontains [string]$timestamp) { $eventTimestamps.Add([string]$timestamp) } } switch ($eventType) { + 'session.usage_checkpoint' { + # Billing is session-cumulative. Cache-break entries are last + # call snapshots, deduplicated by native call identity. + $lastCheckpoint = $data + foreach ($conversation in @(Get-JsonProperty -Object $data -Name 'promptCacheBreakState' -Default @())) { + $models = Get-JsonProperty -Object $conversation -Name 'models' -Default @{} + foreach ($name in @(Get-JsonPropertyNames -Object $models)) { + $call = Get-JsonProperty -Object $models -Name $name + $id = [string](Get-JsonProperty -Object $call -Name 'model_call_id' -Default '') + if ($id) { $checkpointCalls[$id] = $call } + } + } + } 'assistant.message' { $assistantMessageObserved = $true $content = [string](Get-JsonProperty -Object $data -Name 'content' -Default '') @@ -907,6 +938,14 @@ function Read-CopilotEvents { $finalText = [string]::Join("`n", $assistantContents) } $toolCalls = if ($toolStarts -gt 0) { $toolStarts } else { $usageNumToolCalls } + if (-not $usageSeen -and $checkpointCalls.Count -gt 0) { + foreach ($call in $checkpointCalls.Values) { + $usageInput = Add-NullableInt64 -Current $usageInput -Value (Get-JsonProperty -Object $call -Name 'prompt_tokens' -Default $null) + $usageCacheRead = Add-NullableInt64 -Current $usageCacheRead -Value (Get-JsonProperty -Object $call -Name 'cache_read' -Default $null) + $usageCacheWrite = Add-NullableInt64 -Current $usageCacheWrite -Value (Get-JsonProperty -Object $call -Name 'cache_write' -Default $null) + } + $usageSeen = $true + } return [pscustomobject]@{ FinalText = $finalText @@ -916,6 +955,8 @@ function Read-CopilotEvents { UsageOutput = $usageOutput UsageCacheRead = $usageCacheRead UsageCacheWrite = $usageCacheWrite + UsageCheckpoint = $lastCheckpoint + CheckpointCalls = @($checkpointCalls.Values) ToolCalls = $toolCalls SessionError = $sessionError EventCounts = $eventCounts @@ -1018,7 +1059,8 @@ function Invoke-CopilotScriptedExecute { $failureMessage = $null for ($turnIndex = 0; $turnIndex -lt $requestedTurns.Count; $turnIndex++) { - $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $Inputs.Run + $turnSourceRun = if ($null -ne $script:copilotLogicalInputs) { $script:copilotLogicalInputs.Run } else { $Inputs.Run } + $turnText = Get-InteractionTurnText -Turn $requestedTurns[$turnIndex] -RunData $turnSourceRun $arguments = @($baseArguments) $targetSessionId = $null if ($turnIndex -gt 0) { @@ -1239,8 +1281,8 @@ function Invoke-CopilotScriptedExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + paired_arm_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) + grading_material_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) nested_model_execution = $false model_execution_count = 1 same_session_continuation = [bool]$terminalCapture @@ -1261,13 +1303,167 @@ function Invoke-CopilotScriptedExecute { $resultFinalResponse = if ($status -eq 'completed') { $finalText } else { $null } $resultFinalResponseReason = if ($status -eq 'completed') { $null } else { 'native_interaction_incompatible' } $result = New-ExecutionResult -Descriptor $ExecutionDescriptor -Profile $Inputs.Profile -Run $Inputs.Run -Status $status -FinalResponse $resultFinalResponse -FinalResponseReason $resultFinalResponseReason -StartedUtc $firstProcess.StartedUtc.ToString('o') -FinishedUtc $finished.ToString('o') -DurationSeconds $durationSeconds -ExitStatus $exitStatus -Failure $failure -SessionId $capturedSessionId -IsolationCapabilities (Get-CopilotCapabilityMap -Inputs $Inputs -HardFilesystemConfinement $hardFilesystem -ContinuationCapability $continuationCapability) -IsolationMechanisms @($mechanisms) -ResolvedConfiguration ([ordered]@{ status = 'accepted_request'; reason = 'Copilot accepted the requested model alias and configuration; scripted turns retained the exact requested model on every invocation.'; observations = [ordered]@{ model = $Inputs.Profile.Model; observed_models = @($observedModels.ToArray()); continuation_flag = $continuationCapability.Flag } }) -Telemetry $telemetry -Artifacts @($artifacts.ToArray()) -Warnings @($warnings.ToArray()) -Evidence $evidence -AttemptCount 1 - if ($status -eq 'completed') { [void](Assert-InteractionResultEvidence -ExecutionResult $result -RunData $Inputs.Run) } + if ($status -eq 'completed') { + $validationRun = if ($null -ne $script:copilotLogicalInputs) { $script:copilotLogicalInputs.Run } else { $Inputs.Run } + [void](Assert-InteractionResultEvidence -ExecutionResult $result -RunData $validationRun) + } return $result } function Invoke-CopilotExecute { param([Parameter(Mandatory = $true)][object]$Inputs) + # Keep all orchestration data in the parent process. Only repo, prepared + # home, prompt bytes and this arm's candidate cross the projection boundary. + $plan = Get-CopilotProjectionPlan -Inputs $Inputs + $logicalInputs = $Inputs + $script:copilotLogicalInputs = $Inputs + $script:copilotProjection = $plan + $script:copilotBoundaryViolations = [System.Collections.Generic.List[string]]::new() + [void](New-Item -ItemType Directory -Path $plan.Root -ErrorAction Stop) + try { + $physicalRun = $Inputs.Run.PSObject.Copy() + $physicalRun.RunRoot = $plan.Root + foreach ($field in @('WorkingDirectoryPath', 'HomeDirectoryPath', 'SkillDirectoryPath')) { + $source = [string]$Inputs.Run.$field + if ([string]::IsNullOrWhiteSpace($source)) { continue } + $relative = [IO.Path]::GetRelativePath($Inputs.Run.RunRoot, $source) + $destination = Join-Path $plan.Root $relative + Copy-CopilotProjectionTree -Source $source -Destination $destination + $physicalRun.$field = $destination + } + # Future scripted inputs stay parent-owned even when staged in repo/home. + if ($null -ne $Inputs.Run.Interaction) { + foreach ($turn in $Inputs.Run.Interaction.turns) { + $source = [string](Get-JsonProperty -Object $turn -Name source -Default '') + if ($source) { + Assert-SafeRelativePath -RelativePath $source -FieldName 'interaction source' + $path = Join-Path $plan.Root $source + if (Test-Path -LiteralPath $path -PathType Leaf) { Remove-Item -LiteralPath $path -Force } + } + } + } + $physicalRun.PromptPath = Join-Path $plan.Root 'prompt.md' + [IO.File]::WriteAllBytes($physicalRun.PromptPath, $Inputs.Run.PromptBytes) + $projectedFiles = @(Get-ChildItem -LiteralPath $physicalRun.WorkingDirectoryPath -Recurse -Force -File | ForEach-Object { [IO.Path]::GetRelativePath($physicalRun.WorkingDirectoryPath, $_.FullName) }) + $physicalInputs = [pscustomobject]@{ Run = $physicalRun; Profile = $Inputs.Profile } + $result = Invoke-CopilotWithPreparedHome -Inputs $physicalInputs -Action { Invoke-CopilotProjectedExecute -Inputs $physicalInputs } + $result.evidence.execution_paths = [ordered]@{ + projection = 'physical_temp_outside_logical_package' + logical_run_root = $logicalInputs.Run.RunRoot + logical_working_directory = $logicalInputs.Run.WorkingDirectoryPath + logical_home_directory = $logicalInputs.Run.HomeDirectoryPath + physical_run_root = $plan.Root + physical_working_directory = $physicalRun.WorkingDirectoryPath + physical_home_directory = $physicalRun.HomeDirectoryPath + source_repository_root = $plan.SourceRepositoryRoot + projection_proven = $true + hard_filesystem_confinement = [bool]$result.isolation.hard_filesystem_confinement + } + $result.evidence.boundary = [ordered]@{ proof = 'allowlisted_physical_projection'; contradictions = @($script:copilotBoundaryViolations.ToArray()); event_inspection = 'contradiction_detector_not_confinement' } + $capturePath = Join-Path $logicalInputs.Run.RunRoot 'evidence/copilot-events.jsonl' + if (Test-Path -LiteralPath $capturePath) { + $usageWarnings = [System.Collections.Generic.List[string]]::new() + # Reparse the combined native stream once, so resumed cumulative + # checkpoints and repeated call snapshots are never summed twice. + $usage = Read-CopilotEvents -Parsed (ConvertFrom-JsonLines -Text ([IO.File]::ReadAllText($capturePath))) -Warnings $usageWarnings + if ($usage.ParseErrorCount -gt 0) { $script:copilotBoundaryViolations.Add('Unparseable native events prevent complete boundary inspection.') } + $result.evidence.usage_checkpoint = $usage.UsageCheckpoint + $result.evidence.usage_checkpoint_calls = $usage.CheckpointCalls + if ($usage.UsageSeen) { + $buckets = [ordered]@{} + foreach ($mapping in @(@('input_tokens', 'UsageInput'), @('output_tokens', 'UsageOutput'), @('cache_read_tokens', 'UsageCacheRead'), @('cache_write_tokens', 'UsageCacheWrite'))) { + if ($null -ne $usage.($mapping[1])) { $buckets[$mapping[0]] = [int64]$usage.($mapping[1]) } + } + if ($buckets.Count) { $result.telemetry.tokens = New-AvailableMetric -Value $buckets } + } + } + if ($script:copilotBoundaryViolations.Count -gt 0) { + $result.status = 'incompatible' + $result.isolation.status = 'unverified' + $result.isolation.level = 'unsupported' + $result.isolation.hard_filesystem_confinement = $false + $result.exit.failure = New-ExecutionFailure -Code 'isolation_violation' -Message ([string]::Join('; ', $script:copilotBoundaryViolations)) + if ($result.evidence.Contains('delegation')) { + $result.evidence.delegation.paired_arm_visible = $true + $result.evidence.delegation.grading_material_visible = $true + } + } + $result.evidence.boundary.contradictions = @($script:copilotBoundaryViolations | Select-Object -Unique) + # Do not copy runtime links back into the logical package. + Assert-CopilotProjectionTree -Path $physicalRun.WorkingDirectoryPath + Assert-CopilotProjectionTree -Path $logicalInputs.Run.WorkingDirectoryPath + foreach ($file in @(Get-ChildItem -LiteralPath $logicalInputs.Run.WorkingDirectoryPath -Recurse -Force -File)) { + $relative = [IO.Path]::GetRelativePath($logicalInputs.Run.WorkingDirectoryPath, $file.FullName) + if ($projectedFiles -contains $relative -and -not (Test-Path -LiteralPath (Join-Path $physicalRun.WorkingDirectoryPath $relative))) { + if (-not (Test-PathInside -BasePath $logicalInputs.Run.WorkingDirectoryPath -CandidatePath $file.FullName)) { throw 'Unsafe projected output deletion.' } + Remove-Item -LiteralPath $file.FullName -Force + } + } + Copy-CopilotProjectionTree -Source $physicalRun.WorkingDirectoryPath -Destination $logicalInputs.Run.WorkingDirectoryPath + return $result + } finally { + if (-not $script:copilotHomeCleanupSafe) { throw 'Copilot process termination is unproven; projection retained and execution fails closed.' } + if (-not (Test-PathInside -BasePath $plan.Parent -CandidatePath $plan.Root) -or [IO.Path]::GetFileName($plan.Root) -notmatch '^agentic-copilot-projection-[0-9a-f]{32}$') { throw 'Unsafe Copilot projection cleanup path.' } + Remove-Item -LiteralPath $plan.Root -Recurse -Force + $script:copilotLogicalInputs = $null + $script:copilotProjection = $null + } +} + +function Assert-CopilotProjectionTree { + param([string]$Path) + $item = Get-Item -LiteralPath $Path -Force + if ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) { throw "Copilot projection refuses filesystem link '$Path'." } + if ($item.PSIsContainer) { + foreach ($child in Get-ChildItem -LiteralPath $Path -Force) { Assert-CopilotProjectionTree -Path $child.FullName } + } +} + +function Copy-CopilotProjectionTree { + param([string]$Source, [string]$Destination) + Assert-CopilotProjectionTree -Path $Source + [void][IO.Directory]::CreateDirectory($Destination) + foreach ($child in Get-ChildItem -LiteralPath $Source -Force) { Copy-Item -LiteralPath $child.FullName -Destination $Destination -Recurse -Force } +} + +function Get-CopilotProjectionPlan { + param([object]$Inputs) + $sourceRoot = $null + $packageRoot = Split-Path -Parent (Split-Path -Parent $Inputs.Run.RunRoot) + foreach ($start in @($Inputs.Run.RunRoot, $PSScriptRoot)) { + $cursor = $start + while ($cursor) { + if (Test-Path -LiteralPath (Join-Path $cursor '.git')) { $sourceRoot = $cursor; break } + $cursor = Split-Path -Parent $cursor + } + if ($sourceRoot) { break } + } + $parent = [IO.Path]::GetFullPath([IO.Path]::GetTempPath()) + foreach ($forbidden in @($packageRoot, $sourceRoot) | Where-Object { $_ }) { + if (Test-PathInside -BasePath $forbidden -CandidatePath $parent) { throw 'Copilot cannot establish a physical projection outside package/source ancestry.' } + } + # A linked temp parent or ambient instruction file would invalidate ancestry + # isolation. Fail closed rather than disable legitimate repo instructions. + $cursor = $parent + while ($cursor) { + $item = Get-Item -LiteralPath $cursor -Force + if ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) { throw 'Copilot projection parent contains a filesystem link.' } + foreach ($instruction in @('AGENTS.md', '.github/copilot-instructions.md', '.github/instructions', '.git')) { + if (Test-Path -LiteralPath (Join-Path $cursor $instruction)) { throw "Copilot projection would inherit ambient '$instruction'." } + } + $cursor = Split-Path -Parent $cursor + } + foreach ($path in @($Inputs.Run.WorkingDirectoryPath, $Inputs.Run.HomeDirectoryPath, $Inputs.Run.SkillDirectoryPath) | Where-Object { $_ }) { Assert-CopilotProjectionTree -Path $path } + foreach ($indirection in @('.git', '.git/objects/info/alternates', '.git/commondir')) { + if (Test-Path -LiteralPath (Join-Path $Inputs.Run.WorkingDirectoryPath $indirection) -PathType Leaf) { throw "Copilot projection refuses external Git indirection '$indirection'." } + } + return [pscustomobject]@{ Root = Join-Path $parent ('agentic-copilot-projection-' + [Guid]::NewGuid().ToString('N')); Parent = $parent; PackageRoot = $packageRoot; SourceRepositoryRoot = $sourceRoot } +} + +function Invoke-CopilotProjectedExecute { + param([Parameter(Mandatory = $true)][object]$Inputs) + $preflight = Get-CopilotPreflight -Inputs $Inputs $started = [DateTime]::UtcNow $sessionId = [Guid]::NewGuid().ToString('D') @@ -1424,8 +1620,8 @@ function Invoke-CopilotExecute { prompt_fidelity = $true prompt_sha256 = [string]$Inputs.Run.PromptHash terminal_result_capture = [bool]$terminalCapture - paired_arm_visible = $false - grading_material_visible = $false + paired_arm_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) + grading_material_visible = ($null -eq $script:copilotProjection -or $script:copilotBoundaryViolations.Count -gt 0) nested_model_execution = $false model_execution_count = 1 } diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index 90d205b..eefa54b 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -9,6 +9,7 @@ Set-StrictMode -Version Latest $runnerRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path . (Join-Path $runnerRoot 'runner-common.ps1') . (Join-Path $runnerRoot 'execution-freeze.ps1') +. (Join-Path $runnerRoot 'github-copilot/isolation.ps1') # Import definitions only: neither adapter dispatch nor a real CLI is invoked. foreach ($file in @('github-copilot/runner.ps1', 'bridge-execution-result.ps1', 'tests/test-runner-conformance.ps1', '../generate-eval-report.ps1')) { @@ -73,6 +74,14 @@ $gitExit = $LASTEXITCODE $record = [ordered]@{ turn = $turn; arguments = $arguments; cwd = (Get-Location).Path; ceiling = $env:GIT_CEILING_DIRECTORIES; cache = $env:COPILOT_CACHE_HOME; xdg = $env:XDG_CACHE_HOME; localappdata = $env:LOCALAPPDATA; appdata = $env:APPDATA; git_exit = $gitExit; git_root = [string]$gitResult; old_cache_exists = Test-Path -LiteralPath (Join-Path $runHome '.copilot-cache'); double_suffix_exists = Test-Path -LiteralPath (Join-Path $env:COPILOT_CACHE_HOME 'copilot') } [IO.File]::AppendAllText((Join-Path (Get-Location).Path 'fake-log.jsonl'), (($record | ConvertTo-Json -Compress) + "`n")) [IO.File]::WriteAllText((Join-Path (Get-Location).Path 'task-output.txt'), 'keep repo output') +$probe = [ordered]@{ metadata = Test-Path ../../eval-metadata.json; paired = (Test-Path ../../with_skill) -or (Test-Path ../../without_skill); candidate = Test-Path ../skill; staged_agents = Get-Content AGENTS.md -Raw; staged_copilot = Get-Content .github/copilot-instructions.md -Raw; ambient = @() } +$cursor = Split-Path -Parent (Get-Location).Path +while ($cursor) { + foreach ($name in @('AGENTS.md', '.github/copilot-instructions.md')) { if (Test-Path -LiteralPath (Join-Path $cursor $name)) { $probe.ambient += [IO.File]::ReadAllText((Join-Path $cursor $name)) } } + $cursor = Split-Path -Parent $cursor +} +[IO.File]::WriteAllText((Join-Path (Get-Location).Path 'projection-probe.json'), ($probe | ConvertTo-Json)) +if ($inputText -eq 'violation') { '{"type":"tool.execution_start","data":{"toolName":"view","arguments":{"path":"../../eval-metadata.json"}}}' } if ($inputText -eq 'timeout') { [Console]::Out.WriteLine('{"type":"session.start","data":{"sessionId":"fixture-session"}}'); [Console]::Out.Flush(); Start-Sleep -Seconds 30 } $text = if ($turn -eq 1) { 'Confirm before generation. Δ' } else { "Generated successfully.`nExact terminal text." } @{ type = 'session.start'; data = @{ sessionId = 'fixture-session' } } | ConvertTo-Json -Compress @@ -90,12 +99,19 @@ if ($inputText -eq 'failure') { exit 7 } $profilePath = Join-Path $iteration 'execution-profile.json' Write-TestJson -Path $profilePath -Value @{ schema = (Get-RunnerSchemaNames).Profile; runner = 'github-copilot'; model = 'fixture-model'; reasoning_effort = $null; configuration_profile = 'isolated-default'; tool_profile = 'default'; timeout_seconds = 15; concurrency = 1 } foreach ($run in @($with, $without)) { + [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'repo/.github')) + [IO.File]::WriteAllText((Join-Path $run.Root 'repo/AGENTS.md'), 'STAGED_INSTRUCTIONS_CANARY') + [IO.File]::WriteAllText((Join-Path $run.Root 'repo/.github/copilot-instructions.md'), 'STAGED_COPILOT_CANARY') [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'home/baseline/empty')) [IO.File]::WriteAllBytes((Join-Path $run.Root 'home/baseline/nested.bin'), [byte[]]@(0, 255, 10, 13, 42)) [IO.File]::WriteAllBytes((Join-Path $run.Root 'home/baseline/empty.bin'), [byte[]]@()) [void][IO.Directory]::CreateDirectory((Join-Path $run.Root 'evidence')) [IO.File]::WriteAllText((Join-Path $run.Root 'evidence/prepared.txt'), 'keep evidence') } + [IO.File]::WriteAllText((Join-Path $testRoot 'AGENTS.md'), 'FORBIDDEN_SOURCE_INSTRUCTIONS_CANARY') + [void][IO.Directory]::CreateDirectory((Join-Path $testRoot '.github')) + [IO.File]::WriteAllText((Join-Path $testRoot '.github/copilot-instructions.md'), 'FORBIDDEN_COPILOT_CANARY') + [IO.File]::WriteAllText((Join-Path (Split-Path -Parent $with.Root) 'eval-metadata.json'), 'FORBIDDEN_GRADING_CANARY') $inputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $with.Path; Profile = Resolve-ExecutionProfile -ProfilePath $profilePath } $baseline = Get-CopilotHomeBaseline -HomePath $inputs.Run.HomeDirectoryPath $baselineHash = Get-TestTreeHash -Root $inputs.Run.HomeDirectoryPath @@ -117,18 +133,19 @@ if ($inputText -eq 'failure') { exit 7 } $records = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo/fake-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json }) Assert-Equal 2 $records.Count 'both turns execute with shared runtime state' foreach ($record in $records) { - Assert-Equal $with.Root $record.ceiling 'exact child environment ceiling is the manifest RunRoot' + Assert-Equal $result.evidence.execution_paths.physical_run_root $record.ceiling 'child ceiling is the physical run root' + Assert-True (-not (Test-PathInside -BasePath $testRoot -CandidatePath $record.cwd)) 'physical cwd excludes source and package ancestry' Assert-True ($record.git_exit -ne 0) 'child git cannot discover outer/.git from non-Git staged repo' - Assert-Equal $inputs.Run.WorkingDirectoryPath $record.cwd 'child working directory retained' + Assert-Equal $result.evidence.execution_paths.physical_working_directory $record.cwd 'child uses projected working directory' $argsList = [string[]]$record.arguments - Assert-Equal $inputs.Run.WorkingDirectoryPath $argsList[[Array]::IndexOf($argsList, '-C') + 1] '-C retained' + Assert-Equal $record.cwd $argsList[[Array]::IndexOf($argsList, '-C') + 1] '-C agrees with process cwd' Assert-Equal 'fixture-model' $argsList[[Array]::IndexOf($argsList, '--model') + 1] 'model lock retained on every turn' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.cache/copilot') $record.cache 'complete Copilot cache override, no duplicate suffix' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.cache') $record.xdg 'XDG cache root' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.cache/copilot') $record.cache 'complete Copilot cache override, no duplicate suffix' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.cache') $record.xdg 'XDG cache root' Assert-True (-not $record.old_cache_exists -and -not $record.double_suffix_exists) 'no old or double-suffixed cache is created during execution' if ($IsWindows) { Assert-Equal $record.xdg $record.localappdata 'Windows cache fallback converges' - Assert-Equal (Join-Path $inputs.Run.HomeDirectoryPath '.config') $record.appdata 'Windows config root isolated' + Assert-Equal (Join-Path $result.evidence.execution_paths.physical_home_directory '.config') $record.appdata 'Windows config root isolated' } } Assert-True (@($records[1].arguments) -contains '--resume=fixture-session') 'exact session continuation retained' @@ -148,12 +165,13 @@ if ($inputText -eq 'failure') { exit 7 } if ($LASTEXITCODE -ne 0) { throw 'Could not initialize staged fixture Git repository.' } $singleInputs = [pscustomobject]@{ Run = Resolve-RunContract -RunPath $without.Path; Profile = $inputs.Profile } $singleBaselineHash = Get-TestTreeHash -Root $singleInputs.Run.HomeDirectoryPath - foreach ($scenario in @('completed', 'failure', 'timeout')) { + foreach ($scenario in @('completed', 'failure', 'timeout', 'violation')) { $singleInputs.Run.PromptBytes = [Text.Encoding]::UTF8.GetBytes($scenario) $singleInputs.Profile.TimeoutSeconds = if ($scenario -eq 'timeout') { 2 } else { 15 } $singleResult = Invoke-CopilotWithPreparedHome -Inputs $singleInputs -Action { Invoke-CopilotExecute -Inputs $singleInputs } - $expectedStatus = if ($scenario -eq 'failure') { 'failed' } elseif ($scenario -eq 'timeout') { 'timed_out' } else { 'completed' } + $expectedStatus = if ($scenario -eq 'violation') { 'incompatible' } elseif ($scenario -eq 'failure') { 'failed' } elseif ($scenario -eq 'timeout') { 'timed_out' } else { 'completed' } Assert-Equal $expectedStatus $singleResult.status "$scenario terminal status" + [void](Assert-ExecutionResult -Result $singleResult) Assert-Equal $singleBaselineHash (Get-TestTreeHash -Root $singleInputs.Run.HomeDirectoryPath) "$scenario restores prepared baseline" Assert-True (-not (Test-Path -LiteralPath (Join-Path $without.Root 'home/.cache'))) "$scenario removes generated cache" Assert-Equal 'keep repo output' ([IO.File]::ReadAllText((Join-Path $without.Root 'repo/task-output.txt'))) "$scenario preserves repo output" @@ -162,9 +180,58 @@ if ($inputText -eq 'failure') { exit 7 } $singleRecords = @(Get-Content -LiteralPath (Join-Path $without.Root 'repo/fake-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json }) foreach ($record in $singleRecords) { Assert-Equal 0 $record.git_exit 'Git still discovers staged repo/.git' - Assert-Equal ((Join-Path $without.Root 'repo').Replace('\', '/')) ($record.git_root.Replace('\', '/')) 'Git returns staged root' + Assert-Equal ($record.cwd.Replace('\', '/')) ($record.git_root.Replace('\', '/')) 'Git returns projected staged root' Assert-True ($record.cache -ne $records[0].cache) 'paired arms never share cache' } + Assert-True $singleResult.evidence.delegation.grading_material_visible 'contradiction prevents false invisibility claim' + $forged = $singleResult | ConvertTo-Json -Depth 100 | ConvertFrom-Json + $forged.status = 'completed' + $forged.evidence.delegation.paired_arm_visible = $false + $forged.evidence.delegation.grading_material_visible = $false + Assert-Rejected { Assert-CopilotCapturedBoundary -Raw $forged -RunData $singleInputs.Run } 'bridge rejects transcript contradiction despite false invisibility flags' + $proof = [pscustomobject]@{ Root = 'C:/temp/projection'; PackageRoot = 'C:/source/.bot/package'; SourceRepositoryRoot = 'C:/source' } + foreach ($path in @('../../eval-metadata.json', '../../with_skill/repo', '../../without_skill/repo', '../../results/arm.json', '../../grading.json', '../../execution-freeze.json', '../../orchestration-state.json', '../../report.html', 'C:\source\AGENTS.md')) { + Assert-True (@(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = $path } } -Projection $proof).Count -gt 0) "captured forbidden access rejected: $path" + } + Assert-Equal 0 @(Find-CopilotBoundaryContradictions -Data @{ arguments = @{ path = 'src/Widget.cs' } } -Projection $proof).Count 'ordinary staged source is allowed' + $warnings = [Collections.Generic.List[string]]::new() + $checkpoint = @{ type = 'session.usage_checkpoint'; data = @{ totalPremiumRequests = 0.33; totalNanoAiu = 10; promptCacheBreakState = @(@{ models = @{ model = @{ model_call_id = 'call-1'; prompt_tokens = 100; cache_read = 70; cache_write = 20; tool_tokens = 15 } } }) } } + $last = $checkpoint | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $last.data.totalPremiumRequests = 0.66 + $last.data.totalNanoAiu = 20 + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, $last, $last); Errors = @() } -Warnings $warnings + Assert-Equal 100 $usage.UsageInput 'repeated call snapshots are not summed' + Assert-Equal 70 $usage.UsageCacheRead 'cache read bucket retained without duplication' + Assert-Equal 20 $usage.UsageCacheWrite 'cache write bucket retained without duplication' + Assert-Equal 0.66 $usage.UsageCheckpoint.totalPremiumRequests 'final cumulative premium request counter is authoritative' + Assert-Equal 20 $usage.UsageCheckpoint.totalNanoAiu 'final cumulative nano AI units retained without currency conversion' + Assert-True ($null -eq $usage.UsageOutput) 'unexposed output tokens remain unavailable' + Assert-Equal 15 $usage.CheckpointCalls[0].tool_tokens 'tool schema tokens retained in native evidence only' + $secondCall = $last | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $secondCall.data.promptCacheBreakState[0].models.model.model_call_id = 'call-2' + $secondCall.data.promptCacheBreakState[0].models.model.prompt_tokens = 120 + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, $last, $secondCall, $secondCall); Errors = @() } -Warnings $warnings + Assert-Equal 220 $usage.UsageInput 'distinct native calls count once each' + $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, @{ type = 'assistant.usage'; data = @{ inputTokens = 7; outputTokens = 3 } }); Errors = @() } -Warnings $warnings + Assert-Equal 7 $usage.UsageInput 'native assistant usage takes precedence over cache snapshots' + Assert-Equal 3 $usage.UsageOutput 'actual exposed output count retained' + $savedTemp = $env:TEMP; $savedTmp = $env:TMP + try { + $env:TEMP = $testRoot; $env:TMP = $testRoot + Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'temp inside source ancestry fails closed' + } finally { $env:TEMP = $savedTemp; $env:TMP = $savedTmp } + $runtimeLink = Join-Path $singleInputs.Run.WorkingDirectoryPath 'forbidden-link' + New-Item -ItemType $linkType -Path $runtimeLink -Target $testRoot | Out-Null + try { Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'linked projection input fails closed' } + finally { (Get-Item -LiteralPath $runtimeLink -Force).Delete() } + foreach ($arm in @($with, $without)) { + $probe = Get-Content (Join-Path $arm.Root 'repo/projection-probe.json') -Raw | ConvertFrom-Json + Assert-True (-not $probe.metadata -and -not $probe.paired) 'parent metadata and sibling arms are unreachable through projected ancestry' + Assert-Equal 0 @($probe.ambient).Count 'source AGENTS and Copilot instructions are excluded' + Assert-Equal 'STAGED_INSTRUCTIONS_CANARY' $probe.staged_agents 'staged AGENTS preserved identically' + Assert-Equal 'STAGED_COPILOT_CANARY' $probe.staged_copilot 'staged Copilot instructions preserved identically' + Assert-Equal ($arm.Root -eq $with.Root) $probe.candidate 'candidate material is only projected for with_skill' + } $transcript = Get-PortableTranscript -Raw $result -RunData $inputs.Run foreach ($text in @((Get-InteractionTurnText -Turn $inputs.Run.Interaction.turns[0] -RunData $inputs.Run), 'Confirm before generation. Δ', 'Yes, generate the key.', "Generated successfully.`nExact terminal text.", "Session: fixture-session`nSame session: true")) { diff --git a/scripts/eval-runners/tests/test-eval-request.ps1 b/scripts/eval-runners/tests/test-eval-request.ps1 index 84c51f1..81841e7 100644 --- a/scripts/eval-runners/tests/test-eval-request.ps1 +++ b/scripts/eval-runners/tests/test-eval-request.ps1 @@ -17,6 +17,7 @@ function Invoke-FakeHost($Decision) { # The host's only input is the actual canonical handoff file, never an arm prompt. Assert-True (Test-Path -LiteralPath $Decision.prompt_path -PathType Leaf) 'Host received a missing handoff.' Assert-True ([IO.Path]::GetFileName($Decision.prompt_path) -ceq 'RUN-THIS.prompt.md') 'Host received an arm.' + Assert-True (Test-Path -LiteralPath (Join-Path (Split-Path -Parent $Decision.prompt_path) '.external-handoff-started')) 'External handoff must be reserved before the host launch, for every runner.' $script:dispatches.Add($Decision.prompt_path) } } diff --git a/scripts/eval-runners/tests/test-integrity-finalization.ps1 b/scripts/eval-runners/tests/test-integrity-finalization.ps1 index a32bb4d..e27ad94 100644 --- a/scripts/eval-runners/tests/test-integrity-finalization.ps1 +++ b/scripts/eval-runners/tests/test-integrity-finalization.ps1 @@ -191,7 +191,7 @@ function New-TestGradingDocument { assertion_index = $index assertion = [string]$assertions[$index] passed = $true - evidence = 'deterministic grading-isolation fixture evidence' + evidence = "Source: output`nQuote: $((Read-TestJson -Path $record.ResultPath).output)`nReason: The captured fixture response establishes assertion $index for this deterministic transport case." }) } } @@ -417,7 +417,7 @@ for ($index = 0; $index -lt $count; $index++) { eval_name = $evalName prompt = "fixture prompt $evalId" expected_output = 'fixture output' - assertions = @($assertion) + assertions = @($assertion, 'the response contains the fixture completion content') }) $runs = [ordered]@{} foreach ($configuration in @('with_skill', 'without_skill')) { @@ -431,7 +431,7 @@ for ($index = 0; $index -lt $count; $index++) { eval_name = $evalName configuration = $configuration execution_status = 'unrun' - grading = @([ordered]@{ text = $assertion; passed = $null; evidence = '' }) + grading = @([ordered]@{ text = $assertion; passed = $null; evidence = '' }, [ordered]@{ text = 'the response contains the fixture completion content'; passed = $null; evidence = '' }) }) $runs[$configuration] = [ordered]@{ mode = $configuration @@ -596,6 +596,25 @@ for ($index = 0; $index -lt $count; $index++) { Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'invalid grading entry validation' -ExpectedText 'passed must be a boolean' Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'invalid grading validation' + foreach ($badEvidence in @('', " `t`n", 'Evaluation completed with output', "Source: output`nQuote: fabricated unavailable observation`nReason: This establishes the assertion.", "Source: output`nQuote: $((Read-TestJson -Path $records[0].ResultPath).output)`nReason: Evaluation completed with output")) { + $bad = Copy-TestGradingDocument -Document $validGrading + $bad.grading[0].evidence = $badEvidence + Write-TestJson -Path $gradingPath -Value $bad + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'non-evidentiary PASS rejected' + Assert-TestFileHashSnapshot -Expected $validationSnapshot -Message 'evidence rejection preserves frozen execution' + } + $repeated = Copy-TestGradingDocument -Document $validGrading + $repeated.grading[1].evidence = $repeated.grading[0].evidence + Write-TestJson -Path $gradingPath -Value $repeated + Assert-ToolFails -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'repeated PASS evidence rejected' -ExpectedText 'Repeated PASS evidence' + + $artifactEvidence = Copy-TestGradingDocument -Document $validGrading + $firstRecord = @($records | Sort-Object EvalId, Configuration)[0] + $eventLine = @(Get-Content (Join-Path (Split-Path -Parent $firstRecord.RunManifestPath) 'evidence/fixture-events.jsonl'))[0] + $artifactEvidence.grading[0].evidence = "Source: evidence/fixture-events.jsonl`nQuote: $eventLine`nReason: The native event records the fixture response for this assertion." + Write-TestJson -Path $gradingPath -Value $artifactEvidence + Assert-ToolPasses -Invocation (Invoke-TestTool -Path $validationScript -Arguments $validationArguments) -Description 'frozen native artifact citation accepted' + Write-TestJson -Path $gradingPath -Value $validGrading $validValidation = Invoke-TestTool -Path $validationScript -Arguments $validationArguments Assert-ToolPasses -Invocation $validValidation -Description 'valid grading validation' diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 019a416..0916901 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -1329,8 +1329,8 @@ exit 2 Assert-Equal 'explicit_environment' $execution.copilot_authentication_source 'Copilot uses explicit environment authentication in the token fixture' Assert-Equal 3 @($execution.copilot_auth_names_present).Count 'Copilot process receives all protected token variables without logging values' Assert-True ([string]::IsNullOrWhiteSpace([string]$execution.gh_config_dir)) 'Copilot explicit-token path does not forward host GH_CONFIG_DIR' - Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is run-local' - Assert-True (Test-PathInside -BasePath (Join-Path $with.Root 'home') -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the run''s isolated home' + Assert-True (Test-PathInside -BasePath $resultWith.evidence.execution_paths.physical_home_directory -CandidatePath ([string]$execution.copilot_cache_home)) 'Copilot cache is projection-local' + Assert-True (Test-PathInside -BasePath $resultWith.evidence.execution_paths.physical_home_directory -CandidatePath ([string]$execution.copilot_home)) 'Copilot COPILOT_HOME is the projected isolated home' Assert-Equal 'stdin' $resultWith.evidence.prompt_delivery 'Copilot result records stdin prompt delivery' Assert-Equal 'COPILOT_GITHUB_TOKEN' $resultWith.evidence.credential.github_token_variable 'Copilot follows explicit token precedence' Assert-True (-not $resultWith.evidence.credential.github_cli_config_forwarded) 'Copilot result records that GH_CONFIG_DIR was not forwarded with an explicit token' diff --git a/scripts/prepare-skill-evals.ps1 b/scripts/prepare-skill-evals.ps1 index c160999..80fce77 100644 --- a/scripts/prepare-skill-evals.ps1 +++ b/scripts/prepare-skill-evals.ps1 @@ -935,6 +935,8 @@ function Get-EvalWorkspaceOption { if ($EvalEntry.PSObject.Properties.Name -contains 'workspace' -and $null -ne $EvalEntry.workspace) { $workspace = $EvalEntry.workspace if ($workspace.PSObject.Properties.Name -contains 'git' -and $null -ne $workspace.git) { + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario -Scenario $workspace.git $wantsGit = [bool]$workspace.git } } @@ -1043,8 +1045,12 @@ function Initialize-GitWorkspace { '-c', 'user.name=Eval Harness', '-c', 'user.email=eval-harness@localhost', '-c', 'commit.gpgsign=false', + '-c', 'core.hooksPath=', + '-c', 'init.templateDir=', '-c', 'core.autocrlf=false' ) + $previousAuthorDate = $env:GIT_AUTHOR_DATE + $previousCommitterDate = $env:GIT_COMMITTER_DATE $env:GIT_AUTHOR_DATE = '2020-01-01T00:00:00Z' $env:GIT_COMMITTER_DATE = '2020-01-01T00:00:00Z' try { @@ -1062,8 +1068,8 @@ function Initialize-GitWorkspace { } & git @identity -C $RepoDirectory tag 'v1.0.0' 2>$null | Out-Null } finally { - Remove-Item Env:GIT_AUTHOR_DATE -ErrorAction SilentlyContinue - Remove-Item Env:GIT_COMMITTER_DATE -ErrorAction SilentlyContinue + $env:GIT_AUTHOR_DATE = $previousAuthorDate + $env:GIT_COMMITTER_DATE = $previousCommitterDate } } @@ -1550,6 +1556,8 @@ function Invoke-PrepareMode { } if ($workspaceOption.Git) { Initialize-GitWorkspace -RepoDirectory $repoDir + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Add-EvalGitScenario -RepoDirectory $repoDir -Scenario $evalEntry.workspace.git } $homeDir = Join-Path $runDir $runDirectoryNames.Home @@ -1868,6 +1876,7 @@ function New-RunnerPrompt { [void]$builder.AppendLine('Only after Phase 1 returns a successful terminal JSON summary, invoke the deterministic manifest bridge to validate the freeze and populate the canonical result paths before grading:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$manifestBridgePath`" -IterationDirectory `"$IterationDirectory`" -RequireComplete -RequireParallelDispatch") [void]$builder.AppendLine('Only if that bridge succeeds, reveal the grading key in `eval-metadata.json` to the Grader. The Grader may author exactly one package-root `grading.json` with schema `codebeltnet/agentic/eval-grading/1`; each entry contains only `eval_id`, `eval_name`, `configuration`, `assertion_index`, `assertion`, `passed`, and `evidence`. It must not edit raw execution results, canonical non-grading fields, hashes, paths, telemetry, or orchestration state.') + [void]$builder.AppendLine('Before creating grading.json, the Grader MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`; that guidance is authoritative during Phase 2. Uncertain or unverified expectations FAIL. Every assertion requires specific evidence. For PASS, evidence must use three newline-separated fields: `Source: output` (or a manifest-recorded run artifact path), `Quote: `, and `Reason: `. Do not reuse generic completion statements or identical evidence across assertions. FAIL evidence must explain what is missing or contradicted. Deterministic validation checks provenance and shape; it does not replace the Grader judgment required by grader.md.') [void]$builder.AppendLine('To display the authoritative top-level grading skeleton, run:') [void]$builder.AppendLine("pwsh -NoProfile -NonInteractive -File `"$gradingValidatorPath`" -ShowSkeleton") [void]$builder.AppendLine('Write `grading.json`, then validate it before finalization:') diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 2119cc4..f8c5964 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -648,8 +648,9 @@ Add-ValidationResult -Results $results -Name 'All repo-managed skills include va if ($eval.workspace -isnot [System.Management.Automation.PSCustomObject]) { throw "$relativeEvalPath eval $($eval.id) has a non-object 'workspace'" } - if ($eval.workspace.PSObject.Properties.Name -contains 'git' -and $eval.workspace.git -isnot [bool]) { - throw "$relativeEvalPath eval $($eval.id) must declare 'workspace.git' as a boolean" + if ($eval.workspace.PSObject.Properties.Name -contains 'git') { + . (Join-Path $PSScriptRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario -Scenario $eval.workspace.git } } } @@ -1277,6 +1278,18 @@ Add-ValidationResult -Results $results -Name 'Token normalization and benchmark if ($LASTEXITCODE -ne 0) { throw "Token reporting regression failed: $($output -join [Environment]::NewLine)" } } +Add-ValidationResult -Results $results -Name 'Copilot physical boundaries and checkpoint telemetry remain deterministic' -Group 'Runners' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-copilot-boundaries.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Copilot boundary regression failed: $($output -join [Environment]::NewLine)" } +} + +Add-ValidationResult -Results $results -Name 'Change-impact default resolution has paired deterministic Git history' -Group 'Preparation' -Action { + if (-not [string]::IsNullOrWhiteSpace($Ref)) { return } + $output = & pwsh -NoProfile -NonInteractive -File (Join-Path $repoRoot 'scripts/eval-runners/tests/test-change-impact-workspace.ps1') 2>&1 + if ($LASTEXITCODE -ne 0) { throw "Git scenario regression failed: $($output -join [Environment]::NewLine)" } +} + Add-ValidationResult -Results $results -Name 'Runner-owned orchestration remains deterministic' -Group 'Runners' -Action { if (-not [string]::IsNullOrWhiteSpace($Ref)) { return @@ -2359,7 +2372,7 @@ $argumentsPath = Join-Path $PSScriptRoot 'arguments.txt' assertion_index = $assertionIndex assertion = [string]$metadataForGrade.assertions[$assertionIndex] passed = $true - evidence = 'validator evidence' + evidence = "Source: output`nQuote: $(([IO.File]::ReadAllText((Join-Path $iterationDirectory $entryToGrade.runs.$configuration.result)) | ConvertFrom-Json).output)`nReason: The fixture response supplies the observed content for assertion $assertionIndex in this deterministic transport test." }) } } From 611b449b12b7f661d3ef66d1aae4bcfe17e0b456 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 18:59:30 +0200 Subject: [PATCH 11/18] =?UTF-8?q?=E2=9C=85=20add=20evaluation=20configurat?= =?UTF-8?q?ion=20for=20dotnet-change-impact=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive eval cases for dotnet-change-impact skill including test fixtures for API compatibility analysis, breaking change detection, and version impact assessment. Evaluation cases cover .NET library scenarios with fixture files demonstrating package changes and dependency impacts. --- skills/dotnet-change-impact/evals/evals.json | 13 +++++++++++++ .../files/default-resolution/src/Example.csproj | 5 +++++ .../evals/files/default-resolution/src/Widget.cs | 7 +++++++ 3 files changed, 25 insertions(+) create mode 100644 skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj create mode 100644 skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs diff --git a/skills/dotnet-change-impact/evals/evals.json b/skills/dotnet-change-impact/evals/evals.json index 9794de7..c4941f7 100644 --- a/skills/dotnet-change-impact/evals/evals.json +++ b/skills/dotnet-change-impact/evals/evals.json @@ -91,6 +91,19 @@ }, { "id": 9, + "files": ["evals/files/default-resolution/src/Widget.cs", "evals/files/default-resolution/src/Example.csproj"], + "workspace": { + "git": { + "base_branch": "trunk", + "feature_branch": "feature/remove-legacy-api", + "commits": [ + { + "message": "Remove the legacy public Parse overload", + "files": { "src/Widget.cs": "namespace Example;\n\npublic class Widget\n{\n public string Name { get; set; } = string.Empty;\n}\n" } + } + ] + } + }, "prompt": "Use dotnet-change-impact to classify the version bump for this branch.", "expected_output": "Current-branch default-resolution behavior: inspect the local Git repository, resolve the current branch against the upstream/default base branch, collect commits and net diff, then classify those changes with structured reasoning. Must not ask the user to provide change details first.", "expectations": [ diff --git a/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj b/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj new file mode 100644 index 0000000..ec2cce1 --- /dev/null +++ b/skills/dotnet-change-impact/evals/files/default-resolution/src/Example.csproj @@ -0,0 +1,5 @@ + + + net8.0 + + diff --git a/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs b/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs new file mode 100644 index 0000000..184db81 --- /dev/null +++ b/skills/dotnet-change-impact/evals/files/default-resolution/src/Widget.cs @@ -0,0 +1,7 @@ +namespace Example; + +public class Widget +{ + public string Name { get; set; } = string.Empty; + public static Widget Parse(string value) => new Widget { Name = value }; +} From 944dfac45aff195e43d0519d58a78dff9e374024 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 19:06:39 +0200 Subject: [PATCH 12/18] =?UTF-8?q?=E2=9C=85=20add=20test=20for=20eval=20git?= =?UTF-8?q?=20workspace=20scenarios?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Test deterministic package preparation for dotnet-change-impact eval 9 with real git repository staging. Validates paired git history consistency, feature branch setup, default branch fallback resolution, meaningful API diffs, and safety checks for declarative git scenarios including path-traversal rejection. --- .../tests/test-change-impact-workspace.ps1 | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 scripts/eval-runners/tests/test-change-impact-workspace.ps1 diff --git a/scripts/eval-runners/tests/test-change-impact-workspace.ps1 b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 new file mode 100644 index 0000000..23bf0f4 --- /dev/null +++ b/scripts/eval-runners/tests/test-change-impact-workspace.ps1 @@ -0,0 +1,45 @@ +# Real package preparation with a fake model catalog; never launches a provider. +[CmdletBinding()] +param() +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +$scriptsRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +$workspace = Join-Path ([IO.Path]::GetTempPath()) ('change-impact-workspace-' + [guid]::NewGuid().ToString('N')) +[void][IO.Directory]::CreateDirectory($workspace) +function Assert-True($Condition, $Message) { if (-not $Condition) { throw $Message } } +try { + $catalog = Join-Path $workspace 'models.json' + [IO.File]::WriteAllText($catalog, '{"models":[{"id":"fixture-model"}]}') + $prompt = & (Join-Path $scriptsRoot 'prepare-skill-evals.ps1') -Skill dotnet-change-impact -Eval 9 -Runner github-copilot -Model fixture-model -ModelCatalogPath $catalog -OutputRoot $workspace -PassThru + $package = Split-Path -Parent $prompt + $handoff = [IO.File]::ReadAllText($prompt) + Assert-True ($handoff.Contains('MUST read and follow the exact packaged `tools/skill-creator/agents/grader.md`')) 'Handoff must require the exact packaged grader before grading.' + $manifest = Get-Content (Join-Path $package 'manifest.json') -Raw | ConvertFrom-Json + $heads = @(); $diffs = @(); $refs = @() + foreach ($arm in @('with_skill', 'without_skill')) { + $repo = Join-Path (Split-Path -Parent (Join-Path $package $manifest.evals[0].runs.$arm.run_manifest)) 'repo' + Assert-True (Test-Path (Join-Path $repo '.git') -PathType Container) 'Eval 9 must stage a real .git repository.' + Assert-True ((& git -C $repo branch --show-current) -eq 'feature/remove-legacy-api') 'Feature branch missing.' + Assert-True ((& git -C $repo symbolic-ref refs/remotes/origin/HEAD --short) -eq 'origin/trunk') 'Non-main default branch fallback missing.' + Assert-True ((& git -C $repo rev-list --count origin/HEAD..HEAD) -eq '1') 'Feature commit history missing.' + $diff = (& git -C $repo diff origin/HEAD...HEAD) -join "`n" + Assert-True ($diff.Contains('- public static Widget Parse(string value)')) 'Meaningful public API removal missing.' + Assert-True ([string]::IsNullOrWhiteSpace((& git -C $repo status --porcelain) -join '')) 'Staged repository must be clean.' + $heads += & git -C $repo rev-parse HEAD + $diffs += $diff + $refs += ((& git -C $repo show-ref) -join "`n") + } + Assert-True ($heads[0] -ceq $heads[1] -and $diffs[0] -ceq $diffs[1] -and $refs[0] -ceq $refs[1]) 'Paired Git history, refs and diff must be identical.' + . (Join-Path $scriptsRoot 'eval-git-workspace.ps1') + Assert-EvalGitScenario $true + Assert-EvalGitScenario $false + foreach ($invalid in @('{"base_branch":"main","feature_branch":"main","commits":[]}', '{"base_branch":"main","feature_branch":"feature","commits":[{"message":"unsafe","files":{"../escape":"bad"}}]}')) { + $rejected = $false + try { Assert-EvalGitScenario ($invalid | ConvertFrom-Json) } catch { $rejected = $true } + Assert-True $rejected 'Unsafe declarative scenario must be rejected.' + } + 'PASS: eval 9 deterministic package Git history, pairing, grader handoff, boolean compatibility and unsafe-path rejection; no models.' +} finally { + if (-not ([IO.Path]::GetFileName($workspace) -match '^change-impact-workspace-[0-9a-f]{32}$')) { throw 'Unsafe test cleanup.' } + Remove-Item -LiteralPath $workspace -Recurse -Force +} From 05e132ba8fd3e4b7db9468f61a3cb882c80c102d Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 19:46:15 +0200 Subject: [PATCH 13/18] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20clarify=20git-visual?= =?UTF-8?q?-commits=20scope=20and=20extract=20grouping=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhanced git-visual-commits to fully document scope rules for tracked, staged, unstaged, deleted, renamed, and non-ignored untracked files including contents of new directories. Extracted detailed grouping examples and release-adjacent splitting guidance to a new references/grouping-examples.md for better maintainability. Added three new test cases (26–28) covering untracked file discovery, path reconciliation with hidden status, and explicit scope narrowing. --- skills/git-visual-commits/SKILL.md | 104 +++--------------- skills/git-visual-commits/evals/evals.json | 35 ++++++ .../references/grouping-examples.md | 89 +++++++++++++++ 3 files changed, 139 insertions(+), 89 deletions(-) create mode 100644 skills/git-visual-commits/references/grouping-examples.md diff --git a/skills/git-visual-commits/SKILL.md b/skills/git-visual-commits/SKILL.md index 0700436..547cbbd 100644 --- a/skills/git-visual-commits/SKILL.md +++ b/skills/git-visual-commits/SKILL.md @@ -81,7 +81,7 @@ If the user did **not** say `yolo` or `auto`, and session-level auto mode is not If the user says `git bot commit`, `git commit`, or `git our commit` without narrowing language, treat the request as covering the full current worktree. -- The default scope is **all current changes visible in git status**. +- The default scope is **all current changes visible in git status**: staged, unstaged, deleted, renamed, and non-ignored untracked files, including files inside new directories. - Your job is to group that full worktree into the right number of commits by semantic intent. - Never silently narrow the scope to "just the files from the last thing I worked on", "just the files I touched", or "just the newest skill" unless the user explicitly said to do that. - `yolo` keeps this same full-worktree default. It removes the approval wait; it does not narrow scope. @@ -93,6 +93,8 @@ Narrow scope only when the user explicitly does one of these: - asks for a review/plan for a subset before committing If the user did not narrow scope, do not invent a narrower scope on their behalf. + +Use the expanded status inventory in Step 1 as the scope of record. `git diff`, `git diff --stat`, and `git diff --name-only` omit untracked files; none can replace that inventory. Untracked status alone is neither an exclusion nor a reason to ask permission. Respect Git's ignore rules for untracked files; do not force-add ignored files. Already tracked files remain in scope even if an ignore pattern now matches them. ### Recovery Safety Rule @@ -238,7 +240,9 @@ Commits include a body by default. `no-body` or `tmi` suppresses it for one requ ### Step 1: Review changes -Run `git status` and `git diff` (and `git diff --staged` if there are staged changes) to understand what has changed. +From the repository root, run `git status --short --untracked-files=all` to enumerate individual paths even when Git configuration hides untracked files. Keep this inventory through final verification. For programmatic parsing, use `git status --porcelain=v1 --untracked-files=all -z` and parse NUL-delimited records, including rename pairs; do not split filenames on whitespace. + +Review `git diff` and `git diff --staged` for tracked changes and inspect every untracked file directly with an appropriate text or binary viewer. An empty diff does not mean a clean worktree. Do not stage files just to make them appear in a diff. Unless the user explicitly narrowed scope, inspect the **entire current worktree** and build the commit plan from that full set of changes. Do not default to the last task only. @@ -250,7 +254,7 @@ When resolving that reference, prefer the bundled skill path first instead of tr ### Step 2: Classify changes -Before composing any commit message, bucket every changed file by its **semantic intent** — not just its file type. Read the actual diff for each file and ask: *"What is this change trying to accomplish?"* Two files of the same type (e.g. two test files) may have completely different intents and belong in separate commits. +Before composing any commit message, bucket every path in the Step 1 inventory by its **semantic intent** — not its tracking status or file type. Read the actual diff for tracked files and contents of untracked files and ask: *"What is this change trying to accomplish?"* Two files of the same type (e.g. two test files) may have completely different intents and belong in separate commits. Use the inspected commit-language reference as the meaning source, not your gut. For example, restructuring an existing skill's `SKILL.md`, `FORMS.md`, `references/`, or `evals/` is normally refactor intent and should map to `♻️`; configuration-file changes map to `🔧`; truly new repo or application capabilities map to `✨`. @@ -298,36 +302,9 @@ When more than one file is changed and your first classification puts every chan Ask yourself explicitly: **“Did I actually read the whole `git-visual-commits` skill through EOF in this session before classifying this change?”** A metadata preview, remembered rule, or partial read is a failed answer. If the answer is no or uncertain, read `SKILL.md` from its first line through EOF and restart Step 1 and Step 2. -Then re-check the complete `git status`, `git diff`, and applicable staged diff; enumerate every changed path; explain each path's rationale, audience, and lifecycle; and consider whether any path belongs to a different category such as documentation, configuration, tooling, validation, tests, or release communication. Re-read `references/commit-language.md` before confirming the category and emoji. +Then refresh the Step 1 inventory and review tracked diffs and untracked contents; explain each path's rationale, audience, and lifecycle; and consider whether any path belongs to a different category such as documentation, configuration, tooling, validation, tests, or release communication. Re-read `references/commit-language.md` before confirming the category and emoji. Only keep one category after this audit if every path still has one rationale. Put a visible line in the commit plan such as `Quality gate: 3 files, one category retained; full skill read, full diff review, per-file rationale check, and alternative-category check confirmed.` If any check fails or any file has a materially different intent, split the groups and rerun the normal validation. `yolo` and `auto` do not bypass this gate. - -#### Commit body guidance - -Unless **no-body mode** is active, every commit includes a body explaining the *why*: - -- **Config/environment commits** → explain the operational intent (e.g. "Switch to shared-runner testing strategy with multi-image matrix") -- **Test assertion changes** → explain why the expectation changed (e.g. "net11 changed the default precision for DateTime, updating expected value") -- **Refactors** → explain what motivated the restructuring -- **New features** → explain the purpose and scope -- **Bug fixes** → explain what was broken and how this fixes it - -Common groupings: -- New repo-managed skill or workflow introduction together -- Existing skill refactor or extraction together -- Dependency/version baseline updates together -- Package/publish metadata together -- Config/setup files together (app host, bootstrapping) -- Environment and infrastructure config together (test runners, CI matrix, container settings) -- Documentation publishing fixes together -- Community health or release communication docs together -- New feature or module code together -- Data contracts, types, and interfaces together -- Database models, migrations, and schema changes together -- Test logic and assertions together (when they share the same rationale) -- Documentation and inline comments together - -When in doubt, one commit per "thing that changes" is better than one big commit. #### Mixed-scope guard @@ -339,40 +316,9 @@ This guard runs unconditionally — including in auto-approval mode. Documentation files (`CHANGELOG.md`, `AGENTS.md`, `README.md`, `CONTRIBUTING.md`, release notes) are **separate-by-default**. They only belong in the same commit as non-doc files when the commit is explicitly documentation-focused (e.g. `📝 add api usage guide` where the docs are the point, not a side effect). -#### Release-adjacent splitting rule - -Do not treat "all of this supports the release" as one commit. Release-adjacent work often spans different audiences and lifecycle roles that deserve separate history: - -- **Dependency/version baselines** — version alignment or runner baseline changes -- **Community health/release communication** — changelogs and human-facing repo health docs -- **Package/publish metadata** — package release-note definitions, `.nuget/*/PackageReleaseNotes.txt`, and publish targets; this bucket normally maps to `📦` -- **Documentation publishing** — DocFX navigation, branding, or publishing assets -- **CI/automation** — workflows and helper scripts used only by automation - -These buckets are examples, not a fixed file map. The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together. - -Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`, another diff updates CI scripts or workflow files such as `bump-nuget.py` or `.github/workflows/*.yml`, and another diff updates `CHANGELOG.md` plus `.nuget/*/PackageReleaseNotes.txt`, that is at least three intents: - -- **Build system / dependency baseline** -- **CI or automation** -- **Release communication plus package metadata** - -Do not collapse those into one commit, even if they were edited in the same round and all support the same release. Keep `.nuget/*/PackageReleaseNotes.txt` with the `📦` package/publish commit, not with the `💬` community-health commit. - -#### Repo-aligned grouping example - -When a repo like this one mixes skill changes, scaffold assets, validators, and repo docs, split them by intent: +#### Detailed grouping rules -- **New repo-managed skill** — a newly introduced `skills//` folder and its local `evals/` or `references/` -- **Existing skill refactor** — extracting shared rules, renaming sections, or reorganizing an existing skill -- **Skill contract files** — `SKILL.md`, `FORMS.md`, `references/`, `evals/` -- **Template/runtime files** — `assets/`, scaffold helper scripts -- **Validation/tooling** — validator scripts, repo checks -- **Repo docs/rules** — `README.md`, `AGENTS.md`, `CONTRIBUTING.md` - -Do not merge these into one commit unless the diff is truly single-purpose and the explanation still fits one sentence without using "and". - -If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits. "Related" is not enough — the repo history should make it obvious which commit added the capability and which commit reorganized existing behavior around it. +Read [grouping-examples.md](references/grouping-examples.md) before finalizing groups. Apply its release-adjacent splitting rule and repository examples alongside the guards above. #### Rename vs removal distinction @@ -409,7 +355,7 @@ Even in auto-approval mode, surface the commit buckets explicitly before committ The summary is status output, not a review request. Step 5 is mandatory in the same turn once its preconditions pass: never ask "Proceed with committing these groups?" (or an equivalent question), wait for a reply, or finish with a pending commit plan. -If the user did not narrow scope, the plan you surface must account for the full worktree rather than an arbitrarily chosen subset. +Reconcile the plan's exact path lists against a fresh Step 1 inventory before presenting it. Every in-scope path must belong to a commit group; splitting a file across groups must account for all its changes. Resolve missing paths before proceeding. Show a coverage line such as `Scope: 7 changed paths (3 untracked), 7 planned, 0 omitted.` Counts supplement the path comparison; equal counts alone do not prove coverage. Record explicit user exclusions separately. Apply this check in every mode, including single-file and auto-approved requests. **Otherwise**, wait for the user to confirm or adjust. They may say things like: - "Looks good" → proceed to stage and commit @@ -432,7 +378,8 @@ Before committing, validate each message against its file list: ### Step 5: Stage and commit each group For each group: -1. `git add ` — be precise, don't use `git add .` unless everything belongs in one commit +1. Refresh the inventory after approval and reconcile any changes with the plan. Stage each group's exact paths, including new files and deletions, with `git add -A -- `. Use literal pathspecs for names containing Git wildcard syntax. Do not use tracked-only staging such as `git add -u` or `git commit -a` as a substitute. + Compare `git diff --staged --name-status` and the full staged diff against the group's planned changes before committing. Account for pre-existing staged changes too; resolve missing or extra staged paths without discarding their contents. Keep staging and commits sequential because they share the index. 2. Compose the commit message (see format above) 3. Run `scripts/validate-commit-subject.ps1` again against the exact subject that will be passed to Git. Add `-PrefixMode Required` only for an explicitly requested combo. Do not run Git unless the validator exits successfully. 4. Run the appropriate commit command: @@ -449,35 +396,14 @@ When the body is just one short explanatory paragraph, prefer a single natural p ### Step 6: Verify +After each commit, compare its recorded paths (`git diff-tree --root --no-commit-id --name-status -r HEAD`) with the group, accounting for both sides of renames. After the last group, rerun `git status --short --untracked-files=all` and reconcile remaining paths with the original inventory and explicit exclusions. A full-worktree request is complete only when no in-scope changes remain, including untracked files. If a path was missed, review and include it through the same workflow under the existing authorization; do not silently leave it behind or claim completion. Report any concrete blocker or newly arriving change that prevents completion. + After committing, run `git log --oneline -5` to confirm the commit looks right. Read the stored subject with `git log -1 --format=%s` and run `scripts/validate-commit-subject.ps1` against that exact value. Then always run `git log -1 --format="%an <%ae>"` and verify that the author matches the requested identity mode before reporting success. Also run `git log -1 --format=%B` and verify the stored body contains readable prose with real line breaks, not literal escape sequences such as `\n`, and is not hard-wrapped mid-sentence just to satisfy a column limit. If any verification fails, amend the commit immediately instead of merely warning about it. If verification fails because the commit path used the wrong author or ignored the requested alias, stop treating it as a message-tweaking problem. Correct the tool path first, preserve the worktree safely, and only then repair the commit. --- -## Good Examples - -``` -🎉 begin api project -✨ add submission endpoint module -🐛 handle null optional fields in dto -➕ add validation library -🐛 fix: handle null optional fields in dto ← only when combo mode was requested -``` - -## Bad Examples (and why) - -``` -feat: add submission endpoint ← "feat:" is not an allowed prefix -✨ Feat: Add Submission Module ← uppercase, "Feat:" not allowed -💬 Update CHANGELOG for v10.0.10 ← uppercase description beginning -💬 update changelog for v10.0.10 ← more than one separator space -📋 update changelog for v10.0.10 ← emoji is absent from the approved reference table -🎉 initial commit with all files ← vague, bundles everything -⚙️ config: setup api ← "config:" is not an allowed prefix -♻️ refactor: reorganize skill wording ← bad default if the user did not ask for the combo -``` - ## Branching (for reference) When explicitly requested, use `[version]/[description]` (for example `v1.2.0/admin-dashboard`). Do not create, rename, or delete branches unless the user explicitly asks. diff --git a/skills/git-visual-commits/evals/evals.json b/skills/git-visual-commits/evals/evals.json index 6ab5043..7fbda95 100644 --- a/skills/git-visual-commits/evals/evals.json +++ b/skills/git-visual-commits/evals/evals.json @@ -283,6 +283,41 @@ "Treats the full current worktree as scope because the user did not narrow it", "Does not replace bot identity with a human-authored commit plus a Co-authored-by trailer" ] + }, + { + "id": 26, + "prompt": "Please git commit, yolo. The worktree has an unstaged change to src/Router.cs, a staged change to config/routes.json, and new untracked src/RoutePolicy.cs and assets/route-map.svg. git diff --stat only lists src/Router.cs. All four files belong to this routing change.", + "expected_output": "A complete commit workflow accounting for all four paths, inspecting untracked contents and reconciling inventory, plan, index, recorded commits, and final status.", + "expectations": [ + "Uses git status --short --untracked-files=all or its NUL-delimited porcelain equivalent as the scope inventory instead of a diff summary", + "Inspects and includes both src/RoutePolicy.cs and assets/route-map.svg without asking permission merely because they are untracked", + "Accounts for the pre-existing staged config/routes.json change when verifying each commit group", + "Compares exact planned paths with the inventory and reports no omitted in-scope paths", + "Checks staged contents and recorded commit paths against each group and verifies no in-scope tracked or untracked changes remain" + ] + }, + { + "id": 27, + "prompt": "Please review my working tree and show the commit plan only. I changed README.md and added a new assets folder containing logo.svg and dark theme/banner.svg. Git is configured with status.showUntrackedFiles=no. A scratch file under ignored .cache/ also exists.", + "expected_output": "A read-only plan that explicitly discovers both new files, handles the path containing spaces, covers the README change, and excludes the ignored scratch file.", + "expectations": [ + "Overrides hidden untracked status with --untracked-files=all and enumerates individual files in new directories", + "Inspects both untracked assets and includes assets/logo.svg and assets/dark theme/banner.svg in the plan", + "Preserves the space-containing path as one filename", + "Excludes the ignored .cache file without force-adding it", + "Does not stage or commit for a plan-only request" + ] + }, + { + "id": 28, + "prompt": "Please git commit only the README.md change, yolo. src/Router.cs is modified and src/RoutePolicy.cs is untracked too; leave both alone.", + "expected_output": "A README-only commit that records the explicit exclusions and reports the remaining source changes accurately.", + "expectations": [ + "Limits the commit to README.md because the user explicitly narrowed scope", + "Leaves both src/Router.cs and src/RoutePolicy.cs out of the commit and preserves their contents", + "Reconciles final status against the explicit exclusions instead of requiring an entirely clean worktree", + "Does not claim the entire worktree is clean when excluded paths remain" + ] } ] } diff --git a/skills/git-visual-commits/references/grouping-examples.md b/skills/git-visual-commits/references/grouping-examples.md new file mode 100644 index 0000000..689962f --- /dev/null +++ b/skills/git-visual-commits/references/grouping-examples.md @@ -0,0 +1,89 @@ +# Commit grouping examples + +#### Release-adjacent splitting rule + +Do not treat "all of this supports the release" as one commit. Release-adjacent work often spans different audiences and lifecycle roles that deserve separate history: + +- **Dependency/version baselines** — version alignment or runner baseline changes +- **Community health/release communication** — changelogs and human-facing repo health docs +- **Package/publish metadata** — package release-note definitions, `.nuget/*/PackageReleaseNotes.txt`, and publish targets; this bucket normally maps to `📦` +- **Documentation publishing** — DocFX navigation, branding, or publishing assets +- **CI/automation** — workflows and helper scripts used only by automation + +These buckets are examples, not a fixed file map. The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together. + +Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`, another diff updates CI scripts or workflow files such as `bump-nuget.py` or `.github/workflows/*.yml`, and another diff updates `CHANGELOG.md` plus `.nuget/*/PackageReleaseNotes.txt`, that is at least three intents: + +- **Build system / dependency baseline** +- **CI or automation** +- **Release communication plus package metadata** + +Do not collapse those into one commit, even if they were edited in the same round and all support the same release. Keep `.nuget/*/PackageReleaseNotes.txt` with the `📦` package/publish commit, not with the `💬` community-health commit. + +#### Repo-aligned grouping example + +When a repo like this one mixes skill changes, scaffold assets, validators, and repo docs, split them by intent: + +- **New repo-managed skill** — a newly introduced `skills//` folder and its local `evals/` or `references/` +- **Existing skill refactor** — extracting shared rules, renaming sections, or reorganizing an existing skill +- **Skill contract files** — `SKILL.md`, `FORMS.md`, `references/`, `evals/` +- **Template/runtime files** — `assets/`, scaffold helper scripts +- **Validation/tooling** — validator scripts, repo checks +- **Repo docs/rules** — `README.md`, `AGENTS.md`, `CONTRIBUTING.md` + +Do not merge these into one commit unless the diff is truly single-purpose and the explanation still fits one sentence without using "and". + +If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits. "Related" is not enough — the repo history should make it obvious which commit added the capability and which commit reorganized existing behavior around it. + + +#### Commit body guidance + +Unless **no-body mode** is active, every commit includes a body explaining the *why*: + +- **Config/environment commits** → explain the operational intent (e.g. "Switch to shared-runner testing strategy with multi-image matrix") +- **Test assertion changes** → explain why the expectation changed (e.g. "net11 changed the default precision for DateTime, updating expected value") +- **Refactors** → explain what motivated the restructuring +- **New features** → explain the purpose and scope +- **Bug fixes** → explain what was broken and how this fixes it + +Common groupings: +- New repo-managed skill or workflow introduction together +- Existing skill refactor or extraction together +- Dependency/version baseline updates together +- Package/publish metadata together +- Config/setup files together (app host, bootstrapping) +- Environment and infrastructure config together (test runners, CI matrix, container settings) +- Documentation publishing fixes together +- Community health or release communication docs together +- New feature or module code together +- Data contracts, types, and interfaces together +- Database models, migrations, and schema changes together +- Test logic and assertions together (when they share the same rationale) +- Documentation and inline comments together + +When in doubt, one commit per "thing that changes" is better than one big commit. + + +## Good Examples + +``` +🎉 begin api project +✨ add submission endpoint module +🐛 handle null optional fields in dto +➕ add validation library +🐛 fix: handle null optional fields in dto ← only when combo mode was requested +``` + +## Bad Examples (and why) + +``` +feat: add submission endpoint ← "feat:" is not an allowed prefix +✨ Feat: Add Submission Module ← uppercase, "Feat:" not allowed +💬 Update CHANGELOG for v10.0.10 ← uppercase description beginning +💬 update changelog for v10.0.10 ← more than one separator space +📋 update changelog for v10.0.10 ← emoji is absent from the approved reference table +🎉 initial commit with all files ← vague, bundles everything +⚙️ config: setup api ← "config:" is not an allowed prefix +♻️ refactor: reorganize skill wording ← bad default if the user did not ask for the combo +``` + From 66d2fdcbb68280a5751f89a58eb9825fb62bc8d0 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 19:46:30 +0200 Subject: [PATCH 14/18] =?UTF-8?q?=F0=9F=94=A8=20update=20skill=20validatio?= =?UTF-8?q?n=20to=20reference=20grouping=20examples?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated validate-skill-templates.ps1 to check for the new references/grouping-examples.md file and verify its content contains the release-adjacent splitting rules and repo-aligned grouping examples that were extracted from SKILL.md. --- scripts/validate-skill-templates.ps1 | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index f8c5964..b4da584 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -2774,6 +2774,7 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces $commitLanguage = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/references/commit-language.md' -GitRef $Ref $subjectValidator = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/scripts/validate-commit-subject.ps1' -GitRef $Ref $subjectTests = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/scripts/test-commit-subject.ps1' -GitRef $Ref + $groupingExamples = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/git-visual-commits/references/grouping-examples.md' -GitRef $Ref $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'automatic trigger for this skill, not as a casual hint.' @@ -2838,17 +2839,17 @@ Add-ValidationResult -Results $results -Name 'Git visual commits skill enforces Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '#### Single-category context quality gate' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Exactly one changed file is the only fast-path exception; skip this gate for that case.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Did I actually read the whole `git-visual-commits` skill through EOF in this session before classifying this change?' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Then re-check the complete `git status`, `git diff`, and applicable staged diff; enumerate every changed path' + Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Then refresh the Step 1 inventory and review tracked diffs and untracked contents' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Put a visible line in the commit plan such as `Quality gate:' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '`yolo` and `auto` do not bypass this gate.' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '#### Release-adjacent splitting rule' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`,' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'Keep `.nuget/*/PackageReleaseNotes.txt` with the' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together.' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle '#### Release-adjacent splitting rule' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'Concrete example: if one diff updates `Directory.Build.targets`, `Directory.Packages.props`, or `testenvironments.json`,' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'Keep `.nuget/*/PackageReleaseNotes.txt` with the' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'The rule is the abstraction: split by purpose and audience, not by the fact that the changes landed together.' Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'do not collapse "new skill introduced" and "existing skill refactored" into one commit' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle '**New repo-managed skill**' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'a newly introduced `skills//` folder and its local `evals/` or `references/`' - Assert-Contains -Name 'git-visual-commits/SKILL.md' -Content $skill -Needle 'If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits.' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle '**New repo-managed skill**' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'a newly introduced `skills//` folder and its local `evals/` or `references/`' + Assert-Contains -Name 'git-visual-commits/references/grouping-examples.md' -Content $groupingExamples -Needle 'If a commit both introduces a brand-new skill and refactors an existing skill to support it, prefer separate commits.' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle '### Allowed Prefixes' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle '### Emoji Selection' Assert-Contains -Name 'git-visual-commits/references/commit-language.md' -Content $commitLanguage -Needle 'Gitmoji First, Fallback Second' From 469b4a309d98a17468256ffd1a83670cf3e490b4 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 19:46:42 +0200 Subject: [PATCH 15/18] =?UTF-8?q?=F0=9F=92=AC=20document=20git-visual-comm?= =?UTF-8?q?its=20scope=20and=20inventory=20features?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated README.md to document new git-visual-commits capabilities: full inventory of tracked, staged, unstaged, and non-ignored untracked files; individual file enumeration inside new directories; exact path reconciliation before staging; and final remaining-change verification. Updated skill table description to reflect these improvements. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index fbf3803..59e69cb 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ Each `SKILL.md` description is lean activation metadata. The catalog below expla | Skill | Description | |-------|-------------| -| [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping unless narrowed, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | +| [git-visual-commits](skills/git-visual-commits/SKILL.md) | AI-driven git commit workflow with authoritative routing for `git bot commit`, `git commit`, and `git our commit`, including the exact `Please do a git bot commit yolo` form. It locks the requested identity, treats yolo/auto only as scoped auto-approval modifiers, never as the commit message, and does not hand commit execution to changelog or release-note skills. It uses deterministically validated emoji-first subjects, optional conventional prefixes only on explicit request, full-worktree semantic grouping of tracked and non-ignored untracked files unless narrowed, exact path reconciliation before staging, and a final remaining-change check, a visible multi-file single-category quality gate, commit bodies by default, and post-commit identity/body verification. Multi-file plans that initially collapse to one category also require a visible full-context quality gate; one-file changes keep the fast path. Stack-agnostic. | | [git-keep-a-changelog](skills/git-keep-a-changelog/SKILL.md) | Git-aware Keep a Changelog companion selected only for explicit changelog or release-note intent. Bare yolo/auto and commit-execution requests such as `git bot commit yolo` do not activate it; those words modify autonomy only after changelog intent is established. Bundled deterministic resolvers separate branch-unique commit history from merge-base-to-`HEAD` net diffs, exclude the previous-release or comparison boundary, fail on base-history bleed, and classify explicit path-backed release entities as `Added`, `Removed`, `Changed`, or `Unchanged`. The skill establishes each user-facing release entity against the base before section classification. It asks a mandatory `Yes / No / Custom` question before including pending worktree changes in ordinary concrete-release drafts, includes staged, unstaged, and untracked work automatically only in scoped yolo/auto mode, creates missing changelogs, writes SemVer-aware highlights, maintains compare-link footers, preserves natural prose wrapping, and curates surviving outcomes instead of dumping raw commit logs. | | [git-nuget-release-notes](skills/git-nuget-release-notes/SKILL.md) | Git-aware NuGet release-notes companion for .NET repos that keep cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Discovers packable `src/` projects, resolves concrete package version and availability, creates missing files when needed, reduces each package to its surviving base-to-`HEAD` delta before classifying history, and establishes each package capability against the base so pre-release refinements and fixes to a new capability remain one `ADDED` New Feature. It writes per-package `ALM` / `Breaking Changes` / `New Features` / `Improvements` / `Bug Fixes` style notes from final package state plus supporting commit context instead of dumping commit subjects. | | [git-nuget-readme](skills/git-nuget-readme/SKILL.md) | Git-aware NuGet README companion for .NET repos that advertise a package from `src/`. Resolves the real packable project the README should sell, combines git history with actual package metadata, source capabilities, and relevant tests when feasible, preserves honest badge/docs/contributing sections, and writes a forthcoming, adoption-friendly `README.md` with repo-derived branding, clear value, install, framework-support, and quick-start guidance. | @@ -284,6 +284,8 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith Commit messages are the most-read documentation in any codebase — yet they're usually an afterthought. "fix stuff", "wip", "address PR feedback" tells you nothing six months later. Writing good commits takes discipline, and when you're in flow, it's the first thing that slips. +**git-visual-commits** includes staged, unstaged, and non-ignored untracked files by default. It inventories individual files inside new directories, reconciles every path with the commit plan and staged changes, and checks for leftovers before reporting completion. Explicitly narrowed requests preserve excluded changes. + **git-visual-commits** handles the entire commit workflow— staging, diffing, crafting the message, choosing the right emoji — so every commit is consistent and meaningful without breaking your flow. Whether the agent authors the commit (`git bot commit`), you do (`git commit`), or you worked on it together (`git our commit`), the quality is the same. - **Gitmoji-first** — visual commit categories that are scannable at a glance From ab54ffa39b96a5879f9cfc780d89873e25d2075d Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 8 Sep 2026 21:34:59 +0200 Subject: [PATCH 16/18] =?UTF-8?q?=F0=9F=90=9B=20fix=20eval=20runner=20cros?= =?UTF-8?q?s-platform=20path=20and=20environment=20support?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add proper handling for Unix-style temporary directories and environment variables. Runner now respects RunPath property for projected inputs that use /tmp directly. Tests isolate TMPDIR alongside TEMP/TMP, and GitHub CLI config path is now platform-aware to handle Unix vs Windows conventions correctly. --- scripts/eval-runners/runner-common.ps1 | 15 +++++++++++++-- .../tests/test-copilot-boundaries.ps1 | 6 +++--- .../tests/test-runner-conformance.ps1 | 6 ++++-- 3 files changed, 20 insertions(+), 7 deletions(-) diff --git a/scripts/eval-runners/runner-common.ps1 b/scripts/eval-runners/runner-common.ps1 index 89b26be..279d38c 100644 --- a/scripts/eval-runners/runner-common.ps1 +++ b/scripts/eval-runners/runner-common.ps1 @@ -1470,8 +1470,19 @@ function Assert-PhaseOneEvidenceWritable { # package-level freeze exists, refusing to build another result prevents a # direct runner invocation (or the orchestrator-owned recorder) from # truncating or replacing frozen raw evidence. - $runRoot = [System.IO.Path]::GetFullPath([string]$Run.RunRoot) - $iterationDirectory = Split-Path -Parent (Split-Path -Parent $runRoot) + # Projected runner inputs may replace RunRoot with a physical directory + # directly under /tmp. Derive the package location from the original + # manifest path so freeze checks remain valid on every platform. + $runPath = [string](Get-JsonProperty -Object $Run -Name 'RunPath' -Default '') + $runRoot = if ([string]::IsNullOrWhiteSpace($runPath)) { + [System.IO.Path]::GetFullPath([string]$Run.RunRoot) + } else { + [System.IO.Path]::GetDirectoryName([System.IO.Path]::GetFullPath($runPath)) + } + $iterationDirectory = [System.IO.Path]::GetDirectoryName([System.IO.Path]::GetDirectoryName($runRoot)) + if ([string]::IsNullOrWhiteSpace($iterationDirectory)) { + throw 'Cannot derive the eval package directory from the run contract.' + } $freezeRelativePath = 'execution-freeze.json' $manifestPath = Join-Path $iterationDirectory 'manifest.json' if (Test-Path -LiteralPath $manifestPath -PathType Leaf) { diff --git a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 index eefa54b..87789b3 100644 --- a/scripts/eval-runners/tests/test-copilot-boundaries.ps1 +++ b/scripts/eval-runners/tests/test-copilot-boundaries.ps1 @@ -215,11 +215,11 @@ if ($inputText -eq 'failure') { exit 7 } $usage = Read-CopilotEvents -Parsed @{ Events = @($checkpoint, @{ type = 'assistant.usage'; data = @{ inputTokens = 7; outputTokens = 3 } }); Errors = @() } -Warnings $warnings Assert-Equal 7 $usage.UsageInput 'native assistant usage takes precedence over cache snapshots' Assert-Equal 3 $usage.UsageOutput 'actual exposed output count retained' - $savedTemp = $env:TEMP; $savedTmp = $env:TMP + $savedTemp = $env:TEMP; $savedTmp = $env:TMP; $savedTmpDir = $env:TMPDIR try { - $env:TEMP = $testRoot; $env:TMP = $testRoot + $env:TEMP = $testRoot; $env:TMP = $testRoot; $env:TMPDIR = $testRoot Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'temp inside source ancestry fails closed' - } finally { $env:TEMP = $savedTemp; $env:TMP = $savedTmp } + } finally { $env:TEMP = $savedTemp; $env:TMP = $savedTmp; $env:TMPDIR = $savedTmpDir } $runtimeLink = Join-Path $singleInputs.Run.WorkingDirectoryPath 'forbidden-link' New-Item -ItemType $linkType -Path $runtimeLink -Target $testRoot | Out-Null try { Assert-Rejected { Get-CopilotProjectionPlan -Inputs $singleInputs } 'linked projection input fails closed' } diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 0916901..5684a33 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -130,6 +130,7 @@ $recordedOldUserProfile = $env:USERPROFILE $recordedOldAppData = $env:APPDATA $recordedOldLocalAppData = $env:LOCALAPPDATA $recordedOldXdgConfigHome = $env:XDG_CONFIG_HOME +$recordedOldTmpDir = $env:TMPDIR $recordedOldFixtures = $env:AGENTIC_RECORDED_FIXTURES try { $fakeBin = Join-Path $recordedRoot 'bin' @@ -2021,8 +2022,8 @@ exit 2 $copilotFreshAppData = Join-Path $copilotFreshBoundaryRoot 'appdata' $copilotFreshLocalAppData = Join-Path $copilotFreshBoundaryRoot 'localappdata' $copilotFreshHostAppData = Join-Path $copilotFreshBoundaryRoot 'host-appdata' - $copilotFreshHostGhConfig = Join-Path $copilotFreshHostAppData 'GitHub CLI' - New-Item -ItemType Directory -Path $copilotFreshHome, $copilotFreshXdg, $copilotFreshAppData, $copilotFreshLocalAppData, (Join-Path $copilotFreshXdg 'gh'), $copilotFreshHostGhConfig -Force | Out-Null + $copilotFreshHostGhConfig = if ($IsWindows) { Join-Path $copilotFreshHostAppData 'GitHub CLI' } else { Join-Path $copilotFreshXdg 'gh' } + New-Item -ItemType Directory -Path $copilotFreshHome, $copilotFreshXdg, $copilotFreshAppData, $copilotFreshLocalAppData, $copilotFreshHostGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $copilotFreshHostGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) $env:HOME = $copilotFreshHome $env:USERPROFILE = $copilotFreshHome @@ -2122,6 +2123,7 @@ exit 2 $env:APPDATA = $recordedOldAppData $env:LOCALAPPDATA = $recordedOldLocalAppData $env:XDG_CONFIG_HOME = $recordedOldXdgConfigHome + $env:TMPDIR = $recordedOldTmpDir $env:AGENTIC_RECORDED_FIXTURES = $recordedOldFixtures if (Test-Path -LiteralPath $recordedRoot) { Remove-Item -LiteralPath $recordedRoot -Recurse -Force } } From 9cd1c5b68f580002462a7156e1ccb37b6e6d523c Mon Sep 17 00:00:00 2001 From: gimlichael Date: Tue, 8 Sep 2026 23:10:46 +0200 Subject: [PATCH 17/18] =?UTF-8?q?=F0=9F=90=9B=20fix=20explicit=20gh=5Fconf?= =?UTF-8?q?ig=5Fdir=20auth=20selection?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make explicit GH_CONFIG_DIR authoritative during trusted GitHub CLI token resolution for Copilot runs. If that selected configuration cannot resolve a token, preflight now fails closed instead of silently probing other config roots or ambient identity. Add deterministic regressions that prove explicit GH_CONFIG_DIR success, explicit GH_CONFIG_DIR fail-closed behavior, no fallback probing to alternate configs, and no token value leakage in evidence or output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../eval-runners/github-copilot/runner.ps1 | 57 ++++++++++++++++--- .../tests/test-runner-conformance.ps1 | 56 +++++++++++++++++- 2 files changed, 103 insertions(+), 10 deletions(-) diff --git a/scripts/eval-runners/github-copilot/runner.ps1 b/scripts/eval-runners/github-copilot/runner.ps1 index 49d61ee..0a66709 100644 --- a/scripts/eval-runners/github-copilot/runner.ps1 +++ b/scripts/eval-runners/github-copilot/runner.ps1 @@ -220,6 +220,12 @@ function Add-CopilotGhConfigCandidate { } catch { } } +function Get-CopilotExplicitGhConfigDirectory { + $configured = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') + if ([string]::IsNullOrWhiteSpace($configured)) { return $null } + return [string]$configured +} + function Get-CopilotGhConfigDirectories { # GH_CONFIG_DIR is an authentication-state exception to the isolated # Copilot configuration roots. Build documented/default candidates and keep @@ -281,16 +287,27 @@ function Get-CopilotGitHubCliToken { } } + $explicitGhConfigDirectory = Get-CopilotExplicitGhConfigDirectory $candidateConfigDirectories = [System.Collections.Generic.List[string]]::new() - foreach ($directory in @(Get-CopilotGhConfigDirectories)) { - if (-not [string]::IsNullOrWhiteSpace([string]$directory) -and -not $candidateConfigDirectories.Contains([string]$directory)) { - $candidateConfigDirectories.Add([string]$directory) + $useAmbientAttempt = $true + if (-not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory)) { + # An explicit GH_CONFIG_DIR selects the GitHub CLI identity. It is + # authoritative: do not search other config roots when it fails. + $candidateConfigDirectories.Add([string]$explicitGhConfigDirectory) + $useAmbientAttempt = $false + } else { + foreach ($directory in @(Get-CopilotGhConfigDirectories)) { + if (-not [string]::IsNullOrWhiteSpace([string]$directory) -and -not $candidateConfigDirectories.Contains([string]$directory)) { + $candidateConfigDirectories.Add([string]$directory) + } } } - # Keep one ambient attempt with GH_CONFIG_DIR removed so an existing host - # default can still resolve even when no candidate was discovered. - $candidateConfigDirectories.Add('') + if ($useAmbientAttempt) { + # Keep one ambient attempt with GH_CONFIG_DIR removed so an existing + # host default can still resolve when no explicit config was supplied. + $candidateConfigDirectories.Add('') + } $probeRoot = Resolve-RunnerProbeTempRoot $probeDirectory = Join-Path $probeRoot ('agentic-gh-token-probe-' + [Guid]::NewGuid().ToString('N')) @@ -318,6 +335,7 @@ function Get-CopilotGitHubCliToken { return [pscustomobject]@{ Token = $token GhConfigDirectory = if ([string]::IsNullOrWhiteSpace([string]$candidateDirectory)) { $null } else { [string]$candidateDirectory } + GhConfigExplicit = -not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory) } } return $null @@ -329,6 +347,8 @@ function Get-CopilotGitHubCliToken { } function Resolve-CopilotAuthentication { + $explicitGhConfigDirectory = Get-CopilotExplicitGhConfigDirectory + $explicitGhConfigProvided = -not [string]::IsNullOrWhiteSpace([string]$explicitGhConfigDirectory) $tokenVariable = Get-CopilotTokenVariable if (-not [string]::IsNullOrWhiteSpace($tokenVariable)) { return [pscustomobject]@{ @@ -337,6 +357,8 @@ function Resolve-CopilotAuthentication { TokenValue = $null GitHubCliTokenResolved = $false GitHubCliConfigDirectory = $null + GitHubCliConfigExplicit = $false + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided NonInteractiveReady = $true } } @@ -349,6 +371,8 @@ function Resolve-CopilotAuthentication { TokenValue = [string]$githubCliToken.Token GitHubCliTokenResolved = $true GitHubCliConfigDirectory = [string](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigDirectory' -Default $null) + GitHubCliConfigExplicit = [bool](Get-JsonProperty -Object $githubCliToken -Name 'GhConfigExplicit' -Default $false) + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided NonInteractiveReady = $true } } @@ -360,6 +384,8 @@ function Resolve-CopilotAuthentication { TokenValue = $null GitHubCliTokenResolved = $false GitHubCliConfigDirectory = $null + GitHubCliConfigExplicit = $false + ExplicitGhConfigProvided = [bool]$explicitGhConfigProvided NonInteractiveReady = $false } } @@ -677,11 +703,22 @@ function Get-CopilotPreflight { if ($authState.Source -eq 'environment') { $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail "Authentication is available through the explicit $($authState.TokenVariable) environment variable; Copilot OS-keychain and GitHub CLI state are not copied into the run.")) } elseif ($authState.Source -eq 'github_cli_token') { - $configDetail = if ([string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory)) { 'ambient/default' } else { 'resolved GH_CONFIG_DIR candidate' } + $configDetail = if ([bool]$authState.GitHubCliConfigExplicit) { + 'explicit GH_CONFIG_DIR' + } elseif ([string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory)) { + 'ambient/default' + } else { + 'platform-discovered GH_CONFIG_DIR candidate' + } $checks.Add((New-PreflightCheck -Name 'authentication' -Status passed -Detail ("GitHub CLI fallback resolved a token in the trusted runner from a {0} source; only a protected token environment variable will be passed to Copilot." -f $configDetail))) } else { - $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) - $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or ensure `gh auth token` can resolve in this host context.') + if ([bool]$authState.ExplicitGhConfigProvided) { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. GH_CONFIG_DIR was explicitly provided, but `gh auth token` could not resolve that selected configuration. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. GH_CONFIG_DIR was explicitly provided, but `gh auth token` could not resolve that selected configuration. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or fix the selected GH_CONFIG_DIR authentication state.') + } else { + $checks.Add((New-PreflightCheck -Name 'authentication' -Status failed -Detail 'No supported non-interactive Copilot authentication source was resolved. Required source: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN or trusted GitHub CLI token fallback.')) + $reasons.Add('No supported non-interactive Copilot authentication source was resolved before Phase 1. Provide COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN, or ensure `gh auth token` can resolve in this host context.') + } } if ($platform -notin @('linux', 'macos')) { @@ -749,6 +786,8 @@ function Get-CopilotPreflight { noninteractive_ready = [bool]$authState.NonInteractiveReady github_cli_token_resolved = [bool]$authState.GitHubCliTokenResolved github_cli_config_candidate_used = -not [string]::IsNullOrWhiteSpace([string]$authState.GitHubCliConfigDirectory) + github_cli_config_explicit = [bool]$authState.GitHubCliConfigExplicit + explicit_gh_config_dir_provided = [bool]$authState.ExplicitGhConfigProvided token_value_observed = $false } } diff --git a/scripts/eval-runners/tests/test-runner-conformance.ps1 b/scripts/eval-runners/tests/test-runner-conformance.ps1 index 5684a33..fa20d33 100644 --- a/scripts/eval-runners/tests/test-runner-conformance.ps1 +++ b/scripts/eval-runners/tests/test-runner-conformance.ps1 @@ -969,6 +969,12 @@ if ($harness -eq 'codex') { $fakeGh = @' [CmdletBinding()] param([Parameter(ValueFromRemainingArguments = $true)][string[]]$RemainingArguments) +$logPath = Join-Path (Split-Path -Parent $MyInvocation.MyCommand.Path) 'gh-auth-token-probe.jsonl' +$record = [ordered]@{ + args = @($RemainingArguments | ForEach-Object { [string]$_ }) + gh_config_dir = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') +} +[System.IO.File]::AppendAllText($logPath, (($record | ConvertTo-Json -Compress) + [Environment]::NewLine), [System.Text.UTF8Encoding]::new($false)) if ($RemainingArguments.Count -eq 2 -and $RemainingArguments[0] -eq 'auth' -and $RemainingArguments[1] -eq 'token') { $config = [Environment]::GetEnvironmentVariable('GH_CONFIG_DIR') if (-not [string]::IsNullOrWhiteSpace($config) -and (Test-Path -LiteralPath (Join-Path $config 'auth-marker.txt') -PathType Leaf)) { @@ -993,6 +999,7 @@ exit 2 New-Item -ItemType Directory -Path $recordedGhConfig -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $recordedGhConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [System.Text.UTF8Encoding]::new($false)) $env:GH_CONFIG_DIR = $recordedGhConfig + $ghAuthProbeLogPath = Join-Path $fakeBin 'gh-auth-token-probe.jsonl' $ambientCopilotHome = Join-Path $recordedRoot 'ambient-copilot-home' New-Item -ItemType Directory -Path $ambientCopilotHome -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $ambientCopilotHome 'copilot-instructions.md'), '# ambient-personal-instruction-not-logged', [System.Text.UTF8Encoding]::new($false)) @@ -2075,6 +2082,53 @@ exit 2 Assert-True (-not $copilotGhResult.evidence.credential.github_cli_config_forwarded) 'Copilot GitHub CLI fallback does not forward host GH_CONFIG_DIR' $ghRecords = @(Get-Content -LiteralPath (Join-Path $with.Root 'repo\copilot-fake-cli-log.jsonl') | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) -and @($_.copilot_auth_names_present).Count -eq 1 -and @($_.copilot_auth_names_present) -contains 'GH_TOKEN' -and [string]::IsNullOrWhiteSpace([string]$_.gh_config_dir) }) Assert-Equal ($ghFallbackMatchesBefore + 1) $ghRecords.Count 'Copilot fake observes one additional protected GH_TOKEN execution produced by trusted GitHub CLI fallback' + Assert-True (($copilotGhResult | ConvertTo-Json -Depth 100) -notmatch 'recorded-gh-fallback-token-not-logged') 'Copilot GitHub CLI fallback result never exposes token values' + + # Explicit GH_CONFIG_DIR is authoritative. If it cannot resolve auth, do not + # consult discovered/ambient configurations even when they are valid. + $copilotExplicitMissingConfig = Join-Path $recordedRoot 'copilot-explicit-missing-gh-config' + if (Test-Path -LiteralPath $copilotExplicitMissingConfig) { + Remove-Item -LiteralPath $copilotExplicitMissingConfig -Recurse -Force + } + $copilotExplicitMissingRoot = Join-Path $recordedRoot 'copilot-explicit-missing' + $copilotExplicitMissingHome = Join-Path $copilotExplicitMissingRoot 'home' + $copilotExplicitMissingXdg = Join-Path $copilotExplicitMissingRoot 'xdg' + $copilotExplicitMissingAppData = Join-Path $copilotExplicitMissingRoot 'host-appdata' + $copilotExplicitMissingLocalAppData = Join-Path $copilotExplicitMissingRoot 'localappdata' + $copilotExplicitMissingFallbackConfig = if ($IsWindows) { Join-Path $copilotExplicitMissingAppData 'GitHub CLI' } else { Join-Path $copilotExplicitMissingXdg 'gh' } + New-Item -ItemType Directory -Path $copilotExplicitMissingHome, $copilotExplicitMissingXdg, $copilotExplicitMissingAppData, $copilotExplicitMissingLocalAppData, $copilotExplicitMissingFallbackConfig -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $copilotExplicitMissingFallbackConfig 'auth-marker.txt'), 'fixture auth state without a credential value', [Text.UTF8Encoding]::new($false)) + $env:HOME = $copilotExplicitMissingHome + $env:USERPROFILE = $copilotExplicitMissingHome + $env:APPDATA = $copilotExplicitMissingAppData + $env:LOCALAPPDATA = $copilotExplicitMissingLocalAppData + $env:XDG_CONFIG_HOME = $copilotExplicitMissingXdg + $env:COPILOT_HOME = (Join-Path $copilotExplicitMissingRoot 'copilot-home') + New-Item -ItemType Directory -Path $env:COPILOT_HOME -Force | Out-Null + $env:GH_CONFIG_DIR = $copilotExplicitMissingConfig + if (Test-Path -LiteralPath $ghAuthProbeLogPath -PathType Leaf) { Remove-Item -LiteralPath $ghAuthProbeLogPath -Force } + $nativeExecutionsBeforeExplicitMissing = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + $copilotExplicitMissingPreflight = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command preflight -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'incompatible' $copilotExplicitMissingPreflight.status 'Copilot explicit GH_CONFIG_DIR without a valid token fails closed' + Assert-True (@($copilotExplicitMissingPreflight.checks | Where-Object { $_.name -eq 'authentication' -and $_.status -eq 'failed' }).Count -eq 1) 'Copilot explicit GH_CONFIG_DIR failure is detected in preflight' + Assert-True ([bool]$copilotExplicitMissingPreflight.protocol_observations.authentication.explicit_gh_config_dir_provided) 'Copilot preflight records that GH_CONFIG_DIR was explicitly provided' + Assert-True (-not [bool]$copilotExplicitMissingPreflight.protocol_observations.authentication.noninteractive_ready) 'Copilot explicit GH_CONFIG_DIR failure does not report non-interactive readiness' + Assert-True (@($copilotExplicitMissingPreflight.reasons | Where-Object { $_ -match 'GH_CONFIG_DIR was explicitly provided' }).Count -eq 1) 'Copilot explicit GH_CONFIG_DIR failure reports selected-identity resolution failure' + $copilotExplicitMissingResult = Invoke-AdapterJson -RunnerPath (Join-Path $runnerRoot 'github-copilot\runner.ps1') -Command execute -RunPath $with.Path -ProfilePath $recordedProfiles['copilot'] + Assert-Equal 'incompatible' $copilotExplicitMissingResult.status 'Copilot explicit GH_CONFIG_DIR failure remains terminal in execute' + Assert-Equal 'incompatible' ([string]$copilotExplicitMissingResult.evidence.preflight.status) 'Copilot explicit GH_CONFIG_DIR execute result preserves incompatible preflight evidence' + $nativeExecutionsAfterExplicitMissing = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) + Assert-Equal $nativeExecutionsBeforeExplicitMissing.Count $nativeExecutionsAfterExplicitMissing.Count 'Copilot explicit GH_CONFIG_DIR rejection occurs before model execution' + $ghAuthProbeRecords = @() + if (Test-Path -LiteralPath $ghAuthProbeLogPath -PathType Leaf) { + $ghAuthProbeRecords = @(Get-Content -LiteralPath $ghAuthProbeLogPath | Where-Object { -not [string]::IsNullOrWhiteSpace([string]$_) } | ForEach-Object { $_ | ConvertFrom-Json }) + } + $ghAuthTokenProbes = @($ghAuthProbeRecords | Where-Object { @($_.args).Count -eq 2 -and $_.args[0] -eq 'auth' -and $_.args[1] -eq 'token' }) + Assert-Equal 2 $ghAuthTokenProbes.Count 'Copilot explicit GH_CONFIG_DIR executes exactly one GitHub CLI token probe per preflight invocation' + Assert-True (@($ghAuthTokenProbes | Where-Object { [string]$_.gh_config_dir -ne $copilotExplicitMissingConfig }).Count -eq 0) 'Copilot explicit GH_CONFIG_DIR failure never probes another GitHub CLI configuration' + Assert-True (@($ghAuthTokenProbes | Where-Object { [string]$_.gh_config_dir -eq $copilotExplicitMissingFallbackConfig }).Count -eq 0) 'Copilot explicit GH_CONFIG_DIR failure does not consult a valid discovered fallback configuration' + $explicitFailureEvidenceText = ($copilotExplicitMissingPreflight | ConvertTo-Json -Depth 100) + ($copilotExplicitMissingResult | ConvertTo-Json -Depth 100) + Assert-True ($explicitFailureEvidenceText -notmatch 'recorded-gh-fallback-token-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot explicit GH_CONFIG_DIR failure never exposes token values' $copilotNoAuthHome = Join-Path $recordedRoot 'copilot-no-auth-home' New-Item -ItemType Directory -Path $copilotNoAuthHome -Force | Out-Null @@ -2098,7 +2152,7 @@ exit 2 Assert-Equal 'incompatible' ([string]$copilotNoAuthResult.evidence.preflight.status) 'Copilot no-auth execute result preserves the incompatible preflight evidence' $nativeExecutionsAfterNoAuth = @(Get-Content -LiteralPath $copilotLogPath | ForEach-Object { $_ | ConvertFrom-Json } | Where-Object { [bool](Get-JsonProperty -Object $_ -Name 'stdin_received' -Default $false) }) Assert-Equal $nativeExecutionsBeforeNoAuth.Count $nativeExecutionsAfterNoAuth.Count 'Copilot no-auth rejection does not start a model execution process' - Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' + Assert-True (($copilotNoAuthResult | ConvertTo-Json -Depth 100) -notmatch 'ambient-profile-not-logged|recorded-gh-fallback-token-not-logged|recorded-copilot-canary|recorded-gh-canary|recorded-github-canary') 'Copilot authentication fixtures never expose credential values' $env:HOME = $recordedOldHome $env:USERPROFILE = $recordedOldUserProfile $env:APPDATA = $recordedOldAppData From b9bccbc82936a7e734c7ff79aa0edbe45a4f463b Mon Sep 17 00:00:00 2001 From: gimlichael Date: Tue, 8 Sep 2026 23:10:54 +0200 Subject: [PATCH 18/18] =?UTF-8?q?=F0=9F=92=AC=20clarify=20eval=20git=20fix?= =?UTF-8?q?ture=20isolation=20policy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clarify that temporary Git repos, branches, and throwaway commits are forbidden in the real source working tree while deterministic synthetic Git history remains allowed only inside isolated disposable eval fixtures under approved workspace roots. Keep the rule narrow by preserving the existing anti-pollution guardrails and adding deterministic wording checks so this distinction remains explicit across AGENTS, README, and validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- AGENTS.md | 4 +++- README.md | 2 +- scripts/validate-skill-templates.ps1 | 2 ++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 924ac91..fe5f7bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,9 @@ Eval workspaces and test repositories must **never** become part of this reposit - `.bot/-workspace/` — the default. `.gitignore` covers `.bot/*`, so git never sees what lands there, and harnesses that refuse to work outside the repository folder still have somewhere to go. - `$env:TEMP/-workspace/` on Windows, `/tmp/-workspace/` on Unix — for anything that has no reason to sit next to the source. -Anywhere else inside the repository is forbidden, including a `-workspace/` at the root. So are temporary git repos, test branches, throwaway commits, and local config overrides such as git aliases. +Anywhere else inside the repository is forbidden, including a `-workspace/` at the root. In the real `codebeltnet/agentic` source working tree, temporary git repos, test branches, throwaway commits, and local config overrides such as git aliases are forbidden. + +Deterministic synthetic Git history is allowed only when it is the fixture being tested inside an explicitly permitted isolated eval/test repository under `.bot/-workspace/` or the approved OS temp workspace. Those fixture repositories must stay isolated, disposable, and uncommitted to this source repository. `scripts/prepare-skill-evals.ps1` enforces this: it writes to `.bot/` by default, refuses an `-OutputRoot` that is inside the repository but outside `.bot/`, and refuses `.bot/` itself if git has stopped ignoring it. diff --git a/README.md b/README.md index 59e69cb..75cdea7 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ One repo-wide convention matters especially for scaffolding skills: prefer dynam Another repo rule is intentionally strict: every repo-managed skill ships with its own `evals/evals.json`. These files are versioned review specifications whose prompts, fixtures, and expected outcomes are validated locally; they are not instructions to launch model sessions. -Git scenarios may use `workspace.git: true` for a staged initial commit, or a declarative object with `base_branch`, `feature_branch`, and `commits` (each containing a `message` and repository-relative `files` map; string values write UTF-8 content and null deletes a file). Preparation creates local `origin/` and `origin/HEAD` refs before applying feature commits, with fixed identity and timestamps. No remote access or setup commands are required. `dotnet-change-impact` eval 9 uses this to exercise default-base resolution against real paired Git history. +Git scenarios may use `workspace.git: true` for a staged initial commit, or a declarative object with `base_branch`, `feature_branch`, and `commits` (each containing a `message` and repository-relative `files` map; string values write UTF-8 content and null deletes a file). Preparation creates local `origin/` and `origin/HEAD` refs before applying feature commits, with fixed identity and timestamps. No remote access or setup commands are required. `dotnet-change-impact` eval 9 uses this to exercise default-base resolution against real paired Git history. This exception is only for isolated disposable eval fixtures under the approved workspace roots; throwaway Git history is still forbidden in the real source working tree. Copilot eval workers now require physical projections outside package/source ancestry, preserving staged repository instructions while excluding ambient ancestor instructions. Captured boundary contradictions invalidate execution. Phase 2 explicitly follows the packaged Anthropic grader and requires assertion-specific, source-backed PASS evidence; generic completion statements cannot finalize. See the [runner integrity contract](scripts/eval-runners/README.md) for the projection limits and checkpoint telemetry rules. diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index b4da584..aedb05e 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1533,6 +1533,8 @@ Add-ValidationResult -Results $results -Name 'Skill evaluation prepares portable Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'git does not ignore it' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '`.bot/-workspace/` — the default.' Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Anywhere else inside the repository is forbidden' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Deterministic synthetic Git history is allowed only when it is the fixture being tested' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'throwaway Git history is still forbidden in the real source working tree' Assert-Contains -Name 'scripts/prepare-skill-evals.ps1' -Content $prepare -Needle 'It did not run them, and nothing here will.' if (-not [string]::IsNullOrWhiteSpace($Ref)) {