V0.9.1/evals - #39
Open
gimlichael wants to merge 104 commits into
Open
Conversation
Update AGENTS.md, CONTRIBUTING.md, README.md, and CHANGELOG.md to document the new portable evaluation handoff process, eval runner infrastructure, and updated automation prohibition. Clarify the roles of Eval Orchestrator, Eval Runner, Grader, and Human Reviewer in the skill evaluation workflow.
Add portable evaluation runner framework with support for multiple harnesses: Codex, OpenCode, and deterministic Fake runner. Includes contract schemas, runner adapters, result bridging, and conformance tests. Refactor eval preparation and report generation scripts to support the new runner abstraction.
Greptile SummaryThe PR introduces a harness-neutral evaluation-runner boundary with package preparation, runner resolution, preflight, execution, evidence freezing, grading, and reporting support.
Confidence Score: 1/5The PR does not appear safe to merge because both Codex and OpenCode can accept pragmatic-isolation results whose workers retain unrestricted access to host-side grading and sibling-arm data. Without hard filesystem confinement, both adapters continue executing model processes that can access arbitrary host paths while recording sibling-arm and grading-material visibility as false, allowing contaminated runs to pass as blind evaluation evidence. Files Needing Attention: scripts/eval-runners/codex/runner.ps1; scripts/eval-runners/opencode/runner.ps1 Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[Repository preparation] --> B[Prepared eval package]
B --> C[External orchestrator]
C --> D[Selected Eval Runner]
D --> E[Runner-owned paired arms]
E --> F[Execution results]
F --> G[Immutable evidence freeze]
G --> H[Deterministic bridge]
H --> I[Grading and reports]
Reviews (46): Last reviewed commit: "🧱 enable scripted copilot transcripts w..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Update CONTRIBUTING.md and README.md to document runner infrastructure, installation procedures, and eval orchestrator workflow.
Add isolation capability assessment and preflight validation to detect unsupported runner configurations. Implement Cline runner adapter for eval orchestration. Enhance common runner utilities with sandbox path mapping, external command versioning, and environment management. Expand conformance tests with additional event fixtures.
Add GitHub Copilot CLI as a supported Eval Runner alongside Cline, Codex, and OpenCode. The runner handles Copilot-specific authentication (GitHub tokens), isolation constraints, and JSONL-based event output parsing. Includes conformance tests and reference documentation for the runner protocol compliance.
Extend prepare-skill-evals.ps1 and validate-skill-templates.ps1 to support GitHub Copilot runner configuration and execution profile selection. Enables dynamic runner resolution and integration with the new GitHub Copilot CLI evaluation path.
Establish GitHub Copilot with claude-haiku-4.5 as the Codebelt reference evaluation configuration for economical and stable comparison across evals. Update eval isolation terminology from 'hermetic' to 'isolated' and clarify Windows support as a first-class pragmatic evaluation target. Document filesystem confinement as a confidence enhancement rather than a universal prerequisite, enabling broader platform coverage.
Modernize the GitHub Copilot eval runner to deliver the prepared prompt through stdin instead of the --prompt argument. This improves byte fidelity and supports larger prompts without hitting command-line length limits. Update authentication handling to follow Copilot's normal precedence: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN environment variables, then OS keychain, then GitHub CLI fallback through GH_CONFIG_DIR. Remove --no-custom-instructions so repository-owned instructions remain visible to both paired arms. Add COPILOT_CACHE_HOME isolation. Refine --secret-env-vars to filter all listed token variables from shell and MCP child environments. Update preflight to report conditional native keychain readiness without requiring a live model request. Extend runner conformance tests to validate stdin prompt delivery with byte-exact verification, multiple token variables, repository instruction visibility, and authentication source detection.
The codex runner writes to the evidence directory but was not ensuring it existed first. Adding defensive directory creation prevents file-not-found errors when the evidence directory hasn't been created yet.
Change from defensive directory creation to explicit validation. The conformance test now requires the output parent directory to already exist and exits with a clear error message if it doesn't, improving error clarity when the directory structure is misconfigured.
Restructured AGENTS.md Skill Authoring section for clarity, brevity, and progressive disclosure. Updated README.md skill catalog to note that descriptions are lean activation metadata. Optimized all 21 repo-managed skill descriptions to be concise, trigger-oriented metadata following the new authoring standard. Changes align description quality with specification requirements and improve skill discoverability.
Added Layered Capability Classification section to distinguish classification of independently selectable adapters and runners from their parent directory or framework. This ensures that new child adapters are correctly classified as Added rather than grouped with pre-existing framework changes. Added comprehensive eval case covering framework refinement, new adapter introduction, defect repair, and layered state classification to validate the enhanced guidance.
Extended git-keep-a-changelog skill validation to verify the presence and correctness of the new Layered Capability Classification section and its guidance on adapter/runner boundary classification. Added checks for section presence, guidance on child adapter states, refinement classification, and avoidance of repeat classification patterns.
Regenerated [0.9.1] release entry from current git state (2026-08-22) to capture all commits through HEAD, including GitHub Copilot CLI runner support, Cline runner addition, skill description optimization, and AGENTS.md authoring guidance restructuring. Updated release highlight, Added/Changed/Fixed sections, and compare link to reflect the complete v0.9.1 release scope.
Updated repository guidelines, contributor expectations, and evaluation process documentation to distinguish Eval Runners, Orchestrators, and Graders while clarifying the explicit external-handoff boundary. All 21 repo-managed skill descriptions refactored for trigger-oriented activation metadata following progressive disclosure. Improved section organization in AGENTS.md for skill authoring, form handling, and dynamic defaults.
Added entry documenting the harness model resolution workflow, removal of the redundant provider field from execution-profile.json, and introduction of scripts/Get-HarnessModels.ps1 for current model discovery with Codebelt Reference verification and platform-specific filtering.
Modernized eval runner implementations across all harness types (GitHub Copilot, Codex, OpenCode, Cline) with updated contract schemas. Added scripts/Get-HarnessModels.ps1 for discovering current model selectors per harness with Codebelt Reference verification and platform-specific filtering. Improved prepare-skill-evals.ps1 to resolve Harness + Model before package generation, updated validation and conformance testing, and refined execution-profile.json to remove redundant provider field while treating model selectors as runner-native opaque strings.
Added manifest-paths.ps1 and bridge-manifest-results.ps1 for safe manifest path resolution and result artifact bridging. Updated prepare-skill-evals.ps1 with improved error handling and result collection. Enhanced generate-eval-report.ps1 for report generation and grading workflow. Expanded validate-skill-templates.ps1 with comprehensive fixture validation. Improved conformance testing in test-runner-conformance.ps1 to cover all eval runner contracts.
Update repository guidance in AGENTS.md, CONTRIBUTING.md, and README.md to explicitly clarify that the eval completion gate must be satisfied before a package is presented as successfully completed. Incomplete or unrun evaluation packages must be flagged as incomplete, not misrepresented as passing.
Update the generated README.md content in prepare-skill-evals.ps1 to reflect the clarified eval completion gate behavior. Packages generated by this script now include text clarifying that incomplete or unrun arms must be reported as such and that -CollectResults exits non-zero when the completion gate is not satisfied.
Implements deterministic queue and state management for eval workers. Adds native delegation surface with full-capability and model-lock guarantees. Each eval arm executes in an isolated harness-native worker with working-directory isolation, result capture, and fresh session constraints. Supports concurrent worker coordination with configurable capacity slots and rejection handling without eval attempt increment.
Adds native worker terminal evidence requirements and validation logic. Enhances delegation capability assessment with status tracking (supported/conditional/unsupported). Implements terminal evidence collection in orchestration tests including session tracking, model observation, directory isolation verification, and prompt fidelity checks. Updates result bridging to validate native worker evidence when required. Expands runner descriptor validation and adds comprehensive terminal evidence audit trail.
Update SKILL.md and evals.json to explicitly clarify that when the user provides 'yolo' or 'auto' in an explicit commit request, the skill should complete the workflow in the same turn after required checks pass, rather than awaiting additional approval or returning a pending plan.
Add validation assertions to ensure the git-visual-commits SKILL.md contains required phrases about completing the commit workflow in the same turn and that the plan summary is status output rather than a review request.
Update Git Operations Safeguards section to explicitly state that yolo or auto on an explicit commit request counts as approval to complete the commit workflow in the same turn, and clarifies that such approval must be attached to the same explicit commit request rather than treated as a standalone approval modifier.
Move Get-RunnerPreflightTimeoutSeconds from invoke-runner-owned-arms.ps1 to runner-common.ps1 as the authoritative source, making it the single point of truth for the fixed 120-second model-free preflight timeout. Update all callers in prepare-skill-evals.ps1 to use this centralized function. Fix PowerShell collection handling in runner-common to preserve empty HashSets on return. Add test coverage for collection preservation and variable preflight timeout calculations.
OpenCode discovery no longer filters models by availability. Instead, it mirrors every model exposed by all configured providers and returns availability only as presentation metadata. When OpenCode is selected without an explicit model, interactive preparation must present all discovered selectors and wait for the user's choice; the first, free, recommended, or previous model may not be selected automatically.
Remove the Get-PolicyName and Select-ModelsByPolicy functions that filtered OpenCode discovery to free models only. All harnesses now return all available models without filtering by availability. Update OpenCode CLI invocation from 'opencode models opencode --verbose' to 'opencode models --verbose' to avoid duplicating the provider name. This is a breaking change: OpenCode discovery now returns paid and unknown-availability models alongside free models.
Update test fixtures and assertions to validate OpenCode discovery now returns all models regardless of availability. Replace fixture model selectors with provider/model format (e.g., 'provider-free/free-model', 'provider-paid/Paid.Model'). Change OpenCode test expectations from filtering free-only to verifying all models are returned. Add test cases for paid and unknown-availability models. Add tests to verify the 'opencode models --verbose' command format. Add live-execution tests with a fake OpenCode CLI to validate provider and availability metadata are preserved. Remove the old test that expected failure when no free models were available.
Restructure the workflow to enforce a confirmation gate where the skill presents resolved parameters and explicitly asks the user to confirm before generating or writing the .snk file. Update evals to validate this behavior.
Fix OpenCode test assertions for platform-specific environment variables on non-Windows platforms. Fix PATH separator handling in model discovery. Add comprehensive validation rules for strong-name signing confirmation gate pattern.
Implement shared observability primitives for eval runners: a mandatory heartbeat cadence, lifecycle vocabulary, activity tracking that tees captured output to evidence files while exposing safe metadata, and a STDERR-only progress writer that never contaminates STDOUT protocol. Observability never blocks completion, never weakens watchdog enforcement, and never prints secrets or model content.
Pass observability context (runner id, phase, worker metadata) to Invoke-RunnerProcess for all model-process launches across Codex, GitHub Copilot, and OpenCode runners. Each runner's model-CLI lifecycle is now tracked and relayed to the shared observability system.
Add dedicated test module for observability regression testing and update existing runner tests to work with new progress tracking and activity streams. Coverage includes lifecycle state transitions, heartbeat emission, activity metadata accuracy, and STDERR relay correctness.
Add deterministic regression gate for runner observability and update Phase 1 validator to separate STDERR observability output from STDOUT machine protocol. This ensures heartbeats and progress signals never corrupt the summary JSON parsing, maintaining validator determinism.
Document the runner observability system: shared primitives, activity tracking, heartbeat semantics, lifecycle vocabulary, progress persistence, and security guarantees. Explain how observability maintains determinism while providing live feedback without blocking completion or weakening watchdog enforcement.
The freebuff runner is planned and blocked. Remove references from repository-level documentation to keep guidance current.
Add incremental stream activity tracking (events and bytes) to Invoke-RunnerProcess for live heartbeat metadata. Refactor stdout/stderr capture from ReadToEndAsync to CopyToAsync through activity streams. Add comprehensive observability tests for active inner processes, OpenCode streaming, Codex app-server protocol events, and synchronous preflight completion. Remove freebuff-readiness.md (planned/blocked runner). Update validator assertion for current runner support.
Refactor git-keep-a-changelog eval case 22 to reflect current runner support landscape. Replace Cline adapter references with OpenCode. Remove assertion for freebuff-readiness (planned runner). Update SKILL.md example to show GitHub Copilot and OpenCode as independently selectable runners instead of historical options.
Enhance runner-eval infrastructure with better heartbeat tracking and activity monitoring. Track lastProtocolActivityUtc in app-server protocol loops and emit it in progress events. Implement bounded heartbeat emission that prevents timeout stalls during I/O waits. Add outputDrainCompleted field to diagnose child process output draining. Remove stderr tail sanitization in favor of structured state tracking. Comprehensive test coverage for Codex app-server observability in both success and timeout scenarios.
New-PreflightWorkerSummary in invoke-runner-owned-arms.ps1 was building operator-facing failure reasons by concatenating raw child Stdout, Stderr, and ParseError exception text. Any of those may carry secrets, model output, or malformed content that should never appear in human-facing progress or terminal summaries. Extracted the function (plus Get-PreflightGateSummary) into a new preflight-summary.ps1 module that enforces the boundary structurally: only safe, enumerable facts are used in reasons — exit status, timeout state, termination observed, whether valid JSON was returned, and the runner's own structured reasons. Raw child output is never an input to operator summary construction. The extraction also makes both functions independently dot-sourceable for unit tests without running the full Phase 1 orchestration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
During active OpenCode execution (iteration 10) the operator console received both the parent runner heartbeat and the relay's own periodic liveness at nearly the same cadence, producing unnecessary visual noise without adding information. The fix separates the two channels: structured JSONL evidence still records every parent and relay event, but the human console suppresses a parent periodic heartbeat line when the nested relay has recently demonstrated that the same logical worker is alive. The parent heartbeat resumes on the console once the relay has been quiet for a full heartbeat interval, so a genuinely silent worker remains externally observable. Mechanism: Send-RunnerChildRelay now returns the count of relay events it emitted. Invoke-RunnerChildHeartbeatTick uses that count to track LastRelayActivityUtc on the child. When the periodic heartbeat interval elapses, the event is written to the JSONL log unconditionally but passed to Write-RunnerProgress with the new LogOnly switch when relay was recently active, suppressing only the console line. Terminal events, state transitions, and first-activity announcements are never suppressed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two new deterministic test suites, both model-free and network-free. test-preflight-summary.ps1 tests New-PreflightWorkerSummary directly by constructing synthetic invocation records with secret-like STDERR content, arbitrary STDOUT, and malformed JSON parse errors. It proves that none of that raw content escapes into the returned summary or reasons, that the safe failure classification (invalid JSON, no JSON, exit status) still appears, that compatible preflights remain unaffected, and that runner-provided structured reasons are forwarded. test-progress-coalescing.ps1 drives synthetic child processes through the real process primitive with relay sentinels on their STDERR. It proves that meaningful lifecycle events always appear on the operator console, that a quiet worker still produces periodic parent heartbeats, that active relay suppresses near-adjacent parent periodic heartbeats from the console while the JSONL log retains all events, that parent heartbeats resume after relay goes quiet, that terminal events are never suppressed by coalescing, and that the JSONL evidence contains both parent and relay event streams. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds two new validation gates to the repository deterministic validation suite: one for the preflight raw-output boundary tests and one for the progress coalescing tests. Both are gated on local worktree runs and skipped on ref-based checks. The validator now surfaces them as named checks alongside the existing runner observability suite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract grading validation logic into a dedicated eval-grading-contract.ps1 module. Standardize JSON file writing across all eval-runners scripts using Write-RunnerJsonFile. Add validate-eval-grading.ps1 as a deterministic pre-finalization validation helper. This consolidates shared validation rules and ensures consistent JSON handling across the evaluation infrastructure.
Extract eval-runner identity resolution into Assert-PackageRunnerIdentity to eliminate duplication and provide a single source of truth for runner name, path, profile, and descriptor. Simplify callers (bridge-manifest-results, freeze-execution-evidence, invoke-runner-owned-arms, finalize-eval-package) to use the centralized function. Add runner identity validation to package preparation and template validation to enforce contract compliance across all eval-runner operations.
Detect policy-blocked tool signals in Codex structured results (status=blocked/denied/policy_blocked; error.code matching policy rejection patterns). Track item status and error metadata through command execution results. Introduce Get-CodexBehavioralCapabilityFailure and Test-CodexPolicyBlockedToolSignal functions to identify when workspace operations were rejected by effective runtime policy. Record behavioral capability evidence and failures in execution result for proper diagnosis and incompatibility reporting.
The structured tool-result detector could not observe the iteration-12 failure shape, where the Codex runtime tool-router rejects every execution attempt below the app-server protocol layer and emits no commandExecution items. The rejection appeared only in runtime STDERR. Adds Get-CodexStderrPolicyRejections, which recognises the proven Codex tool-router family (codex_core provenance + exec_command failed/CreateProcess + Rejected/blocked by policy) on a per-line basis. Bare occurrences of 'blocked by policy' without that provenance are ignored. Classifies as globally incompatible only when STDERR rejections are observed and no successful workspace operation was recorded from the structured event stream, preserving the distinction between a globally broken execution environment and a legitimate outside-workspace isolation denial. Raw STDERR is never surfaced in operator-facing failure messages; only the rejection count and stable failure code are reported. Deterministic fixtures cover the iteration-12 STDERR-only shape, an outside-workspace isolation denial with a usable workspace, and STDERR false-positive protection, in addition to preserving all existing structured-item and prose-false-positive coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add case-insensitive runner validation and normalize harness intent. Enhance Get-HarnessModels.ps1 to parse GitHub Copilot help config for model discovery and support additional environment variables for harness home/config directories. Improve prepare-skill-evals.ps1 with model validation, clearer documentation, and default model resolution for Codex and GitHub Copilot. Update eval-runners infrastructure with refined runner configuration, improved isolation controls, and updated runner implementations. Enhance test fixtures and validation for eval harness conformance.
Update AGENTS.md with clarifications on eval script execution, model discovery, and runner configuration. Expand guidance on harness normalization and default model resolution. Standardize script invocation examples to use -NonInteractive flag. Update README.md and CONTRIBUTING.md to reflect current evaluation workflow and infrastructure improvements. Enhance eval-runners/README.md with details on native delegation, harness permissions, and model discovery mechanisms across runners.
Extend Codex runner to enforce native skill isolation during eval execution. This prevents eval prompts from accessing the Codex skill catalog, the candidate skill config, or any native skills, ensuring blind execution for unbiased skill evaluation. Update eval preparation to extract and track candidate skill names from SKILL.md frontmatter, enabling runners to suppress skill references in eval prompts. Add comprehensive isolation verification, path normalization utilities, and regression tests for conformance and observability.
Refactor native skill verification from config/read based probes to discovery-probe based approach with defense-in-depth session configuration. Discovery probe identifies ambient native skills; suppression selectors are generated for each discovered skill and applied via session configuration. Verification uses defense_in_depth_session_config method to confirm isolation without relying on authoritative config/read responses. Enhance test assertions to verify: discovery/behavioral context consistency, per-skill suppression selector application, newly enabled ambient skill detection, and ambient skill access observation. Tests now validate that native skill suppression is applied correctly across discovery and behavioral phases. This provides more robust verification that evaluated code cannot access ambient native skills.
Consolidate deterministic token normalization logic across bridge-execution-result and generate-eval-report. Add Get-NormalizedTotalTokens to handle cache-aware token calculations with fallback validation. Fix Codex runner UTF-8 encoding for process I/O to prevent character corruption. Enhance test-runner-conformance and validate-skill-templates with regression checks for token reporting consistency and strong-name output isolation.
Change eval #2 key output destination from absolute path C:\temp\keys to relative path under repository. This ensures arm-local isolation when paired runs execute concurrently with pragmatic filesystem confinement, and validates that each arm can override the destination independently without path conflicts.
Add infrastructure to capture and validate scripted Copilot multi-turn session transcripts. Introduces home directory baseline/restore cycle for test isolation, safe process invocation patterns, and portable transcript extraction from scripted interactions. New test suite validates transcript ordering, session identity continuity, and integrity across paired captures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a harness-agnostic Eval Runner execution boundary, clarifying and formalizing how evaluation packages are prepared, executed, and reported. The main goal is to ensure that all repository automation remains model-free and deterministic, with only explicit, human-directed external orchestration allowed to invoke model-backed evaluation runs. The update adds new runner protocol tools, schemas, and adapters, and updates documentation to reflect these changes and the strict separation between preparation and execution.
Eval Runner Protocol and Execution Boundary:
scripts/eval-runners/directory containing the common Eval Runner protocol (describe,preflight,execute), schemas forexecution-profile.jsonandexecution-result.json, a deterministic fake runner (for conformance), and initial adapters for Codex and OpenCode.execution-profile.json, runner protocol tools), keeping runner selection outside ofevals/evals.jsonand maintaining compatibility with Anthropic's report formats. [1] [2]Repository Automation and Execution Separation:
AGENTS.md,CONTRIBUTING.md,README.md) to distinguish roles (Eval Runner, Eval Orchestrator, Grader, Human Reviewer) and clarify the separation between deterministic preparation and external execution. [1] [2]Evaluation and Reporting Workflow Updates:
execution-result.json), and that the deterministic bridge produces the existing result shape for grading and reporting. [1] [2]Methodology and Compatibility:
incompatibleand no fallback or substitution provided.Changelog:
0.9.1) inCHANGELOG.mdsummarizing these protocol, workflow, and documentation changes.