From 7e1ffd62286db3d7c539b6bca5d77b5223a3a98f Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sat, 4 Jul 2026 23:23:43 -0600 Subject: [PATCH 01/10] =?UTF-8?q?docs(fleet):=20Slice=201=20spec=20?= =?UTF-8?q?=E2=80=94=20round-trip=20proof=20+=20prompt=20hardening=20(flee?= =?UTF-8?q?t-does-the-labor=20track)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 --- ...eet-labor-slice1-roundtrip-proof-design.md | 238 ++++++++++++++++++ 1 file changed, 238 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md diff --git a/docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md b/docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md new file mode 100644 index 0000000..d5b9f54 --- /dev/null +++ b/docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md @@ -0,0 +1,238 @@ +# Fleet Does the Labor — Slice 1: Round-Trip Proof + Prompt Hardening + +**Date:** 2026-07-04 · **Status:** design (approved shape, pending spec review) · +**Track:** "Fleet does the labor" (make the conductor farm coding work to non-Claude instruments) + +## Problem + +Baton's whole thesis is *command-and-control for a fleet of coding LLMs*, but +today the fleet only demonstrably works with Claude. The plumbing to invoke +other instruments exists and is real: + +- `fleet.yaml` carries a roster of 8 providers with invoke templates + (`claude`, `codex exec`, `agy`, `gh copilot`, `ollama`, a remote Ollama box, + two LM Studio pins, `gh models`). +- `Invoke-Fleet` / `Invoke-Fleet-Cli` / the http branch really shell out, with a + stdin-safe path for large/quoted prompts. +- `Select-Capability` routes by capability + cost; the conductor plans a task + DAG and routes each task to a chosen provider. + +But two things are unproven, and one seam is empty: + +1. **No proven end-to-end round-trip.** `fleet doctor` only checks *"is the + binary on PATH / is the base_url up."* It never sends a prompt and confirms a + coherent answer comes back. The repo `fleet.yaml` is a shared **seed**; the + live per-box roster (`~/.baton/fleet.yaml`, box-private) is where real + templates live and may be wrong or untested for a given machine. +2. **Prompt fragility.** The legacy CLI dispatch path interpolates the prompt via + `Invoke-Expression` — quote-fragile and subject to the 965-byte argument + ceiling. Only `stdin: true` providers survive real coding prompts. +3. **The executor seam is empty.** `Invoke-TaskViaFleet` is documented + *"Non-destructive by construction — it never touches the repo; real + code/merge execution is wired by a box via `-Spawner`."* So the conductor + routes, calls, records *which* model it chose and whether it exited 0 — then + discards the output. Nothing turns a model's work into a repo change. + +**This spec covers Slice 1 only:** prove the round-trip and harden the prompt +path, so that a later slice can build the executor on a pipe that is known to +work. Slice 1 does **not** apply repo changes, build the executor, or change +routing. The `-Spawner` executor (gap 3) is Slice 2, a separate spec. + +## Goal + +A user can run one command and get, per **enabled** instrument on their box, an +honest verdict: *this model actually answered a real prompt* — or *it didn't, +and here's why.* And the dispatch pipe is hardened so the real prompts a future +executor sends won't be mangled. + +## Scope & non-goals + +**In scope:** + +- A live round-trip probe over the enabled roster, surfaced through `fleet doctor`. +- A deterministic, judge-free pass criterion (a canary token). +- Making the stdin dispatch path the default for CLI providers so real prompts + survive. +- Plain-English + `--json` legibility of the results. +- Hermetic tests (fake dispatcher; never touches real CLIs, network, or + `~/.baton`). + +**Out of scope (later slices / separate specs):** + +- The `-Spawner` executor that applies repo changes (Slice 2). +- Any routing / `Select-Capability` change. +- Driving Baton *from* Codex/Gemini (the Command Center Codex-adapter follow-on). +- Per-provider latency benchmarking, sample-output capture, quality scoring. + +## Approach (chosen) + +**Extend `fleet doctor` with a `--live` probe** rather than adding a new command +or a `/baton:go` preflight. Doctor already iterates enabled providers and reports +`ok | skip | err`; the live probe is a second, opt-in pass over the same roster. +One health surface answers both "is my fleet reachable?" and "does my fleet +actually answer?" + +Rejected alternatives: + +- **New `/baton:fleet test` command** — a whole new surface that overlaps + doctor's job. A dedicated command is a reasonable *later* affordance once + there's more per-provider detail to show; not needed now. +- **Preflight inside `/baton:go`** — couples proof to execution and taxes every + run. Premature. + +## Design + +### 1. Surface & modes + +`scripts/fleet-doctor.ps1` gains a `-Live` switch (slash surface: `--live`) and +keeps `-Json`. + +- **Default (no `-Live`):** today's behavior verbatim — reachability probe + (binary on PATH; `base_url` / env-URL reachable). Byte-for-byte unchanged. +- **`--live`:** for each **enabled** provider, run the reachability check first; + if it passes, run a live canary round-trip. Disabled providers are `skip`. +- Exit code: `0` if every enabled provider is `live_ok`; `1` if any enabled + provider fails the live probe. (Matches doctor's existing all-ok/any-bad + contract.) + +### 2. The canary round-trip contract + +A new pure-ish helper (in `fleet-lib.ps1` or a small `fleet-probe-lib.ps1`, +implementer's call at plan time) sends a fixed canary prompt and classifies the +result. The probe dispatches through **`Invoke-Fleet`** (not `Invoke-Fleet-Cli` +directly) so both `kind: cli` and `kind: http` providers — including the local +LM Studio / Ollama boxes — are covered by the same code path. A `-Dispatcher` +scriptblock seam is injected for tests. + +- **Canary prompt (constant):** `Reply with exactly the word PONG and nothing else.` +- **Canary token (constant):** `PONG`. +- **Pass (`live_ok`):** dispatch returns exit 0 **and** stdout contains `PONG` + (case-insensitive, substring) **and** it completed within the probe timeout. +- **Fail (`live_fail`):** with a single-word reason: + - `not-on-PATH` — reachability check failed (cli binary missing). + - `unreachable` — reachability check failed (http base_url / env URL down). + - `timeout` — exceeded the probe timeout. + - `nonzero-exit` — dispatch exit code ≠ 0. + - `no-canary` — exit 0 but stdout lacks the token (e.g. the CLI printed its + help text, or the seed template is wrong for this box). **This is the payoff + line** — it distinguishes "the pipe/template is wrong" from "the model is + down." +- **Skip (`skip`):** provider disabled in `fleet.yaml`. + +**Timeout.** A probe timeout (default 60s; overridable via a `-TimeoutS` param / +`--timeout`) is measured and enforced. `Invoke-Fleet-Cli` does not currently +enforce its `TimeoutS` param, so the probe must wrap the dispatch in an enforced +timeout guard (e.g. a job/async wait) rather than assume the callee honors it; +http providers already carry `timeout_s`. The probe records elapsed seconds per +provider for the report. + +**Result shape (per provider):** + +``` +@{ name; kind; enabled; reachable = $true|$false; live = 'live_ok'|'live_fail'|'skip'; + reason = ; elapsed_s = |$null } +``` + +### 3. Prompt robustness (gap 2) + +Make the **stdin path the default** for `kind: cli` dispatch in +`Invoke-Fleet-Cli`: when a provider's resolved command is a clean token list +(exe + args, no shell metacharacters requiring interpolation), pass the prompt +via the existing temp-file→stdin mechanism instead of interpolating `{{prompt}}`. +This immunizes real (large, quote-heavy) prompts against the 965-byte ceiling and +quote mangling — the foundation Slice 2's executor depends on. + +- Providers already marked `stdin: true` are unchanged. +- Providers whose template still *requires* `{{prompt}}` interpolation (a shell + form that can't take stdin) keep the legacy path; the change is opportunistic, + not forced, so no seed template silently breaks. +- The canary probe itself **always** uses the stdin path. +- This must not regress the existing `Invoke-Fleet` cli tests; where a seed + template's semantics would change, prefer adding `stdin: true` to that seed + entry over rewriting dispatch behavior invisibly. + +> **Open implementation note for the plan:** the exact predicate for "clean token +> list, safe to send via stdin" must be pinned to a concrete, tested rule (e.g. +> "template has no `{{prompt}}` placeholder AND no shell operators +> `| > < & ; $(` ") so behavior is deterministic and covered by a unit test. The +> writing-plans step resolves this to exact code + test cases. + +### 4. Legibility + +Human report (doctor `--live`), one row per provider, plain English: + +``` +PROVIDER REACHABLE LIVE DETAIL +codex yes live_ok 1.2s +gemini-antigravity yes live_ok 3.4s +ollama-local yes live_fail timeout>60s +gh-copilot yes live_fail no-canary (returned help text, not an answer) +lm-studio yes live_ok 2.1s +github-models — skip disabled in fleet.yaml +``` + +`--json` emits the array of result shapes above for programmatic use (e.g. a +future dashboard tile). + +### 5. Testing (hermetic) + +- A **fake `-Dispatcher`** is injected into the probe so the suite never invokes + a real CLI, touches the network, or reads real `~/.baton`. The fake returns + canned `@{ stdout; stderr; exit_code }` tuples to exercise every branch: + `live_ok`, `nonzero-exit`, `no-canary`, `timeout` (simulated), `skip` for + disabled, and both `not-on-PATH` / `unreachable` reachability fails. +- Temp `fleet.yaml` fixtures; temp `BATON_HOME`; `try/finally` restore. Never + touch real `~/.baton`, `~/.claude`, `D:\Dev\Grimdex`, or `D:\dev`. +- Stdin-default dispatch gets unit tests for the "clean token list" predicate + (both directions) and a regression assert that `stdin: true` providers and + interpolation-required providers are unchanged. +- The live mode against real box CLIs is a **manual** diagnostic, not part of the + automated suite. + +### 6. Deploy & docs + +- If a new `fleet-probe-lib.ps1` is introduced, add it to the `bootstrap.ps1` + deploy manifest **and** add a `test-bootstrap.ps1` deploy assert (the v1.8.0 + coach-lib omission lesson: every new deployed script gets a deploy assert). +- `commands/` doc for `fleet doctor` updated to document `--live` / `--timeout` + / `--json`. +- `AGENTS.md`: one line noting `fleet doctor --live` as the model-agnostic way to + verify any box's roster actually answers. +- Plugin version bump (minor) at release. + +## House rules (§11, per project standing rules) + +- Every shell command arg < 965 bytes; large prompts go via file/stdin. +- CLI errors: `[Console]::Error.WriteLine(...)` + `exit 2` (never `Write-Error` + under `Stop`). Doctor keeps its existing exit-code contract. +- All file writes `utf8NoBOM`. +- `ConvertFrom-Json` auto-parses ISO dates to `DateTime` — re-stringify on + round-trip. `ConvertTo-Json` needs `-InputObject @(...)` for guaranteed arrays. +- Never name PS vars `$args/$input/$event/$matches/$host/$pid`. +- Unary-comma flatten `,([object[]]$x)` only on direct-assignment returns; use + `@($x)` when callers pipe. +- Guard `0/0` NaN in any elapsed/utilization math. +- Box-private: never write real roster/endpoint values into the shared seed + `fleet.yaml`; placeholder hosts only. The live probe reads the box-private + live roster at run time. + +## Decisions made + +- **Prove the round-trip before building the executor** — cheap de-risking slice + first; the executor (gap 3) is designed against a pipe known to work. +- **Extend `fleet doctor --live`** rather than a new command or a `go` preflight — + one health surface, minimal new code. +- **Canary-token pass criterion** (`PONG`), judge-free — catches help-text / + wrong-template / garbage responses, not just exit 0. +- **Stdin path as the CLI dispatch default** — hardens the pipe for the real + prompts Slice 2 will send. + +## Out-of-scope follow-ons (named, not built here) + +- **Slice 2 — the `-Spawner` executor:** send the task to the chosen instrument, + capture its output/edits, turn that into an applied repo change (agentic tools + edit in-place; chat models emit a diff Baton applies, or file-edit tasks route + only to agentic instruments — resolved in Slice 2's spec), verify via the + existing acceptance gate, work on a branch/worktree for reversibility. +- Per-provider latency/quality benchmarking; sample-output capture. +- A dashboard tile consuming `fleet doctor --live --json`. From c55a49fb06dffe0dfe3b83aaf289d841bbbe3d6b Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sat, 4 Jul 2026 23:38:37 -0600 Subject: [PATCH 02/10] =?UTF-8?q?docs(fleet):=20Slice=201=20implementation?= =?UTF-8?q?=20plan=20=E2=80=94=20round-trip=20proof=20+=20prompt=20hardeni?= =?UTF-8?q?ng?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 --- ...7-04-fleet-labor-slice1-roundtrip-proof.md | 614 ++++++++++++++++++ 1 file changed, 614 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-04-fleet-labor-slice1-roundtrip-proof.md diff --git a/docs/superpowers/plans/2026-07-04-fleet-labor-slice1-roundtrip-proof.md b/docs/superpowers/plans/2026-07-04-fleet-labor-slice1-roundtrip-proof.md new file mode 100644 index 0000000..3906290 --- /dev/null +++ b/docs/superpowers/plans/2026-07-04-fleet-labor-slice1-roundtrip-proof.md @@ -0,0 +1,614 @@ +# Fleet Does the Labor — Slice 1: Round-Trip Proof + Prompt Hardening — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Give Baton a `fleet doctor --live` probe that proves each enabled instrument actually answers a real prompt, and harden the CLI dispatch path so real (large/quoted) prompts survive. + +**Architecture:** A new pure-plus-seamed `fleet-probe-lib.ps1` sends a fixed canary prompt through `Invoke-Fleet` to every enabled provider and classifies the result (canary-token match, judge-free). `fleet-doctor.ps1` gains a `--live` pass over the same roster. Separately, `Invoke-Fleet-Cli` switches to the existing stdin dispatch path for providers whose template ends in a standalone quoted `"{{prompt}}"`, immunizing real prompts against the 965-byte / quote-mangling wall. + +**Tech Stack:** PowerShell 7 (pwsh), the existing `fleet-lib.ps1` dispatch, `Start-ThreadJob` (bundled ThreadJob module) for the probe timeout guard. + +## Global Constraints + +Every task's requirements implicitly include these (spec §11): + +- Every shell command arg < 965 bytes; large prompts go via file/stdin. +- CLI/script errors: `[Console]::Error.WriteLine(...)` + `exit 2` (never `Write-Error` under `Stop`). `fleet-doctor.ps1` keeps its existing exit-code contract (0 = all enabled ok, 1 = any enabled bad). +- All file writes use `-Encoding utf8NoBOM`. +- `ConvertFrom-Json` auto-parses ISO dates to `[datetime]` — re-stringify on round-trip. `ConvertTo-Json` guaranteed-array output uses `-InputObject @(...)`. +- Never name PowerShell vars `$args` / `$input` / `$event` / `$matches` / `$host` / `$pid`. +- Unary-comma flatten `,([object[]]$x)` ONLY on direct-assignment returns; use `@($x)` when callers pipe. +- Guard `0/0` NaN in any division (this slice has none — assert none is introduced). +- Box-private: never write real roster/endpoint values into the shared seed `fleet.yaml`; placeholder hosts only. The live probe reads the box-private live roster at run time. +- Tests are hermetic: temp `fleet.yaml` fixtures, temp `BATON_HOME`, `try/finally` restore. NEVER touch real `~/.baton`, `~/.claude`, `D:\Dev\Grimdex`, or `D:\dev`. +- The canary prompt constant is exactly `Reply with exactly the word PONG and nothing else.` and the token constant is exactly `PONG`. + +## File Structure + +- **Create** `scripts/fleet-probe-lib.ps1` — canary constants, `Test-FleetCanary` (pure classifier), `Test-ProviderReachable` (reachability, injectable URL probe), `Invoke-FleetProbe` (per-provider live round-trip with timeout guard + injectable dispatcher). +- **Create** `scripts/test-fleet-probe-lib.ps1` — unit tests for the three functions (fake dispatcher / injected URL probe; every branch). +- **Modify** `scripts/fleet-lib.ps1` — `Invoke-Fleet-Cli` gains `Test-StdinSafe` predicate + stdin-default routing. +- **Modify** `scripts/test-fleet-dispatch.ps1` — add predicate + stdin-default regression asserts. +- **Modify** `scripts/fleet-doctor.ps1` — `-Live`, `-TimeoutS`, `-Json` live rendering + exit contract. +- **Modify** `scripts/test-fleet-doctor.ps1` — end-to-end `--live` asserts. +- **Modify** `scripts/bootstrap.ps1` — add `fleet-probe-lib.ps1` to the deploy manifest. +- **Modify** `scripts/test-bootstrap.ps1` — add a deploy assert for `fleet-probe-lib.ps1`. +- **Modify** `commands/fleet.md` — document `doctor --live [--timeout ] [--json]`. +- **Modify** `AGENTS.md` — one line: `fleet doctor --live` as the model-agnostic roster verification. +- **Modify** `.claude-plugin/plugin.json` — version bump `1.9.0` → `1.10.0-rc.1`. + +--- + +### Task 1: Canary classifier + reachability (`fleet-probe-lib.ps1` part 1) + +**Files:** +- Create: `scripts/fleet-probe-lib.ps1` +- Test: `scripts/test-fleet-probe-lib.ps1` + +**Interfaces:** +- Produces: + - `$script:FleetCanaryPrompt` = `'Reply with exactly the word PONG and nothing else.'` + - `$script:FleetCanaryToken` = `'PONG'` + - `Test-FleetCanary -Output -ExitCode -TimedOut ` → `@{ live = 'live_ok'|'live_fail'; reason = }`. Reason values: `timeout`, `nonzero-exit`, `no-canary`, or `$null` when `live_ok`. + - `Test-ProviderReachable -Provider [-UrlProbe ]` → `@{ reachable = ; reason = $null|'not-on-PATH'|'unreachable' }`. `-UrlProbe` takes a URL string, returns `$true` reachable / `$false` not; default does `Invoke-WebRequest -Method Head -TimeoutSec 5 -UseBasicParsing`. + +- [ ] **Step 1: Write the failing test** + +Create `scripts/test-fleet-probe-lib.ps1`: + +```powershell +#!/usr/bin/env pwsh +# Tests for scripts/fleet-probe-lib.ps1 — canary classifier, reachability, live probe. +$ErrorActionPreference = 'Stop' +. (Join-Path $PSScriptRoot 'fleet-probe-lib.ps1') + +$failures = 0 +function Assert($label, $cond) { + if ($cond) { Write-Host "PASS $label" -ForegroundColor Green } + else { Write-Host "FAIL $label" -ForegroundColor Red; $script:failures++ } +} + +# --- Test-FleetCanary (pure) --- +$c1 = Test-FleetCanary -Output 'PONG' -ExitCode 0 -TimedOut $false +Assert "canary: exact token -> live_ok" ($c1.live -eq 'live_ok' -and $null -eq $c1.reason) +$c2 = Test-FleetCanary -Output 'the answer is pong.' -ExitCode 0 -TimedOut $false +Assert "canary: case-insensitive substring -> live_ok" ($c2.live -eq 'live_ok') +$c3 = Test-FleetCanary -Output 'usage: codex [options]' -ExitCode 0 -TimedOut $false +Assert "canary: exit 0 but no token -> no-canary" ($c3.live -eq 'live_fail' -and $c3.reason -eq 'no-canary') +$c4 = Test-FleetCanary -Output 'PONG' -ExitCode 3 -TimedOut $false +Assert "canary: nonzero exit beats token -> nonzero-exit" ($c4.live -eq 'live_fail' -and $c4.reason -eq 'nonzero-exit') +$c5 = Test-FleetCanary -Output '' -ExitCode 0 -TimedOut $true +Assert "canary: timeout beats all -> timeout" ($c5.live -eq 'live_fail' -and $c5.reason -eq 'timeout') + +# --- Test-ProviderReachable --- +$cliOk = Test-ProviderReachable -Provider @{ name='p'; kind='cli'; command_template='pwsh -NoProfile -Command "x"' } +Assert "reachable: pwsh on PATH -> reachable" ($cliOk.reachable -eq $true -and $null -eq $cliOk.reason) +$cliNo = Test-ProviderReachable -Provider @{ name='p'; kind='cli'; command_template='definitely-not-a-real-binary-xyz foo' } +Assert "reachable: missing binary -> not-on-PATH" ($cliNo.reachable -eq $false -and $cliNo.reason -eq 'not-on-PATH') +$httpOk = Test-ProviderReachable -Provider @{ name='h'; kind='http'; base_url='http://x' } -UrlProbe { param($u) $true } +Assert "reachable: http probe true -> reachable" ($httpOk.reachable -eq $true) +$httpNo = Test-ProviderReachable -Provider @{ name='h'; kind='http'; base_url='http://x' } -UrlProbe { param($u) $false } +Assert "reachable: http probe false -> unreachable" ($httpNo.reachable -eq $false -and $httpNo.reason -eq 'unreachable') + +if ($failures -gt 0) { Write-Host "`n$failures failure(s)" -ForegroundColor Red; exit 1 } +Write-Host "`nAll tests passed." -ForegroundColor Green +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `pwsh -NoProfile -File scripts/test-fleet-probe-lib.ps1` +Expected: FAIL — `fleet-probe-lib.ps1` does not exist / functions not defined. + +- [ ] **Step 3: Write minimal implementation** + +Create `scripts/fleet-probe-lib.ps1`: + +```powershell +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Fleet round-trip probe (Slice 1). Sends a canary prompt to an enabled provider + and classifies whether it actually answered. Judge-free: a deterministic token. +.NOTES + Pure classifier (Test-FleetCanary) + reachability (Test-ProviderReachable) + + live round-trip (Invoke-FleetProbe, added in Task 2). Diagnostic only — never + mutates state, never throws on a provider failure. +#> +. "$PSScriptRoot/baton-home.ps1" +. "$PSScriptRoot/fleet-lib.ps1" # Read-Fleet, Invoke-Fleet, Get-FleetProvider + +$script:FleetCanaryPrompt = 'Reply with exactly the word PONG and nothing else.' +$script:FleetCanaryToken = 'PONG' + +function Test-FleetCanary { + <# Pure. Classify a dispatch result into a live verdict + reason. + Precedence: timeout > nonzero-exit > token-match. #> + param( + [string]$Output, + [int]$ExitCode = 0, + [bool]$TimedOut = $false + ) + if ($TimedOut) { return @{ live = 'live_fail'; reason = 'timeout' } } + if ($ExitCode -ne 0) { return @{ live = 'live_fail'; reason = 'nonzero-exit' } } + if (([string]$Output).ToUpperInvariant().Contains($script:FleetCanaryToken)) { + return @{ live = 'live_ok'; reason = $null } + } + return @{ live = 'live_fail'; reason = 'no-canary' } +} + +function Test-ProviderReachable { + <# Is the provider's transport up? cli -> binary on PATH; http -> base_url HEAD. + -UrlProbe injects the reachability check for tests. Returns @{reachable;reason}. #> + param( + [Parameter(Mandatory)][hashtable]$Provider, + [scriptblock]$UrlProbe + ) + if (-not $UrlProbe) { + $UrlProbe = { + param($url) + try { Invoke-WebRequest -Uri $url -Method Head -TimeoutSec 5 -UseBasicParsing | Out-Null; return $true } + catch { return $false } + } + } + if ($Provider.kind -eq 'cli') { + $bin = ([string]$Provider.command_template -split '\s+')[0] + if (Get-Command $bin -ErrorAction SilentlyContinue) { return @{ reachable = $true; reason = $null } } + return @{ reachable = $false; reason = 'not-on-PATH' } + } + if ($Provider.kind -eq 'http') { + if (& $UrlProbe ([string]$Provider.base_url)) { return @{ reachable = $true; reason = $null } } + return @{ reachable = $false; reason = 'unreachable' } + } + # Unknown kind: treat as unreachable rather than throwing. + return @{ reachable = $false; reason = 'unreachable' } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `pwsh -NoProfile -File scripts/test-fleet-probe-lib.ps1` +Expected: PASS — all 9 asserts green. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/fleet-probe-lib.ps1 scripts/test-fleet-probe-lib.ps1 +git commit -m "feat(fleet): canary classifier + reachability probe (Slice 1 part 1)" +``` + +--- + +### Task 2: Live round-trip with timeout guard (`fleet-probe-lib.ps1` part 2) + +**Files:** +- Modify: `scripts/fleet-probe-lib.ps1` (append `Invoke-FleetProbe`) +- Test: `scripts/test-fleet-probe-lib.ps1` (append live-probe asserts) + +**Interfaces:** +- Consumes: `Test-FleetCanary`, `Test-ProviderReachable`, `$script:FleetCanaryPrompt` (Task 1); `Invoke-Fleet` (fleet-lib). +- Produces: + - `Invoke-FleetProbe -Provider [-TimeoutS =60] [-FleetPath ] [-Dispatcher ] [-UrlProbe ]` → per-provider result hashtable: + `@{ name; kind; enabled; reachable; live; reason; elapsed_s }`. + - `live` ∈ `live_ok | live_fail | skip`. + - `reason` ∈ `$null | disabled | not-on-PATH | unreachable | timeout | nonzero-exit | no-canary | dispatch-error`. + - Disabled provider → `@{ live='skip'; reason='disabled'; reachable=$null; elapsed_s=$null }`. + - `-Dispatcher` contract (for tests): a scriptblock invoked as `& $Dispatcher $Provider $FleetPath $CanaryPrompt $ScriptRoot`, returning `@{ stdout=; exit_code= }` (matching `Invoke-Fleet`'s shape). It runs inside a `Start-ThreadJob`, so a fake may `Start-Sleep` to exercise the timeout path. + +- [ ] **Step 1: Write the failing test** + +Append to `scripts/test-fleet-probe-lib.ps1` BEFORE the final tally block: + +```powershell +# --- Invoke-FleetProbe (live round-trip) --- +# Fake dispatchers returning the Invoke-Fleet shape @{stdout;exit_code}. +$okDisp = { param($prov,$fp,$canary,$root) @{ stdout = 'PONG'; exit_code = 0 } } +$noTokDisp = { param($prov,$fp,$canary,$root) @{ stdout = 'help text here'; exit_code = 0 } } +$failDisp = { param($prov,$fp,$canary,$root) @{ stdout = ''; exit_code = 7 } } +$slowDisp = { param($prov,$fp,$canary,$root) Start-Sleep -Seconds 5; @{ stdout = 'PONG'; exit_code = 0 } } +$throwDisp = { param($prov,$fp,$canary,$root) throw 'boom' } + +$enabledCli = @{ name='w'; kind='cli'; enabled=$true; command_template='pwsh -NoProfile -Command "x"' } + +$rSkip = Invoke-FleetProbe -Provider @{ name='d'; kind='cli'; enabled=$false; command_template='pwsh x' } +Assert "probe: disabled -> skip" ($rSkip.live -eq 'skip' -and $rSkip.reason -eq 'disabled') + +$rOk = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $okDisp +Assert "probe: token -> live_ok" ($rOk.live -eq 'live_ok' -and $rOk.reachable -eq $true) +Assert "probe: live_ok records elapsed" ($rOk.elapsed_s -ge 0) + +$rNo = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $noTokDisp +Assert "probe: no token -> no-canary" ($rNo.live -eq 'live_fail' -and $rNo.reason -eq 'no-canary') + +$rFail = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $failDisp +Assert "probe: nonzero exit -> nonzero-exit" ($rFail.reason -eq 'nonzero-exit') + +$rSlow = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $slowDisp -TimeoutS 1 +Assert "probe: slow dispatch -> timeout" ($rSlow.reason -eq 'timeout') + +$rThrow = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $throwDisp +Assert "probe: throwing dispatch -> dispatch-error" ($rThrow.reason -eq 'dispatch-error') + +$rUnreach = Invoke-FleetProbe -Provider @{ name='h'; kind='http'; enabled=$true; base_url='http://x' } -UrlProbe { param($u) $false } +Assert "probe: http down -> unreachable (no dispatch)" ($rUnreach.live -eq 'live_fail' -and $rUnreach.reason -eq 'unreachable') +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `pwsh -NoProfile -File scripts/test-fleet-probe-lib.ps1` +Expected: FAIL — `Invoke-FleetProbe` not defined. + +- [ ] **Step 3: Write minimal implementation** + +Append to `scripts/fleet-probe-lib.ps1`: + +```powershell +function Invoke-FleetProbe { + <# Per-provider live round-trip. Reachability precheck -> dispatch a canary + under an enforced timeout -> classify. Diagnostic: never throws. The + dispatch runs in a Start-ThreadJob so a hung/slow provider is bounded; + a timed-out native child may linger (best-effort Stop-Job) — acceptable + for a diagnostic. -Dispatcher injects for tests. #> + param( + [Parameter(Mandatory)][hashtable]$Provider, + [int]$TimeoutS = 60, + [string]$FleetPath = (Join-Path (Get-BatonHome) 'fleet.yaml'), + [scriptblock]$Dispatcher, + [scriptblock]$UrlProbe + ) + $name = [string]$Provider.name + $kind = [string]$Provider.kind + if ($Provider.enabled -ne $true) { + return @{ name = $name; kind = $kind; enabled = $false; reachable = $null; live = 'skip'; reason = 'disabled'; elapsed_s = $null } + } + $reach = Test-ProviderReachable -Provider $Provider -UrlProbe $UrlProbe + if (-not $reach.reachable) { + return @{ name = $name; kind = $kind; enabled = $true; reachable = $false; live = 'live_fail'; reason = $reach.reason; elapsed_s = $null } + } + if (-not $Dispatcher) { + $Dispatcher = { + param($prov, $fleetPath, $canary, $scriptRoot) + . (Join-Path $scriptRoot 'fleet-lib.ps1') + Invoke-Fleet -Name ([string]$prov.name) -Prompt $canary -Path $fleetPath -NoJournal + } + } + $sw = [System.Diagnostics.Stopwatch]::StartNew() + $timedOut = $false; $output = ''; $exit = 0; $errored = $false + $threadJob = $null + try { + $threadJob = Start-ThreadJob -ScriptBlock $Dispatcher -ArgumentList $Provider, $FleetPath, $script:FleetCanaryPrompt, $PSScriptRoot + $done = Wait-Job -Job $threadJob -Timeout $TimeoutS + if (-not $done) { + $timedOut = $true + Stop-Job -Job $threadJob -ErrorAction SilentlyContinue + } else { + $disp = Receive-Job -Job $threadJob -ErrorAction Stop + $output = [string]$disp.stdout + $exit = [int]$disp.exit_code + } + } catch { + $errored = $true + } finally { + if ($threadJob) { Remove-Job -Job $threadJob -Force -ErrorAction SilentlyContinue } + $sw.Stop() + } + $elapsed = [int]$sw.Elapsed.TotalSeconds + if ($errored) { + return @{ name = $name; kind = $kind; enabled = $true; reachable = $true; live = 'live_fail'; reason = 'dispatch-error'; elapsed_s = $elapsed } + } + $verdict = Test-FleetCanary -Output $output -ExitCode $exit -TimedOut $timedOut + return @{ name = $name; kind = $kind; enabled = $true; reachable = $true; live = $verdict.live; reason = $verdict.reason; elapsed_s = $elapsed } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `pwsh -NoProfile -File scripts/test-fleet-probe-lib.ps1` +Expected: PASS — all asserts green (the timeout case takes ~1s). + +- [ ] **Step 5: Commit** + +```bash +git add scripts/fleet-probe-lib.ps1 scripts/test-fleet-probe-lib.ps1 +git commit -m "feat(fleet): live round-trip probe with timeout guard (Slice 1 part 2)" +``` + +--- + +### Task 3: stdin-default for clean-tail CLI templates (`Invoke-Fleet-Cli`) + +**Files:** +- Modify: `scripts/fleet-lib.ps1` (add `Test-StdinSafe`; branch in `Invoke-Fleet-Cli`) +- Test: `scripts/test-fleet-dispatch.ps1` (append asserts) + +**Interfaces:** +- Produces: `Test-StdinSafe -Provider ` → `[bool]`. `$true` when the provider is not already `stdin:true` AND its `command_template` ends in a standalone quoted prompt token (regex `\s+(["'])\{\{prompt\}\}\1\s*$`) AND the template with that trailing token removed contains no shell operators (`|`, `>`, `<`, `&`, `;`, backtick, or `$(`). +- Behavior: when `Test-StdinSafe` is true, `Invoke-Fleet-Cli` strips the trailing quoted `{{prompt}}`, resolves `{{model}}` in the remainder, tokenizes it, and pipes the raw prompt via the existing temp-file→stdin mechanism. When false, the legacy interpolation path runs unchanged. Providers already `stdin:true` keep their current stdin behavior. + +**Why this predicate:** it captures the real interpolating providers (`claude -p "{{prompt}}"`, `codex exec "{{prompt}}"`, `agy --print "{{prompt}}"`, `gh copilot suggest "{{prompt}}"`, `ollama run {{model}} "{{prompt}}"`) while leaving embedded-prompt templates — including the test stubs `pwsh -NoProfile -Command "Write-Output hello-{{prompt}}"` — on the legacy path, so `test-fleet-dispatch.ps1`'s `hello-world` / `m123:p` assertions do not regress. + +- [ ] **Step 1: Write the failing test** + +Append to `scripts/test-fleet-dispatch.ps1` BEFORE its final tally block. (It already dot-sources `fleet-lib.ps1` and defines `Assert` and `$fixture`.) + +```powershell +# --- Test-StdinSafe predicate --- +Assert "stdin-safe: trailing quoted prompt (codex)" (Test-StdinSafe -Provider @{ name='c'; command_template='codex exec "{{prompt}}"' }) +Assert "stdin-safe: trailing quoted prompt with model (ollama)" (Test-StdinSafe -Provider @{ name='o'; command_template='ollama run {{model}} "{{prompt}}"'; model_default='m' }) +Assert "stdin-safe: embedded prompt -> legacy (test stub)" (-not (Test-StdinSafe -Provider @{ name='s'; command_template='pwsh -NoProfile -Command "Write-Output hello-{{prompt}}"' })) +Assert "stdin-safe: shell operator in tail -> legacy" (-not (Test-StdinSafe -Provider @{ name='p'; command_template='foo | bar "{{prompt}}"' })) +Assert "stdin-safe: already stdin:true -> not re-flagged" (-not (Test-StdinSafe -Provider @{ name='h'; stdin=$true; command_template='claude -p --model x' })) + +# --- Regression: embedded-prompt stubs still interpolate --- +$tmpJ = New-TemporaryFile +$rReg = Invoke-Fleet -Name 'stub-cli' -Prompt 'world' -Path $fixture -JournalPath $tmpJ +Assert "regression: stub-cli still outputs hello-world (legacy path)" (($rReg.stdout | Out-String).Trim() -eq 'hello-world') +Remove-Item $tmpJ -ErrorAction SilentlyContinue +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `pwsh -NoProfile -File scripts/test-fleet-dispatch.ps1` +Expected: FAIL — `Test-StdinSafe` not defined. + +- [ ] **Step 3: Write minimal implementation** + +In `scripts/fleet-lib.ps1`, add `Test-StdinSafe` immediately after `Resolve-FleetCommand` (after line 182): + +```powershell +function Test-StdinSafe { + <# True when a cli provider's template can safely pipe the prompt via stdin: + not already stdin, template ends in a standalone quoted {{prompt}}, and the + command minus that tail has no shell operators. Keeps embedded-prompt and + shell-wrapped templates on the legacy interpolation path. #> + param([Parameter(Mandatory)][hashtable]$Provider) + if ($Provider.stdin -eq $true) { return $false } + $template = [string]$Provider.command_template + if (-not $template) { return $false } + if ($template -notmatch '\s+(["''])\{\{prompt\}\}\1\s*$') { return $false } + $head = $template -replace '\s+(["''])\{\{prompt\}\}\1\s*$', '' + if ($head -match '[|><&;`]' -or $head -match '\$\(') { return $false } + return $true +} +``` + +Then in `Invoke-Fleet-Cli`, replace the dispatch-path decision. The current body resolves the command then branches on `$Provider.stdin -eq $true`. Change it so the stdin path is taken when EITHER the provider is `stdin:true` OR `Test-StdinSafe` is true, and in the `Test-StdinSafe` case build the stdin command from the template with the trailing quoted prompt stripped. Concretely, replace the command-resolution + branch prologue: + +```powershell + # Decide dispatch path. stdin:true providers already omit {{prompt}}; + # clean-tail interpolating providers are promoted to stdin (prompt-size / + # quote hardening) by stripping the trailing quoted {{prompt}} token. + $useStdin = ($Provider.stdin -eq $true) -or (Test-StdinSafe -Provider $Provider) + if ($Provider.stdin -eq $true) { + $cmd = Resolve-FleetCommand -Provider $Provider -Prompt '' -Model $Model + } elseif (Test-StdinSafe -Provider $Provider) { + $stripped = ([string]$Provider.command_template) -replace '\s+(["''])\{\{prompt\}\}\1\s*$', '' + $resolvedModel = if ($Model) { $Model } else { $Provider.model_default } + if ($null -ne $resolvedModel) { $stripped = $stripped.Replace('{{model}}', [string]$resolvedModel) } + $cmd = $stripped + } else { + $cmd = Resolve-FleetCommand -Provider $Provider -Prompt $Prompt -Model $Model + } +``` + +…and change the existing `if ($Provider.stdin -eq $true) {` guard around the stdin/legacy blocks to `if ($useStdin) {`. Leave the stdin block body (temp-file write, tokenize `$cmd`, `& $exe @rest`) and the legacy `else` block (`Invoke-Expression $cmd`) otherwise unchanged. + +> **Note for the implementer:** `Resolve-FleetCommand` with `-Prompt ''` for `stdin:true` providers reproduces today's behavior (their templates have no `{{prompt}}`; only `{{model}}` is substituted). Verify the existing stdin providers' tests still pass — do not alter `Resolve-FleetCommand`. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pwsh -NoProfile -File scripts/test-fleet-dispatch.ps1` +Expected: PASS — new predicate + regression asserts green, existing `hello-world` / `m123:p` / http asserts unchanged. + +Run: `pwsh -NoProfile -File scripts/test-fleet-lib.ps1` +Expected: PASS — no regressions. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/fleet-lib.ps1 scripts/test-fleet-dispatch.ps1 +git commit -m "feat(fleet): stdin-default for clean-tail CLI templates (Slice 1 prompt hardening)" +``` + +--- + +### Task 4: `fleet doctor --live` surface + end-to-end tests + +**Files:** +- Modify: `scripts/fleet-doctor.ps1` +- Test: `scripts/test-fleet-doctor.ps1` + +**Interfaces:** +- Consumes: `Invoke-FleetProbe` (Task 2), `Read-Fleet` (fleet-lib). +- Behavior: `fleet-doctor.ps1` gains `-Live` (switch), `-TimeoutS` (int, default 60), keeps `-Json`. Default (no `-Live`) path is byte-for-byte unchanged. With `-Live`: iterate enabled+disabled providers via `Invoke-FleetProbe`; render a live table (human) or the result-array (`-Json`); exit 0 iff every enabled provider is `live_ok`, else 1. + +- [ ] **Step 1: Write the failing test** + +Append to `scripts/test-fleet-doctor.ps1` BEFORE its final tally block: + +```powershell +# --- fleet doctor --live (end-to-end, real Invoke-Fleet against fixture) --- +# Fixture roster: stub-cli/stub-with-model/stub-with-env (echo prompt -> contains +# PONG -> live_ok), stub-disabled (skip), stub-http (localhost:9999 -> unreachable), +# stub-fail (no {{prompt}} -> dispatch throws -> dispatch-error), stub-slow (sleep 10 +# -> timeout at --timeout 3). Mixed roster -> exit 1. +$liveOut = & pwsh -NoProfile -File $doctor -Path $fixture -Live -TimeoutS 3 2>&1 | Out-String +$liveExit = $LASTEXITCODE +Assert "live: stub-cli reports live_ok" ($liveOut -match 'stub-cli\s+.*live_ok') +Assert "live: stub-disabled reports skip" ($liveOut -match 'stub-disabled\s+.*skip') +Assert "live: stub-http reports unreachable" ($liveOut -match 'stub-http\s+.*(live_fail|unreachable)') +Assert "live: exit 1 on a mixed roster" ($liveExit -eq 1) + +# --json shape +$liveJson = & pwsh -NoProfile -File $doctor -Path $fixture -Live -TimeoutS 3 -Json 2>&1 | Out-String +$parsedLive = $liveJson | ConvertFrom-Json +$cliRow = @($parsedLive | Where-Object { $_.name -eq 'stub-cli' }) +Assert "live --json: stub-cli row carries live=live_ok" ($cliRow.Count -eq 1 -and $cliRow[0].live -eq 'live_ok') + +# All-live_ok roster -> exit 0 (hermetic single-provider temp yaml) +$tmpYaml = New-TemporaryFile +@' +providers: + - name: only-ok + kind: cli + enabled: true + cost_tier: free + command_template: 'pwsh -NoProfile -Command "Write-Output PONG-{{prompt}}"' +'@ | Set-Content -LiteralPath $tmpYaml -Encoding utf8NoBOM +$okOut = & pwsh -NoProfile -File $doctor -Path $tmpYaml -Live -TimeoutS 10 2>&1 | Out-String +$okExit = $LASTEXITCODE +Assert "live: all-live_ok roster -> exit 0" ($okExit -eq 0) +Remove-Item $tmpYaml -ErrorAction SilentlyContinue + +# Default (non-live) path unchanged: still reports PATH-based skip/err and exit 1 +$plainOut = & pwsh -NoProfile -File $doctor -Path $fixture 2>&1 | Out-String +Assert "non-live path still reports stub-disabled skip" ($plainOut -match 'stub-disabled\s+skip') +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `pwsh -NoProfile -File scripts/test-fleet-doctor.ps1` +Expected: FAIL — `-Live` / `-TimeoutS` not recognized; no live output. + +- [ ] **Step 3: Write minimal implementation** + +In `scripts/fleet-doctor.ps1`: + +1. Extend `param(...)`: + +```powershell +param( + [string]$Path = $(if ($env:BATON_HOME) { Join-Path $env:BATON_HOME 'fleet.yaml' } else { Join-Path $HOME '.baton/fleet.yaml' }), + [switch]$Json, + [switch]$Live, + [int]$TimeoutS = 60 +) +``` + +2. After the existing `. (Join-Path $PSScriptRoot 'fleet-lib.ps1')`, add: + +```powershell +if ($Live) { . (Join-Path $PSScriptRoot 'fleet-probe-lib.ps1') } +``` + +3. After the `try { $fleet = Read-Fleet ... }` block, add a `--live` branch that returns before the legacy loop: + +```powershell +if ($Live) { + $results = foreach ($p in $fleet) { Invoke-FleetProbe -Provider ([hashtable]$p) -TimeoutS $TimeoutS -FleetPath $Path } + $rows = @($results) + if ($Json) { + ConvertTo-Json -InputObject @($rows) -Depth 4 + } else { + $render = $rows | ForEach-Object { + $reach = if ($null -eq $_.reachable) { '-' } elseif ($_.reachable) { 'yes' } else { 'no' } + $detail = if ($_.reason) { $_.reason } elseif ($null -ne $_.elapsed_s) { "$($_.elapsed_s)s" } else { '' } + [pscustomobject]@{ PROVIDER = $_.name; REACHABLE = $reach; LIVE = $_.live; DETAIL = $detail } + } + $render | Format-Table PROVIDER, REACHABLE, LIVE, DETAIL -AutoSize | Out-String | Write-Host + $enabled = @($fleet | Where-Object { $_.enabled -eq $true }).Count + Write-Host "$enabled enabled provider(s); live round-trip." + } + $anyLiveBad = @($rows | Where-Object { $_.enabled -eq $true -and $_.live -ne 'live_ok' }).Count -gt 0 + if ($anyLiveBad) { exit 1 } else { exit 0 } +} +``` + +(The existing non-live loop and its exit logic remain untouched below this branch.) + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `pwsh -NoProfile -File scripts/test-fleet-doctor.ps1` +Expected: PASS — live asserts + `--json` + exit-0/exit-1 + the unchanged non-live assert all green. (Runs ~3s for the timeout case.) + +- [ ] **Step 5: Commit** + +```bash +git add scripts/fleet-doctor.ps1 scripts/test-fleet-doctor.ps1 +git commit -m "feat(fleet): fleet doctor --live round-trip surface (Slice 1)" +``` + +--- + +### Task 5: Deploy wiring, docs, version bump + +**Files:** +- Modify: `scripts/bootstrap.ps1` +- Modify: `scripts/test-bootstrap.ps1` +- Modify: `commands/fleet.md` +- Modify: `AGENTS.md` +- Modify: `.claude-plugin/plugin.json` + +**Interfaces:** none (integration/config task). + +- [ ] **Step 1: Add the new lib to the deploy manifest** + +In `scripts/bootstrap.ps1`, find the Step 5b manifest array of script basenames (the same list that gained `session-markers-lib.ps1`, `registry-lib.ps1`, `fleet-project.ps1`). Add `'fleet-probe-lib.ps1'` to it. (Do NOT add test-*.ps1 files or hooks.) + +- [ ] **Step 2: Add the deploy assert (v1.8.0 coach-lib omission lesson)** + +In `scripts/test-bootstrap.ps1`, mirror the existing deploy asserts (e.g. the `registry-lib.ps1` one) with: + +```powershell +Assert "deploys fleet-probe-lib.ps1" (Test-Path (Join-Path $deployDir 'fleet-probe-lib.ps1')) +``` + +- [ ] **Step 3: Run the bootstrap test to verify green** + +Run: `pwsh -NoProfile -File scripts/test-bootstrap.ps1` +Expected: PASS — the new deploy assert green, prior count + 1, 0 FAIL. + +- [ ] **Step 4: Document `--live` in the command doc** + +In `commands/fleet.md`: update the front-matter `description`/`argument-hint` to mention `doctor [--live] [--timeout ]`, and in the `doctor` dispatch block add a note + invocation: + +```powershell +& pwsh -NoProfile -File "$HOME/.claude/scripts/fleet-doctor.ps1" -Live -TimeoutS 60 +``` + +with one line explaining: `--live` sends a `PONG` canary to each enabled provider and reports `live_ok | live_fail(reason) | skip`; a `no-canary` reason means the provider ran but didn't answer (often a wrong command template for this box). Plain (no `--live`) stays the fast PATH/reachability check. + +- [ ] **Step 5: Add the AGENTS.md line** + +In `AGENTS.md`, near the fleet/model-agnostic material, add one line: + +> `fleet doctor --live` — harness-neutral way to verify a box's roster actually answers (canary round-trip per enabled provider), not just that the binaries are installed. + +- [ ] **Step 6: Bump the plugin version** + +In `.claude-plugin/plugin.json`, change `"version": "1.9.0"` to `"version": "1.10.0-rc.1"`. + +- [ ] **Step 7: Commit** + +```bash +git add scripts/bootstrap.ps1 scripts/test-bootstrap.ps1 commands/fleet.md AGENTS.md .claude-plugin/plugin.json +git commit -m "chore(fleet): deploy wiring + docs + v1.10.0-rc.1 (Slice 1)" +``` + +--- + +## Global test sweep (run after Task 5, before final review) + +``` +pwsh -NoProfile -File scripts/test-fleet-probe-lib.ps1 +pwsh -NoProfile -File scripts/test-fleet-dispatch.ps1 +pwsh -NoProfile -File scripts/test-fleet-lib.ps1 +pwsh -NoProfile -File scripts/test-fleet-doctor.ps1 +pwsh -NoProfile -File scripts/test-bootstrap.ps1 +pwsh -NoProfile -File scripts/test-conductor-lib.ps1 +``` + +All must be green (0 FAIL) — the last two guard against deploy-manifest and dispatch regressions. + +## Notes for the executor + +- Model ladder: Task 1, 2, 4, 5 are transcription-grade (complete code above) → **haiku**; Task 3 edits hot dispatch code with a regex predicate → **sonnet**. Streamlined ceremony: no per-task reviewers; one **opus** whole-branch review at the end. +- The `Start-ThreadJob` timeout guard may leave a hung native child running after a `timeout` verdict (best-effort `Stop-Job`). This is an accepted Slice-1 limitation for a diagnostic — note it, don't chase it. +- Out of scope (do not build): the `-Spawner` executor that applies repo changes (Slice 2), any routing/`Select-Capability` change, per-provider latency/quality benchmarking. + +## Self-Review + +**Spec coverage:** §Design.1 surface → Task 4. §Design.2 canary contract (prompt/token/pass/reasons/timeout/result-shape) → Tasks 1+2. §Design.3 stdin-default → Task 3. §Design.4 legibility (table + `--json`) → Task 4. §Design.5 hermetic tests (fake dispatcher, temp fixtures) → Tasks 1–4. §Design.6 deploy/docs/bump → Task 5. All covered. (Deviation from spec's 6 reasons: added `dispatch-error` for a thrown dispatch — necessary and documented in Task 2's interface.) + +**Placeholder scan:** no TBD/TODO; every code step carries complete code; the one predicate is pinned to an exact regex with both-direction tests. + +**Type consistency:** `Invoke-FleetProbe` result keys (`name/kind/enabled/reachable/live/reason/elapsed_s`) are consumed identically in Task 4's render/JSON and asserts; `Test-FleetCanary` return (`live/reason`) and `Test-ProviderReachable` return (`reachable/reason`) match their call sites; `Test-StdinSafe` returns `[bool]` used in `Invoke-Fleet-Cli` and tests. From d18572f41fb03505bb1fe12befec89fcf663567d Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sat, 4 Jul 2026 23:40:46 -0600 Subject: [PATCH 03/10] feat(fleet): canary classifier + reachability probe (Slice 1 part 1) --- scripts/fleet-probe-lib.ps1 | 58 ++++++++++++++++++++++++++++++++ scripts/test-fleet-probe-lib.ps1 | 35 +++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 scripts/fleet-probe-lib.ps1 create mode 100644 scripts/test-fleet-probe-lib.ps1 diff --git a/scripts/fleet-probe-lib.ps1 b/scripts/fleet-probe-lib.ps1 new file mode 100644 index 0000000..93cf1bd --- /dev/null +++ b/scripts/fleet-probe-lib.ps1 @@ -0,0 +1,58 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Fleet round-trip probe (Slice 1). Sends a canary prompt to an enabled provider + and classifies whether it actually answered. Judge-free: a deterministic token. +.NOTES + Pure classifier (Test-FleetCanary) + reachability (Test-ProviderReachable) + + live round-trip (Invoke-FleetProbe, added in Task 2). Diagnostic only — never + mutates state, never throws on a provider failure. +#> +. "$PSScriptRoot/baton-home.ps1" +. "$PSScriptRoot/fleet-lib.ps1" # Read-Fleet, Invoke-Fleet, Get-FleetProvider + +$script:FleetCanaryPrompt = 'Reply with exactly the word PONG and nothing else.' +$script:FleetCanaryToken = 'PONG' + +function Test-FleetCanary { + <# Pure. Classify a dispatch result into a live verdict + reason. + Precedence: timeout > nonzero-exit > token-match. #> + param( + [string]$Output, + [int]$ExitCode = 0, + [bool]$TimedOut = $false + ) + if ($TimedOut) { return @{ live = 'live_fail'; reason = 'timeout' } } + if ($ExitCode -ne 0) { return @{ live = 'live_fail'; reason = 'nonzero-exit' } } + if (([string]$Output).ToUpperInvariant().Contains($script:FleetCanaryToken)) { + return @{ live = 'live_ok'; reason = $null } + } + return @{ live = 'live_fail'; reason = 'no-canary' } +} + +function Test-ProviderReachable { + <# Is the provider's transport up? cli -> binary on PATH; http -> base_url HEAD. + -UrlProbe injects the reachability check for tests. Returns @{reachable;reason}. #> + param( + [Parameter(Mandatory)][hashtable]$Provider, + [scriptblock]$UrlProbe + ) + if (-not $UrlProbe) { + $UrlProbe = { + param($url) + try { Invoke-WebRequest -Uri $url -Method Head -TimeoutSec 5 -UseBasicParsing | Out-Null; return $true } + catch { return $false } + } + } + if ($Provider.kind -eq 'cli') { + $bin = ([string]$Provider.command_template -split '\s+')[0] + if (Get-Command $bin -ErrorAction SilentlyContinue) { return @{ reachable = $true; reason = $null } } + return @{ reachable = $false; reason = 'not-on-PATH' } + } + if ($Provider.kind -eq 'http') { + if (& $UrlProbe ([string]$Provider.base_url)) { return @{ reachable = $true; reason = $null } } + return @{ reachable = $false; reason = 'unreachable' } + } + # Unknown kind: treat as unreachable rather than throwing. + return @{ reachable = $false; reason = 'unreachable' } +} diff --git a/scripts/test-fleet-probe-lib.ps1 b/scripts/test-fleet-probe-lib.ps1 new file mode 100644 index 0000000..c6f3506 --- /dev/null +++ b/scripts/test-fleet-probe-lib.ps1 @@ -0,0 +1,35 @@ +#!/usr/bin/env pwsh +# Tests for scripts/fleet-probe-lib.ps1 — canary classifier, reachability, live probe. +$ErrorActionPreference = 'Stop' +. (Join-Path $PSScriptRoot 'fleet-probe-lib.ps1') + +$failures = 0 +function Assert($label, $cond) { + if ($cond) { Write-Host "PASS $label" -ForegroundColor Green } + else { Write-Host "FAIL $label" -ForegroundColor Red; $script:failures++ } +} + +# --- Test-FleetCanary (pure) --- +$c1 = Test-FleetCanary -Output 'PONG' -ExitCode 0 -TimedOut $false +Assert "canary: exact token -> live_ok" ($c1.live -eq 'live_ok' -and $null -eq $c1.reason) +$c2 = Test-FleetCanary -Output 'the answer is pong.' -ExitCode 0 -TimedOut $false +Assert "canary: case-insensitive substring -> live_ok" ($c2.live -eq 'live_ok') +$c3 = Test-FleetCanary -Output 'usage: codex [options]' -ExitCode 0 -TimedOut $false +Assert "canary: exit 0 but no token -> no-canary" ($c3.live -eq 'live_fail' -and $c3.reason -eq 'no-canary') +$c4 = Test-FleetCanary -Output 'PONG' -ExitCode 3 -TimedOut $false +Assert "canary: nonzero exit beats token -> nonzero-exit" ($c4.live -eq 'live_fail' -and $c4.reason -eq 'nonzero-exit') +$c5 = Test-FleetCanary -Output '' -ExitCode 0 -TimedOut $true +Assert "canary: timeout beats all -> timeout" ($c5.live -eq 'live_fail' -and $c5.reason -eq 'timeout') + +# --- Test-ProviderReachable --- +$cliOk = Test-ProviderReachable -Provider @{ name='p'; kind='cli'; command_template='pwsh -NoProfile -Command "x"' } +Assert "reachable: pwsh on PATH -> reachable" ($cliOk.reachable -eq $true -and $null -eq $cliOk.reason) +$cliNo = Test-ProviderReachable -Provider @{ name='p'; kind='cli'; command_template='definitely-not-a-real-binary-xyz foo' } +Assert "reachable: missing binary -> not-on-PATH" ($cliNo.reachable -eq $false -and $cliNo.reason -eq 'not-on-PATH') +$httpOk = Test-ProviderReachable -Provider @{ name='h'; kind='http'; base_url='http://x' } -UrlProbe { param($u) $true } +Assert "reachable: http probe true -> reachable" ($httpOk.reachable -eq $true) +$httpNo = Test-ProviderReachable -Provider @{ name='h'; kind='http'; base_url='http://x' } -UrlProbe { param($u) $false } +Assert "reachable: http probe false -> unreachable" ($httpNo.reachable -eq $false -and $httpNo.reason -eq 'unreachable') + +if ($failures -gt 0) { Write-Host "`n$failures failure(s)" -ForegroundColor Red; exit 1 } +Write-Host "`nAll tests passed." -ForegroundColor Green From 96d816ecd0dea1523ca0b3474c0108d15fbf8b3a Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sat, 4 Jul 2026 23:43:00 -0600 Subject: [PATCH 04/10] feat(fleet): live round-trip probe with timeout guard (Slice 1 part 2) --- scripts/fleet-probe-lib.ps1 | 57 ++++++++++++++++++++++++++++++++ scripts/test-fleet-probe-lib.ps1 | 32 ++++++++++++++++++ 2 files changed, 89 insertions(+) diff --git a/scripts/fleet-probe-lib.ps1 b/scripts/fleet-probe-lib.ps1 index 93cf1bd..d082e52 100644 --- a/scripts/fleet-probe-lib.ps1 +++ b/scripts/fleet-probe-lib.ps1 @@ -56,3 +56,60 @@ function Test-ProviderReachable { # Unknown kind: treat as unreachable rather than throwing. return @{ reachable = $false; reason = 'unreachable' } } + +function Invoke-FleetProbe { + <# Per-provider live round-trip. Reachability precheck -> dispatch a canary + under an enforced timeout -> classify. Diagnostic: never throws. The + dispatch runs in a Start-ThreadJob so a hung/slow provider is bounded; + a timed-out native child may linger (best-effort Stop-Job) — acceptable + for a diagnostic. -Dispatcher injects for tests. #> + param( + [Parameter(Mandatory)][hashtable]$Provider, + [int]$TimeoutS = 60, + [string]$FleetPath = (Join-Path (Get-BatonHome) 'fleet.yaml'), + [scriptblock]$Dispatcher, + [scriptblock]$UrlProbe + ) + $name = [string]$Provider.name + $kind = [string]$Provider.kind + if ($Provider.enabled -ne $true) { + return @{ name = $name; kind = $kind; enabled = $false; reachable = $null; live = 'skip'; reason = 'disabled'; elapsed_s = $null } + } + $reach = Test-ProviderReachable -Provider $Provider -UrlProbe $UrlProbe + if (-not $reach.reachable) { + return @{ name = $name; kind = $kind; enabled = $true; reachable = $false; live = 'live_fail'; reason = $reach.reason; elapsed_s = $null } + } + if (-not $Dispatcher) { + $Dispatcher = { + param($prov, $fleetPath, $canary, $scriptRoot) + . (Join-Path $scriptRoot 'fleet-lib.ps1') + Invoke-Fleet -Name ([string]$prov.name) -Prompt $canary -Path $fleetPath -NoJournal + } + } + $sw = [System.Diagnostics.Stopwatch]::StartNew() + $timedOut = $false; $output = ''; $exit = 0; $errored = $false + $threadJob = $null + try { + $threadJob = Start-ThreadJob -ScriptBlock $Dispatcher -ArgumentList $Provider, $FleetPath, $script:FleetCanaryPrompt, $PSScriptRoot + $done = Wait-Job -Job $threadJob -Timeout $TimeoutS + if (-not $done) { + $timedOut = $true + Stop-Job -Job $threadJob -ErrorAction SilentlyContinue + } else { + $disp = Receive-Job -Job $threadJob -ErrorAction Stop + $output = [string]$disp.stdout + $exit = [int]$disp.exit_code + } + } catch { + $errored = $true + } finally { + if ($threadJob) { Remove-Job -Job $threadJob -Force -ErrorAction SilentlyContinue } + $sw.Stop() + } + $elapsed = [int]$sw.Elapsed.TotalSeconds + if ($errored) { + return @{ name = $name; kind = $kind; enabled = $true; reachable = $true; live = 'live_fail'; reason = 'dispatch-error'; elapsed_s = $elapsed } + } + $verdict = Test-FleetCanary -Output $output -ExitCode $exit -TimedOut $timedOut + return @{ name = $name; kind = $kind; enabled = $true; reachable = $true; live = $verdict.live; reason = $verdict.reason; elapsed_s = $elapsed } +} diff --git a/scripts/test-fleet-probe-lib.ps1 b/scripts/test-fleet-probe-lib.ps1 index c6f3506..3a65a0a 100644 --- a/scripts/test-fleet-probe-lib.ps1 +++ b/scripts/test-fleet-probe-lib.ps1 @@ -31,5 +31,37 @@ Assert "reachable: http probe true -> reachable" ($httpOk.reachable -eq $true) $httpNo = Test-ProviderReachable -Provider @{ name='h'; kind='http'; base_url='http://x' } -UrlProbe { param($u) $false } Assert "reachable: http probe false -> unreachable" ($httpNo.reachable -eq $false -and $httpNo.reason -eq 'unreachable') +# --- Invoke-FleetProbe (live round-trip) --- +# Fake dispatchers returning the Invoke-Fleet shape @{stdout;exit_code}. +$okDisp = { param($prov,$fp,$canary,$root) @{ stdout = 'PONG'; exit_code = 0 } } +$noTokDisp = { param($prov,$fp,$canary,$root) @{ stdout = 'help text here'; exit_code = 0 } } +$failDisp = { param($prov,$fp,$canary,$root) @{ stdout = ''; exit_code = 7 } } +$slowDisp = { param($prov,$fp,$canary,$root) Start-Sleep -Seconds 5; @{ stdout = 'PONG'; exit_code = 0 } } +$throwDisp = { param($prov,$fp,$canary,$root) throw 'boom' } + +$enabledCli = @{ name='w'; kind='cli'; enabled=$true; command_template='pwsh -NoProfile -Command "x"' } + +$rSkip = Invoke-FleetProbe -Provider @{ name='d'; kind='cli'; enabled=$false; command_template='pwsh x' } +Assert "probe: disabled -> skip" ($rSkip.live -eq 'skip' -and $rSkip.reason -eq 'disabled') + +$rOk = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $okDisp +Assert "probe: token -> live_ok" ($rOk.live -eq 'live_ok' -and $rOk.reachable -eq $true) +Assert "probe: live_ok records elapsed" ($rOk.elapsed_s -ge 0) + +$rNo = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $noTokDisp +Assert "probe: no token -> no-canary" ($rNo.live -eq 'live_fail' -and $rNo.reason -eq 'no-canary') + +$rFail = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $failDisp +Assert "probe: nonzero exit -> nonzero-exit" ($rFail.reason -eq 'nonzero-exit') + +$rSlow = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $slowDisp -TimeoutS 1 +Assert "probe: slow dispatch -> timeout" ($rSlow.reason -eq 'timeout') + +$rThrow = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $throwDisp +Assert "probe: throwing dispatch -> dispatch-error" ($rThrow.reason -eq 'dispatch-error') + +$rUnreach = Invoke-FleetProbe -Provider @{ name='h'; kind='http'; enabled=$true; base_url='http://x' } -UrlProbe { param($u) $false } +Assert "probe: http down -> unreachable (no dispatch)" ($rUnreach.live -eq 'live_fail' -and $rUnreach.reason -eq 'unreachable') + if ($failures -gt 0) { Write-Host "`n$failures failure(s)" -ForegroundColor Red; exit 1 } Write-Host "`nAll tests passed." -ForegroundColor Green From e97e0ff8e0cd2480750ea8e0b8f3d15bbba0883d Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sat, 4 Jul 2026 23:45:27 -0600 Subject: [PATCH 05/10] feat(fleet): stdin-default for clean-tail CLI templates (Slice 1 prompt hardening) --- scripts/fleet-lib.ps1 | 32 ++++++++++++++++++++++++++++++-- scripts/test-fleet-dispatch.ps1 | 13 +++++++++++++ 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/scripts/fleet-lib.ps1 b/scripts/fleet-lib.ps1 index 1231d8e..f3d3018 100644 --- a/scripts/fleet-lib.ps1 +++ b/scripts/fleet-lib.ps1 @@ -181,6 +181,21 @@ function Resolve-FleetCommand { return $cmd } +function Test-StdinSafe { + <# True when a cli provider's template can safely pipe the prompt via stdin: + not already stdin, template ends in a standalone quoted {{prompt}}, and the + command minus that tail has no shell operators. Keeps embedded-prompt and + shell-wrapped templates on the legacy interpolation path. #> + param([Parameter(Mandatory)][hashtable]$Provider) + if ($Provider.stdin -eq $true) { return $false } + $template = [string]$Provider.command_template + if (-not $template) { return $false } + if ($template -notmatch '\s+(["''])\{\{prompt\}\}\1\s*$') { return $false } + $head = $template -replace '\s+(["''])\{\{prompt\}\}\1\s*$', '' + if ($head -match '[|><&;`]' -or $head -match '\$\(') { return $false } + return $true +} + function Write-FleetJournalLine { <# Append a `fleet` line to the journal, picking up Plan 3 job/phase tags by reading the state file directly (honors $env:CAO_STATE_PATH). #> @@ -233,7 +248,20 @@ function Invoke-Fleet-Cli { [string]$Model, [int]$TimeoutS = 120 ) - $cmd = Resolve-FleetCommand -Provider $Provider -Prompt $Prompt -Model $Model + # Decide dispatch path. stdin:true providers already omit {{prompt}}; + # clean-tail interpolating providers are promoted to stdin (prompt-size / + # quote hardening) by stripping the trailing quoted {{prompt}} token. + $useStdin = ($Provider.stdin -eq $true) -or (Test-StdinSafe -Provider $Provider) + if ($Provider.stdin -eq $true) { + $cmd = Resolve-FleetCommand -Provider $Provider -Prompt '' -Model $Model + } elseif (Test-StdinSafe -Provider $Provider) { + $stripped = ([string]$Provider.command_template) -replace '\s+(["''])\{\{prompt\}\}\1\s*$', '' + $resolvedModel = if ($Model) { $Model } else { $Provider.model_default } + if ($null -ne $resolvedModel) { $stripped = $stripped.Replace('{{model}}', [string]$resolvedModel) } + $cmd = $stripped + } else { + $cmd = Resolve-FleetCommand -Provider $Provider -Prompt $Prompt -Model $Model + } $saved = @{} if ($Provider.env) { @@ -244,7 +272,7 @@ function Invoke-Fleet-Cli { } $start = Get-Date try { - if ($Provider.stdin -eq $true) { + if ($useStdin) { # Robust path: pass the prompt via stdin instead of interpolating it # into the command string — immune to embedded quotes/backticks/$. # The template is a clean token list (e.g. 'codex exec -') with no diff --git a/scripts/test-fleet-dispatch.ps1 b/scripts/test-fleet-dispatch.ps1 index f3132fa..9375464 100644 --- a/scripts/test-fleet-dispatch.ps1 +++ b/scripts/test-fleet-dispatch.ps1 @@ -63,5 +63,18 @@ Assert "http dispatch exit 0" ($rh.exit_code -eq 0) Assert "http dispatch journaled" (@(Get-Content $tmpJournal2 | Where-Object { $_ -match '\| fleet \| stub-http \|' }).Count -ge 1) Remove-Item $tmpJournal2 -ErrorAction SilentlyContinue +# --- Test-StdinSafe predicate --- +Assert "stdin-safe: trailing quoted prompt (codex)" (Test-StdinSafe -Provider @{ name='c'; command_template='codex exec "{{prompt}}"' }) +Assert "stdin-safe: trailing quoted prompt with model (ollama)" (Test-StdinSafe -Provider @{ name='o'; command_template='ollama run {{model}} "{{prompt}}"'; model_default='m' }) +Assert "stdin-safe: embedded prompt -> legacy (test stub)" (-not (Test-StdinSafe -Provider @{ name='s'; command_template='pwsh -NoProfile -Command "Write-Output hello-{{prompt}}"' })) +Assert "stdin-safe: shell operator in tail -> legacy" (-not (Test-StdinSafe -Provider @{ name='p'; command_template='foo | bar "{{prompt}}"' })) +Assert "stdin-safe: already stdin:true -> not re-flagged" (-not (Test-StdinSafe -Provider @{ name='h'; stdin=$true; command_template='claude -p --model x' })) + +# --- Regression: embedded-prompt stubs still interpolate --- +$tmpJ = New-TemporaryFile +$rReg = Invoke-Fleet -Name 'stub-cli' -Prompt 'world' -Path $fixture -JournalPath $tmpJ +Assert "regression: stub-cli still outputs hello-world (legacy path)" (($rReg.stdout | Out-String).Trim() -eq 'hello-world') +Remove-Item $tmpJ -ErrorAction SilentlyContinue + if ($failures -gt 0) { Write-Host "`n$failures failure(s)" -ForegroundColor Red; exit 1 } Write-Host "`nAll tests passed." -ForegroundColor Green From 3a883b961d6f764f99f2e1b0162995fa83f48b0f Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sat, 4 Jul 2026 23:49:25 -0600 Subject: [PATCH 06/10] feat(fleet): fleet doctor --live round-trip surface (Slice 1) --- scripts/fleet-doctor.ps1 | 25 ++++++++++++++++++++++- scripts/test-fleet-doctor.ps1 | 37 +++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 1 deletion(-) diff --git a/scripts/fleet-doctor.ps1 b/scripts/fleet-doctor.ps1 index 11dc783..172ccba 100644 --- a/scripts/fleet-doctor.ps1 +++ b/scripts/fleet-doctor.ps1 @@ -6,10 +6,13 @@ #> param( [string]$Path = $(if ($env:BATON_HOME) { Join-Path $env:BATON_HOME 'fleet.yaml' } else { Join-Path $HOME '.baton/fleet.yaml' }), - [switch]$Json + [switch]$Json, + [switch]$Live, + [int]$TimeoutS = 60 ) $ErrorActionPreference = 'Stop' . (Join-Path $PSScriptRoot 'fleet-lib.ps1') +if ($Live) { . (Join-Path $PSScriptRoot 'fleet-probe-lib.ps1') } try { $fleet = Read-Fleet -Path $Path @@ -21,6 +24,26 @@ try { Write-Host "fleet doctor: $($_.Exception.Message)" -ForegroundColor Red exit 1 } + +if ($Live) { + $results = foreach ($p in $fleet) { Invoke-FleetProbe -Provider ([hashtable]$p) -TimeoutS $TimeoutS -FleetPath $Path } + $rows = @($results) + if ($Json) { + ConvertTo-Json -InputObject @($rows) -Depth 4 + } else { + $render = $rows | ForEach-Object { + $reach = if ($null -eq $_.reachable) { '-' } elseif ($_.reachable) { 'yes' } else { 'no' } + $detail = if ($_.reason) { $_.reason } elseif ($null -ne $_.elapsed_s) { "$($_.elapsed_s)s" } else { '' } + [pscustomobject]@{ PROVIDER = $_.name; REACHABLE = $reach; LIVE = $_.live; DETAIL = $detail } + } + $render | Format-Table PROVIDER, REACHABLE, LIVE, DETAIL -AutoSize | Out-String | Write-Host + $enabled = @($fleet | Where-Object { $_.enabled -eq $true }).Count + Write-Host "$enabled enabled provider(s); live round-trip." + } + $anyLiveBad = @($rows | Where-Object { $_.enabled -eq $true -and $_.live -ne 'live_ok' }).Count -gt 0 + if ($anyLiveBad) { exit 1 } else { exit 0 } +} + $rows = @() $anyBad = $false diff --git a/scripts/test-fleet-doctor.ps1 b/scripts/test-fleet-doctor.ps1 index aa2352d..70c34ae 100644 --- a/scripts/test-fleet-doctor.ps1 +++ b/scripts/test-fleet-doctor.ps1 @@ -27,5 +27,42 @@ $parsed = $jsonOut | ConvertFrom-Json $tightRow = @($parsed | Where-Object { $_.NAME -eq 'stub-tight' }) Assert "doctor -Json carries class:tight for stub-tight" ($tightRow.Count -eq 1 -and $tightRow[0].class -eq 'tight') +# --- fleet doctor --live (end-to-end, real Invoke-Fleet against fixture) --- +# Fixture roster: stub-cli/stub-with-model/stub-with-env (echo prompt -> contains +# PONG -> live_ok), stub-disabled (skip), stub-http (localhost:9999 -> unreachable), +# stub-fail (no {{prompt}} -> dispatch throws -> dispatch-error), stub-slow (sleep 10 +# -> timeout at --timeout 3). Mixed roster -> exit 1. +$liveOut = & pwsh -NoProfile -File $doctor -Path $fixture -Live -TimeoutS 3 2>&1 | Out-String +$liveExit = $LASTEXITCODE +Assert "live: stub-cli reports live_ok" ($liveOut -match 'stub-cli\s+.*live_ok') +Assert "live: stub-disabled reports skip" ($liveOut -match 'stub-disabled\s+.*skip') +Assert "live: stub-http reports unreachable" ($liveOut -match 'stub-http\s+.*(live_fail|unreachable)') +Assert "live: exit 1 on a mixed roster" ($liveExit -eq 1) + +# --json shape +$liveJson = & pwsh -NoProfile -File $doctor -Path $fixture -Live -TimeoutS 3 -Json 2>&1 | Out-String +$parsedLive = $liveJson | ConvertFrom-Json +$cliRow = @($parsedLive | Where-Object { $_.name -eq 'stub-cli' }) +Assert "live --json: stub-cli row carries live=live_ok" ($cliRow.Count -eq 1 -and $cliRow[0].live -eq 'live_ok') + +# All-live_ok roster -> exit 0 (hermetic single-provider temp yaml) +$tmpYaml = New-TemporaryFile +@' +providers: + - name: only-ok + kind: cli + enabled: true + cost_tier: free + command_template: 'pwsh -NoProfile -Command "Write-Output PONG-{{prompt}}"' +'@ | Set-Content -LiteralPath $tmpYaml -Encoding utf8NoBOM +$okOut = & pwsh -NoProfile -File $doctor -Path $tmpYaml -Live -TimeoutS 10 2>&1 | Out-String +$okExit = $LASTEXITCODE +Assert "live: all-live_ok roster -> exit 0" ($okExit -eq 0) +Remove-Item $tmpYaml -ErrorAction SilentlyContinue + +# Default (non-live) path unchanged: still reports PATH-based skip/err and exit 1 +$plainOut = & pwsh -NoProfile -File $doctor -Path $fixture 2>&1 | Out-String +Assert "non-live path still reports stub-disabled skip" ($plainOut -match 'stub-disabled\s+skip') + if ($failures -gt 0) { Write-Host "`n$failures failure(s)" -ForegroundColor Red; exit 1 } Write-Host "`nAll tests passed." -ForegroundColor Green From 8536cd72853fe5d809876c15191df6638a26fd86 Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sat, 4 Jul 2026 23:52:36 -0600 Subject: [PATCH 07/10] chore(fleet): deploy wiring + docs + v1.10.0-rc.1 (Slice 1) --- .claude-plugin/plugin.json | 2 +- AGENTS.md | 3 +++ commands/fleet.md | 8 ++++---- scripts/bootstrap.ps1 | 2 +- scripts/test-bootstrap.ps1 | 1 + 5 files changed, 10 insertions(+), 6 deletions(-) diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 7188b1f..c4836a1 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "baton", "displayName": "Baton", - "version": "1.9.0", + "version": "1.10.0-rc.1", "description": "Pass the baton. Conduct the fleet. Claude Code as command-and-control for a fleet of coding LLMs — capability routing, cost engine, jobs, decisions, and a knowledge base.", "author": { "name": "Kevin Rank", "url": "https://github.com/Ryfter" }, "repository": "https://github.com/Ryfter/baton", diff --git a/AGENTS.md b/AGENTS.md index ead65ad..a1d26bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,3 +31,6 @@ Active-session detection and resume pointers use a neutral marker contract under `$BATON_HOME/sessions/` (`{agent,session_id,cwd,started_at}`). The Claude adapter (SessionStart/SessionEnd hooks) ships now; a Codex lifecycle adapter writing the same marker shape is the documented follow-on. + +Fleet health (model-agnostic): +- `fleet doctor --live` — harness-neutral way to verify a box's roster actually answers (canary round-trip per enabled provider), not just that the binaries are installed. diff --git a/commands/fleet.md b/commands/fleet.md index 94148fb..0237d95 100644 --- a/commands/fleet.md +++ b/commands/fleet.md @@ -1,6 +1,6 @@ --- description: Manage and invoke the LLM fleet. `doctor` health-checks providers, `test` dispatches a prompt to one provider, `list` shows the registry. -argument-hint: doctor | test "" [--model ] | list +argument-hint: doctor [--live] [--timeout ] | test "" [--model ] | list --- # /baton:fleet @@ -15,13 +15,13 @@ Operate the fleet defined in `$BATON_HOME/fleet.yaml` (default `~/.baton/fleet.y 2. **Dispatch by subcommand:** - **`doctor`** — run: + **`doctor`** — run (with `--live` to enable canary probes): ```powershell - & pwsh -NoProfile -File "$HOME/.claude/scripts/fleet-doctor.ps1" + & pwsh -NoProfile -File "$HOME/.claude/scripts/fleet-doctor.ps1" -Live -TimeoutS 60 ``` - Echo the table to the user. + Echo the table to the user. With `--live`, each enabled provider receives a `PONG` canary and reports `live_ok`, `live_fail()`, or `skip`; a `no-canary` reason means the provider ran but didn't answer (often a wrong command template for this box). Plain (no `--live`) performs the fast PATH/reachability check only. **`list`** — run: diff --git a/scripts/bootstrap.ps1 b/scripts/bootstrap.ps1 index 9c6aff2..96c3cd5 100644 --- a/scripts/bootstrap.ps1 +++ b/scripts/bootstrap.ps1 @@ -256,7 +256,7 @@ if (-not (Test-Path $scriptsDst)) { if ($DryRun) { Write-Ok "[dry-run] would create $scriptsDst" } else { New-Item -ItemType Directory -Force -Path $scriptsDst | Out-Null; Write-Ok "created $scriptsDst" } } -foreach ($script in @('baton-home.ps1', 'job-lib.ps1', 'consolidate-lessons.ps1', 'parse-otel.ps1', 'fleet-lib.ps1', 'fleet-doctor.ps1', 'fleet-ensemble.ps1', 'routing-lib.ps1', 'saturation-lib.ps1', 'effective-cost-lib.ps1', 'routing-dispatch.ps1', 'routing-learn.ps1', 'routing-calibrate.ps1', 'routing-cascade.ps1', 'prime-hours.ps1', 'six-hats-lib.ps1', 'council-lib.ps1', 'code-lib.ps1', 'kb-lib.ps1', 'decisions-lib.ps1', 'consolidate-decisions.ps1', 'cost-lib.ps1', 'runs-lib.ps1', 'statusline-feed.ps1', 'fleet-runs-bridge.ps1', 'fleet-orchestrate.ps1', 'fleet-backlog.ps1', 'run-backlog.ps1', 'fleet-models.ps1', 'triage-lib.ps1', 'fleet-triage.ps1', 'usage-lib.ps1', 'fleet-usage.ps1', 'projects-lib.ps1', 'fleet-projects.ps1', 'research-gate-lib.ps1', 'fleet-research-gate.ps1', 'conductor-lib.ps1', 'fleet-go.ps1', 'cost-resolver-lib.ps1', 'prompt-pool-lib.ps1', 'optimize-prompt-lib.ps1', 'fleet-optimize-prompt.ps1', 'memory-lib.ps1', 'fleet-memory.ps1', 'worker-lib.ps1', 'fleet-worker.ps1', 'gate-lib.ps1', 'fleet-gate.ps1', 'fleet-effective-cost.ps1', 'idea-lib.ps1', 'start-lib.ps1', 'coach-lib.ps1', 'session-markers-lib.ps1', 'registry-lib.ps1', 'fleet-project.ps1')) { +foreach ($script in @('baton-home.ps1', 'job-lib.ps1', 'consolidate-lessons.ps1', 'parse-otel.ps1', 'fleet-lib.ps1', 'fleet-doctor.ps1', 'fleet-ensemble.ps1', 'routing-lib.ps1', 'saturation-lib.ps1', 'effective-cost-lib.ps1', 'routing-dispatch.ps1', 'routing-learn.ps1', 'routing-calibrate.ps1', 'routing-cascade.ps1', 'prime-hours.ps1', 'six-hats-lib.ps1', 'council-lib.ps1', 'code-lib.ps1', 'kb-lib.ps1', 'decisions-lib.ps1', 'consolidate-decisions.ps1', 'cost-lib.ps1', 'runs-lib.ps1', 'statusline-feed.ps1', 'fleet-runs-bridge.ps1', 'fleet-orchestrate.ps1', 'fleet-backlog.ps1', 'run-backlog.ps1', 'fleet-models.ps1', 'triage-lib.ps1', 'fleet-triage.ps1', 'usage-lib.ps1', 'fleet-usage.ps1', 'projects-lib.ps1', 'fleet-projects.ps1', 'research-gate-lib.ps1', 'fleet-research-gate.ps1', 'conductor-lib.ps1', 'fleet-go.ps1', 'cost-resolver-lib.ps1', 'prompt-pool-lib.ps1', 'optimize-prompt-lib.ps1', 'fleet-optimize-prompt.ps1', 'memory-lib.ps1', 'fleet-memory.ps1', 'worker-lib.ps1', 'fleet-worker.ps1', 'gate-lib.ps1', 'fleet-gate.ps1', 'fleet-effective-cost.ps1', 'idea-lib.ps1', 'start-lib.ps1', 'coach-lib.ps1', 'session-markers-lib.ps1', 'registry-lib.ps1', 'fleet-project.ps1', 'fleet-probe-lib.ps1')) { $src = Join-Path $repoRoot "scripts\$script" $dst = Join-Path $scriptsDst $script Copy-WithPrompt $src $dst "lib script: $script" -Force diff --git a/scripts/test-bootstrap.ps1 b/scripts/test-bootstrap.ps1 index 1f6fd8d..9b6eac9 100644 --- a/scripts/test-bootstrap.ps1 +++ b/scripts/test-bootstrap.ps1 @@ -61,6 +61,7 @@ Assert "deploys fleet-gate script" ($out -match 'fleet-gate\.ps1') Assert "would deploy start-lib.ps1" ($out -match 'start-lib\.ps1') Assert "deploys coach-lib script (v1.8.0 footers need it on deployed boxes)" ($out -match 'coach-lib\.ps1') Assert "deploys registry-lib script (roster/resolution needed on deployed boxes)" ($out -match 'registry-lib\.ps1') +Assert "deploys fleet-probe-lib script (canary round-trip needed on deployed boxes)" ($out -match 'fleet-probe-lib\.ps1') Assert "deploys session-markers-lib script (active detection needs it on-box)" ($out -match 'session-markers-lib\.ps1') Assert "deploys fleet-project script (/baton:project CLI)" ($out -match 'fleet-project\.ps1') Assert "would deploy cost-resolver-lib.ps1" ($out -match 'cost-resolver-lib\.ps1') From c6b9764bfbc7c18b37731214f0d6782de10558a3 Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sun, 5 Jul 2026 01:17:19 -0600 Subject: [PATCH 08/10] feat(fleet): 4-challenge canary battery (closes echo-back false-pass) --- scripts/fleet-doctor.ps1 | 5 +++- scripts/fleet-probe-lib.ps1 | 51 ++++++++++++++++++++++++-------- scripts/test-fleet-doctor.ps1 | 30 +++++++++++++------ scripts/test-fleet-probe-lib.ps1 | 32 ++++++++++++-------- 4 files changed, 83 insertions(+), 35 deletions(-) diff --git a/scripts/fleet-doctor.ps1 b/scripts/fleet-doctor.ps1 index 172ccba..1132542 100644 --- a/scripts/fleet-doctor.ps1 +++ b/scripts/fleet-doctor.ps1 @@ -33,7 +33,10 @@ if ($Live) { } else { $render = $rows | ForEach-Object { $reach = if ($null -eq $_.reachable) { '-' } elseif ($_.reachable) { 'yes' } else { 'no' } - $detail = if ($_.reason) { $_.reason } elseif ($null -ne $_.elapsed_s) { "$($_.elapsed_s)s" } else { '' } + $detail = if ($_.reason) { $_.reason } + elseif ($null -ne $_.score -and $null -ne $_.elapsed_s) { "$($_.score)/$($script:FleetCanaryChallenges.Count)`u{00B7}$($_.elapsed_s)s" } + elseif ($null -ne $_.elapsed_s) { "$($_.elapsed_s)s" } + else { '' } [pscustomobject]@{ PROVIDER = $_.name; REACHABLE = $reach; LIVE = $_.live; DETAIL = $detail } } $render | Format-Table PROVIDER, REACHABLE, LIVE, DETAIL -AutoSize | Out-String | Write-Host diff --git a/scripts/fleet-probe-lib.ps1 b/scripts/fleet-probe-lib.ps1 index d082e52..3dba498 100644 --- a/scripts/fleet-probe-lib.ps1 +++ b/scripts/fleet-probe-lib.ps1 @@ -11,23 +11,48 @@ . "$PSScriptRoot/baton-home.ps1" . "$PSScriptRoot/fleet-lib.ps1" # Read-Fleet, Invoke-Fleet, Get-FleetProvider -$script:FleetCanaryPrompt = 'Reply with exactly the word PONG and nothing else.' -$script:FleetCanaryToken = 'PONG' +# The canary battery: 4 trivial challenges. Only PONG's answer appears in the +# prompt text (the instruction-following check); 42/PARIS/COLD do NOT, so a +# template that merely echoes the prompt scores 1/4 and correctly fails. +$script:FleetCanaryChallenges = @( + @{ ask = 'Reply with the word PONG.'; token = 'PONG' } + @{ ask = 'What is 6 times 7?'; token = '42' } + @{ ask = 'What is the capital of France?'; token = 'PARIS' } + @{ ask = 'What is the opposite of the word HOT?'; token = 'COLD' } +) + +# Combined single-dispatch prompt (one round-trip per provider), built from the +# challenge asks so it stays in sync with the tokens above. +$script:FleetCanaryPrompt = @( + 'Answer all four questions. Reply with one answer per line, each a single word or number, with no other text:' + for ($n = 0; $n -lt $script:FleetCanaryChallenges.Count; $n++) { + "$($n + 1). $($script:FleetCanaryChallenges[$n].ask)" + } +) -join "`n" function Test-FleetCanary { - <# Pure. Classify a dispatch result into a live verdict + reason. - Precedence: timeout > nonzero-exit > token-match. #> + <# Pure. Score a dispatch result against the canary battery. + Precedence: timeout > nonzero-exit > token score. + Returns @{ live; reason; score } where score = tokens matched (0..N), + or $null when timed out / nonzero exit (no clean answer to score). #> param( [string]$Output, [int]$ExitCode = 0, [bool]$TimedOut = $false ) - if ($TimedOut) { return @{ live = 'live_fail'; reason = 'timeout' } } - if ($ExitCode -ne 0) { return @{ live = 'live_fail'; reason = 'nonzero-exit' } } - if (([string]$Output).ToUpperInvariant().Contains($script:FleetCanaryToken)) { - return @{ live = 'live_ok'; reason = $null } + $total = $script:FleetCanaryChallenges.Count + if ($TimedOut) { return @{ live = 'live_fail'; reason = 'timeout'; score = $null } } + if ($ExitCode -ne 0) { return @{ live = 'live_fail'; reason = 'nonzero-exit'; score = $null } } + $up = ([string]$Output).ToUpperInvariant() + $score = 0 + $missing = @() + foreach ($ch in $script:FleetCanaryChallenges) { + if ($up.Contains(([string]$ch.token).ToUpperInvariant())) { $score++ } + else { $missing += [string]$ch.token } } - return @{ live = 'live_fail'; reason = 'no-canary' } + if ($score -eq $total) { return @{ live = 'live_ok'; reason = $null; score = $score } } + if ($score -eq 0) { return @{ live = 'live_fail'; reason = 'no-canary'; score = 0 } } + return @{ live = 'live_fail'; reason = "canary $score/$total (missing: $($missing -join ', '))"; score = $score } } function Test-ProviderReachable { @@ -73,11 +98,11 @@ function Invoke-FleetProbe { $name = [string]$Provider.name $kind = [string]$Provider.kind if ($Provider.enabled -ne $true) { - return @{ name = $name; kind = $kind; enabled = $false; reachable = $null; live = 'skip'; reason = 'disabled'; elapsed_s = $null } + return @{ name = $name; kind = $kind; enabled = $false; reachable = $null; live = 'skip'; reason = 'disabled'; elapsed_s = $null; score = $null } } $reach = Test-ProviderReachable -Provider $Provider -UrlProbe $UrlProbe if (-not $reach.reachable) { - return @{ name = $name; kind = $kind; enabled = $true; reachable = $false; live = 'live_fail'; reason = $reach.reason; elapsed_s = $null } + return @{ name = $name; kind = $kind; enabled = $true; reachable = $false; live = 'live_fail'; reason = $reach.reason; elapsed_s = $null; score = $null } } if (-not $Dispatcher) { $Dispatcher = { @@ -108,8 +133,8 @@ function Invoke-FleetProbe { } $elapsed = [int]$sw.Elapsed.TotalSeconds if ($errored) { - return @{ name = $name; kind = $kind; enabled = $true; reachable = $true; live = 'live_fail'; reason = 'dispatch-error'; elapsed_s = $elapsed } + return @{ name = $name; kind = $kind; enabled = $true; reachable = $true; live = 'live_fail'; reason = 'dispatch-error'; elapsed_s = $elapsed; score = $null } } $verdict = Test-FleetCanary -Output $output -ExitCode $exit -TimedOut $timedOut - return @{ name = $name; kind = $kind; enabled = $true; reachable = $true; live = $verdict.live; reason = $verdict.reason; elapsed_s = $elapsed } + return @{ name = $name; kind = $kind; enabled = $true; reachable = $true; live = $verdict.live; reason = $verdict.reason; elapsed_s = $elapsed; score = $verdict.score } } diff --git a/scripts/test-fleet-doctor.ps1 b/scripts/test-fleet-doctor.ps1 index 70c34ae..eb9c0a0 100644 --- a/scripts/test-fleet-doctor.ps1 +++ b/scripts/test-fleet-doctor.ps1 @@ -34,18 +34,24 @@ Assert "doctor -Json carries class:tight for stub-tight" ($tightRow.Count -eq 1 # -> timeout at --timeout 3). Mixed roster -> exit 1. $liveOut = & pwsh -NoProfile -File $doctor -Path $fixture -Live -TimeoutS 3 2>&1 | Out-String $liveExit = $LASTEXITCODE -Assert "live: stub-cli reports live_ok" ($liveOut -match 'stub-cli\s+.*live_ok') +Assert "live: echo stub scores partial (echo hole closed)" ($liveOut -match 'stub-cli\s+.*live_fail') Assert "live: stub-disabled reports skip" ($liveOut -match 'stub-disabled\s+.*skip') Assert "live: stub-http reports unreachable" ($liveOut -match 'stub-http\s+.*(live_fail|unreachable)') Assert "live: exit 1 on a mixed roster" ($liveExit -eq 1) -# --json shape -$liveJson = & pwsh -NoProfile -File $doctor -Path $fixture -Live -TimeoutS 3 -Json 2>&1 | Out-String -$parsedLive = $liveJson | ConvertFrom-Json -$cliRow = @($parsedLive | Where-Object { $_.name -eq 'stub-cli' }) -Assert "live --json: stub-cli row carries live=live_ok" ($cliRow.Count -eq 1 -and $cliRow[0].live -eq 'live_ok') - -# All-live_ok roster -> exit 0 (hermetic single-provider temp yaml) +# All-live_ok roster -> exit 0 (hermetic single-provider temp yaml; must emit +# all 4 canary tokens or the echo hole would sink it to a partial score). +# NOTE: the brief's suggested single-line-echo template +# ('pwsh -NoProfile -Command "Write-Output PONG-42-PARIS-COLD-{{prompt}}"') +# does not survive reconciliation: the combined canary prompt is multi-line +# ("1. ...\n2. ..."), and interpolating it unquoted into a child -Command +# breaks the child's parser (newlines become statement separators; "1." reads +# as a number literal). Fixed by using a template whose trailing +# `"{{prompt}}"` is a Test-StdinSafe-eligible quoted tail (see fleet-lib.ps1 +# Test-StdinSafe): the tail gets stripped and the real prompt is instead piped +# via stdin (and ignored), while the head — with zero embedded spaces in its +# final token so the naive whitespace tokenizer in Invoke-Fleet-Cli's stdin +# dispatch splits it cleanly — deterministically emits all 4 tokens. $tmpYaml = New-TemporaryFile @' providers: @@ -53,11 +59,17 @@ providers: kind: cli enabled: true cost_tier: free - command_template: 'pwsh -NoProfile -Command "Write-Output PONG-{{prompt}}"' + command_template: 'pwsh -NoProfile -Command Write-Output("PONG-42-PARIS-COLD") "{{prompt}}"' '@ | Set-Content -LiteralPath $tmpYaml -Encoding utf8NoBOM $okOut = & pwsh -NoProfile -File $doctor -Path $tmpYaml -Live -TimeoutS 10 2>&1 | Out-String $okExit = $LASTEXITCODE Assert "live: all-live_ok roster -> exit 0" ($okExit -eq 0) + +# --json shape (moved onto the all-4-token provider; stub-cli is no longer live_ok) +$okJson = & pwsh -NoProfile -File $doctor -Path $tmpYaml -Live -TimeoutS 10 -Json 2>&1 | Out-String +$okParsed = $okJson | ConvertFrom-Json +$okRow = @($okParsed | Where-Object { $_.name -eq 'only-ok' }) +Assert "live --json: only-ok row live_ok score 4" ($okRow.Count -eq 1 -and $okRow[0].live -eq 'live_ok' -and $okRow[0].score -eq 4) Remove-Item $tmpYaml -ErrorAction SilentlyContinue # Default (non-live) path unchanged: still reports PATH-based skip/err and exit 1 diff --git a/scripts/test-fleet-probe-lib.ps1 b/scripts/test-fleet-probe-lib.ps1 index 3a65a0a..605daf1 100644 --- a/scripts/test-fleet-probe-lib.ps1 +++ b/scripts/test-fleet-probe-lib.ps1 @@ -9,17 +9,19 @@ function Assert($label, $cond) { else { Write-Host "FAIL $label" -ForegroundColor Red; $script:failures++ } } -# --- Test-FleetCanary (pure) --- -$c1 = Test-FleetCanary -Output 'PONG' -ExitCode 0 -TimedOut $false -Assert "canary: exact token -> live_ok" ($c1.live -eq 'live_ok' -and $null -eq $c1.reason) -$c2 = Test-FleetCanary -Output 'the answer is pong.' -ExitCode 0 -TimedOut $false -Assert "canary: case-insensitive substring -> live_ok" ($c2.live -eq 'live_ok') -$c3 = Test-FleetCanary -Output 'usage: codex [options]' -ExitCode 0 -TimedOut $false -Assert "canary: exit 0 but no token -> no-canary" ($c3.live -eq 'live_fail' -and $c3.reason -eq 'no-canary') -$c4 = Test-FleetCanary -Output 'PONG' -ExitCode 3 -TimedOut $false -Assert "canary: nonzero exit beats token -> nonzero-exit" ($c4.live -eq 'live_fail' -and $c4.reason -eq 'nonzero-exit') -$c5 = Test-FleetCanary -Output '' -ExitCode 0 -TimedOut $true -Assert "canary: timeout beats all -> timeout" ($c5.live -eq 'live_fail' -and $c5.reason -eq 'timeout') +# --- Test-FleetCanary (battery) --- +$c1 = Test-FleetCanary -Output 'PONG 42 PARIS COLD' -ExitCode 0 -TimedOut $false +Assert "canary: all 4 tokens -> live_ok score 4" ($c1.live -eq 'live_ok' -and $c1.score -eq 4) +$c2 = Test-FleetCanary -Output 'the answer is pong, 42, paris, and cold' -ExitCode 0 -TimedOut $false +Assert "canary: case-insensitive all 4 -> live_ok" ($c2.live -eq 'live_ok') +$c3 = Test-FleetCanary -Output 'PONG 42' -ExitCode 0 -TimedOut $false +Assert "canary: partial 2/4 -> live_fail with score+missing" ($c3.live -eq 'live_fail' -and $c3.score -eq 2 -and $c3.reason -match 'canary 2/4' -and $c3.reason -match 'PARIS' -and $c3.reason -match 'COLD') +$c4 = Test-FleetCanary -Output 'usage: codex [options]' -ExitCode 0 -TimedOut $false +Assert "canary: zero tokens -> no-canary score 0" ($c4.live -eq 'live_fail' -and $c4.reason -eq 'no-canary' -and $c4.score -eq 0) +$c5 = Test-FleetCanary -Output 'PONG 42 PARIS COLD' -ExitCode 3 -TimedOut $false +Assert "canary: nonzero exit beats full score -> nonzero-exit" ($c5.live -eq 'live_fail' -and $c5.reason -eq 'nonzero-exit') +$c6 = Test-FleetCanary -Output '' -ExitCode 0 -TimedOut $true +Assert "canary: timeout beats all -> timeout" ($c6.live -eq 'live_fail' -and $c6.reason -eq 'timeout') # --- Test-ProviderReachable --- $cliOk = Test-ProviderReachable -Provider @{ name='p'; kind='cli'; command_template='pwsh -NoProfile -Command "x"' } @@ -33,7 +35,8 @@ Assert "reachable: http probe false -> unreachable" ($httpNo.reachable -eq $fals # --- Invoke-FleetProbe (live round-trip) --- # Fake dispatchers returning the Invoke-Fleet shape @{stdout;exit_code}. -$okDisp = { param($prov,$fp,$canary,$root) @{ stdout = 'PONG'; exit_code = 0 } } +$okDisp = { param($prov,$fp,$canary,$root) @{ stdout = 'PONG 42 PARIS COLD'; exit_code = 0 } } +$partDisp = { param($prov,$fp,$canary,$root) @{ stdout = 'PONG 42'; exit_code = 0 } } $noTokDisp = { param($prov,$fp,$canary,$root) @{ stdout = 'help text here'; exit_code = 0 } } $failDisp = { param($prov,$fp,$canary,$root) @{ stdout = ''; exit_code = 7 } } $slowDisp = { param($prov,$fp,$canary,$root) Start-Sleep -Seconds 5; @{ stdout = 'PONG'; exit_code = 0 } } @@ -47,9 +50,14 @@ Assert "probe: disabled -> skip" ($rSkip.live -eq 'skip' -and $rSkip.reas $rOk = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $okDisp Assert "probe: token -> live_ok" ($rOk.live -eq 'live_ok' -and $rOk.reachable -eq $true) Assert "probe: live_ok records elapsed" ($rOk.elapsed_s -ge 0) +Assert "probe: full battery -> live_ok" ($rOk.live -eq 'live_ok' -and $rOk.score -eq 4) + +$rPart = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $partDisp +Assert "probe: partial battery -> live_fail 2/4" ($rPart.live -eq 'live_fail' -and $rPart.score -eq 2 -and $rPart.reason -match '2/4') $rNo = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $noTokDisp Assert "probe: no token -> no-canary" ($rNo.live -eq 'live_fail' -and $rNo.reason -eq 'no-canary') +Assert "probe: no tokens -> no-canary" ($rNo.reason -eq 'no-canary' -and $rNo.score -eq 0) $rFail = Invoke-FleetProbe -Provider $enabledCli -Dispatcher $failDisp Assert "probe: nonzero exit -> nonzero-exit" ($rFail.reason -eq 'nonzero-exit') From 5bec3e66add08ddf6bf5a2d5514f2491ce3f6a9a Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sun, 5 Jul 2026 01:23:24 -0600 Subject: [PATCH 09/10] =?UTF-8?q?fix(fleet):=20stdin:true=20dispatch=20reg?= =?UTF-8?q?ression=20=E2=80=94=20resolve=20template=20without=20empty-prom?= =?UTF-8?q?pt=20binding?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 3 routed stdin:true providers through Resolve-FleetCommand -Prompt '', which a mandatory [string] param rejects, breaking every stdin:true provider (github-models, box-private reviewer pairs). Resolve {{model}} inline for both stdin paths instead. Add a stdin:true fixture provider + round-trip asserts — the gap that let this through review. Co-Authored-By: Claude Opus 4.8 --- scripts/fixtures/fleet-sample.yaml | 10 ++++++++++ scripts/fleet-lib.ps1 | 16 +++++++++------- scripts/test-fleet-dispatch.ps1 | 8 ++++++++ scripts/test-fleet-lib.ps1 | 2 +- 4 files changed, 28 insertions(+), 8 deletions(-) diff --git a/scripts/fixtures/fleet-sample.yaml b/scripts/fixtures/fleet-sample.yaml index 710aab4..26d5605 100644 --- a/scripts/fixtures/fleet-sample.yaml +++ b/scripts/fixtures/fleet-sample.yaml @@ -53,3 +53,13 @@ providers: cost_tier: local usage_class: tight command_template: 'pwsh -NoProfile -Command "Write-Output tight-{{prompt}}"' + + # stdin:true provider — the prompt is piped via stdin, template carries no + # {{prompt}}. Guards the regression where the stdin path called + # Resolve-FleetCommand with an empty prompt and threw. Echoes stdin back. + - name: stub-stdin + kind: cli + enabled: true + cost_tier: local + stdin: true + command_template: 'pwsh -NoProfile -Command [Console]::In.ReadToEnd()' diff --git a/scripts/fleet-lib.ps1 b/scripts/fleet-lib.ps1 index f3d3018..56f1153 100644 --- a/scripts/fleet-lib.ps1 +++ b/scripts/fleet-lib.ps1 @@ -250,15 +250,17 @@ function Invoke-Fleet-Cli { ) # Decide dispatch path. stdin:true providers already omit {{prompt}}; # clean-tail interpolating providers are promoted to stdin (prompt-size / - # quote hardening) by stripping the trailing quoted {{prompt}} token. + # quote hardening) by stripping the trailing quoted {{prompt}} token. For + # both stdin cases we resolve {{model}} inline — NOT via Resolve-FleetCommand, + # whose mandatory -Prompt would reject the empty prompt a stdin dispatch uses. $useStdin = ($Provider.stdin -eq $true) -or (Test-StdinSafe -Provider $Provider) - if ($Provider.stdin -eq $true) { - $cmd = Resolve-FleetCommand -Provider $Provider -Prompt '' -Model $Model - } elseif (Test-StdinSafe -Provider $Provider) { - $stripped = ([string]$Provider.command_template) -replace '\s+(["''])\{\{prompt\}\}\1\s*$', '' + if ($useStdin) { + # stdin:true templates carry no {{prompt}}; Test-StdinSafe templates end in + # a standalone quoted {{prompt}} that we strip. Both then resolve {{model}}. + $cmd = if ($Provider.stdin -eq $true) { [string]$Provider.command_template } + else { ([string]$Provider.command_template) -replace '\s+(["''])\{\{prompt\}\}\1\s*$', '' } $resolvedModel = if ($Model) { $Model } else { $Provider.model_default } - if ($null -ne $resolvedModel) { $stripped = $stripped.Replace('{{model}}', [string]$resolvedModel) } - $cmd = $stripped + if ($null -ne $resolvedModel) { $cmd = $cmd.Replace('{{model}}', [string]$resolvedModel) } } else { $cmd = Resolve-FleetCommand -Provider $Provider -Prompt $Prompt -Model $Model } diff --git a/scripts/test-fleet-dispatch.ps1 b/scripts/test-fleet-dispatch.ps1 index 9375464..691bf5b 100644 --- a/scripts/test-fleet-dispatch.ps1 +++ b/scripts/test-fleet-dispatch.ps1 @@ -76,5 +76,13 @@ $rReg = Invoke-Fleet -Name 'stub-cli' -Prompt 'world' -Path $fixture -JournalPat Assert "regression: stub-cli still outputs hello-world (legacy path)" (($rReg.stdout | Out-String).Trim() -eq 'hello-world') Remove-Item $tmpJ -ErrorAction SilentlyContinue +# --- Regression: stdin:true provider round-trips via stdin (guards the empty-prompt +# Resolve-FleetCommand rejection that broke real stdin providers) --- +$tmpJs = New-TemporaryFile +$rStdin = Invoke-Fleet -Name 'stub-stdin' -Prompt 'HELLO-VIA-STDIN' -Path $fixture -JournalPath $tmpJs +Assert "stdin:true provider dispatches without throwing" ($rStdin.exit_code -eq 0) +Assert "stdin:true provider receives the prompt on stdin" (($rStdin.stdout | Out-String) -match 'HELLO-VIA-STDIN') +Remove-Item $tmpJs -ErrorAction SilentlyContinue + if ($failures -gt 0) { Write-Host "`n$failures failure(s)" -ForegroundColor Red; exit 1 } Write-Host "`nAll tests passed." -ForegroundColor Green diff --git a/scripts/test-fleet-lib.ps1 b/scripts/test-fleet-lib.ps1 index a97e536..a3befa8 100644 --- a/scripts/test-fleet-lib.ps1 +++ b/scripts/test-fleet-lib.ps1 @@ -12,7 +12,7 @@ function Assert($label, $cond) { # --- Read-Fleet --- $fleet = Read-Fleet -Path $fixture -Assert "Read-Fleet returns 8 providers" ($fleet.Count -eq 8) +Assert "Read-Fleet returns 9 providers" ($fleet.Count -eq 9) Assert "first provider name is stub-cli" ($fleet[0].name -eq 'stub-cli') Assert "stub-cli kind is cli" ($fleet[0].kind -eq 'cli') Assert "stub-cli enabled is boolean true" ($fleet[0].enabled -eq $true) From eac461e3144a288a77eeeca37be1447685a8e1d0 Mon Sep 17 00:00:00 2001 From: Kevin Rank Date: Sun, 5 Jul 2026 01:24:23 -0600 Subject: [PATCH 10/10] =?UTF-8?q?docs(fleet):=20spec=20=E2=80=94=20canary?= =?UTF-8?q?=20battery=20(4=20challenges)=20replaces=20single=20PONG=20toke?= =?UTF-8?q?n?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 --- ...eet-labor-slice1-roundtrip-proof-design.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md b/docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md index d5b9f54..663c939 100644 --- a/docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md +++ b/docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md @@ -104,10 +104,19 @@ directly) so both `kind: cli` and `kind: http` providers — including the local LM Studio / Ollama boxes — are covered by the same code path. A `-Dispatcher` scriptblock seam is injected for tests. -- **Canary prompt (constant):** `Reply with exactly the word PONG and nothing else.` -- **Canary token (constant):** `PONG`. -- **Pass (`live_ok`):** dispatch returns exit 0 **and** stdout contains `PONG` - (case-insensitive, substring) **and** it completed within the probe timeout. +- **Canary battery (constant):** a set of 4 trivial challenges, dispatched as one + combined numbered prompt (a single round-trip). Only the first challenge's + answer appears in the prompt text; the other three do **not**, so a template + that merely echoes the prompt cannot score them: + 1. *Reply with the word PONG.* → token `PONG` (literal instruction-following) + 2. *What is 6 times 7?* → token `42` (echo-proof) + 3. *What is the capital of France?* → token `PARIS` (echo-proof) + 4. *What is the opposite of the word HOT?* → token `COLD` (echo-proof) +- **Score:** count of the 4 tokens present in stdout (case-insensitive substring). +- **Pass (`live_ok`):** dispatch returns exit 0 **and** all 4 tokens are present + (score 4/4) **and** it completed within the probe timeout. A partial score + (`live_fail` reason `canary 2/4 (missing: 42, PARIS)`) tells you exactly how a + provider misbehaved; an echo-back scores 1/4 and correctly fails. - **Fail (`live_fail`):** with a single-word reason: - `not-on-PATH` — reachability check failed (cli binary missing). - `unreachable` — reachability check failed (http base_url / env URL down). @@ -222,8 +231,10 @@ future dashboard tile). first; the executor (gap 3) is designed against a pipe known to work. - **Extend `fleet doctor --live`** rather than a new command or a `go` preflight — one health surface, minimal new code. -- **Canary-token pass criterion** (`PONG`), judge-free — catches help-text / - wrong-template / garbage responses, not just exit 0. +- **Canary battery pass criterion** (4 challenges scored k/4; `PONG`/`42`/`PARIS`/ + `COLD`), judge-free — catches help-text / wrong-template / garbage / echo-back + responses, not just exit 0. (Revised from a single `PONG` token during review: + the token lived inside its own prompt, so an echoing template false-passed.) - **Stdin path as the CLI dispatch default** — hardens the pipe for the real prompts Slice 2 will send.