From 7c6ffe3f8c0891c4106da92c7e3c22fa776ee49b Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Tue, 2 Jun 2026 21:49:43 -0700 Subject: [PATCH 01/11] docs: agent-framework benchmark plan (GTA build target, gx10 vLLM) Plan to benchmark 13 OSS agent frameworks against this harness by giving each the same autonomous build task (a GTA-style game) over one shared gx10 vLLM /v1 endpoint. Two tiers: turnkey coding agents (apples-to-apples) and build-your-own frameworks (scaffold-bound, flagged). Harness invariants carried over: adapter boundary per framework, append-first attributed results store, metrics-as-config. Repeatable (pinned + N-run distributions), recordable (full artifact capture + replay), adjustable (scorer plugins listed in metrics.yaml). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/agent-framework-bench.md | 211 ++++++++++++++++++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 docs/agent-framework-bench.md diff --git a/docs/agent-framework-bench.md b/docs/agent-framework-bench.md new file mode 100644 index 0000000..b755b75 --- /dev/null +++ b/docs/agent-framework-bench.md @@ -0,0 +1,211 @@ +# Agent-Framework Benchmark Plan + +Compare open-source agent frameworks **similar to this harness** by giving each the +same autonomous build task — **build a GTA-style game** — against a single shared +remote model (**gx10**). The harness enters as the home-team baseline. + +The benchmark is designed to be **repeatable** (pinned everything, N runs, full env +manifest), **recordable** (every transcript, diff, and artifact captured for replay), +and **adjustable** (metrics are pluggable scorers listed in a config — add/remove a +metric without touching the runner). + +> Scope decisions locked with the requester: build target = *build a GTA-style game*; +> remote model = *gx10 over an OpenAI-compatible `/v1` endpoint (vLLM)*; candidate +> language = *any, best-in-class*. + +--- + +## 1. Candidate libraries + +Two buckets. **Turnkey coding agents** drive their own multi-step build loop, exactly +like this harness does — apples-to-apples. **Build-your-own frameworks** need a thin +GTA-builder scaffold written per library, so they measure *your scaffold quality* as +much as the library; include them as a second tier and flag the caveat in results. + +Every candidate below is confirmed to support a custom OpenAI-compatible `base_url` +(directly or via LiteLLM), so all can be repointed at gx10 with config only. + +### Tier 1 — turnkey coding agents (primary comparands) + +| Framework | Repo | gx10 wiring | Notes | +|---|---|---|---| +| **This harness** | (local) | native vLLM / OpenAI adapter | baseline / reference entry | +| **OpenHands** (ex-OpenDevin) | https://github.com/All-Hands-AI/OpenHands | LiteLLM → `base_url` | best open SWE-bench scaffold; Docker sandbox; fully autonomous | +| **Aider** | https://github.com/Aider-AI/aider | `--openai-api-base` / LiteLLM | veteran, diff-based edits, terminal-native, scriptable | +| **OpenCode** | https://github.com/sst/opencode | OpenAI-compatible provider in `opencode.json` | terminal agent, 75+ providers, easy headless | +| **Goose** (Block) | https://github.com/block/goose | built-in OpenAI provider → vLLM `/v1` | Apache-2.0, MCP extensions, vLLM guide exists | +| **mini-SWE-agent** | https://github.com/SWE-agent/mini-swe-agent | LiteLLM | minimal canonical reference loop; great control | +| **SWE-agent** | https://github.com/SWE-agent/SWE-agent | LiteLLM | research scaffold; heavier than mini | + +Optional / harder to headless: **Cline** (https://github.com/cline/cline, VS-Code-bound), +**Codex CLI** (https://github.com/openai/codex, supports custom provider base_url). + +### Tier 2 — build-your-own frameworks (need a per-lib GTA-builder scaffold) + +| Framework | Repo | gx10 wiring | +|---|---|---| +| **smolagents** (HF) — `CodeAgent` | https://github.com/huggingface/smolagents | `OpenAIServerModel(api_base=...)` | +| **Pydantic-AI** | https://github.com/pydantic/pydantic-ai | `OpenAIProvider(base_url=...)` | +| **LangGraph** | https://github.com/langchain-ai/langgraph | `ChatOpenAI(base_url=...)` | +| **AutoGen / AG2** | https://github.com/microsoft/autogen · https://github.com/ag2ai/ag2 | OpenAI-compatible `config_list` | +| **CrewAI** | https://github.com/crewAIInc/crewAI | LiteLLM / `base_url` | +| **Agno** | https://github.com/agno-agi/agno | OpenAI-compatible model class | + +**Recommended starting set:** harness + OpenHands + Aider + OpenCode + Goose + +mini-SWE-agent (Tier 1), plus smolagents `CodeAgent` as the one Tier-2 control. Six to +seven entries keeps a full N-run matrix tractable; add more once the rig is proven. + +--- + +## 2. The benchmark product — GTA-style game + +A fixed spec file (`spec/gta-spec.md`) is the **only** task input, identical for every +framework. It defines a top-down 2D GTA-style game with graded milestones so partial +success is measurable, not pass/fail: + +1. **M1 — window + loop**: game window opens, runs headless, exits cleanly. +2. **M2 — player car**: a controllable car renders; arrow/WASD move + rotate. +3. **M3 — world**: top-down tiled map / roads larger than viewport; camera follows car. +4. **M4 — physics**: acceleration, steering, friction (not instant teleport). +5. **M5 — NPC traffic**: ≥1 AI-driven vehicle moving on the map. +6. **M6 — collision**: car vs world/NPC collision detected and resolved. +7. **M7 — objective**: a mission/score loop (reach waypoint, pickup, or wanted level) + HUD. + +Pin the stack in the spec to remove a free variable (recommend **Python + pygame** — +trivially headless-testable via `SDL_VIDEODRIVER=dummy`, no GPU/browser needed). The +spec is git-tracked; its commit SHA goes in every result row so a spec change is never +silently mixed into a results series. + +--- + +## 3. gx10 as the shared remote model + +One model endpoint for all frameworks isolates *framework* as the only variable. + +- gx10 serves the model via **vLLM** exposing OpenAI `/v1/chat/completions` over Tailscale. +- Every adapter gets the same `base_url`, `model` id, and a dummy API key. +- **Pin and record** in the run manifest: model id + quant, `max_model_len`, + `temperature` (0 for max determinism), `top_p`, `seed` (if the vLLM build honors it), + and a hash of the server launch args. +- **Serialize runs** (or hard rate-limit) so concurrent agents don't contend on gx10 + and skew wall-clock / latency numbers. +- Health-gate before each run: probe `/v1/models`; abort the batch if the served model + id ≠ the pinned id (catches a silently swapped endpoint). + +--- + +## 4. Harness architecture + +Mirror this repo's load-bearing patterns: an **adapter boundary** per framework, an +**append-first attributed store** for results, and **config-as-data** for metrics. + +``` +agent-bench/ + spec/gta-spec.md # the one task input (git SHA recorded per run) + bench.yaml # frameworks, runs-per-framework, gx10 endpoint, active metrics + metrics.yaml # ordered list of scorer plugins to run (the adjustable surface) + adapters/ # one thin wrapper per framework — the only framework-specific code + base.py # Adapter protocol: prepare(workspace) -> invoke(spec) -> Result + openhands.py aider.py opencode.py goose.py mini_swe.py smolagents_codeagent.py ... + scorers/ # pluggable metric collectors (drop-in, listed in metrics.yaml) + builds.py runs_headless.py feature_checklist.py cost.py process.py quality_judge.py + runner.py # orchestrates the matrix; framework-agnostic + store.py # SQLite + JSONL, append-first, attributed, timestamped + results/ # per-run artifacts: transcript, diff, repo tarball, scores.json + report.py # aggregate -> markdown/CSV tables + per-metric distributions +``` + +**Adapter protocol** (the framework boundary — keep framework SDKs *only* in here): + +```python +class Adapter(Protocol): + name: str + version: str # pinned, recorded in manifest + def prepare(self, workspace: Path) -> None: ... # fresh, isolated build dir + def invoke(self, spec: str, gx10: Endpoint) -> RunArtifacts: ... + # returns transcript, final diff, token/turn counters, exit status +``` + +**Run loop** (per `(framework, run_idx)`): + +1. Fresh isolated workspace — a git worktree or clean temp dir, so runs never cross-contaminate. +2. Health-gate gx10; stamp the manifest (framework version, model id, spec SHA, params). +3. `adapter.invoke(spec, gx10)` with a wall-clock timeout and a turn cap. +4. Capture artifacts: full stdout/stderr transcript, final `git diff`, repo tarball, + raw token/turn counters from the framework or proxied off gx10. +5. Run every scorer listed in `metrics.yaml` against the artifacts → `scores.json`. +6. Append one row to the store; write artifacts under `results///`. + +--- + +## 5. Repeatable · recordable · adjustable + +**Repeatable** +- Pin framework versions in a lockfile; record each in the manifest. Containerize Tier-1 + agents (most ship a Docker image) so host drift can't leak in. +- Pin gx10 model/params/seed; `temperature=0`. Accept that agent loops stay + non-deterministic even at temp 0 → run **N ≥ 5** per framework and report + **distributions** (success rate, median, IQR), never a single number. +- One command: `bench run --runs 5` replays the whole matrix from `bench.yaml`. +- Fresh workspace per run; spec SHA pinned per row → a spec edit starts a new series, never blends. + +**Recordable** +- Every run persists: transcript, diff, repo tarball, token/turn counters, scores, manifest. +- Append-first store (SQLite + JSONL) keyed by + `(framework, version, run_id, started_at, spec_sha, model_id)`. Nothing is overwritten — + re-runs append, so trends over time are queryable. +- `report.py` regenerates tables/plots from the store at any time; raw artifacts allow + full offline replay and dispute resolution. + +**Adjustable** +- `metrics.yaml` is the dial: an ordered list of scorer plugin names (+ optional weights). + The runner loads and runs *only* what's listed. +- Add a metric = drop a `scorers/.py` implementing the `Scorer` protocol + (`score(artifacts) -> dict[str, float|bool]`) and add its name to `metrics.yaml`. + No runner edit, no schema migration — new keys just appear in `scores.json`. +- Remove/disable a metric = delete its line. Old rows keep their historical keys; the + store is schemaless JSON per row for scores. + +```yaml +# metrics.yaml — illustrative; swap freely (the requester sets the real set) +metrics: + - builds # does the generated project import/compile? + - runs_headless # SDL dummy driver: boots and exits without crash? + - feature_checklist # M1..M7 milestones probed statically + at runtime + - cost # prompt/completion tokens, wall-clock, gx10 GPU-seconds + - process # agent turns, tool calls, files touched, diff LOC, retries, interventions + - quality_judge # optional LLM-judge (0-5) on code quality + playability +``` + +--- + +## 6. Build order + +1. **Skeleton + store + one adapter** (Aider — simplest to script) + 2 cheap scorers + (`builds`, `runs_headless`). Prove one row end-to-end against gx10. +2. **Write `spec/gta-spec.md`** (M1–M7) and the `feature_checklist` scorer that probes them. +3. **Add Tier-1 adapters**: OpenHands, OpenCode, Goose, mini-SWE-agent. +4. **Add `cost` + `process` scorers** (parse counters / proxy gx10 token usage). +5. **N-run matrix + `report.py`** distributions; lock the manifest/repeatability story. +6. **Tier-2 scaffold** (smolagents `CodeAgent`) + `quality_judge`; expand candidates as needed. + +## 7. Known risks / caveats + +- **Tier-2 measures the scaffold, not just the library** — report it in its own section; don't rank it head-to-head with turnkey agents. +- **Non-determinism** survives `temperature=0`; only distributions over N runs are trustworthy. +- **Sandboxing**: agents run/install arbitrary generated code — isolate every run (container or throwaway worktree), never the host. +- **Cline / IDE-bound agents** resist headless automation — keep optional until a scripted entrypoint exists. +- **gx10 contention** skews latency — serialize runs or the wall-clock metric is noise. + +--- + +### Sources + +- [Best Open Source Coding Agents 2026 — Open Source AI Review](https://www.opensourceaireview.com/blog/best-open-source-coding-agents-in-2026-reviewed-ranked) +- [OpenHands vs SWE-agent (2026) — CodeSOTA](https://www.codesota.com/agentic/openhands-vs-swe-agent) +- [OpenHands evaluation harness](https://github.com/OpenHands/benchmarks) +- [OpenCode docs — config / providers](https://opencode.ai/docs/config/) +- [Goose — Configure LLM Provider](https://goose-docs.ai/docs/getting-started/providers/) · [block/goose vLLM provider](https://github.com/block/goose) +- [Pydantic-AI — OpenAI-compatible models](https://ai.pydantic.dev/models/openai/) +- [Open-source agent framework comparison — Langfuse](https://langfuse.com/blog/2025-03-19-ai-agent-comparison) +- [Python AI agent library comparison 2026](https://jangwook.net/en/blog/en/python-ai-agent-library-comparison-2026/) From 98f83e116392b381b6f12c05b86e20f953249812 Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Tue, 2 Jun 2026 22:00:33 -0700 Subject: [PATCH 02/11] agent-bench: scaffold runner + aider adapter + three scorers End-to-end rig wired against the plan: framework-agnostic matrix runner, append-first SQLite+JSONL results store, metrics-as-config (metrics.yaml -> scorer registry), gx10 /v1/models health-gate with pinned-model check. - adapters/: Adapter protocol + aider adapter (drives `aider` non-interactively at gx10 via OPENAI_API_BASE; snapshots git diff as the artifact) - scorers/: builds (py_compile), runs_headless (SDL dummy boot window), feature_checklist (static M1..M7 probe) - spec/gta-spec.md: the one task input, 7 graded milestones, pygame stack - bench.yaml pinned to gx10 model Qwen/Qwen3-Coder-30B-A3B-Instruct - store.py append-first, schemaless per-run scores blob; report.py discovers metric keys from rows Smoke-tested: scorers -> store -> report path runs; ruff clean. Remaining adapters (OpenHands/OpenCode/Goose/mini-SWE) and cost/process scorers are TODO. Co-Authored-By: Claude Opus 4.8 (1M context) --- agent-bench/.gitignore | 3 + agent-bench/README.md | 43 ++++++ agent-bench/adapters/__init__.py | 1 + agent-bench/adapters/aider.py | 97 +++++++++++++ agent-bench/adapters/base.py | 52 +++++++ agent-bench/bench.yaml | 23 +++ agent-bench/metrics.yaml | 10 ++ agent-bench/report.py | 72 ++++++++++ agent-bench/runner.py | 172 +++++++++++++++++++++++ agent-bench/scorers/__init__.py | 2 + agent-bench/scorers/base.py | 20 +++ agent-bench/scorers/builds.py | 28 ++++ agent-bench/scorers/feature_checklist.py | 45 ++++++ agent-bench/scorers/runs_headless.py | 68 +++++++++ agent-bench/spec/gta-spec.md | 33 +++++ agent-bench/store.py | 95 +++++++++++++ 16 files changed, 764 insertions(+) create mode 100644 agent-bench/.gitignore create mode 100644 agent-bench/README.md create mode 100644 agent-bench/adapters/__init__.py create mode 100644 agent-bench/adapters/aider.py create mode 100644 agent-bench/adapters/base.py create mode 100644 agent-bench/bench.yaml create mode 100644 agent-bench/metrics.yaml create mode 100644 agent-bench/report.py create mode 100644 agent-bench/runner.py create mode 100644 agent-bench/scorers/__init__.py create mode 100644 agent-bench/scorers/base.py create mode 100644 agent-bench/scorers/builds.py create mode 100644 agent-bench/scorers/feature_checklist.py create mode 100644 agent-bench/scorers/runs_headless.py create mode 100644 agent-bench/spec/gta-spec.md create mode 100644 agent-bench/store.py diff --git a/agent-bench/.gitignore b/agent-bench/.gitignore new file mode 100644 index 0000000..7596101 --- /dev/null +++ b/agent-bench/.gitignore @@ -0,0 +1,3 @@ +# Per-run artifacts + results store are generated, not tracked. +results/ +__pycache__/ diff --git a/agent-bench/README.md b/agent-bench/README.md new file mode 100644 index 0000000..781f22a --- /dev/null +++ b/agent-bench/README.md @@ -0,0 +1,43 @@ +# agent-bench + +Benchmark rig: give each agent framework the same autonomous build task — a +GTA-style game — against one shared gx10 vLLM endpoint, then score the result. + +See `../docs/agent-framework-bench.md` for the full plan. + +## Layout + +``` +spec/gta-spec.md the one task input (M1..M7 milestones) +bench.yaml frameworks, runs-per-framework, gx10 endpoint, paths +metrics.yaml ordered scorer plugins to run (the adjustable surface) +adapters/ one thin wrapper per framework (framework SDKs live ONLY here) +scorers/ pluggable metric collectors (drop-in, listed in metrics.yaml) +runner.py framework-agnostic matrix runner +store.py append-first SQLite + JSONL results store +report.py aggregate -> tables + per-metric distributions +results/ per-run artifacts (transcript, diff, scores.json) +``` + +## Run + +```bash +cd agent-bench +python runner.py --runs 5 # full matrix from bench.yaml +python runner.py --framework aider --runs 1 +python report.py # tables from the store +``` + +## Extend + +- **New framework**: add `adapters/.py` implementing the `Adapter` protocol; + register it in `ADAPTERS` (runner.py). +- **New metric**: add `scorers/.py` implementing the `Scorer` protocol; + register it in `SCORERS` (runner.py) and list its name in `metrics.yaml`. + No runner edit beyond the registry line; scores are schemaless JSON per row. + +## Status + +Scaffold. Wired end-to-end with the **aider** adapter and three scorers +(`builds`, `runs_headless`, `feature_checklist`). Remaining adapters (OpenHands, +OpenCode, Goose, mini-SWE-agent) and cost/process scorers are TODO — see the plan. diff --git a/agent-bench/adapters/__init__.py b/agent-bench/adapters/__init__.py new file mode 100644 index 0000000..54596f5 --- /dev/null +++ b/agent-bench/adapters/__init__.py @@ -0,0 +1 @@ +"""Framework adapters. Framework SDKs / CLIs are invoked only from here.""" diff --git a/agent-bench/adapters/aider.py b/agent-bench/adapters/aider.py new file mode 100644 index 0000000..0de0830 --- /dev/null +++ b/agent-bench/adapters/aider.py @@ -0,0 +1,97 @@ +"""Aider adapter — drives the `aider` CLI non-interactively against gx10. + +https://github.com/Aider-AI/aider +""" + +from __future__ import annotations + +import subprocess +import time +from pathlib import Path + +from .base import Endpoint, RunArtifacts + +# Pin and record. Bump deliberately; the value lands in every result row's manifest. +AIDER_VERSION = "0.86.1" + + +class AiderAdapter: + name = "aider" + version = AIDER_VERSION + + def prepare(self, workspace: Path) -> None: + workspace.mkdir(parents=True, exist_ok=True) + _git(workspace, "init", "-q") + _git(workspace, "commit", "-q", "--allow-empty", "-m", "bench: empty baseline") + + def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> RunArtifacts: + env_extra = { + "OPENAI_API_BASE": gx10.base_url, + "OPENAI_API_KEY": gx10.api_key, + } + cmd = [ + "aider", + "--model", + f"openai/{gx10.model}", + "--yes-always", # non-interactive: accept edits/commits + "--no-auto-commits", # we snapshot the diff ourselves + "--no-gitignore", + "--no-check-update", + "--no-show-model-warnings", + "--map-tokens", + "1024", + "--message", + spec, # the whole task as a single instruction + ] + started = time.perf_counter() + proc = _run(cmd, cwd=workspace, timeout_s=timeout_s, env_extra=env_extra) + duration = time.perf_counter() - started + + _git(workspace, "add", "-A") + diff = _capture(["git", "diff", "--cached"], cwd=workspace) + + transcript = f"$ {' '.join(cmd)}\n\n[stdout]\n{proc.stdout}\n\n[stderr]\n{proc.stderr}" + return RunArtifacts( + exit_ok=proc.returncode == 0, + transcript=transcript, + diff=diff, + duration_s=duration, + extra={"returncode": proc.returncode, "timed_out": proc.returncode == _TIMEOUT_RC}, + ) + + +_TIMEOUT_RC = -999 + + +def _git(cwd: Path, *args: str) -> None: + _run(["git", *args], cwd=cwd, timeout_s=60, env_extra={}) + + +def _run( + cmd: list[str], *, cwd: Path, timeout_s: int, env_extra: dict[str, str] +) -> subprocess.CompletedProcess[str]: + import os + + env = {**os.environ, **env_extra} + try: + return subprocess.run( # noqa: S603 — fixed argv, no shell; benchmark runner by design + cmd, + cwd=cwd, + env=env, + capture_output=True, + text=True, + timeout=timeout_s, + check=False, + ) + except subprocess.TimeoutExpired as exc: + out = _as_text(exc.stdout or "") + err = _as_text(exc.stderr or "") + f"\n[bench] timed out after {timeout_s}s" + return subprocess.CompletedProcess(cmd, _TIMEOUT_RC, out, err) + + +def _capture(cmd: list[str], *, cwd: Path) -> str: + return _run(cmd, cwd=cwd, timeout_s=60, env_extra={}).stdout + + +def _as_text(value: str | bytes) -> str: + return value.decode(errors="replace") if isinstance(value, bytes) else value diff --git a/agent-bench/adapters/base.py b/agent-bench/adapters/base.py new file mode 100644 index 0000000..7df51d6 --- /dev/null +++ b/agent-bench/adapters/base.py @@ -0,0 +1,52 @@ +"""Adapter boundary. + +Mirrors the harness's model-adapter invariant: framework-specific code (CLIs, +SDKs) lives only inside adapter modules. The runner speaks `Adapter` + +`RunArtifacts` and nothing else. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Protocol, runtime_checkable + + +@dataclass(frozen=True) +class Endpoint: + """The shared gx10 vLLM endpoint handed to every framework.""" + + base_url: str + model: str + api_key: str = "gx10" + temperature: float = 0.0 + + +@dataclass +class RunArtifacts: + """Everything one run produces, captured for scoring and replay.""" + + exit_ok: bool + transcript: str + diff: str + duration_s: float + tokens_prompt: int | None = None + tokens_completion: int | None = None + turns: int | None = None + extra: dict[str, object] = field(default_factory=dict) + + +@runtime_checkable +class Adapter(Protocol): + """One framework under test.""" + + name: str + version: str + + def prepare(self, workspace: Path) -> None: + """Set up a fresh, isolated build directory (e.g. git init).""" + ... + + def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> RunArtifacts: + """Drive the framework to build the spec; return captured artifacts.""" + ... diff --git a/agent-bench/bench.yaml b/agent-bench/bench.yaml new file mode 100644 index 0000000..ce8c29b --- /dev/null +++ b/agent-bench/bench.yaml @@ -0,0 +1,23 @@ +# agent-bench run config. One gx10 endpoint for every framework so the only +# variable under test is the framework itself. +gx10: + base_url: "http://gx10:8000/v1" # vLLM OpenAI-compatible endpoint over Tailscale + model: "Qwen/Qwen3-Coder-30B-A3B-Instruct" # pinned; runner aborts if /v1/models disagrees + api_key: "gx10" # dummy; vLLM ignores but OpenAI clients require one + temperature: 0.0 # max determinism (loops still vary -> N runs) + +runs_per_framework: 5 +timeout_s: 1800 # per-run wall-clock cap +turn_cap: 60 # advisory; adapters that support it pass it through + +spec: "spec/gta-spec.md" +metrics_file: "metrics.yaml" +results_dir: "results" +store_db: "results/bench.sqlite3" + +frameworks: + - aider + # - openhands # TODO + # - opencode # TODO + # - goose # TODO + # - mini_swe # TODO diff --git a/agent-bench/metrics.yaml b/agent-bench/metrics.yaml new file mode 100644 index 0000000..5b9ce6b --- /dev/null +++ b/agent-bench/metrics.yaml @@ -0,0 +1,10 @@ +# The adjustable surface. Runner runs ONLY the scorers listed here, in order. +# Add a metric: drop scorers/.py, register it in runner.SCORERS, add its +# name below. Remove a metric: delete its line. Historical rows keep their keys. +metrics: + - builds # generated project compiles (py_compile over all .py) + - runs_headless # boots under SDL dummy driver without an immediate crash + - feature_checklist # static probe of milestones M1..M7 + # - cost # TODO: prompt/completion tokens, wall-clock, gx10 GPU-seconds + # - process # TODO: turns, tool calls, files touched, diff LOC, retries + # - quality_judge # TODO: LLM-judge (0-5) code quality + playability diff --git a/agent-bench/report.py b/agent-bench/report.py new file mode 100644 index 0000000..cf80210 --- /dev/null +++ b/agent-bench/report.py @@ -0,0 +1,72 @@ +"""Aggregate the results store into per-framework distributions. + +Reports survive any metric set: it discovers metric keys from the rows, so adding +or removing a scorer changes the report with no code change here. +""" + +from __future__ import annotations + +import argparse +import statistics +from pathlib import Path + +import yaml +from store import ResultsStore + +ROOT = Path(__file__).resolve().parent + + +def main() -> int: + ap = argparse.ArgumentParser(description="agent-bench report") + ap.add_argument("--config", default=str(ROOT / "bench.yaml")) + args = ap.parse_args() + cfg = yaml.safe_load(Path(args.config).read_text()) + store = ResultsStore(ROOT / cfg["store_db"]) + rows = store.all_rows() + if not rows: + print("no runs recorded yet") + return 0 + + by_fw: dict[str, list[dict[str, object]]] = {} + for r in rows: + by_fw.setdefault(str(r["framework"]), []).append(r) + + for fw, fw_rows in sorted(by_fw.items()): + n = len(fw_rows) + durations = [float(r["duration_s"]) for r in fw_rows] + exit_ok = sum(int(bool(r["exit_ok"])) for r in fw_rows) + print(f"\n=== {fw} (n={n}, version={fw_rows[0]['version']}) ===") + print(f" exit_ok : {exit_ok}/{n}") + print( + f" duration_s : median {statistics.median(durations):.1f} " + f"min {min(durations):.1f} max {max(durations):.1f}" + ) + _report_metrics(fw_rows) + return 0 + + +def _report_metrics(fw_rows: list[dict[str, object]]) -> None: + keys: list[str] = [] + for r in fw_rows: + for k in dict(r["scores"]): # type: ignore[call-overload] + if k not in keys: + keys.append(k) + n = len(fw_rows) + for key in keys: + vals = [dict(r["scores"]).get(key) for r in fw_rows] # type: ignore[call-overload] + present = [v for v in vals if v is not None] + if all(isinstance(v, bool) for v in present): + true_n = sum(1 for v in present if v) + print(f" {key:<20}: {true_n}/{n} true") + elif all(isinstance(v, (int, float)) for v in present): + nums = [float(v) for v in present] # type: ignore[arg-type] + print( + f" {key:<20}: median {statistics.median(nums):.2f} " + f"min {min(nums):.2f} max {max(nums):.2f}" + ) + else: + print(f" {key:<20}: {present[:3]}{' ...' if len(present) > 3 else ''}") + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/agent-bench/runner.py b/agent-bench/runner.py new file mode 100644 index 0000000..d826c94 --- /dev/null +++ b/agent-bench/runner.py @@ -0,0 +1,172 @@ +"""Framework-agnostic matrix runner. + +For each (framework, run_idx): fresh isolated workspace -> health-gate gx10 -> +adapter.invoke -> run the metrics.yaml scorers -> append one attributed row + +persist artifacts. Knows nothing framework-specific; that lives in adapters/. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import shutil +import sys +import urllib.error +import urllib.request +from pathlib import Path + +import yaml +from adapters.aider import AiderAdapter +from adapters.base import Adapter, Endpoint, RunArtifacts +from scorers.base import Scorer, Scores +from scorers.builds import BuildsScorer +from scorers.feature_checklist import FeatureChecklistScorer +from scorers.runs_headless import RunsHeadlessScorer +from store import ResultsStore, RunRecord + +ROOT = Path(__file__).resolve().parent + +# Registries. Add a framework / metric by adding a line here. +ADAPTERS: dict[str, type[Adapter]] = { + "aider": AiderAdapter, +} +SCORERS: dict[str, type[Scorer]] = { + "builds": BuildsScorer, + "runs_headless": RunsHeadlessScorer, + "feature_checklist": FeatureChecklistScorer, +} + + +def main() -> int: + ap = argparse.ArgumentParser(description="agent-bench matrix runner") + ap.add_argument("--config", default=str(ROOT / "bench.yaml")) + ap.add_argument("--framework", help="run only this framework (default: all in config)") + ap.add_argument("--runs", type=int, help="override runs_per_framework") + ap.add_argument("--no-health-gate", action="store_true", help="skip gx10 /v1/models check") + args = ap.parse_args() + + cfg = yaml.safe_load(Path(args.config).read_text()) + gx10 = Endpoint( + base_url=cfg["gx10"]["base_url"], + model=cfg["gx10"]["model"], + api_key=cfg["gx10"].get("api_key", "gx10"), + temperature=float(cfg["gx10"].get("temperature", 0.0)), + ) + runs = args.runs or int(cfg["runs_per_framework"]) + timeout_s = int(cfg["timeout_s"]) + spec_path = ROOT / cfg["spec"] + spec = spec_path.read_text() + spec_sha = hashlib.sha256(spec.encode()).hexdigest()[:12] + metrics = yaml.safe_load((ROOT / cfg["metrics_file"]).read_text())["metrics"] + store = ResultsStore(ROOT / cfg["store_db"]) + results_dir = ROOT / cfg["results_dir"] + + frameworks = [args.framework] if args.framework else list(cfg["frameworks"]) + _validate(frameworks, metrics) + + if not args.no_health_gate and not _health_ok(gx10): + print(f"[bench] ABORT: gx10 health-gate failed for {gx10.base_url} / {gx10.model}") + return 2 + + scorers = [SCORERS[m]() for m in metrics] + for fw in frameworks: + adapter = ADAPTERS[fw]() + for run_idx in range(runs): + print(f"[bench] {fw} run {run_idx + 1}/{runs} ...") + _do_run( + adapter=adapter, + gx10=gx10, + spec=spec, + spec_sha=spec_sha, + timeout_s=timeout_s, + run_idx=run_idx, + scorers=scorers, + results_dir=results_dir, + store=store, + ) + print(f"[bench] done. store: {ROOT / cfg['store_db']}") + return 0 + + +def _do_run( + *, + adapter: Adapter, + gx10: Endpoint, + spec: str, + spec_sha: str, + timeout_s: int, + run_idx: int, + scorers: list[Scorer], + results_dir: Path, + store: ResultsStore, +) -> None: + run_dir = results_dir / adapter.name / str(run_idx) + workspace = run_dir / "workspace" + if workspace.exists(): + shutil.rmtree(workspace) # fresh, isolated — never cross-contaminate runs + workspace.mkdir(parents=True) + + adapter.prepare(workspace) + artifacts = adapter.invoke(spec, gx10, workspace, timeout_s) + + scores: Scores = {} + for scorer in scorers: + scores.update(scorer.score(workspace, artifacts)) + + _persist_artifacts(run_dir, artifacts, scores) + store.append( + RunRecord( + framework=adapter.name, + version=adapter.version, + run_idx=run_idx, + spec_sha=spec_sha, + model_id=gx10.model, + exit_ok=artifacts.exit_ok, + duration_s=artifacts.duration_s, + scores=dict(scores), + manifest={ + "base_url": gx10.base_url, + "temperature": gx10.temperature, + "timeout_s": timeout_s, + "tokens_prompt": artifacts.tokens_prompt, + "tokens_completion": artifacts.tokens_completion, + "turns": artifacts.turns, + }, + ) + ) + + +def _persist_artifacts(run_dir: Path, artifacts: RunArtifacts, scores: Scores) -> None: + (run_dir / "transcript.txt").write_text(artifacts.transcript) + (run_dir / "result.diff").write_text(artifacts.diff) + (run_dir / "scores.json").write_text(json.dumps(scores, indent=2)) + + +def _validate(frameworks: list[str], metrics: list[str]) -> None: + unknown_fw = [f for f in frameworks if f not in ADAPTERS] + unknown_m = [m for m in metrics if m not in SCORERS] + if unknown_fw: + raise SystemExit(f"unknown framework(s): {unknown_fw}; known: {list(ADAPTERS)}") + if unknown_m: + raise SystemExit(f"unknown metric(s): {unknown_m}; known: {list(SCORERS)}") + + +def _health_ok(gx10: Endpoint) -> bool: + """Probe /v1/models; require the served model id to match the pinned one.""" + url = gx10.base_url.rstrip("/") + "/models" + req = urllib.request.Request(url, headers={"Authorization": f"Bearer {gx10.api_key}"}) # noqa: S310 — http(s) to the configured gx10 endpoint + try: + with urllib.request.urlopen(req, timeout=10) as resp: # noqa: S310 + served = {m.get("id") for m in json.load(resp).get("data", [])} + except (urllib.error.URLError, TimeoutError, ValueError) as exc: + print(f"[bench] health probe error: {exc}") + return False + if gx10.model not in served: + print(f"[bench] served models {served} do not include pinned {gx10.model!r}") + return False + return True + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/agent-bench/scorers/__init__.py b/agent-bench/scorers/__init__.py new file mode 100644 index 0000000..a2e8abf --- /dev/null +++ b/agent-bench/scorers/__init__.py @@ -0,0 +1,2 @@ +"""Pluggable metric collectors. Each scorer reads a finished run's workspace + +artifacts and returns a flat dict of metric -> value.""" diff --git a/agent-bench/scorers/base.py b/agent-bench/scorers/base.py new file mode 100644 index 0000000..51634bb --- /dev/null +++ b/agent-bench/scorers/base.py @@ -0,0 +1,20 @@ +"""Scorer protocol. Add a metric by dropping a module here and registering it.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Protocol, runtime_checkable + +from adapters.base import RunArtifacts + +ScoreValue = float | int | bool | str +Scores = dict[str, ScoreValue] + + +@runtime_checkable +class Scorer(Protocol): + name: str + + def score(self, workspace: Path, artifacts: RunArtifacts) -> Scores: + """Return metric keys -> values for this run. Keys are namespaced by caller.""" + ... diff --git a/agent-bench/scorers/builds.py b/agent-bench/scorers/builds.py new file mode 100644 index 0000000..5dc5d2c --- /dev/null +++ b/agent-bench/scorers/builds.py @@ -0,0 +1,28 @@ +"""`builds` — does the generated project compile? py_compile over every .py.""" + +from __future__ import annotations + +import py_compile +from pathlib import Path + +from adapters.base import RunArtifacts + +from scorers.base import Scores + + +class BuildsScorer: + name = "builds" + + def score(self, workspace: Path, artifacts: RunArtifacts) -> Scores: + py_files = [p for p in workspace.rglob("*.py") if ".git" not in p.parts] + failed: list[str] = [] + for path in py_files: + try: + py_compile.compile(str(path), doraise=True) + except py_compile.PyCompileError: + failed.append(str(path.relative_to(workspace))) + return { + "builds": len(py_files) > 0 and not failed, + "py_files": len(py_files), + "compile_failures": len(failed), + } diff --git a/agent-bench/scorers/feature_checklist.py b/agent-bench/scorers/feature_checklist.py new file mode 100644 index 0000000..81f11c0 --- /dev/null +++ b/agent-bench/scorers/feature_checklist.py @@ -0,0 +1,45 @@ +"""`feature_checklist` — static probe of milestones M1..M7. + +Cheap, honest heuristic: scan generated source for signals that each milestone +was attempted. This is a *static* approximation — it detects presence of the +relevant APIs/patterns, not that they work. Runtime probes (driving the game and +asserting behavior) are a TODO; see the plan's quality section. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +from adapters.base import RunArtifacts + +from scorers.base import Scores + +# milestone -> regex signals (any match -> attempted). Deliberately conservative. +MILESTONES: dict[str, list[str]] = { + "m1_window_loop": [r"set_mode", r"pygame\.event\.get", r"while .*:"], + "m2_player_car": [r"K_(UP|DOWN|LEFT|RIGHT|w|a|s|d)\b", r"get_pressed"], + "m3_world": [r"camera|offset|scroll", r"blit"], + "m4_physics": [r"accel|velocity|friction|momentum"], + "m5_npc": [r"npc|traffic|enemy|ai_", r"class .*(Car|Vehicle|NPC)"], + "m6_collision": [r"colliderect|collide|Rect\(", r"collision"], + "m7_objective": [r"score|mission|waypoint|wanted|objective", r"render.*font|HUD|hud"], +} + + +class FeatureChecklistScorer: + name = "feature_checklist" + + def score(self, workspace: Path, artifacts: RunArtifacts) -> Scores: + src = "\n".join( + p.read_text(errors="replace") for p in workspace.rglob("*.py") if ".git" not in p.parts + ) + scores: Scores = {} + reached = 0 + for milestone, patterns in MILESTONES.items(): + hit = all(re.search(pat, src, re.IGNORECASE) for pat in patterns) + scores[milestone] = hit + reached += int(hit) + scores["milestones_reached"] = reached + scores["milestones_total"] = len(MILESTONES) + return scores diff --git a/agent-bench/scorers/runs_headless.py b/agent-bench/scorers/runs_headless.py new file mode 100644 index 0000000..4d744e3 --- /dev/null +++ b/agent-bench/scorers/runs_headless.py @@ -0,0 +1,68 @@ +"""`runs_headless` — boot main.py under the SDL dummy driver. + +A pygame game loops forever, so "ran" means: started and survived a short window +without crashing. We launch it, wait `BOOT_WINDOW_S`, and treat *still-running* +(killed by us) or a clean exit as success; an early non-zero exit with a traceback +is failure. +""" + +from __future__ import annotations + +import os +import signal +import subprocess +import time +from pathlib import Path + +from adapters.base import RunArtifacts + +from scorers.base import Scores + +BOOT_WINDOW_S = 8.0 +ENTRYPOINTS = ("main.py", "game.py", "run.py") + + +class RunsHeadlessScorer: + name = "runs_headless" + + def score(self, workspace: Path, artifacts: RunArtifacts) -> Scores: + entry = next((workspace / e for e in ENTRYPOINTS if (workspace / e).exists()), None) + if entry is None: + return {"runs_headless": False, "reason": "no entrypoint"} + + env = {**os.environ, "SDL_VIDEODRIVER": "dummy", "SDL_AUDIODRIVER": "dummy"} + proc = subprocess.Popen( # noqa: S603 — fixed argv, no shell; sandboxed bench run + ["python", entry.name], # noqa: S607 — `python` from the bench venv on PATH + cwd=workspace, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + start_new_session=True, + ) + deadline = time.perf_counter() + BOOT_WINDOW_S + while time.perf_counter() < deadline: + if proc.poll() is not None: + break + time.sleep(0.1) + + survived = proc.poll() is None + if survived: + _kill_group(proc) + return {"runs_headless": True, "reason": "survived boot window"} + + out = proc.stdout.read() if proc.stdout else "" + crashed = proc.returncode != 0 or "Traceback" in out + return { + "runs_headless": not crashed, + "reason": "clean early exit" if not crashed else "crash", + "exit_code": proc.returncode, + } + + +def _kill_group(proc: subprocess.Popen[str]) -> None: + try: + os.killpg(os.getpgid(proc.pid), signal.SIGKILL) + except (ProcessLookupError, PermissionError): + proc.kill() + proc.wait(timeout=10) diff --git a/agent-bench/spec/gta-spec.md b/agent-bench/spec/gta-spec.md new file mode 100644 index 0000000..02ea3b0 --- /dev/null +++ b/agent-bench/spec/gta-spec.md @@ -0,0 +1,33 @@ +# Build task: top-down GTA-style game + +Build a small top-down 2D Grand-Theft-Auto-style driving game in **Python using +pygame**, in this directory. Entry point must be `main.py` runnable with +`python main.py`. Keep all game code importable without launching a window +(guard the loop under `if __name__ == "__main__":`). + +The game must run **headless-testable**: it must import and initialize cleanly when +the environment sets `SDL_VIDEODRIVER=dummy` (no real display). + +Implement these milestones in order. Each is graded independently, so a partial +result still counts — do as many as you can, correctly. + +- **M1 — window + loop**: open a game window, run a main loop, exit cleanly on quit. +- **M2 — player car**: a car sprite renders and is controllable — WASD / arrow keys + accelerate, brake, and steer (rotate) the car. +- **M3 — world**: a top-down tiled map / road network larger than the viewport; the + camera scrolls to follow the player car. +- **M4 — physics**: acceleration, steering, and friction — momentum, not instant + teleport-style movement. +- **M5 — NPC traffic**: at least one AI-driven vehicle that moves around the map on + its own. +- **M6 — collision**: collision detection and resolution between the player car and + the world and/or NPC vehicles. +- **M7 — objective**: a mission / score loop (reach a waypoint, pick up a target, or a + wanted-level mechanic) plus an on-screen HUD showing score or state. + +Constraints: + +- Pure Python + `pygame` only. No network calls, no asset downloads — generate any + needed shapes/sprites in code. +- Single self-contained project rooted here. `main.py` is the entry point. +- Favor correctness and runnability over scope: a clean M1–M4 beats a crashing M1–M7. diff --git a/agent-bench/store.py b/agent-bench/store.py new file mode 100644 index 0000000..31003cc --- /dev/null +++ b/agent-bench/store.py @@ -0,0 +1,95 @@ +"""Append-first, attributed, timestamped results store — SQLite + JSONL mirror. + +Same philosophy as the harness stores: nothing is overwritten. Re-runs append, +so a results series over time is queryable. Per-run scores are stored as a JSON +blob so adding/removing a metric needs no schema migration. +""" + +from __future__ import annotations + +import json +import sqlite3 +from dataclasses import asdict, dataclass, field +from datetime import UTC, datetime +from pathlib import Path + +_SCHEMA = """ +CREATE TABLE IF NOT EXISTS runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + framework TEXT NOT NULL, + version TEXT NOT NULL, + run_idx INTEGER NOT NULL, + started_at TEXT NOT NULL, + spec_sha TEXT NOT NULL, + model_id TEXT NOT NULL, + exit_ok INTEGER NOT NULL, + duration_s REAL NOT NULL, + scores_json TEXT NOT NULL, + manifest_json TEXT NOT NULL +); +""" + + +@dataclass +class RunRecord: + framework: str + version: str + run_idx: int + spec_sha: str + model_id: str + exit_ok: bool + duration_s: float + scores: dict[str, object] + manifest: dict[str, object] + started_at: str = field(default_factory=lambda: datetime.now(UTC).isoformat()) + + +class ResultsStore: + def __init__(self, db_path: Path) -> None: + self._db_path = db_path + self._jsonl = db_path.with_suffix(".jsonl") + db_path.parent.mkdir(parents=True, exist_ok=True) + with self._connect() as conn: + conn.executescript(_SCHEMA) + + def _connect(self) -> sqlite3.Connection: + conn = sqlite3.connect(self._db_path) + conn.execute("PRAGMA busy_timeout = 5000") + return conn + + def append(self, rec: RunRecord) -> int: + row = ( + rec.framework, + rec.version, + rec.run_idx, + rec.started_at, + rec.spec_sha, + rec.model_id, + int(rec.exit_ok), + rec.duration_s, + json.dumps(rec.scores), + json.dumps(rec.manifest), + ) + with self._connect() as conn: + cur = conn.execute( + "INSERT INTO runs (framework, version, run_idx, started_at, spec_sha, " + "model_id, exit_ok, duration_s, scores_json, manifest_json) " + "VALUES (?,?,?,?,?,?,?,?,?,?)", + row, + ) + run_id = int(cur.lastrowid or 0) + with self._jsonl.open("a") as fh: + fh.write(json.dumps({"id": run_id, **asdict(rec)}) + "\n") + return run_id + + def all_rows(self) -> list[dict[str, object]]: + with self._connect() as conn: + conn.row_factory = sqlite3.Row + rows = conn.execute("SELECT * FROM runs ORDER BY framework, run_idx").fetchall() + out: list[dict[str, object]] = [] + for r in rows: + d = dict(r) + d["scores"] = json.loads(d.pop("scores_json")) + d["manifest"] = json.loads(d.pop("manifest_json")) + out.append(d) + return out From 4c4523953937fa7a331db0c908cfdb8f6bf629df Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Tue, 2 Jun 2026 22:06:38 -0700 Subject: [PATCH 03/11] agent-bench: opencode + goose adapters, cost scorer, shared _proc Adds two more turnkey-agent adapters and the first cost metric. - adapters/_proc.py: extracted shared subprocess + git helpers (run, init_repo, snapshot_diff, transcript_of); aider refactored onto it (no behavior change) - adapters/opencode.py: `opencode run --model gx10/ --format json`; gx10 wired as a custom @ai-sdk/openai-compatible provider written to a per-workspace opencode.json - adapters/goose.py: `goose run --no-session --quiet -t `; gx10 base_url decomposed into OPENAI_HOST + OPENAI_BASE_PATH; keyring disabled, key from env - scorers/cost.py: tokens_{prompt,completion,total}, wall_clock_s, tokens_per_s, cost_usd (+cost_source). Tokens from artifact fields, else transcript regex fallback (JSON usage + aider k/M lines). $ priced via BENCH_PRICE_*_PER_MTOK env (default $0 for self-hosted gx10) - runner registries + bench.yaml + metrics.yaml updated Smoke-tested: registries load (3 adapters, 4 scorers), goose endpoint split and opencode provider-config shape verified; ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) --- agent-bench/adapters/_proc.py | 58 ++++++++++++++++++++ agent-bench/adapters/aider.py | 55 +++---------------- agent-bench/adapters/goose.py | 72 +++++++++++++++++++++++++ agent-bench/adapters/opencode.py | 71 +++++++++++++++++++++++++ agent-bench/bench.yaml | 4 +- agent-bench/metrics.yaml | 2 +- agent-bench/runner.py | 6 +++ agent-bench/scorers/cost.py | 91 ++++++++++++++++++++++++++++++++ 8 files changed, 307 insertions(+), 52 deletions(-) create mode 100644 agent-bench/adapters/_proc.py create mode 100644 agent-bench/adapters/goose.py create mode 100644 agent-bench/adapters/opencode.py create mode 100644 agent-bench/scorers/cost.py diff --git a/agent-bench/adapters/_proc.py b/agent-bench/adapters/_proc.py new file mode 100644 index 0000000..bca9c1c --- /dev/null +++ b/agent-bench/adapters/_proc.py @@ -0,0 +1,58 @@ +"""Subprocess + git helpers shared by every CLI-driven adapter. + +Adapters shell out to a framework's CLI under a fixed argv (no shell), capture +stdout/stderr, and snapshot the resulting git diff as the run artifact. +""" + +from __future__ import annotations + +import os +import subprocess +from pathlib import Path + +TIMEOUT_RC = -999 + + +def run( + cmd: list[str], *, cwd: Path, timeout_s: int, env_extra: dict[str, str] | None = None +) -> subprocess.CompletedProcess[str]: + """Run argv to completion; on timeout return a synthetic result with TIMEOUT_RC.""" + env = {**os.environ, **(env_extra or {})} + try: + return subprocess.run( # noqa: S603 — fixed argv, no shell; benchmark runner by design + cmd, + cwd=cwd, + env=env, + capture_output=True, + text=True, + timeout=timeout_s, + check=False, + ) + except subprocess.TimeoutExpired as exc: + out = as_text(exc.stdout or "") + err = as_text(exc.stderr or "") + f"\n[bench] timed out after {timeout_s}s" + return subprocess.CompletedProcess(cmd, TIMEOUT_RC, out, err) + + +def git(cwd: Path, *args: str) -> None: + run(["git", *args], cwd=cwd, timeout_s=60) + + +def init_repo(workspace: Path) -> None: + """Fresh git repo with an empty baseline commit, so `git diff` captures all work.""" + workspace.mkdir(parents=True, exist_ok=True) + git(workspace, "init", "-q") + git(workspace, "commit", "-q", "--allow-empty", "-m", "bench: empty baseline") + + +def snapshot_diff(workspace: Path) -> str: + git(workspace, "add", "-A") + return run(["git", "diff", "--cached"], cwd=workspace, timeout_s=60).stdout + + +def transcript_of(cmd: list[str], proc: subprocess.CompletedProcess[str]) -> str: + return f"$ {' '.join(cmd)}\n\n[stdout]\n{proc.stdout}\n\n[stderr]\n{proc.stderr}" + + +def as_text(value: str | bytes) -> str: + return value.decode(errors="replace") if isinstance(value, bytes) else value diff --git a/agent-bench/adapters/aider.py b/agent-bench/adapters/aider.py index 0de0830..c6543ec 100644 --- a/agent-bench/adapters/aider.py +++ b/agent-bench/adapters/aider.py @@ -5,10 +5,10 @@ from __future__ import annotations -import subprocess import time from pathlib import Path +from ._proc import TIMEOUT_RC, init_repo, run, snapshot_diff, transcript_of from .base import Endpoint, RunArtifacts # Pin and record. Bump deliberately; the value lands in every result row's manifest. @@ -20,9 +20,7 @@ class AiderAdapter: version = AIDER_VERSION def prepare(self, workspace: Path) -> None: - workspace.mkdir(parents=True, exist_ok=True) - _git(workspace, "init", "-q") - _git(workspace, "commit", "-q", "--allow-empty", "-m", "bench: empty baseline") + init_repo(workspace) def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> RunArtifacts: env_extra = { @@ -44,54 +42,13 @@ def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> spec, # the whole task as a single instruction ] started = time.perf_counter() - proc = _run(cmd, cwd=workspace, timeout_s=timeout_s, env_extra=env_extra) + proc = run(cmd, cwd=workspace, timeout_s=timeout_s, env_extra=env_extra) duration = time.perf_counter() - started - _git(workspace, "add", "-A") - diff = _capture(["git", "diff", "--cached"], cwd=workspace) - - transcript = f"$ {' '.join(cmd)}\n\n[stdout]\n{proc.stdout}\n\n[stderr]\n{proc.stderr}" return RunArtifacts( exit_ok=proc.returncode == 0, - transcript=transcript, - diff=diff, + transcript=transcript_of(cmd, proc), + diff=snapshot_diff(workspace), duration_s=duration, - extra={"returncode": proc.returncode, "timed_out": proc.returncode == _TIMEOUT_RC}, + extra={"returncode": proc.returncode, "timed_out": proc.returncode == TIMEOUT_RC}, ) - - -_TIMEOUT_RC = -999 - - -def _git(cwd: Path, *args: str) -> None: - _run(["git", *args], cwd=cwd, timeout_s=60, env_extra={}) - - -def _run( - cmd: list[str], *, cwd: Path, timeout_s: int, env_extra: dict[str, str] -) -> subprocess.CompletedProcess[str]: - import os - - env = {**os.environ, **env_extra} - try: - return subprocess.run( # noqa: S603 — fixed argv, no shell; benchmark runner by design - cmd, - cwd=cwd, - env=env, - capture_output=True, - text=True, - timeout=timeout_s, - check=False, - ) - except subprocess.TimeoutExpired as exc: - out = _as_text(exc.stdout or "") - err = _as_text(exc.stderr or "") + f"\n[bench] timed out after {timeout_s}s" - return subprocess.CompletedProcess(cmd, _TIMEOUT_RC, out, err) - - -def _capture(cmd: list[str], *, cwd: Path) -> str: - return _run(cmd, cwd=cwd, timeout_s=60, env_extra={}).stdout - - -def _as_text(value: str | bytes) -> str: - return value.decode(errors="replace") if isinstance(value, bytes) else value diff --git a/agent-bench/adapters/goose.py b/agent-bench/adapters/goose.py new file mode 100644 index 0000000..9c3420e --- /dev/null +++ b/agent-bench/adapters/goose.py @@ -0,0 +1,72 @@ +"""Goose adapter — drives the `goose run` CLI (Block) against gx10. + +https://github.com/block/goose · https://goose-docs.ai/docs/getting-started/providers/ + +gx10 is wired via goose's OpenAI provider pointed at a custom host. goose splits +the endpoint into OPENAI_HOST (scheme://host:port) + OPENAI_BASE_PATH (the request +path), so the bench's single `base_url` (a `/v1` root) is decomposed here. + +`goose run --no-session -t ` runs one shot and exits — no session file. +NOTE: goose's provider/keyring env names shift across releases; verify +GOOSE_PROVIDER / GOOSE_DISABLE_KEYRING against the installed goose before a real run. +""" + +from __future__ import annotations + +import time +from pathlib import Path +from urllib.parse import urlsplit + +from ._proc import TIMEOUT_RC, init_repo, run, snapshot_diff, transcript_of +from .base import Endpoint, RunArtifacts + +GOOSE_VERSION = "1.x" # pin to the installed release before a real run + + +class GooseAdapter: + name = "goose" + version = GOOSE_VERSION + + def prepare(self, workspace: Path) -> None: + init_repo(workspace) + + def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> RunArtifacts: + host, base_path = _split_endpoint(gx10.base_url) + env_extra = { + "GOOSE_PROVIDER": "openai", + "GOOSE_MODEL": gx10.model, + "GOOSE_DISABLE_KEYRING": "1", # read the key from env, not the OS keyring + "OPENAI_API_KEY": gx10.api_key, + "OPENAI_HOST": host, + "OPENAI_BASE_PATH": base_path, + } + cmd = [ + "goose", + "run", + "--no-session", # headless: no session file + "--quiet", # print only the model response + "--model", + gx10.model, + "-t", + spec, + ] + started = time.perf_counter() + proc = run(cmd, cwd=workspace, timeout_s=timeout_s, env_extra=env_extra) + duration = time.perf_counter() - started + + return RunArtifacts( + exit_ok=proc.returncode == 0, + transcript=transcript_of(cmd, proc), + diff=snapshot_diff(workspace), + duration_s=duration, + extra={"returncode": proc.returncode, "timed_out": proc.returncode == TIMEOUT_RC}, + ) + + +def _split_endpoint(base_url: str) -> tuple[str, str]: + """`http://gx10:8000/v1` -> ("http://gx10:8000", "v1/chat/completions").""" + parts = urlsplit(base_url) + host = f"{parts.scheme}://{parts.netloc}" + prefix = parts.path.strip("/") + base_path = f"{prefix}/chat/completions" if prefix else "v1/chat/completions" + return host, base_path diff --git a/agent-bench/adapters/opencode.py b/agent-bench/adapters/opencode.py new file mode 100644 index 0000000..051c49a --- /dev/null +++ b/agent-bench/adapters/opencode.py @@ -0,0 +1,71 @@ +"""OpenCode adapter — drives the `opencode run` CLI against gx10. + +https://github.com/sst/opencode · https://opencode.ai/docs/cli/ + +gx10 is wired as a custom OpenAI-compatible provider written into the workspace's +`opencode.json` (the `@ai-sdk/openai-compatible` npm provider). `opencode run` +completes after the single message and exits — ideal for headless benchmarking. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path + +from ._proc import TIMEOUT_RC, init_repo, run, snapshot_diff, transcript_of +from .base import Endpoint, RunArtifacts + +OPENCODE_VERSION = "0.6.x" # pin to the installed release before a real run +_PROVIDER_ID = "gx10" + + +class OpencodeAdapter: + name = "opencode" + version = OPENCODE_VERSION + + def prepare(self, workspace: Path) -> None: + init_repo(workspace) + + def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> RunArtifacts: + _write_config(workspace, gx10) + # provider/model: opencode splits on the first '/', so the model id may itself + # contain slashes (e.g. "Qwen/Qwen3-Coder-..."). + model_ref = f"{_PROVIDER_ID}/{gx10.model}" + cmd = [ + "opencode", + "run", + "--model", + model_ref, + "--format", + "json", # raw events -> token usage parseable by the cost scorer + spec, + ] + # opencode reads opencode.json from the cwd; keep the dummy key in env too. + env_extra = {"OPENAI_API_KEY": gx10.api_key} + started = time.perf_counter() + proc = run(cmd, cwd=workspace, timeout_s=timeout_s, env_extra=env_extra) + duration = time.perf_counter() - started + + return RunArtifacts( + exit_ok=proc.returncode == 0, + transcript=transcript_of(cmd, proc), + diff=snapshot_diff(workspace), + duration_s=duration, + extra={"returncode": proc.returncode, "timed_out": proc.returncode == TIMEOUT_RC}, + ) + + +def _write_config(workspace: Path, gx10: Endpoint) -> None: + config = { + "$schema": "https://opencode.ai/config.json", + "provider": { + _PROVIDER_ID: { + "npm": "@ai-sdk/openai-compatible", + "name": "gx10", + "options": {"baseURL": gx10.base_url, "apiKey": gx10.api_key}, + "models": {gx10.model: {"name": gx10.model}}, + } + }, + } + (workspace / "opencode.json").write_text(json.dumps(config, indent=2)) diff --git a/agent-bench/bench.yaml b/agent-bench/bench.yaml index ce8c29b..beeeda7 100644 --- a/agent-bench/bench.yaml +++ b/agent-bench/bench.yaml @@ -17,7 +17,7 @@ store_db: "results/bench.sqlite3" frameworks: - aider + - opencode + - goose # - openhands # TODO - # - opencode # TODO - # - goose # TODO # - mini_swe # TODO diff --git a/agent-bench/metrics.yaml b/agent-bench/metrics.yaml index 5b9ce6b..251c9ba 100644 --- a/agent-bench/metrics.yaml +++ b/agent-bench/metrics.yaml @@ -5,6 +5,6 @@ metrics: - builds # generated project compiles (py_compile over all .py) - runs_headless # boots under SDL dummy driver without an immediate crash - feature_checklist # static probe of milestones M1..M7 - # - cost # TODO: prompt/completion tokens, wall-clock, gx10 GPU-seconds + - cost # prompt/completion tokens, wall-clock, tokens/s, optional $ (env-priced) # - process # TODO: turns, tool calls, files touched, diff LOC, retries # - quality_judge # TODO: LLM-judge (0-5) code quality + playability diff --git a/agent-bench/runner.py b/agent-bench/runner.py index d826c94..18ed4f1 100644 --- a/agent-bench/runner.py +++ b/agent-bench/runner.py @@ -19,8 +19,11 @@ import yaml from adapters.aider import AiderAdapter from adapters.base import Adapter, Endpoint, RunArtifacts +from adapters.goose import GooseAdapter +from adapters.opencode import OpencodeAdapter from scorers.base import Scorer, Scores from scorers.builds import BuildsScorer +from scorers.cost import CostScorer from scorers.feature_checklist import FeatureChecklistScorer from scorers.runs_headless import RunsHeadlessScorer from store import ResultsStore, RunRecord @@ -30,11 +33,14 @@ # Registries. Add a framework / metric by adding a line here. ADAPTERS: dict[str, type[Adapter]] = { "aider": AiderAdapter, + "opencode": OpencodeAdapter, + "goose": GooseAdapter, } SCORERS: dict[str, type[Scorer]] = { "builds": BuildsScorer, "runs_headless": RunsHeadlessScorer, "feature_checklist": FeatureChecklistScorer, + "cost": CostScorer, } diff --git a/agent-bench/scorers/cost.py b/agent-bench/scorers/cost.py new file mode 100644 index 0000000..35b620c --- /dev/null +++ b/agent-bench/scorers/cost.py @@ -0,0 +1,91 @@ +"""`cost` — token accounting + wall-clock + optional USD pricing. + +gx10 is a self-hosted local model, so the default dollar cost is $0. Pricing is +overridable without touching code via BENCH_PRICE_PROMPT_PER_MTOK / +BENCH_PRICE_COMPLETION_PER_MTOK (USD per 1M tokens). +""" + +from __future__ import annotations + +import os +import re +from pathlib import Path + +from adapters.base import RunArtifacts + +from scorers.base import Scores + +# JSON usage blocks, e.g. `"prompt_tokens": 1234` / `"completion_tokens": 56`. +_PROMPT_JSON_RE = re.compile(r'"?prompt_tokens"?\s*[:=]\s*(\d+)') +_COMPLETION_JSON_RE = re.compile(r'"?completion_tokens"?\s*[:=]\s*(\d+)') + +# aider-style summary, e.g. `Tokens: 1.2k sent, 345 received` (k/M suffixes). +_AIDER_RE = re.compile( + r"Tokens:\s*([\d.]+)\s*([kKmM]?)\s*sent,\s*([\d.]+)\s*([kKmM]?)\s*received", +) + +_SUFFIX_MULTIPLIER = {"": 1, "k": 1_000, "m": 1_000_000} + + +def _scale(value: str, suffix: str) -> int: + """Turn a `1.2` + `k` pair into an int token count.""" + return round(float(value) * _SUFFIX_MULTIPLIER[suffix.lower()]) + + +def _parse_transcript(transcript: str) -> tuple[int, int]: + """Best-effort token recovery from a transcript. + + Conservative: returns (0, 0) when nothing matches. Sums every JSON usage + block found, then falls back to the last aider summary line if no JSON + block carried any tokens. + """ + prompt = sum(int(m) for m in _PROMPT_JSON_RE.findall(transcript)) + completion = sum(int(m) for m in _COMPLETION_JSON_RE.findall(transcript)) + if prompt or completion: + return prompt, completion + + # No JSON usage; try aider's running summary. Take the last line so the + # final cumulative count wins over earlier partial reports. + matches = _AIDER_RE.findall(transcript) + if matches: + sent_val, sent_suffix, recv_val, recv_suffix = matches[-1] + return _scale(sent_val, sent_suffix), _scale(recv_val, recv_suffix) + + return 0, 0 + + +class CostScorer: + name = "cost" + + def score(self, workspace: Path, artifacts: RunArtifacts) -> Scores: + # (1) Trust structured artifact fields when the adapter populated them. + if artifacts.tokens_prompt is not None or artifacts.tokens_completion is not None: + prompt = artifacts.tokens_prompt or 0 + completion = artifacts.tokens_completion or 0 + source = "artifacts" + else: + # (2) Fall back to scraping the transcript for usage lines. + prompt, completion = _parse_transcript(artifacts.transcript) + source = "transcript" if (prompt or completion) else "none" + + total = prompt + completion + duration = artifacts.duration_s + tokens_per_s = completion / duration if duration > 0 else 0.0 + + cost_usd = 0.0 + price_p = os.environ.get("BENCH_PRICE_PROMPT_PER_MTOK") + price_c = os.environ.get("BENCH_PRICE_COMPLETION_PER_MTOK") + if price_p is not None or price_c is not None: + cost_usd = prompt / 1e6 * float(price_p or 0.0) + completion / 1e6 * float( + price_c or 0.0 + ) + + return { + "tokens_prompt": prompt, + "tokens_completion": completion, + "tokens_total": total, + "wall_clock_s": duration, + "tokens_per_s": tokens_per_s, + "cost_usd": cost_usd, + "cost_source": source, + } From 6d5c55a51df15161d56f7cf8e6c49a24cfa18b58 Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Tue, 2 Jun 2026 22:12:55 -0700 Subject: [PATCH 04/11] agent-bench: exact token extraction from opencode --format json MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit opencode adapter now parses the JSONL event stream and populates RunArtifacts.tokens_prompt / tokens_completion / turns directly, so the cost scorer reports cost_source="artifacts" (exact) instead of falling back to the transcript regex. - _parse_usage sums step_finish events: part.tokens.input -> prompt, output+reasoning -> completion; cache read/write + opencode's own part.cost captured in extra. turns = step_finish count. - Per-step semantics (AI-SDK convention; undocumented by opencode) — per-step breakdown stashed in extra.per_step_tokens for audit; one-line switch to max if a release turns out to report cumulative totals. - Tolerant reader: JSONL or single top-level array, skips non-JSON lines. Verified: multi-event sample -> prompt 2000 / completion 550 / turns 2; cost scorer flips to cost_source="artifacts"; array + empty + garbage inputs handled; ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) --- agent-bench/adapters/opencode.py | 101 ++++++++++++++++++++++++++++++- 1 file changed, 100 insertions(+), 1 deletion(-) diff --git a/agent-bench/adapters/opencode.py b/agent-bench/adapters/opencode.py index 051c49a..2514261 100644 --- a/agent-bench/adapters/opencode.py +++ b/agent-bench/adapters/opencode.py @@ -11,7 +11,10 @@ import json import time +from collections.abc import Iterator +from dataclasses import dataclass, field from pathlib import Path +from typing import Any from ._proc import TIMEOUT_RC, init_repo, run, snapshot_diff, transcript_of from .base import Endpoint, RunArtifacts @@ -47,15 +50,111 @@ def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> proc = run(cmd, cwd=workspace, timeout_s=timeout_s, env_extra=env_extra) duration = time.perf_counter() - started + usage = _parse_usage(proc.stdout) return RunArtifacts( exit_ok=proc.returncode == 0, transcript=transcript_of(cmd, proc), diff=snapshot_diff(workspace), duration_s=duration, - extra={"returncode": proc.returncode, "timed_out": proc.returncode == TIMEOUT_RC}, + tokens_prompt=usage.tokens_prompt, + tokens_completion=usage.tokens_completion, + turns=usage.turns, + extra={ + "returncode": proc.returncode, + "timed_out": proc.returncode == TIMEOUT_RC, + "opencode_cost_usd": usage.cost_usd, + "cache_read_tokens": usage.cache_read, + "cache_write_tokens": usage.cache_write, + "per_step_tokens": usage.per_step, # audit trail; see _parse_usage note + }, ) +@dataclass +class _Usage: + tokens_prompt: int = 0 + tokens_completion: int = 0 + turns: int = 0 + cost_usd: float = 0.0 + cache_read: int = 0 + cache_write: int = 0 + per_step: list[dict[str, int]] = field(default_factory=list) + + +def _parse_usage(stdout: str) -> _Usage: + """Extract exact token usage from `opencode run --format json` (JSONL) output. + + opencode emits one JSON object per line; `step_finish` events carry + `part.tokens.{input,output,reasoning,cache.{read,write}}` and `part.cost`. + Token/cost are summed across step_finish events (per-step semantics, AI-SDK + convention — undocumented by opencode, so the per-step breakdown is kept in + `extra.per_step_tokens` for audit; if a release reports cumulative totals + instead, switch the sum to a max here). Tolerant: skips non-JSON lines and + also accepts a single top-level JSON array. + """ + usage = _Usage() + for event in _iter_events(stdout): + if not _is_step_finish(event): + continue + part = event.get("part", {}) + tokens = part.get("tokens", {}) if isinstance(part, dict) else {} + if not isinstance(tokens, dict): + continue + prompt = _as_int(tokens.get("input")) + output = _as_int(tokens.get("output")) + reasoning = _as_int(tokens.get("reasoning")) + cache = tokens.get("cache", {}) if isinstance(tokens.get("cache"), dict) else {} + usage.tokens_prompt += prompt + usage.tokens_completion += output + reasoning # reasoning is generated output + usage.cache_read += _as_int(cache.get("read")) + usage.cache_write += _as_int(cache.get("write")) + usage.cost_usd += _as_float(part.get("cost")) + usage.turns += 1 + usage.per_step.append({"input": prompt, "output": output, "reasoning": reasoning}) + return usage + + +def _iter_events(stdout: str) -> Iterator[dict[str, Any]]: + text = stdout.strip() + if not text: + return + # Some releases may emit a single JSON array rather than JSONL. + if text[0] == "[": + try: + arr = json.loads(text) + except json.JSONDecodeError: + arr = [] + for obj in arr: + if isinstance(obj, dict): + yield obj + return + for line in text.splitlines(): + line = line.strip() + if not line or line[0] != "{": + continue + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(obj, dict): + yield obj + + +def _is_step_finish(event: dict[str, Any]) -> bool: + if event.get("type") in {"step_finish", "step-finish"}: + return True + part = event.get("part") + return isinstance(part, dict) and part.get("type") == "step-finish" + + +def _as_int(value: object) -> int: + return int(value) if isinstance(value, (int, float)) else 0 + + +def _as_float(value: object) -> float: + return float(value) if isinstance(value, (int, float)) else 0.0 + + def _write_config(workspace: Path, gx10: Endpoint) -> None: config = { "$schema": "https://opencode.ai/config.json", From 2d374a7b35ec67b59f34f77e5497cf34eee1b6be Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Tue, 2 Jun 2026 22:16:31 -0700 Subject: [PATCH 05/11] agent-bench: process scorer (diff LOC, turns, tool calls, tracebacks) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fifth scorer: how the agent worked, not whether it passed. - Deterministic from the unified diff: files_touched, files_created, lines_added/removed, diff_loc. - Soft signals from artifacts, reported as 0 (never guessed) when the framework doesn't surface them: turns (from artifacts.turns — exact for opencode), tool_calls (counts opencode `"type":"tool_use"` JSONL markers), tracebacks. - Registered in runner.SCORERS + metrics.yaml. Verified: sample 2-file diff -> files 2 / created 1 / +4 -1 / loc 5; turns 2, tool_calls 2, tracebacks 1; 5 scorers load; ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) --- agent-bench/metrics.yaml | 2 +- agent-bench/runner.py | 2 ++ agent-bench/scorers/process.py | 50 ++++++++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 1 deletion(-) create mode 100644 agent-bench/scorers/process.py diff --git a/agent-bench/metrics.yaml b/agent-bench/metrics.yaml index 251c9ba..d49cf61 100644 --- a/agent-bench/metrics.yaml +++ b/agent-bench/metrics.yaml @@ -6,5 +6,5 @@ metrics: - runs_headless # boots under SDL dummy driver without an immediate crash - feature_checklist # static probe of milestones M1..M7 - cost # prompt/completion tokens, wall-clock, tokens/s, optional $ (env-priced) - # - process # TODO: turns, tool calls, files touched, diff LOC, retries + - process # files touched/created, diff LOC, turns, tool calls, tracebacks # - quality_judge # TODO: LLM-judge (0-5) code quality + playability diff --git a/agent-bench/runner.py b/agent-bench/runner.py index 18ed4f1..558de3d 100644 --- a/agent-bench/runner.py +++ b/agent-bench/runner.py @@ -25,6 +25,7 @@ from scorers.builds import BuildsScorer from scorers.cost import CostScorer from scorers.feature_checklist import FeatureChecklistScorer +from scorers.process import ProcessScorer from scorers.runs_headless import RunsHeadlessScorer from store import ResultsStore, RunRecord @@ -41,6 +42,7 @@ "runs_headless": RunsHeadlessScorer, "feature_checklist": FeatureChecklistScorer, "cost": CostScorer, + "process": ProcessScorer, } diff --git a/agent-bench/scorers/process.py b/agent-bench/scorers/process.py new file mode 100644 index 0000000..dcd3f1f --- /dev/null +++ b/agent-bench/scorers/process.py @@ -0,0 +1,50 @@ +"""`process` — how the agent worked, not whether it succeeded. + +Strong signal is the unified diff (deterministic): files touched/created and +lines added/removed. Softer signals (turns, tool calls, tracebacks) come from +the captured artifacts and are framework-dependent — reported as 0 when the +framework doesn't surface them, never guessed. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +from adapters.base import RunArtifacts + +from scorers.base import Scores + +_DIFF_GIT = re.compile(r"^diff --git ", re.MULTILINE) +_NEW_FILE = re.compile(r"^new file mode ", re.MULTILINE) +# Tool-call markers: opencode emits `"type":"tool_use"` JSONL; tolerate spacing/dashes. +_TOOL_USE = re.compile(r'"type"\s*:\s*"tool[_-]?use"') +_TRACEBACK = re.compile(r"^Traceback \(most recent call last\):", re.MULTILINE) + + +class ProcessScorer: + name = "process" + + def score(self, workspace: Path, artifacts: RunArtifacts) -> Scores: + added, removed = _count_diff_lines(artifacts.diff) + return { + "files_touched": len(_DIFF_GIT.findall(artifacts.diff)), + "files_created": len(_NEW_FILE.findall(artifacts.diff)), + "lines_added": added, + "lines_removed": removed, + "diff_loc": added + removed, + "turns": artifacts.turns if artifacts.turns is not None else 0, + "tool_calls": len(_TOOL_USE.findall(artifacts.transcript)), + "tracebacks": len(_TRACEBACK.findall(artifacts.transcript)), + } + + +def _count_diff_lines(diff: str) -> tuple[int, int]: + """Added/removed content lines in a unified diff, excluding +++/--- headers.""" + added = removed = 0 + for line in diff.splitlines(): + if line.startswith("+") and not line.startswith("+++"): + added += 1 + elif line.startswith("-") and not line.startswith("---"): + removed += 1 + return added, removed From 1393a8c5ecd80ae550895a60ce5c6a4ef3367454 Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Tue, 2 Jun 2026 23:14:23 -0700 Subject: [PATCH 06/11] agent-bench: pivot to HTML5-canvas web game + live gx10 validation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Benchmark product is now a top-down GTA2-style web game (index.html + game.js, vanilla JS, canvas 2D) instead of pygame. Validated end-to-end against live gx10 (gx10-5fb9, vLLM 0.21.0, Qwen/Qwen3-Coder-30B-A3B-Instruct) via the aider adapter. Spec + scorers reworked for the web stack: - spec/gta-spec.md: self-contained HTML/CSS/JS GTA2 spec, milestones M1..M8, hard tech constraints (one requestAnimationFrame loop, no modules/classes, canvas-2D only, zero console errors). - builds: node --check over every .js + index.html has +`. No build step, no bundler, no external assets. +- `game.js` — all game code in **plain browser JavaScript**. -Implement these milestones in order. Each is graded independently, so a partial -result still counts — do as many as you can, correctly. +Hard tech constraints (the game is graded against these): -- **M1 — window + loop**: open a game window, run a main loop, exit cleanly on quit. -- **M2 — player car**: a car sprite renders and is controllable — WASD / arrow keys - accelerate, brake, and steer (rotate) the car. -- **M3 — world**: a top-down tiled map / road network larger than the viewport; the - camera scrolls to follow the player car. -- **M4 — physics**: acceleration, steering, and friction — momentum, not instant - teleport-style movement. -- **M5 — NPC traffic**: at least one AI-driven vehicle that moves around the map on - its own. -- **M6 — collision**: collision detection and resolution between the player car and - the world and/or NPC vehicles. -- **M7 — objective**: a mission / score loop (reach a waypoint, pick up a target, or a - wanted-level mechanic) plus an on-screen HUD showing score or state. +- **One `requestAnimationFrame` loop** named `gameLoop`: read input → update + fixed-`dt` physics → render. No `setInterval`, no second loop. +- **No ES modules, no `class`, no `async`, no IIFE.** Top-level `let`/`const` only. +- Read `canvas.width`/`canvas.height` from JS; do not hardcode dimensions twice. +- All drawing is HTML5 canvas 2D (`getContext('2d')`). No DOM-element sprites, no WebGL. +- No network calls, no asset downloads — generate all shapes/colors in code. +- The page must load with **zero console errors** and the loop must run every frame. -Constraints: +Implement these milestones in order. Each is graded independently — a clean +M1–M5 beats a crashing M1–M8. Favor correctness and runnability over scope. -- Pure Python + `pygame` only. No network calls, no asset downloads — generate any - needed shapes/sprites in code. -- Single self-contained project rooted here. `main.py` is the entry point. -- Favor correctness and runnability over scope: a clean M1–M4 beats a crashing M1–M7. +- **M1 — project shape + loop**: `index.html` canvas skeleton + `game.js` with the + `gameLoop` requestAnimationFrame callback and `keydown`/`keyup` listeners that + populate a `keys` object (use `event.code`, not `event.key`). +- **M2 — world map**: a tile grid stored as a multi-line string (alphabet e.g. + `.`=road, `=`=sidewalk, `B`=building), 32px tiles, ~40×30 → larger than the + viewport. Render by iterating the grid through a single `drawTile(char,x,y)`. + Must show streets + buildings, not a blank canvas. +- **M3 — player car**: a `player = { x, y, angle, speed, ... }` object literal. + Render with the `save → translate(x,y) → rotate(angle) → draw → restore` + transform (body, roof, windshield, headlights, wheels). Spawn on a road tile. +- **M4 — driving + collision**: `dt`-based motion. Forward accelerates; reverse + brakes then reverses; friction returns speed to 0. **Steering proportional to + speed** (stationary car cannot turn). Building collision via center-vs-tile + (`floor(x/32), floor(y/32)`) — revert position + small bounce on hit. +- **M5 — camera**: a `camera = {x,y}` holding the screen's top-left world coord; + `ctx.translate(-camera.x,-camera.y)` once around the world render. Follow the + player and clamp to map bounds. Map scrolls as the car drives. +- **M6 — pedestrians + run-over + score**: a `peds` array spawned on sidewalk + tiles; they wander and panic near the moving car. Running one over (proximity + + speed) kills it and adds to `score`. +- **M7 — HUD**: after the world pass, reset the transform + (`ctx.setTransform(1,0,0,1,0,0)`) and draw screen-space HUD: `SCORE: ` and a + health bar. HUD must not scroll with the camera. +- **M8 — police + wanted (stretch)**: a `wanted` level that rises on crimes; a + `police` array of cop cars that spawn by wanted level and chase the player. + +Constraints recap: pure HTML/CSS/JS, `index.html` + `game.js`, canvas 2D only, +one `requestAnimationFrame` loop, no modules/classes/async, zero console errors. From 8ee2193a054fa3282850eb232c3571e00287fc2a Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Mon, 15 Jun 2026 09:27:14 -0700 Subject: [PATCH 07/11] agent-bench: resolve html entry to absolute before as_uri() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RunsHeadlessScorer._find_entry returned whatever path it was handed. Path.as_uri() rejects relative paths, so a relative workspace raised "relative path can't be expressed as a file URI" — caught by the broad except and reported as a bogus "playwright error", masking the real load-timeout verdict. Resolve the entry to absolute so a hung page (e.g. an unbounded init loop) records cleanly as runs_headless:false with reason "load timeout (page blocks main thread)". Co-Authored-By: Claude Opus 4.8 (1M context) --- agent-bench/scorers/runs_headless.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/agent-bench/scorers/runs_headless.py b/agent-bench/scorers/runs_headless.py index 1f96be9..051f6cc 100644 --- a/agent-bench/scorers/runs_headless.py +++ b/agent-bench/scorers/runs_headless.py @@ -57,10 +57,12 @@ def score(self, workspace: Path, artifacts: RunArtifacts) -> Scores: @staticmethod def _find_entry(workspace: Path) -> Path | None: + # Resolve to absolute: Path.as_uri() (used below) rejects relative paths. index = workspace / "index.html" if index.exists(): - return index - return next((p for p in workspace.rglob("*.html") if ".git" not in p.parts), None) + return index.resolve() + hit = next((p for p in workspace.rglob("*.html") if ".git" not in p.parts), None) + return hit.resolve() if hit is not None else None @staticmethod def _run(entry: Path) -> Scores: From b53cbc5cb63ea2ece6b5cdd5895697c64b5c1a06 Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Mon, 15 Jun 2026 09:30:21 -0700 Subject: [PATCH 08/11] agent-bench: Bar-A tracer-bullet close (gx10 FP8 pin + clean status) gx10 now serves the FP8 quant; repin the model id so the health gate passes (was aborting on Qwen3-Coder-30B-A3B-Instruct vs ...-FP8). Validated the full loop end-to-end: one live aider run -> all five scorers -> report.py renders a clean single-schema distribution (39.2 tok/s on FP8, 6/8 milestones, runs_headless correctly false on aider's unbounded init-loop hang). Stale mixed-schema rows from the pygame->web pivot were cleared from the store first. README status rewritten to reflect: rig proven on one framework; remaining work is an N-run matrix + a second comparand (mini-SWE next). Co-Authored-By: Claude Opus 4.8 (1M context) --- agent-bench/README.md | 21 ++++++++++++++++----- agent-bench/bench.yaml | 2 +- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/agent-bench/README.md b/agent-bench/README.md index b7bb24e..ffc977b 100644 --- a/agent-bench/README.md +++ b/agent-bench/README.md @@ -57,9 +57,20 @@ python report.py # tables from the store ## Status -Validated end-to-end: **aider** adapter builds the web game against gx10 -(`Qwen/Qwen3-Coder-30B-A3B-Instruct`); all five scorers run, incl. the headless -browser. The reworked scorer suite scores the reference build -(`scratch/workspace/index.html` + `game.js`) at 8/8 milestones + runs_headless ✅. +**Tracer bullet complete (1 framework, full loop).** The **aider** adapter builds +the web game against gx10 (`Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`), all five +scorers run incl. the headless browser, and `report.py` renders a clean +single-schema distribution. The headless scorer is hardened: a build that hangs +the main thread (e.g. an unbounded init loop) records cleanly as +`runs_headless: false, reason: "load timeout …"` instead of wedging the run. + +The current aider build is itself a true-positive failure the rig caught: an +unbounded `while(!validPosition)` pedestrian-spawn loop against a map with no +matching tile → infinite loop on load. `builds` + `feature_checklist` pass +(6/8 milestones), `runs_headless` correctly fails. + +Not yet done (see plan §6): a real N-run matrix and a second comparand. **opencode** + **goose** adapters exist but opencode 1.x hangs on run-init in this -env; goose CLI not installed. OpenHands / mini-SWE adapters are TODO — see the plan. +env and the goose CLI isn't installed; **OpenHands** / **mini-SWE** adapters and a +**harness** baseline entry are TODO. mini-SWE is the cheapest next comparand +(pure LiteLLM, no sandbox). diff --git a/agent-bench/bench.yaml b/agent-bench/bench.yaml index 1fbc74e..3479647 100644 --- a/agent-bench/bench.yaml +++ b/agent-bench/bench.yaml @@ -2,7 +2,7 @@ # variable under test is the framework itself. gx10: base_url: "http://gx10-5fb9:8000/v1" # vLLM OpenAI-compatible endpoint over Tailscale - model: "Qwen/Qwen3-Coder-30B-A3B-Instruct" # pinned; runner aborts if /v1/models disagrees + model: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" # pinned; runner aborts if /v1/models disagrees api_key: "gx10" # dummy; vLLM ignores but OpenAI clients require one temperature: 0.0 # max determinism (loops still vary -> N runs) From 9bcd2f5e5146bad06114a8f81ec407bf7d2d9534 Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Mon, 15 Jun 2026 11:01:24 -0700 Subject: [PATCH 09/11] agent-bench: mini-SWE-agent adapter (headless via Python API) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second comparand. mini-SWE-agent's `mini` CLI is interactive-only — it wires asyncio stdin readers and crashes on a non-tty (OSError [Errno 22] in _add_reader), even with --yolo/--agent-class default. So drive its documented Python API (DefaultAgent + LocalEnvironment + LitellmModel, per the project's own run/hello_world.py) through a thin driver run by the tool venv's interpreter: - adapters/_mini_driver.py: builds the agent rooted at the workspace, runs the task to a terminal state (Submitted / LimitsExceeded / TimeExceeded), emits a metrics JSON (turns=n_calls, summed prompt/completion tokens, cost, exit_status) + a readable transcript. - adapters/mini_swe.py: locates the tool venv python via the `mini` shim's shebang (stable, no uv-cache hash; falls back to `uv tool run`), shells the driver through the shared _proc helpers, and populates RunArtifacts.tokens_*/turns so the cost + process scorers read source="artifacts" rather than scraping the transcript. minisweagent stays out of the bench's deps — it's an external CLI on PATH (uv tool install mini-swe-agent), same convention as aider. Validated end-to-end against gx10 FP8: builds + runs_headless both pass, 7/8 milestones, 17 turns, 140k tokens, ~926s. First real cross-framework contrast vs aider (1-shot, 82s, 6/8, runs_headless false on its init-loop hang) — exactly the comparison the bench exists to produce. Co-Authored-By: Claude Opus 4.8 (1M context) --- agent-bench/README.md | 38 +++++---- agent-bench/adapters/_mini_driver.py | 111 ++++++++++++++++++++++++++ agent-bench/adapters/mini_swe.py | 115 +++++++++++++++++++++++++++ agent-bench/bench.yaml | 2 +- agent-bench/runner.py | 2 + 5 files changed, 251 insertions(+), 17 deletions(-) create mode 100644 agent-bench/adapters/_mini_driver.py create mode 100644 agent-bench/adapters/mini_swe.py diff --git a/agent-bench/README.md b/agent-bench/README.md index ffc977b..16b5c32 100644 --- a/agent-bench/README.md +++ b/agent-bench/README.md @@ -57,20 +57,26 @@ python report.py # tables from the store ## Status -**Tracer bullet complete (1 framework, full loop).** The **aider** adapter builds -the web game against gx10 (`Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`), all five -scorers run incl. the headless browser, and `report.py` renders a clean -single-schema distribution. The headless scorer is hardened: a build that hangs -the main thread (e.g. an unbounded init loop) records cleanly as -`runs_headless: false, reason: "load timeout …"` instead of wedging the run. - -The current aider build is itself a true-positive failure the rig caught: an -unbounded `while(!validPosition)` pedestrian-spawn loop against a map with no -matching tile → infinite loop on load. `builds` + `feature_checklist` pass -(6/8 milestones), `runs_headless` correctly fails. - -Not yet done (see plan §6): a real N-run matrix and a second comparand. +**Two frameworks live against gx10** (`Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`), +both through all five scorers incl. the headless browser, with `report.py` +rendering clean single-schema distributions. First real contrast (n=1 each): + +| framework | turns | wall-clock | milestones | runs_headless | +|---|---|---|---|---| +| aider | 1-shot | ~82 s | 6/8 | ✗ — build hangs on load | +| mini-swe-agent | 17 | ~926 s | 7/8 | ✅ — game runs (362 rAF ticks, 0 errors) | + +The aider row is a true-positive failure the rig caught: an unbounded +`while(!validPosition)` pedestrian-spawn loop against a map with no matching tile +→ infinite loop on load. The headless scorer is hardened to record that cleanly +as `runs_headless: false, reason: "load timeout …"` instead of wedging the run. + +The **mini-swe-agent** adapter drives the agent's Python API headlessly via +`adapters/_mini_driver.py` (its `mini` CLI is interactive-only — crashes on a +non-tty), run by the tool venv's own interpreter. Install once: +`uv tool install mini-swe-agent`. + +Not yet done (see plan §6): an N-run matrix (currently n=1 per framework). **opencode** + **goose** adapters exist but opencode 1.x hangs on run-init in this -env and the goose CLI isn't installed; **OpenHands** / **mini-SWE** adapters and a -**harness** baseline entry are TODO. mini-SWE is the cheapest next comparand -(pure LiteLLM, no sandbox). +env and the goose CLI isn't installed; an **OpenHands** adapter and a **harness** +baseline entry are TODO. diff --git a/agent-bench/adapters/_mini_driver.py b/agent-bench/adapters/_mini_driver.py new file mode 100644 index 0000000..9ef1353 --- /dev/null +++ b/agent-bench/adapters/_mini_driver.py @@ -0,0 +1,111 @@ +"""Headless driver for mini-SWE-agent, run inside the tool's own interpreter. + +The `mini` CLI is interactive-only (it wires asyncio stdin readers and crashes +on a non-tty), so the sanctioned headless path is the Python API documented in +mini-SWE-agent's own `run/hello_world.py`. This script is that path: build a +DefaultAgent against gx10 via LiteLLM, run the task to a terminal state, and +emit a metrics JSON the adapter reads back. + +It imports `minisweagent` and so must run under the tool venv's python, NOT the +bench's. The adapter locates that interpreter and shells out to this file. Keep +it pure stdlib + minisweagent — it never imports bench code. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +import yaml +from minisweagent import package_dir +from minisweagent.agents.default import DefaultAgent +from minisweagent.environments.local import LocalEnvironment +from minisweagent.exceptions import LimitsExceeded, TimeExceeded +from minisweagent.models.litellm_model import LitellmModel + +# Per-command shell timeout inside the agent's environment. The build task only +# writes files; generous enough that a stray `node`/test invocation won't trip it. +ENV_COMMAND_TIMEOUT_S = 120 + + +def _sum_tokens(messages: list[dict]) -> tuple[int, int]: + """Sum prompt/completion tokens across every model response in the trajectory. + + Matches the cost scorer's convention (sum every usage block), so the + `artifacts` and `transcript` token sources agree. + """ + prompt = completion = 0 + for msg in messages: + usage = (msg.get("extra") or {}).get("response", {}) + usage = usage.get("usage") if isinstance(usage, dict) else None + if isinstance(usage, dict): + prompt += int(usage.get("prompt_tokens") or 0) + completion += int(usage.get("completion_tokens") or 0) + return prompt, completion + + +def _render_transcript(messages: list[dict]) -> str: + """Flatten the message log into a readable transcript for replay.""" + lines: list[str] = [] + for msg in messages: + role = msg.get("role", "?") + content = msg.get("content", "") + if isinstance(content, list): # multimodal — keep the text parts + content = " ".join(p.get("text", "") for p in content if isinstance(p, dict)) + lines.append(f"### {role}\n{content}") + return "\n\n".join(lines) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--task-file", required=True) + ap.add_argument("--model", required=True) + ap.add_argument("--cwd", required=True) + ap.add_argument("--metrics-out", required=True) + ap.add_argument("--step-limit", type=int, default=60) + ap.add_argument("--temperature", type=float, default=0.0) + args = ap.parse_args() + + task = Path(args.task_file).read_text() + agent_cfg = yaml.safe_load((package_dir / "config" / "mini.yaml").read_text())["agent"] + # Non-interactive: never prompt; bound by our step limit; cost limit off + # (litellm can't price the local model, so dollar-cost gating is meaningless). + agent_cfg.update(step_limit=args.step_limit, cost_limit=0.0, mode="yolo") + + model = LitellmModel( + model_name=args.model, + model_kwargs={"temperature": args.temperature, "drop_params": True}, + ) + env = LocalEnvironment(cwd=args.cwd, timeout=ENV_COMMAND_TIMEOUT_S) + agent = DefaultAgent(model, env, **agent_cfg) + + exit_status = "ok" + try: + agent.run(task) + exit_status = (agent.messages[-1].get("extra") or {}).get("exit_status", "ok") + except (LimitsExceeded, TimeExceeded) as exc: + exit_status = type(exc).__name__ + except Exception as exc: # a framework crash is a run outcome, not a driver bug + exit_status = f"error:{type(exc).__name__}: {exc}" + + prompt_tokens, completion_tokens = _sum_tokens(agent.messages) + Path(args.metrics_out).write_text( + json.dumps( + { + "exit_status": exit_status, + "turns": agent.n_calls, + "cost": agent.cost, + "tokens_prompt": prompt_tokens, + "tokens_completion": completion_tokens, + } + ) + ) + # The transcript goes to stdout; the adapter captures it via the subprocess. + print(_render_transcript(agent.messages)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/agent-bench/adapters/mini_swe.py b/agent-bench/adapters/mini_swe.py new file mode 100644 index 0000000..08aeb08 --- /dev/null +++ b/agent-bench/adapters/mini_swe.py @@ -0,0 +1,115 @@ +"""mini-SWE-agent adapter — drives the agent headlessly against gx10. + +https://github.com/SWE-agent/mini-swe-agent + +mini-SWE-agent ships as a standalone CLI (`uv tool install mini-swe-agent`), but +its `mini` command is interactive-only and crashes on a non-tty. So we drive its +Python API through `_mini_driver.py`, executed by the tool venv's own +interpreter (minisweagent is never a dependency of the bench itself). The driver +runs the agent in a LocalEnvironment rooted at the workspace; we snapshot the +resulting git diff and read a metrics JSON the driver writes. +""" + +from __future__ import annotations + +import json +import shutil +import tempfile +import time +from pathlib import Path + +from ._proc import TIMEOUT_RC, init_repo, run, snapshot_diff, transcript_of +from .base import Endpoint, RunArtifacts + +# Pin and record. Bump deliberately; the value lands in every result row's manifest. +MINI_VERSION = "2.4.1" +# Bound the agent's turns. Mirrors bench.yaml's advisory turn_cap; the runner's +# wall-clock timeout is the outer backstop. +MINI_STEP_LIMIT = 60 + +_DRIVER = Path(__file__).resolve().parent / "_mini_driver.py" + + +class MiniSweAdapter: + name = "mini-swe-agent" + version = MINI_VERSION + + def prepare(self, workspace: Path) -> None: + init_repo(workspace) + + def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> RunArtifacts: + interpreter = _tool_interpreter() + env_extra = { + "OPENAI_API_BASE": gx10.base_url, + "OPENAI_API_KEY": gx10.api_key, + "MSWEA_COST_TRACKING": "ignore_errors", # local model has no litellm price + "MSWEA_SILENT_STARTUP": "1", + } + # Pass the (large) spec and collect metrics via temp files — no argv escaping, + # and kept out of the workspace so they never pollute the diff snapshot. + with tempfile.TemporaryDirectory(prefix="mini-bench-") as tmp: + task_file = Path(tmp) / "task.txt" + task_file.write_text(spec) + metrics_file = Path(tmp) / "metrics.json" + + cmd = [ + *interpreter, + str(_DRIVER), + "--task-file", + str(task_file), + "--model", + f"openai/{gx10.model}", + "--cwd", + str(workspace), + "--metrics-out", + str(metrics_file), + "--step-limit", + str(MINI_STEP_LIMIT), + "--temperature", + str(gx10.temperature), + ] + started = time.perf_counter() + proc = run(cmd, cwd=workspace, timeout_s=timeout_s, env_extra=env_extra) + duration = time.perf_counter() - started + metrics = _read_metrics(metrics_file) + + timed_out = proc.returncode == TIMEOUT_RC + return RunArtifacts( + exit_ok=proc.returncode == 0, + transcript=transcript_of(cmd, proc), + diff=snapshot_diff(workspace), + duration_s=duration, + tokens_prompt=metrics.get("tokens_prompt"), + tokens_completion=metrics.get("tokens_completion"), + turns=metrics.get("turns"), + extra={ + "returncode": proc.returncode, + "timed_out": timed_out, + "exit_status": metrics.get("exit_status"), + "model_cost": metrics.get("cost"), + }, + ) + + +def _read_metrics(path: Path) -> dict: + """Driver metrics, or empty on timeout/crash before it could write them.""" + try: + return json.loads(path.read_text()) + except (FileNotFoundError, ValueError): + return {} + + +def _tool_interpreter() -> list[str]: + """Locate the tool venv python that has minisweagent installed. + + The `mini` console-script shim's shebang points straight at it (stable, no + uv cache hash). Fall back to `uv tool run` if the shim isn't found. + """ + shim = shutil.which("mini") + if shim: + first_line = Path(shim).read_text(errors="replace").splitlines()[:1] + if first_line and first_line[0].startswith("#!"): + interpreter = first_line[0][2:].strip() + if interpreter and Path(interpreter).exists(): + return [interpreter] + return ["uv", "tool", "run", "--from", "mini-swe-agent", "python"] diff --git a/agent-bench/bench.yaml b/agent-bench/bench.yaml index 3479647..415e619 100644 --- a/agent-bench/bench.yaml +++ b/agent-bench/bench.yaml @@ -17,7 +17,7 @@ store_db: "results/bench.sqlite3" frameworks: - aider + - mini-swe-agent - opencode - goose # - openhands # TODO - # - mini_swe # TODO diff --git a/agent-bench/runner.py b/agent-bench/runner.py index 558de3d..8665b22 100644 --- a/agent-bench/runner.py +++ b/agent-bench/runner.py @@ -20,6 +20,7 @@ from adapters.aider import AiderAdapter from adapters.base import Adapter, Endpoint, RunArtifacts from adapters.goose import GooseAdapter +from adapters.mini_swe import MiniSweAdapter from adapters.opencode import OpencodeAdapter from scorers.base import Scorer, Scores from scorers.builds import BuildsScorer @@ -36,6 +37,7 @@ "aider": AiderAdapter, "opencode": OpencodeAdapter, "goose": GooseAdapter, + "mini-swe-agent": MiniSweAdapter, } SCORERS: dict[str, type[Scorer]] = { "builds": BuildsScorer, From c755f00b25c0c0ba61d188208a7666537493465f Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Mon, 15 Jun 2026 11:11:12 -0700 Subject: [PATCH 10/11] agent-bench: bring rig up to repo standards (mypy + tests) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prep for merging agent-bench into main. The per-commit hooks scoped mypy to src/, so the bench tree was untyped and untested. Close both gaps: - Type-clean agent-bench under strict mypy and extend the mypy hook to `src agent-bench`. Fixes: typed dict/list generics in the mini driver and metrics reader, an explicit numeric coercion in report.py (no more float(object)), a robust isinstance-guarded token sum, and a typed _opt_int accessor so RunArtifacts fields stay int | None. Drops a stale type: ignore. - Register a minisweagent missing-imports override (it's an external CLI, never a repo dependency — same rationale as the playwright override). - Add tests/agent_bench/: unit coverage for the deterministic surface — feature_checklist milestone probe, cost token accounting (artifacts + transcript-scrape paths), process diff/transcript counters, and the SQLite store round-trip. builds gets a node-guarded smoke test. The external-tool scorers (node, Playwright, gx10) stay covered by the runner end-to-end. A local conftest puts the agent-bench dir on path. Gates green: ruff, ruff format, mypy (src + agent-bench, 215 files), pytest (3905 passed). Co-Authored-By: Claude Opus 4.8 (1M context) --- .pre-commit-config.yaml | 2 +- agent-bench/adapters/_mini_driver.py | 11 ++- agent-bench/adapters/mini_swe.py | 17 +++- agent-bench/report.py | 11 ++- pyproject.toml | 6 ++ tests/agent_bench/conftest.py | 17 ++++ tests/agent_bench/test_agent_bench.py | 135 ++++++++++++++++++++++++++ 7 files changed, 186 insertions(+), 13 deletions(-) create mode 100644 tests/agent_bench/conftest.py create mode 100644 tests/agent_bench/test_agent_bench.py diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index ca95841..7c57793 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -20,7 +20,7 @@ repos: - id: mypy name: mypy - entry: uv run mypy src + entry: uv run mypy src agent-bench language: system types: [python] pass_filenames: false diff --git a/agent-bench/adapters/_mini_driver.py b/agent-bench/adapters/_mini_driver.py index 9ef1353..394900f 100644 --- a/agent-bench/adapters/_mini_driver.py +++ b/agent-bench/adapters/_mini_driver.py @@ -30,7 +30,7 @@ ENV_COMMAND_TIMEOUT_S = 120 -def _sum_tokens(messages: list[dict]) -> tuple[int, int]: +def _sum_tokens(messages: list[dict[str, object]]) -> tuple[int, int]: """Sum prompt/completion tokens across every model response in the trajectory. Matches the cost scorer's convention (sum every usage block), so the @@ -38,22 +38,23 @@ def _sum_tokens(messages: list[dict]) -> tuple[int, int]: """ prompt = completion = 0 for msg in messages: - usage = (msg.get("extra") or {}).get("response", {}) - usage = usage.get("usage") if isinstance(usage, dict) else None + extra = msg.get("extra") + response = extra.get("response") if isinstance(extra, dict) else None + usage = response.get("usage") if isinstance(response, dict) else None if isinstance(usage, dict): prompt += int(usage.get("prompt_tokens") or 0) completion += int(usage.get("completion_tokens") or 0) return prompt, completion -def _render_transcript(messages: list[dict]) -> str: +def _render_transcript(messages: list[dict[str, object]]) -> str: """Flatten the message log into a readable transcript for replay.""" lines: list[str] = [] for msg in messages: role = msg.get("role", "?") content = msg.get("content", "") if isinstance(content, list): # multimodal — keep the text parts - content = " ".join(p.get("text", "") for p in content if isinstance(p, dict)) + content = " ".join(str(p.get("text", "")) for p in content if isinstance(p, dict)) lines.append(f"### {role}\n{content}") return "\n\n".join(lines) diff --git a/agent-bench/adapters/mini_swe.py b/agent-bench/adapters/mini_swe.py index 08aeb08..d7c088d 100644 --- a/agent-bench/adapters/mini_swe.py +++ b/agent-bench/adapters/mini_swe.py @@ -79,9 +79,9 @@ def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> transcript=transcript_of(cmd, proc), diff=snapshot_diff(workspace), duration_s=duration, - tokens_prompt=metrics.get("tokens_prompt"), - tokens_completion=metrics.get("tokens_completion"), - turns=metrics.get("turns"), + tokens_prompt=_opt_int(metrics, "tokens_prompt"), + tokens_completion=_opt_int(metrics, "tokens_completion"), + turns=_opt_int(metrics, "turns"), extra={ "returncode": proc.returncode, "timed_out": timed_out, @@ -91,12 +91,19 @@ def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> ) -def _read_metrics(path: Path) -> dict: +def _opt_int(metrics: dict[str, object], key: str) -> int | None: + """A metric value as int, or None when absent/non-int (bools excluded).""" + value = metrics.get(key) + return value if isinstance(value, int) and not isinstance(value, bool) else None + + +def _read_metrics(path: Path) -> dict[str, object]: """Driver metrics, or empty on timeout/crash before it could write them.""" try: - return json.loads(path.read_text()) + data = json.loads(path.read_text()) except (FileNotFoundError, ValueError): return {} + return data if isinstance(data, dict) else {} def _tool_interpreter() -> list[str]: diff --git a/agent-bench/report.py b/agent-bench/report.py index cf80210..0d41ef7 100644 --- a/agent-bench/report.py +++ b/agent-bench/report.py @@ -16,6 +16,13 @@ ROOT = Path(__file__).resolve().parent +def _as_float(value: object) -> float: + """Coerce a stored metric value to float, rejecting non-numbers explicitly.""" + if isinstance(value, (int, float)): + return float(value) + raise TypeError(f"expected a number, got {type(value).__name__}") + + def main() -> int: ap = argparse.ArgumentParser(description="agent-bench report") ap.add_argument("--config", default=str(ROOT / "bench.yaml")) @@ -33,7 +40,7 @@ def main() -> int: for fw, fw_rows in sorted(by_fw.items()): n = len(fw_rows) - durations = [float(r["duration_s"]) for r in fw_rows] + durations = [_as_float(r["duration_s"]) for r in fw_rows] exit_ok = sum(int(bool(r["exit_ok"])) for r in fw_rows) print(f"\n=== {fw} (n={n}, version={fw_rows[0]['version']}) ===") print(f" exit_ok : {exit_ok}/{n}") @@ -59,7 +66,7 @@ def _report_metrics(fw_rows: list[dict[str, object]]) -> None: true_n = sum(1 for v in present if v) print(f" {key:<20}: {true_n}/{n} true") elif all(isinstance(v, (int, float)) for v in present): - nums = [float(v) for v in present] # type: ignore[arg-type] + nums = [_as_float(v) for v in present] print( f" {key:<20}: median {statistics.median(nums):.2f} " f"min {min(nums):.2f} max {max(nums):.2f}" diff --git a/pyproject.toml b/pyproject.toml index 2258eb6..88e9f0c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -255,6 +255,12 @@ ignore_missing_imports = true module = ["playwright", "playwright.*"] ignore_missing_imports = true +# agent-bench drives mini-SWE-agent through its tool-venv interpreter; minisweagent +# is intentionally not a dependency of this repo (external CLI, like aider). +[[tool.mypy.overrides]] +module = ["minisweagent", "minisweagent.*"] +ignore_missing_imports = true + [[tool.mypy.overrides]] module = [ "tree_sitter", diff --git a/tests/agent_bench/conftest.py b/tests/agent_bench/conftest.py new file mode 100644 index 0000000..46dc995 --- /dev/null +++ b/tests/agent_bench/conftest.py @@ -0,0 +1,17 @@ +"""Put the agent-bench tree on sys.path for its tests. + +agent-bench is run as scripts from its own directory (`python runner.py`), so +its modules (`adapters`, `scorers`, `store`) are top-level — not under the +`harness` package. These tests import them directly, so we prepend the +agent-bench dir here. Localized to this conftest so the rest of the suite is +unaffected. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +_AGENT_BENCH = Path(__file__).resolve().parents[2] / "agent-bench" +if str(_AGENT_BENCH) not in sys.path: + sys.path.insert(0, str(_AGENT_BENCH)) diff --git a/tests/agent_bench/test_agent_bench.py b/tests/agent_bench/test_agent_bench.py new file mode 100644 index 0000000..2a431ba --- /dev/null +++ b/tests/agent_bench/test_agent_bench.py @@ -0,0 +1,135 @@ +"""Unit coverage for the agent-bench rig: pure scorers + the results store. + +These never touch gx10, node, or a browser — they pin the deterministic parts +(regex milestone probe, token/diff accounting, SQLite round-trip). The +external-tool scorers (`builds` via node, `runs_headless` via Playwright) are +exercised end-to-end by the runner, not here; `builds` gets a smoke test only +when node is present. +""" + +from __future__ import annotations + +import shutil +from pathlib import Path + +import pytest +from adapters.base import RunArtifacts +from scorers.builds import BuildsScorer +from scorers.cost import CostScorer +from scorers.feature_checklist import FeatureChecklistScorer +from scorers.process import ProcessScorer +from store import ResultsStore, RunRecord + +# A workspace that hits a known subset of milestones (m1, m2, m5, m7) and +# deliberately misses others, so the count is an exact assertion. +_GAME_JS = """ +const ctx = canvas.getContext('2d'); +let player = { angle: 0 }; +addEventListener('keydown', e => {}); +function loop() { + ctx.setTransform(1, 0, 0, 1, 0, 0); + ctx.translate(-camera.x, -camera.y); + drawTile(); + ctx.fillText('HUD', 10, 10); + requestAnimationFrame(loop); +} +requestAnimationFrame(loop); +""" +_INDEX_HTML = '' + + +def _workspace(tmp_path: Path) -> Path: + (tmp_path / "game.js").write_text(_GAME_JS) + (tmp_path / "index.html").write_text(_INDEX_HTML) + return tmp_path + + +def _artifacts(**kw: object) -> RunArtifacts: + base: dict[str, object] = {"exit_ok": True, "transcript": "", "diff": "", "duration_s": 1.0} + base.update(kw) + return RunArtifacts(**base) # type: ignore[arg-type] + + +def test_feature_checklist_counts_exact_milestones(tmp_path: Path) -> None: + scores = FeatureChecklistScorer().score(_workspace(tmp_path), _artifacts()) + assert scores["m1_loop"] is True # requestAnimationFrame + keydown + assert scores["m7_hud"] is True # fillText + setTransform(1,0,0,1 + assert scores["m8_police"] is False # no police/wanted signals + assert scores["milestones_total"] == 8 + # reached == number of true milestone flags + reached = sum(1 for k, v in scores.items() if k.startswith("m") and v is True) + assert scores["milestones_reached"] == reached + + +def test_cost_trusts_artifact_tokens() -> None: + scores = CostScorer().score( + Path("."), _artifacts(tokens_prompt=100, tokens_completion=50, duration_s=10.0) + ) + assert scores["tokens_total"] == 150 + assert scores["tokens_per_s"] == pytest.approx(5.0) # completion / duration + assert scores["cost_source"] == "artifacts" + + +def test_cost_falls_back_to_transcript_scrape() -> None: + transcript = 'usage {"prompt_tokens": 30, "completion_tokens": 20}' + scores = CostScorer().score(Path("."), _artifacts(transcript=transcript)) + assert (scores["tokens_prompt"], scores["tokens_completion"]) == (30, 20) + assert scores["cost_source"] == "transcript" + + +def test_process_counts_diff_and_transcript_signals() -> None: + diff = ( + "diff --git a/game.js b/game.js\n" + "new file mode 100644\n" + "--- /dev/null\n+++ b/game.js\n" + "+line one\n+line two\n" + ) + transcript = 'Traceback (most recent call last):\n{"type":"tool_use"}' + scores = ProcessScorer().score(Path("."), _artifacts(diff=diff, transcript=transcript, turns=7)) + assert scores["files_created"] == 1 + assert scores["lines_added"] == 2 + assert scores["lines_removed"] == 0 + assert scores["turns"] == 7 # surfaced from artifacts, not guessed + assert scores["tool_calls"] == 1 + assert scores["tracebacks"] == 1 + + +def test_process_reports_zero_turns_when_unknown() -> None: + scores = ProcessScorer().score(Path("."), _artifacts(turns=None)) + assert scores["turns"] == 0 # never guessed + + +def test_store_round_trips_scores_and_mirrors_jsonl(tmp_path: Path) -> None: + store = ResultsStore(tmp_path / "bench.sqlite3") + rec = RunRecord( + framework="aider", + version="0.86.2", + run_idx=0, + spec_sha="abc123", + model_id="qwen", + exit_ok=True, + duration_s=12.5, + scores={"builds": True, "milestones_reached": 6}, + manifest={"temperature": 0.0}, + ) + run_id = store.append(rec) + assert run_id == 1 + + rows = store.all_rows() + assert len(rows) == 1 + row = rows[0] + assert row["framework"] == "aider" + assert row["exit_ok"] == 1 # SQLite stores bool as int + assert row["scores"] == {"builds": True, "milestones_reached": 6} + assert row["manifest"] == {"temperature": 0.0} + + jsonl = (tmp_path / "bench.jsonl").read_text().strip().splitlines() + assert len(jsonl) == 1 + + +@pytest.mark.skipif(shutil.which("node") is None, reason="node not on PATH") +def test_builds_passes_on_valid_game(tmp_path: Path) -> None: + scores = BuildsScorer().score(_workspace(tmp_path), _artifacts()) + assert scores["builds"] is True + assert scores["js_syntax_failures"] == 0 + assert scores["index_html"] is True From b8d721a15f1505c3bb9847eba7fd8ec71f2f0b8f Mon Sep 17 00:00:00 2001 From: Mark Evans Date: Mon, 15 Jun 2026 11:25:47 -0700 Subject: [PATCH 11/11] test: isolate bd_warning_filter tests from inherited GIT_* env Pre-existing flake, surfaced when running the full suite under the pre-push hook: pre-commit's pre-push exports GIT_DIR / GIT_WORK_TREE / GIT_INDEX_FILE, so workspace_is_gitignored's `git check-ignore` resolved against the repo running the hook instead of each test's tmp_path, flipping the gitignored/tracked assertions. test_git_tools.py and test_driver_cli.py already guard against this; the bd_warning_filter tests never did. Add an autouse fixture that strips GIT_* per test (monkeypatch.delenv), mirroring tests/test_git_tools.py::_clean_env. Verified: the file's 13 tests pass both bare and under a simulated GIT_DIR/GIT_WORK_TREE env. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_driver_bd_warning_filter.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/test_driver_bd_warning_filter.py b/tests/test_driver_bd_warning_filter.py index fde1636..3bbb23f 100644 --- a/tests/test_driver_bd_warning_filter.py +++ b/tests/test_driver_bd_warning_filter.py @@ -2,6 +2,7 @@ from __future__ import annotations +import os import subprocess from pathlib import Path @@ -21,6 +22,17 @@ from harness.tools.base import ToolCall, ToolResult, ToolSpec +@pytest.fixture(autouse=True) +def _clean_git_env(monkeypatch: pytest.MonkeyPatch) -> None: + """Strip ``GIT_*`` env before each test. The pre-push hook in pre-commit + exports ``GIT_DIR`` / ``GIT_WORK_TREE`` / ``GIT_INDEX_FILE``; without this, + ``workspace_is_gitignored``'s ``git check-ignore`` resolves against the repo + running the hook instead of the test's tmp dirs and the assertions flip. + Mirrors tests/test_git_tools.py::_clean_env.""" + for key in [k for k in os.environ if k.startswith("GIT_")]: + monkeypatch.delenv(key, raising=False) + + def _spec(name: str = "shell") -> ToolSpec: return ToolSpec(name=name, description="", parameters={}, tier="write")