agent-bench: framework benchmark rig + aider & mini-SWE-agent comparands

.pre-commit-config.yaml

@@ -20,7 +20,7 @@ repos:
 
       - id: mypy
         name: mypy
-        entry: uv run mypy src
+        entry: uv run mypy src agent-bench
         language: system
         types: [python]
         pass_filenames: false

agent-bench/.gitignore

@@ -0,0 +1,3 @@
+# Per-run artifacts + results store are generated, not tracked.
+results/
+__pycache__/

agent-bench/README.md

@@ -0,0 +1,82 @@
+# agent-bench
+
+Benchmark rig: give each agent framework the same autonomous build task — a
+top-down **GTA2-style web game** (HTML5 canvas + vanilla JS: `index.html` +
+`game.js`) — against one shared gx10 vLLM endpoint, then score the result.
+
+See `../docs/agent-framework-bench.md` for the full plan.
+
+## Prerequisites
+
+- **node** (≥18) — JS syntax check in the `builds` scorer (`node --check`).
+- **Playwright + chromium** — the `runs_headless` scorer loads the generated
+  page in a real headless browser. Install once:
+  `uv pip install playwright && uv run playwright install chromium`.
+- A framework CLI on PATH for each adapter you run (`aider`, `opencode`, …).
+
+## Layout
+
+```
+spec/gta-spec.md   the one task input (HTML/CSS/JS GTA2, milestones M1..M8)
+bench.yaml         frameworks, runs-per-framework, gx10 endpoint, paths
+metrics.yaml       ordered scorer plugins to run (the adjustable surface)
+adapters/          one thin wrapper per framework (framework SDKs live ONLY here)
+scorers/           pluggable metric collectors (drop-in, listed in metrics.yaml)
+runner.py          framework-agnostic matrix runner
+store.py           append-first SQLite + JSONL results store
+report.py          aggregate -> tables + per-metric distributions
+results/           per-run artifacts (transcript, diff, scores.json)
+```
+
+## Run
+
+```bash
+cd agent-bench
+python runner.py --runs 5                 # full matrix from bench.yaml
+python runner.py --framework aider --runs 1
+python report.py                          # tables from the store
+```
+
+## Extend
+
+- **New framework**: add `adapters/<name>.py` implementing the `Adapter` protocol;
+  register it in `ADAPTERS` (runner.py).
+- **New metric**: add `scorers/<name>.py` implementing the `Scorer` protocol;
+  register it in `SCORERS` (runner.py) and list its name in `metrics.yaml`.
+  No runner edit beyond the registry line; scores are schemaless JSON per row.
+
+## Scorers (web stack)
+
+- `builds` — `node --check` over every `.js`; `index.html` has `<canvas>` + `<script>`.
+- `runs_headless` — Playwright chromium loads `index.html`, runs ~3s, asserts:
+  zero console errors (`loads_clean`), `requestAnimationFrame` ran (`loop_alive`),
+  canvas drew non-blank pixels (`renders`).
+- `feature_checklist` — static regex probe of milestones M1..M8 over `game.js` + `index.html`.
+- `cost` — tokens / wall-clock / tokens·s⁻¹ / $ (env-priced).
+- `process` — files touched/created, diff LOC, turns, tool calls, tracebacks.
+
+## Status
+
+**Two frameworks live against gx10** (`Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`),
+both through all five scorers incl. the headless browser, with `report.py`
+rendering clean single-schema distributions. First real contrast (n=1 each):
+
+| framework | turns | wall-clock | milestones | runs_headless |
+|---|---|---|---|---|
+| aider | 1-shot | ~82 s | 6/8 | ✗ — build hangs on load |
+| mini-swe-agent | 17 | ~926 s | 7/8 | ✅ — game runs (362 rAF ticks, 0 errors) |
+
+The aider row is a true-positive failure the rig caught: an unbounded
+`while(!validPosition)` pedestrian-spawn loop against a map with no matching tile
+→ infinite loop on load. The headless scorer is hardened to record that cleanly
+as `runs_headless: false, reason: "load timeout …"` instead of wedging the run.
+
+The **mini-swe-agent** adapter drives the agent's Python API headlessly via
+`adapters/_mini_driver.py` (its `mini` CLI is interactive-only — crashes on a
+non-tty), run by the tool venv's own interpreter. Install once:
+`uv tool install mini-swe-agent`.
+
+Not yet done (see plan §6): an N-run matrix (currently n=1 per framework).
+**opencode** + **goose** adapters exist but opencode 1.x hangs on run-init in this
+env and the goose CLI isn't installed; an **OpenHands** adapter and a **harness**
+baseline entry are TODO.

agent-bench/adapters/__init__.py

@@ -0,0 +1 @@
+"""Framework adapters. Framework SDKs / CLIs are invoked only from here."""

agent-bench/adapters/_mini_driver.py

@@ -0,0 +1,112 @@
+"""Headless driver for mini-SWE-agent, run inside the tool's own interpreter.
+
+The `mini` CLI is interactive-only (it wires asyncio stdin readers and crashes
+on a non-tty), so the sanctioned headless path is the Python API documented in
+mini-SWE-agent's own `run/hello_world.py`. This script is that path: build a
+DefaultAgent against gx10 via LiteLLM, run the task to a terminal state, and
+emit a metrics JSON the adapter reads back.
+
+It imports `minisweagent` and so must run under the tool venv's python, NOT the
+bench's. The adapter locates that interpreter and shells out to this file. Keep
+it pure stdlib + minisweagent — it never imports bench code.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import yaml
+from minisweagent import package_dir
+from minisweagent.agents.default import DefaultAgent
+from minisweagent.environments.local import LocalEnvironment
+from minisweagent.exceptions import LimitsExceeded, TimeExceeded
+from minisweagent.models.litellm_model import LitellmModel
+
+# Per-command shell timeout inside the agent's environment. The build task only
+# writes files; generous enough that a stray `node`/test invocation won't trip it.
+ENV_COMMAND_TIMEOUT_S = 120
+
+
+def _sum_tokens(messages: list[dict[str, object]]) -> tuple[int, int]:
+    """Sum prompt/completion tokens across every model response in the trajectory.
+
+    Matches the cost scorer's convention (sum every usage block), so the
+    `artifacts` and `transcript` token sources agree.
+    """
+    prompt = completion = 0
+    for msg in messages:
+        extra = msg.get("extra")
+        response = extra.get("response") if isinstance(extra, dict) else None
+        usage = response.get("usage") if isinstance(response, dict) else None
+        if isinstance(usage, dict):
+            prompt += int(usage.get("prompt_tokens") or 0)
+            completion += int(usage.get("completion_tokens") or 0)
+    return prompt, completion
+
+
+def _render_transcript(messages: list[dict[str, object]]) -> str:
+    """Flatten the message log into a readable transcript for replay."""
+    lines: list[str] = []
+    for msg in messages:
+        role = msg.get("role", "?")
+        content = msg.get("content", "")
+        if isinstance(content, list):  # multimodal — keep the text parts
+            content = " ".join(str(p.get("text", "")) for p in content if isinstance(p, dict))
+        lines.append(f"### {role}\n{content}")
+    return "\n\n".join(lines)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--task-file", required=True)
+    ap.add_argument("--model", required=True)
+    ap.add_argument("--cwd", required=True)
+    ap.add_argument("--metrics-out", required=True)
+    ap.add_argument("--step-limit", type=int, default=60)
+    ap.add_argument("--temperature", type=float, default=0.0)
+    args = ap.parse_args()
+
+    task = Path(args.task_file).read_text()
+    agent_cfg = yaml.safe_load((package_dir / "config" / "mini.yaml").read_text())["agent"]
+    # Non-interactive: never prompt; bound by our step limit; cost limit off
+    # (litellm can't price the local model, so dollar-cost gating is meaningless).
+    agent_cfg.update(step_limit=args.step_limit, cost_limit=0.0, mode="yolo")
+
+    model = LitellmModel(
+        model_name=args.model,
+        model_kwargs={"temperature": args.temperature, "drop_params": True},
+    )
+    env = LocalEnvironment(cwd=args.cwd, timeout=ENV_COMMAND_TIMEOUT_S)
+    agent = DefaultAgent(model, env, **agent_cfg)
+
+    exit_status = "ok"
+    try:
+        agent.run(task)
+        exit_status = (agent.messages[-1].get("extra") or {}).get("exit_status", "ok")
+    except (LimitsExceeded, TimeExceeded) as exc:
+        exit_status = type(exc).__name__
+    except Exception as exc:  # a framework crash is a run outcome, not a driver bug
+        exit_status = f"error:{type(exc).__name__}: {exc}"
+
+    prompt_tokens, completion_tokens = _sum_tokens(agent.messages)
+    Path(args.metrics_out).write_text(
+        json.dumps(
+            {
+                "exit_status": exit_status,
+                "turns": agent.n_calls,
+                "cost": agent.cost,
+                "tokens_prompt": prompt_tokens,
+                "tokens_completion": completion_tokens,
+            }
+        )
+    )
+    # The transcript goes to stdout; the adapter captures it via the subprocess.
+    print(_render_transcript(agent.messages))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

agent-bench/adapters/_proc.py

@@ -0,0 +1,75 @@
+"""Subprocess + git helpers shared by every CLI-driven adapter.
+
+Adapters shell out to a framework's CLI under a fixed argv (no shell), capture
+stdout/stderr, and snapshot the resulting git diff as the run artifact.
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+
+TIMEOUT_RC = -999
+
+
+def run(
+    cmd: list[str], *, cwd: Path, timeout_s: int, env_extra: dict[str, str] | None = None
+) -> subprocess.CompletedProcess[str]:
+    """Run argv to completion; on timeout return a synthetic result with TIMEOUT_RC."""
+    env = {**os.environ, **(env_extra or {})}
+    try:
+        return subprocess.run(  # noqa: S603 — fixed argv, no shell; benchmark runner by design
+            cmd,
+            cwd=cwd,
+            env=env,
+            capture_output=True,
+            text=True,
+            timeout=timeout_s,
+            check=False,
+        )
+    except subprocess.TimeoutExpired as exc:
+        out = as_text(exc.stdout or "")
+        err = as_text(exc.stderr or "") + f"\n[bench] timed out after {timeout_s}s"
+        return subprocess.CompletedProcess(cmd, TIMEOUT_RC, out, err)
+
+
+def git(cwd: Path, *args: str) -> None:
+    run(["git", *args], cwd=cwd, timeout_s=60)
+
+
+def init_repo(workspace: Path) -> None:
+    """Fresh git repo with an empty baseline commit, so `git diff` captures all work."""
+    workspace.mkdir(parents=True, exist_ok=True)
+    git(workspace, "init", "-q")
+    git(workspace, "commit", "-q", "--allow-empty", "-m", "bench: empty baseline")
+
+
+# Framework scratch + bench-injected files that are NOT model build output.
+# Excluded from the snapshot so diff-based metrics (files, LOC) measure the game,
+# not aider's chat history / cache or the opencode provider config we write in.
+_DIFF_EXCLUDES = (
+    ".aider*",
+    ".opencode*",
+    "opencode.json",  # written by the opencode adapter, not the model
+    ".gitignore",  # aider auto-creates this
+    "node_modules",
+    "*.lock",
+    "bun.lock",
+)
+
+
+def snapshot_diff(workspace: Path) -> str:
+    excludes = [f":(exclude){pat}" for pat in _DIFF_EXCLUDES]
+    git(workspace, "add", "-A", "--", ".", *excludes)
+    return run(
+        ["git", "diff", "--cached", "--", ".", *excludes], cwd=workspace, timeout_s=60
+    ).stdout
+
+
+def transcript_of(cmd: list[str], proc: subprocess.CompletedProcess[str]) -> str:
+    return f"$ {' '.join(cmd)}\n\n[stdout]\n{proc.stdout}\n\n[stderr]\n{proc.stderr}"
+
+
+def as_text(value: str | bytes) -> str:
+    return value.decode(errors="replace") if isinstance(value, bytes) else value

agent-bench/adapters/aider.py

@@ -0,0 +1,54 @@
+"""Aider adapter — drives the `aider` CLI non-interactively against gx10.
+
+https://github.com/Aider-AI/aider
+"""
+
+from __future__ import annotations
+
+import time
+from pathlib import Path
+
+from ._proc import TIMEOUT_RC, init_repo, run, snapshot_diff, transcript_of
+from .base import Endpoint, RunArtifacts
+
+# Pin and record. Bump deliberately; the value lands in every result row's manifest.
+AIDER_VERSION = "0.86.2"
+
+
+class AiderAdapter:
+    name = "aider"
+    version = AIDER_VERSION
+
+    def prepare(self, workspace: Path) -> None:
+        init_repo(workspace)
+
+    def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> RunArtifacts:
+        env_extra = {
+            "OPENAI_API_BASE": gx10.base_url,
+            "OPENAI_API_KEY": gx10.api_key,
+        }
+        cmd = [
+            "aider",
+            "--model",
+            f"openai/{gx10.model}",
+            "--yes-always",  # non-interactive: accept edits/commits
+            "--no-auto-commits",  # we snapshot the diff ourselves
+            "--no-gitignore",
+            "--no-check-update",
+            "--no-show-model-warnings",
+            "--map-tokens",
+            "1024",
+            "--message",
+            spec,  # the whole task as a single instruction
+        ]
+        started = time.perf_counter()
+        proc = run(cmd, cwd=workspace, timeout_s=timeout_s, env_extra=env_extra)
+        duration = time.perf_counter() - started
+
+        return RunArtifacts(
+            exit_ok=proc.returncode == 0,
+            transcript=transcript_of(cmd, proc),
+            diff=snapshot_diff(workspace),
+            duration_s=duration,
+            extra={"returncode": proc.returncode, "timed_out": proc.returncode == TIMEOUT_RC},
+        )

agent-bench/adapters/base.py

@@ -0,0 +1,52 @@
+"""Adapter boundary.
+
+Mirrors the harness's model-adapter invariant: framework-specific code (CLIs,
+SDKs) lives only inside adapter modules. The runner speaks `Adapter` +
+`RunArtifacts` and nothing else.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Protocol, runtime_checkable
+
+
+@dataclass(frozen=True)
+class Endpoint:
+    """The shared gx10 vLLM endpoint handed to every framework."""
+
+    base_url: str
+    model: str
+    api_key: str = "gx10"
+    temperature: float = 0.0
+
+
+@dataclass
+class RunArtifacts:
+    """Everything one run produces, captured for scoring and replay."""
+
+    exit_ok: bool
+    transcript: str
+    diff: str
+    duration_s: float
+    tokens_prompt: int | None = None
+    tokens_completion: int | None = None
+    turns: int | None = None
+    extra: dict[str, object] = field(default_factory=dict)
+
+
+@runtime_checkable
+class Adapter(Protocol):
+    """One framework under test."""
+
+    name: str
+    version: str
+
+    def prepare(self, workspace: Path) -> None:
+        """Set up a fresh, isolated build directory (e.g. git init)."""
+        ...
+
+    def invoke(self, spec: str, gx10: Endpoint, workspace: Path, timeout_s: int) -> RunArtifacts:
+        """Drive the framework to build the spec; return captured artifacts."""
+        ...