agent-bench: framework benchmark rig + aider & mini-SWE-agent comparands - #2
Open
PositiveControl wants to merge 11 commits into
Open
agent-bench: framework benchmark rig + aider & mini-SWE-agent comparands#2PositiveControl wants to merge 11 commits into
PositiveControl wants to merge 11 commits into
Conversation
Plan to benchmark 13 OSS agent frameworks against this harness by giving each the same autonomous build task (a GTA-style game) over one shared gx10 vLLM /v1 endpoint. Two tiers: turnkey coding agents (apples-to-apples) and build-your-own frameworks (scaffold-bound, flagged). Harness invariants carried over: adapter boundary per framework, append-first attributed results store, metrics-as-config. Repeatable (pinned + N-run distributions), recordable (full artifact capture + replay), adjustable (scorer plugins listed in metrics.yaml). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
End-to-end rig wired against the plan: framework-agnostic matrix runner, append-first SQLite+JSONL results store, metrics-as-config (metrics.yaml -> scorer registry), gx10 /v1/models health-gate with pinned-model check. - adapters/: Adapter protocol + aider adapter (drives `aider` non-interactively at gx10 via OPENAI_API_BASE; snapshots git diff as the artifact) - scorers/: builds (py_compile), runs_headless (SDL dummy boot window), feature_checklist (static M1..M7 probe) - spec/gta-spec.md: the one task input, 7 graded milestones, pygame stack - bench.yaml pinned to gx10 model Qwen/Qwen3-Coder-30B-A3B-Instruct - store.py append-first, schemaless per-run scores blob; report.py discovers metric keys from rows Smoke-tested: scorers -> store -> report path runs; ruff clean. Remaining adapters (OpenHands/OpenCode/Goose/mini-SWE) and cost/process scorers are TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds two more turnkey-agent adapters and the first cost metric.
- adapters/_proc.py: extracted shared subprocess + git helpers (run, init_repo,
snapshot_diff, transcript_of); aider refactored onto it (no behavior change)
- adapters/opencode.py: `opencode run --model gx10/<model> --format json`;
gx10 wired as a custom @ai-sdk/openai-compatible provider written to a
per-workspace opencode.json
- adapters/goose.py: `goose run --no-session --quiet -t <spec>`; gx10 base_url
decomposed into OPENAI_HOST + OPENAI_BASE_PATH; keyring disabled, key from env
- scorers/cost.py: tokens_{prompt,completion,total}, wall_clock_s, tokens_per_s,
cost_usd (+cost_source). Tokens from artifact fields, else transcript regex
fallback (JSON usage + aider k/M lines). $ priced via BENCH_PRICE_*_PER_MTOK
env (default $0 for self-hosted gx10)
- runner registries + bench.yaml + metrics.yaml updated
Smoke-tested: registries load (3 adapters, 4 scorers), goose endpoint split and
opencode provider-config shape verified; ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
opencode adapter now parses the JSONL event stream and populates RunArtifacts.tokens_prompt / tokens_completion / turns directly, so the cost scorer reports cost_source="artifacts" (exact) instead of falling back to the transcript regex. - _parse_usage sums step_finish events: part.tokens.input -> prompt, output+reasoning -> completion; cache read/write + opencode's own part.cost captured in extra. turns = step_finish count. - Per-step semantics (AI-SDK convention; undocumented by opencode) — per-step breakdown stashed in extra.per_step_tokens for audit; one-line switch to max if a release turns out to report cumulative totals. - Tolerant reader: JSONL or single top-level array, skips non-JSON lines. Verified: multi-event sample -> prompt 2000 / completion 550 / turns 2; cost scorer flips to cost_source="artifacts"; array + empty + garbage inputs handled; ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fifth scorer: how the agent worked, not whether it passed. - Deterministic from the unified diff: files_touched, files_created, lines_added/removed, diff_loc. - Soft signals from artifacts, reported as 0 (never guessed) when the framework doesn't surface them: turns (from artifacts.turns — exact for opencode), tool_calls (counts opencode `"type":"tool_use"` JSONL markers), tracebacks. - Registered in runner.SCORERS + metrics.yaml. Verified: sample 2-file diff -> files 2 / created 1 / +4 -1 / loc 5; turns 2, tool_calls 2, tracebacks 1; 5 scorers load; ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Benchmark product is now a top-down GTA2-style web game (index.html + game.js, vanilla JS, canvas 2D) instead of pygame. Validated end-to-end against live gx10 (gx10-5fb9, vLLM 0.21.0, Qwen/Qwen3-Coder-30B-A3B-Instruct) via the aider adapter. Spec + scorers reworked for the web stack: - spec/gta-spec.md: self-contained HTML/CSS/JS GTA2 spec, milestones M1..M8, hard tech constraints (one requestAnimationFrame loop, no modules/classes, canvas-2D only, zero console errors). - builds: node --check over every .js + index.html has <canvas>+<script>. - runs_headless: Playwright headless chromium — loads index.html, asserts loads_clean / loop_alive (rAF ticked) / renders (canvas non-blank). 10s goto timeout fast-fails a page that blocks the main thread (reports the init-loop case). - feature_checklist: static regex probe of M1..M8 over game.js + index.html. - _proc.snapshot_diff: exclude framework cruft (.aider*, injected opencode.json, .gitignore, node_modules, lockfiles) so diff LOC measures the game, not scratch. - bench.yaml -> live gx10 endpoint; AIDER_VERSION 0.86.2. Each run builds from scratch in its own wiped results/<fw>/<run>/workspace; the reference scratch/workspace is never touched (read-only oracle: scores 8/8 + runs). Live aider run: builds true, 6/8 milestones, 2 files / 436 LOC (cruft excluded). runs_headless correctly False — the model emitted an unbounded init loop (game.js while(!validPosition)) that hangs the browser. Exactly the runtime signal static probes miss. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RunsHeadlessScorer._find_entry returned whatever path it was handed. Path.as_uri() rejects relative paths, so a relative workspace raised "relative path can't be expressed as a file URI" — caught by the broad except and reported as a bogus "playwright error", masking the real load-timeout verdict. Resolve the entry to absolute so a hung page (e.g. an unbounded init loop) records cleanly as runs_headless:false with reason "load timeout (page blocks main thread)". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gx10 now serves the FP8 quant; repin the model id so the health gate passes (was aborting on Qwen3-Coder-30B-A3B-Instruct vs ...-FP8). Validated the full loop end-to-end: one live aider run -> all five scorers -> report.py renders a clean single-schema distribution (39.2 tok/s on FP8, 6/8 milestones, runs_headless correctly false on aider's unbounded init-loop hang). Stale mixed-schema rows from the pygame->web pivot were cleared from the store first. README status rewritten to reflect: rig proven on one framework; remaining work is an N-run matrix + a second comparand (mini-SWE next). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second comparand. mini-SWE-agent's `mini` CLI is interactive-only — it wires asyncio stdin readers and crashes on a non-tty (OSError [Errno 22] in _add_reader), even with --yolo/--agent-class default. So drive its documented Python API (DefaultAgent + LocalEnvironment + LitellmModel, per the project's own run/hello_world.py) through a thin driver run by the tool venv's interpreter: - adapters/_mini_driver.py: builds the agent rooted at the workspace, runs the task to a terminal state (Submitted / LimitsExceeded / TimeExceeded), emits a metrics JSON (turns=n_calls, summed prompt/completion tokens, cost, exit_status) + a readable transcript. - adapters/mini_swe.py: locates the tool venv python via the `mini` shim's shebang (stable, no uv-cache hash; falls back to `uv tool run`), shells the driver through the shared _proc helpers, and populates RunArtifacts.tokens_*/turns so the cost + process scorers read source="artifacts" rather than scraping the transcript. minisweagent stays out of the bench's deps — it's an external CLI on PATH (uv tool install mini-swe-agent), same convention as aider. Validated end-to-end against gx10 FP8: builds + runs_headless both pass, 7/8 milestones, 17 turns, 140k tokens, ~926s. First real cross-framework contrast vs aider (1-shot, 82s, 6/8, runs_headless false on its init-loop hang) — exactly the comparison the bench exists to produce. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prep for merging agent-bench into main. The per-commit hooks scoped mypy to src/, so the bench tree was untyped and untested. Close both gaps: - Type-clean agent-bench under strict mypy and extend the mypy hook to `src agent-bench`. Fixes: typed dict/list generics in the mini driver and metrics reader, an explicit numeric coercion in report.py (no more float(object)), a robust isinstance-guarded token sum, and a typed _opt_int accessor so RunArtifacts fields stay int | None. Drops a stale type: ignore. - Register a minisweagent missing-imports override (it's an external CLI, never a repo dependency — same rationale as the playwright override). - Add tests/agent_bench/: unit coverage for the deterministic surface — feature_checklist milestone probe, cost token accounting (artifacts + transcript-scrape paths), process diff/transcript counters, and the SQLite store round-trip. builds gets a node-guarded smoke test. The external-tool scorers (node, Playwright, gx10) stay covered by the runner end-to-end. A local conftest puts the agent-bench dir on path. Gates green: ruff, ruff format, mypy (src + agent-bench, 215 files), pytest (3905 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pre-existing flake, surfaced when running the full suite under the pre-push hook: pre-commit's pre-push exports GIT_DIR / GIT_WORK_TREE / GIT_INDEX_FILE, so workspace_is_gitignored's `git check-ignore` resolved against the repo running the hook instead of each test's tmp_path, flipping the gitignored/tracked assertions. test_git_tools.py and test_driver_cli.py already guard against this; the bd_warning_filter tests never did. Add an autouse fixture that strips GIT_* per test (monkeypatch.delenv), mirroring tests/test_git_tools.py::_clean_env. Verified: the file's 13 tests pass both bare and under a simulated GIT_DIR/GIT_WORK_TREE env. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds agent-bench — a local-first rig that gives each agent framework the same autonomous build task (a top-down HTML5-canvas web game) against one shared gx10 vLLM endpoint, then scores the result. Framework-agnostic runner + pluggable adapters + pluggable scorers + append-first results store.
Comparands (live, validated end-to-end against gx10
Qwen3-Coder-30B-A3B-Instruct-FP8)The aider row is a true-positive failure the rig caught (unbounded
while(!validPosition)spawn loop).opencode+gooseadapters exist but aren't runnable in this env yet (opencode 1.x hangs on run-init; goose CLI absent).Scorers
builds(node --check) ·runs_headless(Playwright chromium) ·feature_checklist(M1–M8 regex) ·cost(tokens/wall-clock/$) ·process(diff LOC / turns / tool calls / tracebacks).Notable engineering
_mini_driver.py(itsminiCLI is interactive-only — crashes on a non-tty), run by the tool venv's interpreter. minisweagent stays out of repo deps (external CLI, like aider).runs_headless: false, reason: "load timeout …"instead of wedging the run.Standards / gates
src agent-bench.tests/agent_bench/) cover the deterministic surface (scorers + store round-trip); external-tool scorers covered by the runner e2e.test_driver_bd_warning_filter.pynow strips inheritedGIT_*env (the pre-push hook exportsGIT_DIR/GIT_WORK_TREE), mirroringtest_git_tools.py.🤖 Generated with Claude Code