spec: task-grain experiment — paired double-run + sde-telemetry - #72
spec: task-grain experiment — paired double-run + sde-telemetry#72SSFSKIM wants to merge 1 commit into
Conversation
…y collector Pre-registered experiment testing whether writing-plans' task unit should move from smallest-reviewable to largest-ownable. Two arms on one real feature (arkho#9 recommended): baseline doctrine vs coarse-grain treatment (3-5 tasks, interface-frontier boundaries, ~500-line diff cap), single variable, blinded comparative judging, decision rule fixed before any run. scripts/sde-telemetry mines session transcripts (+ subagents/) for per-dispatch role, token, and duration accounting; smoke-tested on a real 8-task run (37 dispatches classified, fixers outnumbered executors 11:8).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e70ca0aa85
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| controller_totals, first, last = scan_transcript(session_jsonl) | ||
| if first and last: | ||
| span = last - first | ||
| print(f"wall-clock: {first.isoformat()} -> {last.isoformat()} ({span})") |
There was a problem hiding this comment.
Each arm first authors its plan and only then starts SDE, but this computes the span from the first to last event in the entire controller transcript. Planning time and any idle time before the first executor dispatch therefore enter a metric whose pre-registered boundary is the first executor dispatch; because the adoption threshold is only 25%, those unrelated periods can determine the result. Locate the first executor dispatch and measure from that phase boundary instead.
Useful? React with 👍 / 👎.
| duration = (a_last - a_first) if a_first and a_last else None | ||
| dispatches.append((role, description, duration)) |
There was a problem hiding this comment.
The collector computes each subagent duration only for the dispatch list and never sums those durations or derives controller turn time. Consequently it cannot produce the experiment's primary metric, defined in the added spec as dispatch durations plus controller turn time; the only aggregate it prints is the explicitly secondary transcript span. Aggregate these durations with controller active time before using the run to make the adoption decision.
Useful? React with 👍 / 👎.
| message = entry.get("message") | ||
| if isinstance(message, dict) and isinstance(message.get("usage"), dict): | ||
| usage = message["usage"] |
There was a problem hiding this comment.
This counts only Claude transcript message.usage, while the prescribed arm runs also include Codex adversarial plan review and the final Codex whole-branch review, which execute through the companion rather than as native subagents and thus do not appear in these message or sidecar totals. Since the decision rule permits adoption based on a 25% reduction in total tokens alone, omitting those potentially arm-dependent review tokens can produce a decision from partial costs; capture that external usage or explicitly exclude token totals from the decision rule.
Useful? React with 👍 / 👎.
| agent_jsonl = meta_path.replace(".meta.json", ".jsonl") | ||
| if os.path.exists(agent_jsonl): |
There was a problem hiding this comment.
When a .meta.json survives but its corresponding agent transcript is missing—for example after copying only the session JSONL and metadata or after partial transcript retention—the dispatch is counted while its tokens and duration are silently omitted. A wholly missing sibling subagents directory similarly produces a plausible-looking zero-dispatch report. Because these incomplete totals feed the experiment's adoption threshold, fail or emit a prominent incomplete-data status instead of silently skipping the missing transcript.
Useful? React with 👍 / 👎.
| ROLE_PATTERNS = [ | ||
| (re.compile(r"implement task", re.I), "executor"), | ||
| (re.compile(r"review task", re.I), "task-reviewer"), | ||
| (re.compile(r"\bfix", re.I), "fixer"), | ||
| ] |
There was a problem hiding this comment.
The fixer pattern matches the prefix fix rather than a complete role word, so unrelated descriptions such as Inspect test fixtures or Investigate fixture data are counted as fixer dispatches. Test-oriented implementation sessions are likely to use such descriptions, and the experiment explicitly treats fixer count as evidence about review-loop overhead. Restrict this classifier to actual fixer wording instead of any word beginning with fix.
Useful? React with 👍 / 👎.
1 participant
What problem are you trying to solve?
The controlled track's task unit is defined as the smallest reviewable unit, producing 8–12 tasks per mid-size feature and paying worker re-orientation cost 20–30× per feature. Real-run telemetry (session b380b6da, the 8-task PR #43 run) shows 37 dispatches with fixers outnumbering executors 11:8. Before Task Right-Sizing may be redefined, the repo's own bar demands eval evidence.
What does this PR change?
docs/doperpowers/specs/2026-08-19-task-grain-experiment.md— pre-registered paired experiment: baseline doctrine vs a coarse-grain treatment (largest-ownable tasks, interface-frontier boundaries, ~500-line diff cap, target 3–5 tasks) on one real feature, blinded comparative judging, decision rule + failure routing fixed before any run. Wave-cadence (V2) held as contingent follow-up.scripts/sde-telemetry— post-hoc collector over session transcripts (+subagents/): per-dispatch role/token/duration accounting. Smoke-tested against two real sessions.No skill content changes; no version bump (nothing plugin-behavioral changes until the experiment produces evidence).
What alternatives did you consider?
Historical-baseline comparison (confounded by feature difficulty), synthetic eval-harness runs (measure compliance, not pipeline economics), 3-arm design (3× cost; V1 subsumes review-count economics) — all recorded with rationale in the spec's Decision Log.
Next step
Human go required before Phase 0: confirms the feature (arkho#9 recommended) and the double-implementation spend. No dispatch before approval.