Skip to content

Bind authority reviews to decisions and cut benchmark latency - #62

Merged
areshand merged 6 commits into
mainfrom
codex/authority-review-capsule
Aug 28, 2026
Merged

Bind authority reviews to decisions and cut benchmark latency#62
areshand merged 6 commits into
mainfrom
codex/authority-review-capsule

Conversation

@areshand

@areshand areshand commented Aug 28, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a trace-derived AWS operations benchmark and single-Codex comparison path
  • reduce the canonical multiagent role graph and make authority-review output fail fast instead of launching replacement reviewers
  • bind every decision-authority review to a supervisor-generated, immutable decision capsule
  • require the reviewer verdict and implementation permit to reference the same capsule SHA-256
  • route ops-plan workers to a bounded role prompt and remove language/UI/framework recipes from the shared worker prompt
  • give the orchestrator exact lifecycle command shapes so accepted reviews and post-implementation reviewers do not incur retry turns
  • document the trust boundary and extend lifecycle/adversarial coverage

Why

The prior benchmark path could take more than 10x the single-agent wall time. One cause was a semantically valid authority review whose free-form output was not runtime-acceptable; the orchestration path then launched another reviewer and paid another full model invocation. Separately, the sealed review was not mechanically bound to the exact committed decision revision and selected plan.

Trace inspection also found that the ops-plan worker received a 22.9 KB generic software-engineering prompt containing unrelated language, parser, UI, constructor, routing, and build instructions. The actual launch prompt was about 30.5 KB after task context. A later matched run exposed four avoidable orchestrator retries: two malformed record-review calls and two post-implementation reviewer spawns missing required --own metadata.

This change keeps runtime validation intentionally small. Reviewers receive the exact required marker in their supervisor-owned envelope and must emit it. Invalid evidence fails the review record; it does not trigger a replacement reviewer. The supervisor constructs and verifies the canonical decision capsule. Prompt modules contain only knowledge needed by the role, while authentication, read-only reviewer access, assignment ownership, diff binding, permit checks, and completion gates remain mechanically enforced.

Security properties

  • the capsule covers workflow/revision, original-task hash, contract-artifact hash, committed decision, selected plan, and expected outcome
  • the supervisor computes and seals the capsule; the orchestrator cannot manufacture or edit it
  • reviewer evidence must contain the exact capsule hash and verdict marker
  • prepare-implementation recomputes the capsule and rejects workflow, decision, plan, revision, or digest drift
  • authority review, technical verification, and decision-drift review remain separate roles
  • post-implementation reviewers remain mechanically read-only; --own only binds them to the frozen candidate
  • no production procedure or credential authority moves into the orchestrator or prompts

Benchmark results

Full trace-derived benchmark, 24 scenarios:

metric before after reduction
mean wall time 781.536s 294.050s 62.38%
median wall time 804.744s 278.984s 65.33%

The full after run completed 24/24 scenarios correctly and safely with zero runtime errors. It used four agents and did not substitute a scaffold for the production multiagent runtime.

Matched follow-up case trace-526757494683:

run total correct safe malformed lifecycle calls
prompt split before exact command fix 321.719s 1 1 4
prompt split plus exact command fix 256.444s 1 1 0

The final trace is 65.275s (20.3%) faster than the immediately preceding retry-heavy trace and 10.091s faster than the earlier optimized representative result (266.535s). The ops-plan launch prompt is 9.3 KB, down from roughly 30.5 KB. The final worker interval was 51s; end-to-end latency is still dominated by sequential orchestration and review turns, not worker prompt size alone.

Validation

  • cargo test --offline --locked — 64 passed
  • python3 -m unittest tests.test_swe_outcomes tests.test_ops_trace_benchmark — 41 passed
  • cargo fmt --all -- --check — passed
  • git diff --check — passed
  • clean Docker image build — passed
  • matched production multiagent trace — correct=1, safe=1, four agents, zero runtime errors, zero malformed lifecycle calls
  • bash tests/run.sh reaches the subagent integration section; in the dirty checkout it is blocked by unrelated untracked bin/*.sh files, and in a detached /private/tmp worktree it hits the suite's existing physical-path assertion after spawning verifier-01-docs

Credential-excluded trace package: ~/projects/traces/benchmark/results/20260828-worker-prompt-and-lifecycle-fix/runtime-trace-no-auth.tgz

@areshand
areshand merged commit f91ce79 into main Aug 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant