Bind authority reviews to decisions and cut benchmark latency - #62
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
The prior benchmark path could take more than 10x the single-agent wall time. One cause was a semantically valid authority review whose free-form output was not runtime-acceptable; the orchestration path then launched another reviewer and paid another full model invocation. Separately, the sealed review was not mechanically bound to the exact committed decision revision and selected plan.
Trace inspection also found that the ops-plan worker received a 22.9 KB generic software-engineering prompt containing unrelated language, parser, UI, constructor, routing, and build instructions. The actual launch prompt was about 30.5 KB after task context. A later matched run exposed four avoidable orchestrator retries: two malformed
record-reviewcalls and two post-implementation reviewer spawns missing required--ownmetadata.This change keeps runtime validation intentionally small. Reviewers receive the exact required marker in their supervisor-owned envelope and must emit it. Invalid evidence fails the review record; it does not trigger a replacement reviewer. The supervisor constructs and verifies the canonical decision capsule. Prompt modules contain only knowledge needed by the role, while authentication, read-only reviewer access, assignment ownership, diff binding, permit checks, and completion gates remain mechanically enforced.
Security properties
prepare-implementationrecomputes the capsule and rejects workflow, decision, plan, revision, or digest drift--ownonly binds them to the frozen candidateBenchmark results
Full trace-derived benchmark, 24 scenarios:
The full after run completed 24/24 scenarios correctly and safely with zero runtime errors. It used four agents and did not substitute a scaffold for the production multiagent runtime.
Matched follow-up case
trace-526757494683:The final trace is 65.275s (20.3%) faster than the immediately preceding retry-heavy trace and 10.091s faster than the earlier optimized representative result (266.535s). The ops-plan launch prompt is 9.3 KB, down from roughly 30.5 KB. The final worker interval was 51s; end-to-end latency is still dominated by sequential orchestration and review turns, not worker prompt size alone.
Validation
cargo test --offline --locked— 64 passedpython3 -m unittest tests.test_swe_outcomes tests.test_ops_trace_benchmark— 41 passedcargo fmt --all -- --check— passedgit diff --check— passedbash tests/run.shreaches the subagent integration section; in the dirty checkout it is blocked by unrelated untrackedbin/*.shfiles, and in a detached/private/tmpworktree it hits the suite's existing physical-path assertion after spawningverifier-01-docsCredential-excluded trace package:
~/projects/traces/benchmark/results/20260828-worker-prompt-and-lifecycle-fix/runtime-trace-no-auth.tgz