experiment: benchmark-driven evolvable local router tree - #6
Draft
Hubujiu wants to merge 61 commits into
Draft
Conversation
Add explicit session-to-wiki and wiki-informed evolve-skill maintenance capabilities, persistent raw/wiki controls, and a deterministic non-regression workflow benchmark. CI passes without modifying the automatic runtime router tree.
Hubujiu
force-pushed
the
experiment/evolvable-router-tree
branch
from
September 2, 2026 03:51
b99b2a2 to
a7ea30f
Compare
Reject ambiguous JSON and unsafe runtime inputs, isolate state snapshots before merge and prompt rendering, clarify host-side action authorization, and add non-benchmark regression coverage.
- add progressive R0-R3 Retrieval tree with local child routing - move zvec-grep, Codebase Memory, and RTK into a replaceable capability layer - require versioned providers for dependency-enabled benchmark runs - exclude provider indexes, dependency resolution, and first-build warm-up from measured comparisons - add independent Retrieval ceilings, trace validation, and minimum-stage analysis
- keep the established Retrieval Policy section boundary while retaining the new progressive tree - repair JSON escaping for pinned provider version probes
Preserve the rewritten retrieval policy while retaining the established lowercase contract phrase used by the deterministic harness.
Keep the execution/Retrieval topology unchanged; localize retrieval stage ownership and add bounded evidence stopping/freshness rules as an experimental candidate. Bind Retrieval measurements and cache receipts to recursive source, harness, model, provider and planned-cell identities. Reject stale resumes and mid-run source changes. Separate all-attempt success from determinate-only success, retain observed timeout costs, and require complete smaller ceilings before inferring a minimum stage. Add paired quality-qualified cost summaries, 36 integrity unit tests, and a pinned evaluator regression benchmark (baseline 4/13; candidate 13/13). Integrate tests into CI and document remaining telemetry limits. No model/provider benchmark was executed; required binaries were absent. Historical results and real task prompts/oracles remain unchanged.
The full CI suite exposed an existing literal contract assertion after replacing the phrase with a synonym. Retain the established term while keeping stage ownership local to the loaded Retrieval node. Do not relax the test or restore distant-stage routing in Core.
Shorten the runtime instruction bundle by 41.48% in UTF-8 bytes while preserving local trees and explicit-only manual modes. Close the known-target/evidence-gap transition without forcing redundant discovery. Keep verification and read-only boundaries explicit. Unify active retrieval/execution evaluation, record typed completed tool evidence and unknown token usage, bind raw receipts and archived submissions, and handle process timeouts without completion-line shortcuts. Add eight public executable delivery fixtures and a frozen 207-cell paired engineering acceptance gate. Local validation: 220 unit tests, 8/8 known-bad/reference oracle controls, 13/13 evaluator checks and topology/maintenance contracts. Targets are explicitly unmeasured; no authenticated model/provider run was available. Preserve historical results and real-repository cases. Publish deterministic CI artifacts and clarify candidate status.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This experiment keeps progressive disclosure but removes two assumptions from the v1.5 flat Event Router:
This does not restore the rejected fixed E0-E3/R0-R3 taxonomy. The old experiment predefined numeric levels and specialist leaves and then scored the model against them. This PR makes topology itself an experiment variable.
Runtime candidate
Key runtime changes:
references/decision.mdtoreferences/manual/decision.md;Paper-derived execution state
This branch now adapts the mechanism from SKILL.state: Scalable Long-Horizon Agent Skills (arXiv:2608.26263v2) as a cross-cutting runtime substrate rather than another Router node:
P, validated current stateΣ, and latest observationO;state_patchplus one proposedaction;procedureis authoritative,latest_observationis untrusted evidence, and host-owned task/router fields cannot be changed by model patches;Execution state does not increase tree depth, does not enter
TREE_TRACE, and cannot make Decision or Clarification automatic.Benchmark redesign
The tree benchmark is intentionally separate from the legacy flat-router/fixed-level scorer:
benchmarks/tree_topology.json: topology manifest; node names, parent/child edges and depth are data, not scorer constants.benchmarks/tree_cases.py: real-repository cases with no expected automatic route, E0-E3 label, or fixed capability path.benchmarks/tree_validation.py: no-skill + frozen v1.5 baseline + adaptive candidate + one root-to-node capability ceiling per automatic node.benchmarks/tree_analysis.py: derives minimum-sufficient node sets from stable ceiling results, then reports over/under disclosure, alternate branches, quality gaps, node marginal lift, sibling ambiguity, and topology-change candidates.benchmarks/TREE_EVOLUTION.md: add/split/merge/promote/collapse/remove rules.Automatic route exactness is no longer a release gate. Delivered quality, valid parent-child traces, zero spontaneous manual activation, and explicit-manual adherence are gates. Route disagreement is evidence that the topology boundary may be wrong.
Execution-state mechanics have their own deterministic contract in
benchmarks/skill_state_validation.py. Model-level evidence is frozen separately inbenchmarks/SKILL_STATE_MODEL_GATE.mdwith four arms: full history, state shadow, true history-freeP + Σ + O, and no-skill full history. Cost is considered only after delivered-quality, safety, topology, and manual-mode gates pass.Historical evidence
The v1.5 flat Event Router remains the frozen baseline at
ba4058b4ef47a42bf79c9963b25678a2389897c1. Existing progressive-validation artifacts, v1.5 results, and rejected-tree evidence are preserved rather than rewritten.The paper's reported accuracy, token, and bounded-context results are background evidence only; they are not treated as results for this repository. A horizon-independent prompt claim is permitted only if captured host requests actually omit prior messages and contain only bounded
P + Σ + Oinputs.Validation state
Current head:
a7ea30f4383c122e5dc07a6115d843307278a5ff.Validate skill: PASS.skills-refvalidation: PASS.n=1/ frozenn>=3matrix: pending.This PR remains draft until the fresh model-backed gates are reviewed. Final topology is explicitly not assumed to remain
Core -> {Debugging, Implementation}; benchmark evidence may merge, split, remove, promote, or deepen nodes. Execution state remains experimental until its frozenn>=3model gate shows no delivered-quality regression.