Harden the user-driven loop with the loop-engineering rules - #3
Draft
drmikecrowe wants to merge 16 commits into
Draft
Harden the user-driven loop with the loop-engineering rules#3drmikecrowe wants to merge 16 commits into
drmikecrowe wants to merge 16 commits into
Conversation
The final gauntlet run and the EVIDENCE draft were the loop's last two self-report surfaces: the author executed the run and narrated its result. Two fresh-context briefs close that, the same maker-and-checker split the spec-intent and adversary agents already apply. The runner executes the entry point once and transcribes a per-layer verdict; it fixes nothing, and a layer with no log is a failed row whatever the exit code says. The scribe drafts EVIDENCE from artifacts alone and holds no Bash, so it can transcribe a number but never produce one; the author's claims enter through FACTS.md as labeled assertions that never upgrade a status. Both required at Tier 3, optional at Tier 2; author-run remains the fallback, recorded in EVIDENCE as a downgrade like the brief path.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
EVIDENCE is model-written and the runner agent only transcribes; nothing non-model attested that the run happened, on this tree, after the last edit. The entry point's exit trap now writes that record on the green and red paths alike, and the exit code distinguishes a failed layer from a violated orchestration contract from a crash. The runner reads the record verbatim and fails a run whose record is absent or mis-bound; the scribe copies the source state from it; the report's consistency check gains a Record line. Loop-engineering VE-9, VE-11, CO-9, CO-10.
The adversary reads potentially hostile repo content with a live tool set, and the spec reviewer reads text the requester controls; neither brief said what to do with a directive found inside it. Now both: an embedded instruction is a finding to report, never an order to follow. Loop-engineering EX-8.
The adversary's 10-call budget was a limit with no consequence: an uncounted or over-budget round could still be averaged into confidence. Now it is a failed round, recorded and rerun. And the last permitted round fixes and re-briefs only the blocking finding, so it cannot be spent on cosmetics while the blocker stands. Loop-engineering CO-4, CO-13.
Every checker in the gauntlet must show a red before its green counts; the entry point that runs them all was the one checker exempt. Two one-off negative controls close that: an absent layer must redden the audit, and a failing layer must redden the exit and the record. Loop-engineering VE-13.
The prose is the mechanism, so a change that alters what the gauntlet accepts ships with the case that fails without it. Loop-engineering DR-4.
Two review rounds failing the same way prove the fix changed nothing; a third round spends budget to learn it again. Compare what failed, not its wording, and escalate to the human with both attempts. Loop-engineering CO-3, read for a human-driven loop.
drmikecrowe
changed the title
A spec is complete only against the kind of request it answers: a bug without a reproduction cannot produce its RED test, a feature without the problem stated apart from the proposal invites building the wrong thing, a UI change without a visual expectation is unfalsifiable. The reviewer classifies first (bug / ui / feature), holds the spec to that kind's required substance, and reports the kind so the author can dispute the classification. Adapted from the ticket-enhancer judge in the reliable-agentic-lab workshop.
Two failure modes from running loops on a schedule: redoing finished work because the task was named again, and silently retrying past a standing escalation only a human can clear. Both are stops, and both are narrated — silence on a wake reads as a hang. Adapted from the reliable-agentic-lab enhancer loop's state check and sticky needs-human label; MAST's unaware-of-done mode.
Post only to a destination the human already made; mark this skill's own comments so a later run never answers itself; never filter by author, which would also drop the human's approval. Adapted from the reliable-agentic-lab enhancer loop's comment marker and never-create-an-issue rules.
The consistency lines and the stagnation compare are set membership and string matching, and a decision computed that way cannot be talked past. Where a script exists, the script's answer is the answer. The stance is the enhancer loop's: ready comes from check_fields.py, stop from check_stop.py, never from the model's own judgment.
Where the request and the codebase do not settle a value, an explicit proposed value the reviewer can veto beats a hole discovered mid-build, and a value read from the code beats one composed. The enhancer doer's investigate-before-you-invent rule, applied to spec drafting.
Construction was one dense bullet plus fallback tables; now it is six steps. The new piece is the output contract: a layer's command is wired in only if it exits nonzero on violation, writes its own log, and emits the number its EVIDENCE row will cite — changed-line coverage must gate and fail, not print a global percentage. A tool that cannot meet the contract goes to the human, never in as a report-only step.
The tooling exchange with the human becomes a structured artifact: one row per layer with the pinned tool, the command, and the number EVIDENCE gets from it. Approving the spec approves the rows not struck; a struck row is UNAVAILABLE, never substituted.
The builder writes the entry point, so the instrument was the one piece of the trust chain no fresh eyes checked. A read-only fresh-context agent now certifies it against the approved table before first use: wiring, fail-closed traits, the output contract, the commissioning reds, and what the certification binds to. It runs nothing and fixes nothing; certification voids on any later edit. EVIDENCE gains a Gauntlet commissioned field.
…ry point Two closures for the instrument-authored-by-the-author gap: the scribe fails the report when the runner's executed command differs from the Entry point field, and the adversary's hunt list opens with the entry point script whenever the diff touches it — every green number downstream rests on it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens the user-driven loop with the applicable rules from the loop-engineering design doc. Based directly on ca70137; depends on nothing after it. Seven commits, one concern each.
The two agents (259b13d)
agents/old-coder-gauntlet.md— a fresh-context runner executes the final gauntlet run: four inputs, one entry-point invocation, per-layer verdict in the closed five-status vocabulary. It fixes nothing; a layer with no log is FAILED whatever the exit code or record claims.agents/old-coder-evidence.md— a fresh-context scribe drafts EVIDENCE from artifacts alone, holds no Bash (it can transcribe a number, never produce one), and writes absent evidence as failing rows. Author claims enter viaFACTS.mdas labeled assertions that never upgrade a status.The hardening (VE/CO/EX/DR rule ids from the doc)
Gauntlet construction, specified
agents/old-coder-gauntlet-verifier(EX-3 at the instrument). The builder writes the entry point, so a read-only fresh-context agent certifies it before first use: wiring against the approved table, fail-closed traits, the output contract, the commissioning reds, and what the certification binds to. Voids on any later edit; EVIDENCE gains aGauntlet commissioned:field.Entry point:header, and the adversary's hunt list opens with the entry-point script whenever the diff touches it.Deliberately not adopted
Outer-loop automation (iteration counting, durable locked state, scheduled drift evaluation) — the human is this loop's controller; those rules activate the day the skill runs on a trigger instead of an ask.
What this does not claim
The entry point's own exit remains the gate; both agents report and decide nothing. The scribe is a model: it removes author-correlation only, and EVIDENCE names who ran and who drafted so a reader can weigh the claim.
Supersedes #2.