test(prompt-ontology): ADR-022 fingerprint/scope/surface/capability harness (#1726) - #1744
Merged
Merged
Conversation
…arness (#1726) Add tests/test_prompt_ontology.py covering the four ADR-022 rule-4/5 assertions over the assembled loop-profile system prompt (release root + a fixture instance workspace): - Fingerprint once: each named rule (skip, mutation surface, branch, runner, identity, budget, final JSON) matches exactly one system block. Regexes were proven against the real release files, not the issue's draft table -- OPERATING.md's real branch sentence and IDENTITY.md's real self-declaration needed widened patterns. A deliberate-duplicate case proves the checker names both offending blocks. - AGENTS.md scope: mutation_policy.agents_md_scope_violations() against a target-shape fixture (compliant), an over-length fixture, a runtime-heading fixture, and the real 192-line instance fixture (xfail(strict), #1730 part 1 pending). - Surface parity: OPERATING.md's Mutation surface section against MUTATION_POLICY.render_bridge_surface_block(), reusing tests/test_operating_md.py's _section helper rather than duplicating it; a mutated copy fails with a diff; "Allowed targets:" is confirmed absent from every other release-root file and system block. - No guidance without capability: every tool name in the assembled prompt's prose is in EXECUTOR_TOOL_NAMES (excluding the loop's own suppressed builtins, mirroring bridge.py's _LOOP_EXCLUDED_SKILLS). Two things are intentionally xfail(strict) pending work this issue does not own: the user-message half of the fingerprint test (#1723 part b), and the AGENTS.md-scope check against the real instance fixture (#1730 part 1). A third, newly discovered while proving the capability assertion, is also xfail(strict): the search_memory status contract is restated in both OPERATING.md's Tools section and nanobot/agent/memory.py's MEMORY_SEARCH_POINTER, a real ADR-022 rule 5 violation this issue cannot fix without touching a protected or shared file it does not own. ADR-022's Test Contract section gets a one-line status note recording which checks pass today and which remain red by design.
…p the budget line out of tool-name prose #1742 removed build_task's rule literals, so the user-message assertion this harness carried as xfail(strict) now passes -- the marker is dropped rather than left to XPASS. The iteration-budget pointer added with that PR said the number is stated in 'your task message', and the bare word 'message' is the name of a tool the executor was never given, which the capability assertion (correctly) rejects; the sentence now points at the task section and the line by name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ozand
force-pushed
the
test/1726-prompt-ontology
branch
from
September 18, 2026 00:26
30b33bd to
1d7d837
Compare
Owner
Author
|
Rebased onto
38 passed, 2 xfailed locally ( |
3 tasks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
tests/test_prompt_ontology.py, the ADR-022 rule-4/5 test harness named in issue #1726: fingerprint-once, AGENTS.md scope, surface parity, and no-guidance-without-capability, all built over the assembled loop-profile system prompt (release root atIDENTITY.md/SOUL.md/goals.md/USER.md/OPERATING.md+ a fixture instance workspace).What passes today
never (checkout|push)does not matchOPERATING.md's real branch sentence, andYou are thedoes not matchIDENTITY.md's real self-declaration (it would also false-positive onSOUL.md's "You are an honest instrument"). Both were widened; see the module docstring for the exact greps.AGENTS.md) fails, naming bothoperatingandagentsand the count.mutation_policy.agents_md_scope_violations()against a compliant target-shape fixture, an over-length fixture, and a runtime-heading fixture.OPERATING.md's## Mutation surfacesection againstMUTATION_POLICY.render_bridge_surface_block()(reusingtests/test_operating_md.py's_sectionhelper rather than duplicating it), a mutated copy fails with a diff, andAllowed targets:is confirmed absent from every other release-root*.mdfile and every other assembled-prompt block.EXECUTOR_TOOL_NAMES, excluding the loop's own suppressed builtins (mirroringbridge.py's local_LOOP_EXCLUDED_SKILLS = ["weather", "tmux", "clawhub", "cron", "summarize", "github"]— without excluding these the fixture skills catalogue includescron, a false positive, not a real one).xfail(strict) — three cases, three reasons
test_fingerprint_user_message_only_budget_and_task—#1723 part (b) pending.build_task's user message still carries its own skip/surface/branch/runner/final-JSON literals; PR fix(bridge): remove build_task cycle-rule literals now that OPERATING.md is loaded (#1723 part b) #1742 is already open for this (fix(bridge): remove build_task cycle-rule literals now that OPERATING.md is loaded). Once it merges, this test flips to a hard failure until the marker is removed.test_agents_md_scope_violations_against_real_instance_fixture—#1730 part 1 pending. The real instanceAGENTS.mdfixture (tests/fixtures/instance_agents_md/AGENTS.md, refreshed viagh api repos/ozand/eeebot-self-evolving/contents/AGENTS.md --jq .content | base64 -d) is 192 lines with 8 runtime headings today, matching the issue's stated baseline exactly.test_search_memory_status_contract_appears_only_in_operating_tools_block— new finding, no issue filed yet. Proving the capability assertion surfaced that thesearch_memorystatus contract ("complete/partial/unavailable") is restated in two places inside the assembled prompt:OPERATING.md's## Toolssection andnanobot/agent/memory.py'sMemoryStore.MEMORY_SEARCH_POINTER(folded into the loop's memory block whenevermemory/index.mdis non-trivial). This is a real ADR-022 rule 5 violation, but fixing it means editing the protectedOPERATING.mdor the sharednanobot/agent/memory.py— both out of this issue's "do not touch" scope — so it'sxfail(strict)with the mechanism named in the reason, flagged here for a follow-up issue.Constraints honored
nanobot/agent/context.py,nanobot/runtime/bridge.py,OPERATING.md, or any other release file.tests/fixtures/instance_agents_md/AGENTS.mdandmemory_index.md(already present untracked from the prior dead attempt) verified byte-identical in content to a freshgh apipull before committing — no refresh needed.python -m pytest tests/test_prompt_ontology.py tests/test_operating_md.py tests/test_ontology_loader.py(Python 3.13, since 3.9/3.11/3.12 on this host lackloguru/pytest) — 26 passed, 3 xfailed, no regressions in the other two files.Test plan
python -m pytest tests/test_prompt_ontology.py tests/test_operating_md.py tests/test_ontology_loader.py— 26 passed, 3 xfailed