Skip to content

test(prompt-ontology): ADR-022 fingerprint/scope/surface/capability harness (#1726) - #1744

Merged
ozand merged 3 commits into
mainfrom
test/1726-prompt-ontology
Sep 18, 2026
Merged

ozand merged 3 commits into
mainfrom
test/1726-prompt-ontology

Conversation

@ozand

@ozand ozand commented Sep 18, 2026

Copy link
Copy Markdown
Owner

Summary

Adds tests/test_prompt_ontology.py, the ADR-022 rule-4/5 test harness named in issue #1726: fingerprint-once, AGENTS.md scope, surface parity, and no-guidance-without-capability, all built over the assembled loop-profile system prompt (release root at IDENTITY.md/SOUL.md/goals.md/USER.md/OPERATING.md + a fixture instance workspace).

What passes today

  • Fingerprint once (system-block half): each of skip / mutation surface / branch / runner / identity / budget / final-JSON matches exactly one system block. Regexes were proven against the real release files, not copied verbatim from the issue's draft table — the issue's never (checkout|push) does not match OPERATING.md's real branch sentence, and You are the does not match IDENTITY.md's real self-declaration (it would also false-positive on SOUL.md's "You are an honest instrument"). Both were widened; see the module docstring for the exact greps.
  • A deliberate-duplicate fixture (skip rule restated in AGENTS.md) fails, naming both operating and agents and the count.
  • AGENTS.md scope: mutation_policy.agents_md_scope_violations() against a compliant target-shape fixture, an over-length fixture, and a runtime-heading fixture.
  • Surface parity: OPERATING.md's ## Mutation surface section against MUTATION_POLICY.render_bridge_surface_block() (reusing tests/test_operating_md.py's _section helper rather than duplicating it), a mutated copy fails with a diff, and Allowed targets: is confirmed absent from every other release-root *.md file and every other assembled-prompt block.
  • No guidance without capability: every tool name in the assembled prompt's prose is in EXECUTOR_TOOL_NAMES, excluding the loop's own suppressed builtins (mirroring bridge.py's local _LOOP_EXCLUDED_SKILLS = ["weather", "tmux", "clawhub", "cron", "summarize", "github"] — without excluding these the fixture skills catalogue includes cron, a false positive, not a real one).

xfail(strict) — three cases, three reasons

  1. test_fingerprint_user_message_only_budget_and_task#1723 part (b) pending. build_task's user message still carries its own skip/surface/branch/runner/final-JSON literals; PR fix(bridge): remove build_task cycle-rule literals now that OPERATING.md is loaded (#1723 part b) #1742 is already open for this (fix(bridge): remove build_task cycle-rule literals now that OPERATING.md is loaded). Once it merges, this test flips to a hard failure until the marker is removed.
  2. test_agents_md_scope_violations_against_real_instance_fixture#1730 part 1 pending. The real instance AGENTS.md fixture (tests/fixtures/instance_agents_md/AGENTS.md, refreshed via gh api repos/ozand/eeebot-self-evolving/contents/AGENTS.md --jq .content | base64 -d) is 192 lines with 8 runtime headings today, matching the issue's stated baseline exactly.
  3. test_search_memory_status_contract_appears_only_in_operating_tools_blocknew finding, no issue filed yet. Proving the capability assertion surfaced that the search_memory status contract ("complete/partial/unavailable") is restated in two places inside the assembled prompt: OPERATING.md's ## Tools section and nanobot/agent/memory.py's MemoryStore.MEMORY_SEARCH_POINTER (folded into the loop's memory block whenever memory/index.md is non-trivial). This is a real ADR-022 rule 5 violation, but fixing it means editing the protected OPERATING.md or the shared nanobot/agent/memory.py — both out of this issue's "do not touch" scope — so it's xfail(strict) with the mechanism named in the reason, flagged here for a follow-up issue.

Constraints honored

  • Did not modify nanobot/agent/context.py, nanobot/runtime/bridge.py, OPERATING.md, or any other release file.
  • tests/fixtures/instance_agents_md/AGENTS.md and memory_index.md (already present untracked from the prior dead attempt) verified byte-identical in content to a fresh gh api pull before committing — no refresh needed.
  • ADR-022's Test Contract section gets a one-line status note.
  • Ran only python -m pytest tests/test_prompt_ontology.py tests/test_operating_md.py tests/test_ontology_loader.py (Python 3.13, since 3.9/3.11/3.12 on this host lack loguru/pytest) — 26 passed, 3 xfailed, no regressions in the other two files.

Test plan

  • python -m pytest tests/test_prompt_ontology.py tests/test_operating_md.py tests/test_ontology_loader.py — 26 passed, 3 xfailed

ozand and others added 2 commits September 18, 2026 03:25
…arness (#1726)

Add tests/test_prompt_ontology.py covering the four ADR-022 rule-4/5
assertions over the assembled loop-profile system prompt (release root +
a fixture instance workspace):

- Fingerprint once: each named rule (skip, mutation surface, branch,
  runner, identity, budget, final JSON) matches exactly one system block.
  Regexes were proven against the real release files, not the issue's
  draft table -- OPERATING.md's real branch sentence and IDENTITY.md's
  real self-declaration needed widened patterns. A deliberate-duplicate
  case proves the checker names both offending blocks.
- AGENTS.md scope: mutation_policy.agents_md_scope_violations() against a
  target-shape fixture (compliant), an over-length fixture, a
  runtime-heading fixture, and the real 192-line instance fixture
  (xfail(strict), #1730 part 1 pending).
- Surface parity: OPERATING.md's Mutation surface section against
  MUTATION_POLICY.render_bridge_surface_block(), reusing
  tests/test_operating_md.py's _section helper rather than duplicating
  it; a mutated copy fails with a diff; "Allowed targets:" is confirmed
  absent from every other release-root file and system block.
- No guidance without capability: every tool name in the assembled
  prompt's prose is in EXECUTOR_TOOL_NAMES (excluding the loop's own
  suppressed builtins, mirroring bridge.py's _LOOP_EXCLUDED_SKILLS).

Two things are intentionally xfail(strict) pending work this issue does
not own: the user-message half of the fingerprint test (#1723 part b),
and the AGENTS.md-scope check against the real instance fixture (#1730
part 1). A third, newly discovered while proving the capability
assertion, is also xfail(strict): the search_memory status contract is
restated in both OPERATING.md's Tools section and
nanobot/agent/memory.py's MEMORY_SEARCH_POINTER, a real ADR-022 rule 5
violation this issue cannot fix without touching a protected or shared
file it does not own.

ADR-022's Test Contract section gets a one-line status note recording
which checks pass today and which remain red by design.
…p the budget line out of tool-name prose

#1742 removed build_task's rule literals, so the user-message assertion
this harness carried as xfail(strict) now passes -- the marker is dropped
rather than left to XPASS. The iteration-budget pointer added with that
PR said the number is stated in 'your task message', and the bare word
'message' is the name of a tool the executor was never given, which the
capability assertion (correctly) rejects; the sentence now points at the
task section and the line by name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ozand
ozand force-pushed the test/1726-prompt-ontology branch from 30b33bd to 1d7d837 Compare September 18, 2026 00:26
@ozand

ozand commented Sep 18, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto 023a38f7 (#1742, #1723 part b) and fixed two things the rebase exposed:

  1. The user-message fingerprint assertion was xfail(strict=True, reason="#1723 part b pending") and it XPASSed once the literals were gone — the marker did exactly what it was for, so it is dropped rather than left to fail as an unexpected pass.
  2. The capability assertion went red on the word message. fix(bridge): remove build_task cycle-rule literals now that OPERATING.md is loaded (#1723 part b) #1742's iteration-budget pointer (added in review, ae10173c) said the number is stated in "your task message", and message is the name of a tool the executor was never given — the assertion is right to reject it. The sentence now points at the task section and names the line instead, and message appears nowhere in OPERATING.md.

38 passed, 2 xfailed locally (tests/test_prompt_ontology.py tests/test_operating_md.py tests/test_ontology_loader.py tests/test_task_prompt_hygiene.py). The two remaining strict xfails are #1730 part 1 (instance AGENTS.md is 192 lines) and the search_memory contract duplication you found — filing that one as its own issue now.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ozand
ozand merged commit 96a450a into main Sep 18, 2026
3 checks passed
@ozand
ozand deleted the test/1726-prompt-ontology branch September 18, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant