Skip to content

feat(e2e): support forbidden recall context - #1432

Merged
PsiACE merged 2 commits into
oceanbase:masterfrom
russell001209-ai:feat/negative-recall-probes
Sep 2, 2026
Merged

feat(e2e): support forbidden recall context#1432
PsiACE merged 2 commits into
oceanbase:masterfrom
russell001209-ai:feat/negative-recall-probes

Conversation

@russell001209-ai

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Refs #1263.

Follow-up to #1272.

Rationale for this change

PR #1272 added declarative recall probes but deferred negative benchmark cases because the evaluator could not express context that must not be returned. Without a negative contract, a probe can satisfy its required fragments while still returning stale or otherwise forbidden context.

What changes are included in this PR?

  • Add an optional forbidden_context contract to recall probes.
  • Match required and forbidden fragments case-insensitively with Unicode normalization.
  • Reject blank fragments and contracts where a forbidden fragment is contained in required context.
  • Treat any forbidden match as a hard acceptance failure, independent of probe_coverage.
  • Record the pre-redaction Boolean verdict for replay scoring and discard matched prepared context before persistence.
  • Preserve compatibility with existing manifests and older replays that do not contain the new field.
  • Document the contract in the Bub README and the English and Chinese RFCs.

The related production substring paths were checked: recall-probe matching is updated here, while expected_memory remains a separate Memory-recording contract and is unchanged.

Are there any user-facing changes?

E2E workload authors can now declare context that a recall probe must not return. Existing manifests keep their previous behavior because forbidden_context defaults to empty. Existing evaluation reports are unchanged unless a workload declares a forbidden fragment. There are no PowerContext Server, Client, or persisted Memory API changes, and no new model calls or scheduled work.

How was this change tested?

  • uv run --locked ruff check e2e/bub
  • uv run --locked ruff format --check e2e/bub
  • uv run --locked ty check --project e2e/bub --python e2e/bub/.venv --python-version 3.12 e2e/bub/src integrations/bub/src
  • uv run --project e2e/bub python -m pytest e2e/bub/tests -q (23 passed)
  • uv run --project e2e/bub powercontext-e2e --help
  • git diff --check

Regression coverage includes a forbidden match with probe_coverage: 0, canonically equivalent Unicode, manifest validation, removal of matched context before replay persistence, and live/offline replay parity for both recorded true and false verdicts.

AI usage statement

OpenAI Codex with GPT-5.6-sol was used for implementation assistance and independent adversarial review. The final diff passed the repository checks listed above, and the final independent review reported no remaining findings.

Comment thread e2e/bub/src/powercontext_e2e/evaluation.py

@PsiACE PsiACE left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@PsiACE
PsiACE merged commit 571ce28 into oceanbase:master Sep 2, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants