Skip to content

backend: report query failures instead of encoding them - #11

Open
dingf3ng wants to merge 1 commit into
tests-assert-resultsfrom
search-failure-contract
Open

backend: report query failures instead of encoding them#11
dingf3ng wants to merge 1 commit into
tests-assert-resultsfrom
search-failure-contract

Conversation

@dingf3ng

@dingf3ng dingf3ng commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Why

CoqInterface.search() is behind every Search / Print / Locate / About / Check / Print Assumptions query. It always returned a str, and encoded its failures as ordinary text:

aux_file not accessible · Query error: … · Empty query · No search term provided · Error executing print: … · Unsupported query type: …

_extract_search_results() went further and swallowed its exception entirely, returning whatever partial list it had — usually [] — as if the search had completed.

The consequence

Nothing downstream could tell "found nothing" from "broke".

CoqCommandSearch._create_search_result scores a result 1.0 unless its text contains the literal phrase "No results found". So a dead aux_file, or an LSP error mid-extraction, reached the LLM as a confident, high-relevance search hit whose content was an error message — and the agent reasoned over it.

What changed

search() and _run_aux_query() now return Optional[str]:

  • success → the output
  • failureNone, with the reason on self.last_error, read back via get_last_error()

That's the signal the class already uses for apply_tactic and reset_by_step. No new convention.

An empty result is a success, not a failure — it still returns "No results found.".

Keeping those apart in the Search path takes care, because _extract_search_results runs in a polling loop and must keep returning its partial list so retries still work. It now records the exception on last_error instead of dropping it, and the Search branch clears last_error before polling, then:

outcome returns
results found the results
empty + error recorded None
empty, no error "No results found."

skip_prefixes is deliberately untouched — it filters proof-goal noise out of legitimate hits, including lines Rocq itself emits starting with Error:. Separate, more ambiguous question.

One judgment call

The None case is handled in _create_search_result rather than at each of the eight CoqCommandSearch call sites, since all eight funnel through it — one place to get right instead of eight. A failed query becomes relevance_score=0.0 with metadata['failed'] set and content "Query failed: <reason>". interactive_session prints that content, so the reason still reaches a human.

Tests

tests/test_search_failures.pyno Rocq needed, runs in 2 seconds:

  • extraction failure → None, cause on last_error
  • genuinely empty search → still "No results found.", last_error unset
  • every malformed-query path → None with the right reason
  • all 8 CoqCommandSearch methods score a failure 0.0; a success stays 1.0

@dingf3ng
dingf3ng force-pushed the search-failure-contract branch from a04e59f to f0d2bbd Compare September 1, 2026 14:17
@dingf3ng
dingf3ng force-pushed the search-failure-contract branch from f0d2bbd to cda38b0 Compare September 2, 2026 04:44
@dingf3ng
dingf3ng force-pushed the search-failure-contract branch from cda38b0 to 001d0c8 Compare September 2, 2026 07:36
CoqInterface.search() is the implementation behind every Search, Print,
Locate, About, Check and Print Assumptions query. It always returned a
str, and encoded its failures as ordinary text: "aux_file not
accessible", "Query error: ...", "Empty query", "No search term
provided", "Error executing print: ...", "Unsupported query type: ...".
_extract_search_results went further and swallowed its exception
entirely, returning whatever partial list it had collected -- usually []
-- as if the search had completed.

Nothing downstream could tell "found nothing" from "broke".
CoqCommandSearch._create_search_result scores a result 1.0 unless its
text contains the literal phrase "No results found", so a dead aux_file
or an LSP error mid-extraction reached the LLM as a confident, high
relevance search hit whose content was an error message. The agent then
reasoned over it.

search() and _run_aux_query() now return Optional[str]: the output on
success, None on failure, with the reason on self.last_error and read
back through get_last_error(). That is the signal the class already uses
for apply_tactic and reset_by_step, so this adds no new convention.

A query that legitimately matches nothing is a success, not a failure,
and still returns "No results found.". Keeping those apart in the Search
path takes some care, because _extract_search_results is called in a
polling loop and has to keep returning its partial list so the retry
still works. It now records the exception on last_error instead of
dropping it, and the Search branch clears last_error before polling, then
distinguishes: results found -> return them; empty with an error
recorded -> None; empty with no error -> "No results found.".

skip_prefixes is deliberately untouched. It filters proof-goal noise out
of legitimate hits, including lines Rocq itself emits starting with
"Error:", and is a separate, more ambiguous question.

The None case is handled in _create_search_result rather than at each of
the eight CoqCommandSearch call sites, since all eight funnel through it.
A failed query becomes relevance 0.0 with metadata['failed'] set and
content "Query failed: <reason>" -- interactive_session prints that
content, so the reason still reaches a human.

tests/test_search_failures.py covers the contract without Rocq: an
extraction failure returns None with the cause on last_error, a genuinely
empty search still returns "No results found." with last_error unset,
every malformed-query path returns None with the right reason, and all
eight CoqCommandSearch methods score a failure 0.0 while a success stays
1.0. Runs in 2 seconds.

The two live test files are adapted to the new contract, and
test_coq_interface_queries gains the failure half: an unsupported command
returns None naming the command, and the session still answers afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dingf3ng
dingf3ng force-pushed the search-failure-contract branch from 001d0c8 to 98c1c3b Compare September 2, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant