backend: report query failures instead of encoding them - #11
Open
dingf3ng wants to merge 1 commit into
Open
Conversation
dingf3ng
force-pushed
the
search-failure-contract
branch
from
September 1, 2026 14:17
a04e59f to
f0d2bbd
Compare
dingf3ng
force-pushed
the
search-failure-contract
branch
from
September 2, 2026 04:44
f0d2bbd to
cda38b0
Compare
dingf3ng
force-pushed
the
search-failure-contract
branch
from
September 2, 2026 07:36
cda38b0 to
001d0c8
Compare
CoqInterface.search() is the implementation behind every Search, Print, Locate, About, Check and Print Assumptions query. It always returned a str, and encoded its failures as ordinary text: "aux_file not accessible", "Query error: ...", "Empty query", "No search term provided", "Error executing print: ...", "Unsupported query type: ...". _extract_search_results went further and swallowed its exception entirely, returning whatever partial list it had collected -- usually [] -- as if the search had completed. Nothing downstream could tell "found nothing" from "broke". CoqCommandSearch._create_search_result scores a result 1.0 unless its text contains the literal phrase "No results found", so a dead aux_file or an LSP error mid-extraction reached the LLM as a confident, high relevance search hit whose content was an error message. The agent then reasoned over it. search() and _run_aux_query() now return Optional[str]: the output on success, None on failure, with the reason on self.last_error and read back through get_last_error(). That is the signal the class already uses for apply_tactic and reset_by_step, so this adds no new convention. A query that legitimately matches nothing is a success, not a failure, and still returns "No results found.". Keeping those apart in the Search path takes some care, because _extract_search_results is called in a polling loop and has to keep returning its partial list so the retry still works. It now records the exception on last_error instead of dropping it, and the Search branch clears last_error before polling, then distinguishes: results found -> return them; empty with an error recorded -> None; empty with no error -> "No results found.". skip_prefixes is deliberately untouched. It filters proof-goal noise out of legitimate hits, including lines Rocq itself emits starting with "Error:", and is a separate, more ambiguous question. The None case is handled in _create_search_result rather than at each of the eight CoqCommandSearch call sites, since all eight funnel through it. A failed query becomes relevance 0.0 with metadata['failed'] set and content "Query failed: <reason>" -- interactive_session prints that content, so the reason still reaches a human. tests/test_search_failures.py covers the contract without Rocq: an extraction failure returns None with the cause on last_error, a genuinely empty search still returns "No results found." with last_error unset, every malformed-query path returns None with the right reason, and all eight CoqCommandSearch methods score a failure 0.0 while a success stays 1.0. Runs in 2 seconds. The two live test files are adapted to the new contract, and test_coq_interface_queries gains the failure half: an unsupported command returns None naming the command, and the session still answers afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dingf3ng
force-pushed
the
search-failure-contract
branch
from
September 2, 2026 07:57
001d0c8 to
98c1c3b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
CoqInterface.search()is behind everySearch/Print/Locate/About/Check/Print Assumptionsquery. It always returned astr, and encoded its failures as ordinary text:aux_file not accessible·Query error: …·Empty query·No search term provided·Error executing print: …·Unsupported query type: …_extract_search_results()went further and swallowed its exception entirely, returning whatever partial list it had — usually[]— as if the search had completed.The consequence
Nothing downstream could tell "found nothing" from "broke".
CoqCommandSearch._create_search_resultscores a result1.0unless its text contains the literal phrase"No results found". So a dead aux_file, or an LSP error mid-extraction, reached the LLM as a confident, high-relevance search hit whose content was an error message — and the agent reasoned over it.What changed
search()and_run_aux_query()now returnOptional[str]:None, with the reason onself.last_error, read back viaget_last_error()That's the signal the class already uses for
apply_tacticandreset_by_step. No new convention.An empty result is a success, not a failure — it still returns
"No results found.".Keeping those apart in the
Searchpath takes care, because_extract_search_resultsruns in a polling loop and must keep returning its partial list so retries still work. It now records the exception onlast_errorinstead of dropping it, and the Search branch clearslast_errorbefore polling, then:None"No results found."skip_prefixesis deliberately untouched — it filters proof-goal noise out of legitimate hits, including lines Rocq itself emits starting withError:. Separate, more ambiguous question.One judgment call
The
Nonecase is handled in_create_search_resultrather than at each of the eightCoqCommandSearchcall sites, since all eight funnel through it — one place to get right instead of eight. A failed query becomesrelevance_score=0.0withmetadata['failed']set and content"Query failed: <reason>".interactive_sessionprints that content, so the reason still reaches a human.Tests
tests/test_search_failures.py— no Rocq needed, runs in 2 seconds:None, cause onlast_error"No results found.",last_errorunsetNonewith the right reasonCoqCommandSearchmethods score a failure0.0; a success stays1.0