Skip to content

tests: make the query and search tests assert results - #10

Open
dingf3ng wants to merge 1 commit into
agent-scratch-prooffrom
tests-assert-results
Open

tests: make the query and search tests assert results#10
dingf3ng wants to merge 1 commit into
agent-scratch-prooffrom
tests-assert-results

Conversation

@dingf3ng

@dingf3ng dingf3ng commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Why

Both test files reported success while checking nothing.

test_context_search.py scored successful_searches > 0, where a "success" was any string coming back from coq.search(). It built its interface as CoqInterface(path) — no workspace, no library_paths — so libframac was never mapped, the goal file's statement didn't typecheck, no proof opened, and load() died in coqpyt with IndexError: pop from empty list.

The return value of load() was discarded and "✅ CoqInterface loaded successfully" printed regardless. All 15 queries then returned the string "aux_file not accessible" and each was counted as a hit:

📊 Successful queries: 15/15
📊 Successful tests: 12/12
exit 0, in 12 seconds

Under pytest it was worse — both functions return a bool, which pytest discards:

PytestReturnNotNoneWarning: Test functions should return None, but ... returned <class 'bool'>
2 passed, 2 warnings in 14.58s

Green unconditionally. test_coq_interface_queries.py had the same construction and the same scoring, reporting 23/23.

What changed

Both now build the interface from configs/default_config.json and assert load().

Queries are checked by content, never by size:

query must contain
Search Z.abs Z.abs_0
Locate mult Corelib.Init.Peano.mult
Check (fun x => x + 1) fun x : int => x + 1

Sizes and exact text drift with the Rocq version and with which notations are in scope, so only substrings are asserted. The three searches too large to pin that way assert a size floor instead.

One assertion is load-bearing: Search Z.abs must contain Abs.Abs_pos, which lives in libautorocq/int/Abs.v. It fails if the libframac mapping isn't actually on the load path — the exact defect that made the file vacuous.

Reduction and ranking are tested directly, not through Rocq

Which size band a live query lands in is decided purely by len(content), and real output sits right on the boundaries — Search to_sint32 is 503 characters against a 500-char threshold. Three characters of library drift would silently delete that coverage with nothing failing.

So ResultReducer is driven directly with synthetic input, pinning all four bands, both sides of 500/501 and 1000/1001, and the goal-context ranking. Instant, no coq-lsp.

Verified by mutation

mutation result
remove workspace 3 tests fail — pop from empty list
wrong expected fragment fails, printing the real output
wrong expected reduction band fails, naming both

@dingf3ng
dingf3ng force-pushed the tests-assert-results branch from c942ef9 to 2db7caa Compare September 1, 2026 14:17
@dingf3ng
dingf3ng force-pushed the tests-assert-results branch from 2db7caa to 57cce57 Compare September 2, 2026 04:44
@dingf3ng
dingf3ng force-pushed the tests-assert-results branch from 57cce57 to ff51c9d Compare September 2, 2026 07:36
Both files reported success without checking anything.

test_context_search asserted `successful_searches > 0`, where a "success"
was any string coming back from coq.search(). It built its interface as
CoqInterface(path) with no workspace and no library_paths, so libframac
was never mapped, the goal file's statement did not typecheck, no proof
was opened, and load() died in coqpyt with "pop from empty list". The
return value of load() was discarded and "CoqInterface loaded
successfully" printed regardless. Every one of the 15 queries then
returned the string "aux_file not accessible" and was counted as a hit:
15/15, 12/12, exit 0, in 12 seconds. Under pytest it was worse -- both
functions `return` a bool, which pytest discards, so they were green
unconditionally.

test_coq_interface_queries had the same construction and the same
scoring, reporting 23/23.

Both now build the interface from configs/default_config.json and assert
load(). Queries are checked by content, never by size: `Search Z.abs`
must contain "Z.abs_0", `Locate mult` must contain
"Corelib.Init.Peano.mult", `Check (fun x => x + 1)` must contain
"fun x : int => x + 1". Sizes and exact text drift with the Rocq version
and with which notations are in scope, so only substrings are asserted.
The three searches too large to pin that way assert a size floor instead.

One assertion is load-bearing: `Search Z.abs` must contain "Abs.Abs_pos",
which lives in libautorocq/int/Abs.v. It fails if the libframac mapping
is not actually on the load path, which is the defect that made the file
vacuous in the first place.

test_context_search additionally splits what it was only claiming to
test. Ranking and size reduction are driven directly against
ResultReducer with synthetic input, because which band a live query lands
in is decided purely by len(content) and real output sits close to the
boundaries -- `Search to_sint32` is 503 characters against a 500-char
threshold. Three characters of library drift would silently drop that
coverage with nothing failing. Driving the reducer directly pins all four
bands, both sides of 500/501 and 1000/1001, and the goal-context ranking.

Verified by mutation: removing the workspace fails 3 tests with "pop from
empty list"; changing an expected fragment fails with the real output
printed; changing an expected reduction band fails naming both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dingf3ng
dingf3ng force-pushed the tests-assert-results branch from ff51c9d to 29ca081 Compare September 2, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant