Skip to content

backend: CoqInterface always proves on a copy - #14

Closed
dingf3ng wants to merge 6 commits into
tests-config-fixturefrom
always-scratch
Closed

backend: CoqInterface always proves on a copy#14
dingf3ng wants to merge 6 commits into
tests-config-fixturefrom
always-scratch

Conversation

@dingf3ng

@dingf3ng dingf3ng commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Why

Proving on a scratch copy was arranged by each entry point: main.py and test_folder_batch each built a ScratchProof, swapped the path, and passed source_path so records still named the original.

Every other caller — 17 of the 19 construction sites — got no copy at all. The destructive behaviour was the default.

There's no read-only mode to justify that. load() alone pops the trailing Admitted., clear_all_proof_scripts() rewrites the file, and coqpyt writes every accepted tactic straight to disk. Any CoqInterface built on a file you care about will damage it.

What changed

The copy moves into the constructor, and both knobs disappear with it:

  • work_on_copy is gone — there's nothing to opt out of
  • source_path is gone — the file you pass is the source, so the interface derives it

ScratchProof now has exactly one caller. Nothing else in the tree references it.

What had to move

Two things edited the file before a copy existed, and would otherwise have hit the user's own file:

now runs
Hammer import injection after construction, on coq_interface.file_path
proof cleaning between construction and load(), clean_success threaded back through the components dict

Saving became coq_interface.save_result(), behind a _harvest_proof() helper shared by the signal handler and the normal exit.

Scratch cleanup is on atexit, not in close()load() calls close() to tear down the previous coq-lsp session and would otherwise delete the file out from under itself. ScratchProof.close() only unlinks files, so it's safe at interpreter exit.

Also drops getattr(self.coq, 'source_path', None) in ProofController. The attribute is always set, and the None fallback resolved to proof_file.path — the scratch name, the exact value the feature exists to keep out of records.

Verified end to end

Real run against gpt-4.1:

🎉 Proof completed successfully!
examples/example.v                       a380c035 -> a380c035   (untouched)
examples/autorocq-20260901-153147/example.v   holds the found proof
stray *_autorocq_*.v                     none

Full suite: 36 passed, 2 skipped, 0 errors.

dingf3ng and others added 6 commits September 1, 2026 16:50
The agent proves in place. clean_proof_file() strips the existing tactics
from the target .v, and coqpyt then writes every accepted tactic straight
back to disk. Pointed at a real file, a run destroys it: the original
proof is gone and the working tree is dirty. A single benchmark run
rewrites every .v in AutoRocq-bench that way, which is why the standing
advice after test_folder_batch has been
`git -C AutoRocq-bench checkout -- benchmarks`.

ScratchProof (utils/scratch.py) hands the agent a throwaway copy instead.
The copy is created beside the original, so the workspace resolves
exactly as before -- same _CoqProject, same sibling modules, same library
paths -- and it is given a module-safe generated name. When the run ends
the finished proof is saved into the run's output directory, where it
stays available for independent re-checking rather than being clobbered
by the next run, and the scratch file and its build artifacts are
removed.

Because the file being proved now has a generated name, anything that
reports or records a proof has to be told the original:

  - CoqInterface takes source_path and defaults it to file_path, so a
    non-scratch caller is unaffected.
  - ProofRecorder.start_proof_recording takes proof_file_path; records are
    grouped by file, so a scratch name would scatter them.
  - ProofController passes coq.source_path through.

main.py opens the scratch copy before proving and harvests it on both
exits -- the normal one and the signal handler -- so a Ctrl-C still keeps
whatever the run had proved. test_folder_batch does the same per file.

.gitignore covers *_autorocq_*.v so a scratch file left behind by a hard
kill cannot be mistaken for a source file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Proving on a scratch copy changes the behaviour the README documents. The
quickstart still promised

    "the proof script is saved in the same example.v file"

which was true, and was the reason example.v arrived in the tree already
proven (a4c1e2f committed the output of running that very command). It is
not true any more: the source file is left untouched and the result is
written into the run's output directory.

The rewritten paragraph names where the proof lands and offers --output-dir
for choosing somewhere else. That flag did not exist: main.py read output_dir
from the config file and nothing else, so the sentence documented something
imaginary. Adding it is the smaller fix, and it is the flag a benchmark run
wants -- without it every run drops its output directory next to the .v file,
which for AutoRocq-bench means inside the submodule.

It wins over the config the way every other command line option here already
does, and the directory is created with parents so a path like
/tmp/runs/today/first works.

Checked: --help lists the flag, and setup_output_directory creates a nested
path that does not exist yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both files reported success without checking anything.

test_context_search asserted `successful_searches > 0`, where a "success"
was any string coming back from coq.search(). It built its interface as
CoqInterface(path) with no workspace and no library_paths, so libframac
was never mapped, the goal file's statement did not typecheck, no proof
was opened, and load() died in coqpyt with "pop from empty list". The
return value of load() was discarded and "CoqInterface loaded
successfully" printed regardless. Every one of the 15 queries then
returned the string "aux_file not accessible" and was counted as a hit:
15/15, 12/12, exit 0, in 12 seconds. Under pytest it was worse -- both
functions `return` a bool, which pytest discards, so they were green
unconditionally.

test_coq_interface_queries had the same construction and the same
scoring, reporting 23/23.

Both now build the interface from configs/default_config.json and assert
load(). Queries are checked by content, never by size: `Search Z.abs`
must contain "Z.abs_0", `Locate mult` must contain
"Corelib.Init.Peano.mult", `Check (fun x => x + 1)` must contain
"fun x : int => x + 1". Sizes and exact text drift with the Rocq version
and with which notations are in scope, so only substrings are asserted.
The three searches too large to pin that way assert a size floor instead.

One assertion is load-bearing: `Search Z.abs` must contain "Abs.Abs_pos",
which lives in libautorocq/int/Abs.v. It fails if the libframac mapping
is not actually on the load path, which is the defect that made the file
vacuous in the first place.

test_context_search additionally splits what it was only claiming to
test. Ranking and size reduction are driven directly against
ResultReducer with synthetic input, because which band a live query lands
in is decided purely by len(content) and real output sits close to the
boundaries -- `Search to_sint32` is 503 characters against a 500-char
threshold. Three characters of library drift would silently drop that
coverage with nothing failing. Driving the reducer directly pins all four
bands, both sides of 500/501 and 1000/1001, and the goal-context ranking.

Verified by mutation: removing the workspace fails 3 tests with "pop from
empty list"; changing an expected fragment fails with the real output
printed; changing an expected reduction band fails naming both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CoqInterface.search() is the implementation behind every Search, Print,
Locate, About, Check and Print Assumptions query. It always returned a
str, and encoded its failures as ordinary text: "aux_file not
accessible", "Query error: ...", "Empty query", "No search term
provided", "Error executing print: ...", "Unsupported query type: ...".
_extract_search_results went further and swallowed its exception
entirely, returning whatever partial list it had collected -- usually []
-- as if the search had completed.

Nothing downstream could tell "found nothing" from "broke".
CoqCommandSearch._create_search_result scores a result 1.0 unless its
text contains the literal phrase "No results found", so a dead aux_file
or an LSP error mid-extraction reached the LLM as a confident, high
relevance search hit whose content was an error message. The agent then
reasoned over it.

search() and _run_aux_query() now return Optional[str]: the output on
success, None on failure, with the reason on self.last_error and read
back through get_last_error(). That is the signal the class already uses
for apply_tactic and reset_by_step, so this adds no new convention.

A query that legitimately matches nothing is a success, not a failure,
and still returns "No results found.". Keeping those apart in the Search
path takes some care, because _extract_search_results is called in a
polling loop and has to keep returning its partial list so the retry
still works. It now records the exception on last_error instead of
dropping it, and the Search branch clears last_error before polling, then
distinguishes: results found -> return them; empty with an error
recorded -> None; empty with no error -> "No results found.".

skip_prefixes is deliberately untouched. It filters proof-goal noise out
of legitimate hits, including lines Rocq itself emits starting with
"Error:", and is a separate, more ambiguous question.

The None case is handled in _create_search_result rather than at each of
the eight CoqCommandSearch call sites, since all eight funnel through it.
A failed query becomes relevance 0.0 with metadata['failed'] set and
content "Query failed: <reason>" -- interactive_session prints that
content, so the reason still reaches a human.

tests/test_search_failures.py covers the contract without Rocq: an
extraction failure returns None with the cause on last_error, a genuinely
empty search still returns "No results found." with last_error unset,
every malformed-query path returns None with the right reason, and all
eight CoqCommandSearch methods score a failure 0.0 while a success stays
1.0. Runs in 2 seconds.

The two live test files are adapted to the new contract, and
test_coq_interface_queries gains the failure half: an unsupported command
returns None naming the command, and the session still answers afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Its three tests take a `config` argument, supplied by the __main__ block
at the bottom. Under pytest there is no fixture by that name, so all
three error at setup before any test body runs:

    E  fixture 'config' not found

Adds the fixture. It skips rather than fails when no API key is
configured, because these tests call a live LLM -- an unconfigured
checkout should not go red, and CI should not spend money by default.

    with a key:     3 passed
    without a key:  3 skipped

__main__ passes config explicitly and is unaffected either way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Proving on a scratch copy was arranged by each entry point: main.py and
test_folder_batch each built a ScratchProof, swapped the path, and passed
source_path so records would still name the original. Every other caller
-- 17 of the 19 construction sites -- got no copy at all, and the default
was the destructive one.

There is no read-only mode to justify that. load() alone pops the trailing
"Admitted.", clear_all_proof_scripts() rewrites the file, and coqpyt
writes every accepted tactic straight to disk. Any CoqInterface built on a
file the caller cares about will damage it.

So the copy moves into the constructor, and both knobs disappear with it:

  - work_on_copy is gone: there is nothing to opt out of.
  - source_path is gone: the file you pass IS the source, so the interface
    derives it. The question "when is source_path None?" no longer exists.

ScratchProof now has exactly one caller. Nothing else in the tree
references it.

Two things had to move, because they edited the file before a copy
existed and would otherwise have hit the user's own file: the Hammer
import injection, and proof cleaning. Both now run between construction
and load(), on coq_interface.file_path, with clean_success threaded back
through the components dict. Saving became coq_interface.save_result(),
behind a _harvest_proof() helper the signal handler and the normal exit
both use.

Scratch cleanup is registered with atexit rather than done in close(),
because load() calls close() to tear down the previous coq-lsp session
and would otherwise delete the file out from under itself.
ScratchProof.close() only unlinks files, so it is safe at interpreter
exit.

Also drops getattr(self.coq, 'source_path', None) in ProofController. The
attribute is always set, and the None fallback resolved to
proof_file.path -- the scratch name, the exact value the feature exists to
keep out of records.

Verified end to end against a real run (gpt-4.1):

    🎉 Proof completed successfully!
    examples/example.v                      a380c035 -> a380c035  (untouched)
    examples/autorocq-20260901-153147/example.v   holds the found proof
    stray *_autorocq_*.v                    none

Full suite: 34 passed, 1 skipped, 3 errors -- the errors pre-existing and
fixed separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dingf3ng

dingf3ng commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #9.

Splitting this from #9 meant introducing work_on_copy and source_path in #9 and deleting them here, four PRs later, and rewriting ~40 lines of main.py that #9 had just written. Nobody had reviewed either PR, so there was no reason to make a reviewer read that twice.

#9 now makes the copy in CoqInterface.__init__ directly — no intermediate opt-in API, and main.py is one coherent change instead of two overlapping ones. The end state is byte-identical: git diff between the old and new stack tips is empty.

#15's base has been repointed to #13 (tests-config-fixture), so the chain is unbroken.

🤖 Generated with Claude Code

@dingf3ng dingf3ng closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant