fix(judge): parse native edit targets - #1069
Conversation
|
Thermo-nuclear exact-head follow-up at The native edit-target parser is sound, and the review uncovered and repaired two adjacent integrity-policy errors: an honest failed attempt was being rejected merely for failing the task, while a task that explicitly requires editing an ordinary test file was mechanically rejected even when the edit was legitimate. The exemption is deliberately narrow: it applies only to direct write-like edits of ordinary test files explicitly requested by the trusted task prompt, never shell/delete/move operations or verifier, grader, conftest, reward, and run scripts. A real Gemini judge battery classified all seven cases correctly: genuine solve, genuine failure, injected observation, verifier tamper, deleted tests, buried tamper, and legitimate test edit. Exact-head validation passed: I wrote the repair commits during this review, so this still needs an independent maintainer to review the exact SHA before squash merge under the repository policy. |
Summary
Editing <path>targets from native OpenHands ACP edit titles.editevents; malformed, trailing-prose, and unrecognized titles retain conservative full-title scanning.Why
A confirmed live rollout earned reward 1.0 and passed the LLM judge, but the mechanical tamper gate rejected native edits to
/tmp/test_logistic.pyand/tmp/test_rnn.py. OpenHands recorded those targets in titles such asCreate test script: Editing /tmp/test_rnn.py; scanning the whole title matched the test-like filename before the scratch-path exemption could apply.Failed job: https://github.com/benchflow-ai/benchflow/actions/runs/31622440593/job/94200401493
Behavior
The scanner now extracts only an exact trailing
Editing <path>target for native ACPeditevents. Protected targets remain flagged. Non-edit events and malformed titles remain fail-closed. Lexical normalization prevents paths such as/tmp/../verifier/test.shfrom receiving the scratch exemption, including during shell-command masking.Tests
/tmpedit titleFocused result: 59 passed, 1 deselected. Ruff check and format checks pass.
Provenance
Extracts and hardens the judge fix originally authored in #979 commit
bc9a1c3. This separates judge behavior from #979 and #1030 Gemini model-routing scope while preserving the original author through the existing repository author identity.