Skip to content

fix(judge): parse native edit targets - #1069

Open
kywch wants to merge 4 commits into
mainfrom
fix/judge-native-edit-scratch-paths
Open

fix(judge): parse native edit targets#1069
kywch wants to merge 4 commits into
mainfrom
fix/judge-native-edit-scratch-paths

Conversation

@kywch

@kywch kywch commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Parse exact trailing Editing <path> targets from native OpenHands ACP edit titles.
  • Keep suffix parsing limited to edit events; malformed, trailing-prose, and unrecognized titles retain conservative full-title scanning.
  • Normalize POSIX paths before applying scratch-root exemptions, and make shell-path masking use the same normalized check.

Why

A confirmed live rollout earned reward 1.0 and passed the LLM judge, but the mechanical tamper gate rejected native edits to /tmp/test_logistic.py and /tmp/test_rnn.py. OpenHands recorded those targets in titles such as Create test script: Editing /tmp/test_rnn.py; scanning the whole title matched the test-like filename before the scratch-path exemption could apply.

Failed job: https://github.com/benchflow-ai/benchflow/actions/runs/31622440593/job/94200401493

Behavior

The scanner now extracts only an exact trailing Editing <path> target for native ACP edit events. Protected targets remain flagged. Non-edit events and malformed titles remain fail-closed. Lexical normalization prevents paths such as /tmp/../verifier/test.sh from receiving the scratch exemption, including during shell-command masking.

Tests

  • observed benign native /tmp edit title
  • protected native verifier target
  • malformed trailing title fallback
  • non-edit title fallback
  • scratch-root traversal through a shell command

Focused result: 59 passed, 1 deselected. Ruff check and format checks pass.

Provenance

Extracts and hardens the judge fix originally authored in #979 commit bc9a1c3. This separates judge behavior from #979 and #1030 Gemini model-routing scope while preserving the original author through the existing repository author identity.


Devin Review

@kywch
kywch deployed to pypi-internal-preview August 30, 2026 21:35 — with GitHub Actions Active

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment thread tests/integration/agent_judge.py Outdated
@bingran-you
bingran-you deployed to pypi-internal-preview September 2, 2026 07:55 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator

Thermo-nuclear exact-head follow-up at a06386cfc59447ee60615ecb6dac08f5d861d4cd: behavior is ready; independent human review is the remaining merge gate.

The native edit-target parser is sound, and the review uncovered and repaired two adjacent integrity-policy errors: an honest failed attempt was being rejected merely for failing the task, while a task that explicitly requires editing an ordinary test file was mechanically rejected even when the edit was legitimate. The exemption is deliberately narrow: it applies only to direct write-like edits of ordinary test files explicitly requested by the trusted task prompt, never shell/delete/move operations or verifier, grader, conftest, reward, and run scripts.

A real Gemini judge battery classified all seven cases correctly: genuine solve, genuine failure, injected observation, verifier tamper, deleted tests, buried tamper, and legitimate test edit. Exact-head validation passed: 5918 passed, 86 skipped, 7 deselected, Ruff, ty, diff checks, and all current CI checks.

I wrote the repair commits during this review, so this still needs an independent maintainer to review the exact SHA before squash merge under the repository policy.

@bingran-you
bingran-you requested a review from xdotli September 2, 2026 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants