Skip to content

Fix grader exploits, sandbox agent SQL, add tests/CI, and rewrite docs - #1

Merged
rajdeepchatale merged 2 commits into
mainfrom
polish/grader-fixes-and-docs
Sep 22, 2026
Merged

rajdeepchatale merged 2 commits into
mainfrom
polish/grader-fixes-and-docs

Conversation

@rajdeepchatale

Copy link
Copy Markdown
Owner

Summary

Repo-wide cleanup. Running the original code turned up several grading and safety bugs, which are fixed and covered by tests here. The docs have been rewritten to match what the code actually does.

Bugs fixed

Problem Before After
Writes were penalised but still executed DELETE FROM employees then a wrong query scored 0.97 and ended the episode Read-only DB (SQLite authorizer); destructive queries rejected unexecuted, score 0
DROP TABLE broke the episode Every later step() raised Blocked by the read-only DB
Correct answers didn't end the episode 4 reference queries scored 0.92–0.94, below the 0.95 threshold Episode ends on an exact result match (is_correct)
Duplicate rows ignored Join fan-out got full result credit Multiset comparison
Reward farming A pointless COALESCE added points Bonus only when the task needs it
Runaway queries A recursive CTE blocked the worker indefinitely 2 s / 10,000-row budget, RESOURCE_LIMIT diagnostic
Baseline ignored task_id inference.py ran random tasks logged under the wrong name Targets each task; covers all 14
Hidden task requirements company_medium_2 filtered active employees without saying so Questions state every filter

Tooling

  • 71 pytest tests, covering the grader, the sandbox, episodes, and manifest/registry/baseline consistency
  • GitHub Actions workflow: ruff, pytest and openenv validate on Python 3.10 and 3.12
  • Committed uv.lock, which openenv validate failed without. Declared openai, added project metadata, bumped to 0.2.0.
  • Removed the duplicate server/Dockerfile and the unused server/requirements.txt

Docs

  • README renamed to SQL Query and rewritten, covering episodes, scoring, diagnostics, sandboxing, quickstart and limitations
  • Removed the per-task baseline table because inference.py could not have produced those numbers. Re-run the fixed script to publish real results.
  • CONTRIBUTING rewritten; LICENSE holder updated

Verification

  • ruff check ., pytest (71 passed) and openenv validate . pass locally and in a clean Python 3.10 checkout
  • End-to-end over WebSocket: reset(task_id=...) works, DELETE is blocked, the reference query ends the episode as correct
  • Not verified: docker build, because Docker wasn't running locally

Behaviour changes

Scores aren't comparable with runs made before this change, because the grader now penalises duplicate rows, no longer rewards irrelevant COALESCE, and ends episodes on correctness.

🤖 Generated with Claude Code

rajdeepchatale and others added 2 commits September 22, 2026 08:51
Grading and safety:
- Make the episode database read-only with an SQLite authorizer. Writes
  were penalised but still executed, so DELETE-then-empty-SELECT scored
  0.97 and DROP TABLE could break every later step.
- Reject destructive queries without executing them.
- Cap agent queries at 2s and 10,000 rows so a runaway recursive CTE
  can't block a server worker.
- Compare result rows as multisets so join fan-out duplicates count
  against the query, and compare numbers by value (INTEGER == REAL).
- End the episode when the result matches exactly (new is_correct flag)
  instead of at a 0.95 score that four reference queries couldn't reach.
- Award the COALESCE style bonus only when the task needs it.
- Keep penalty diagnostics when a query also fails to execute; detect
  comma-joined tables; add NOT_ALLOWED and RESOURCE_LIMIT diagnostics.

Environment and baseline:
- reset() accepts seed/episode_id/task_id and rejects unknown task ids
  instead of silently serving a random task.
- inference.py now passes task_id (every run previously got a random task
  logged under the wrong name) and covers all 14 tasks.
- State the "active employees" filter in company_medium_2 and define
  "items returned" in ecommerce_hard_1; both were hidden requirements.

Tooling:
- Add a pytest suite (71 tests) and a GitHub Actions workflow running
  ruff, pytest, and openenv validate on Python 3.10 and 3.12.
- Commit uv.lock (openenv validate requires it); declare openai; add
  project metadata; bump to 0.2.0 since scores changed.
- Remove the duplicate server/Dockerfile and the unused
  server/requirements.txt (it named the wrong package).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- README: name the project SQL Query and document episodes, scoring,
  diagnostics, sandboxing, quickstart, the baseline script, and known
  limitations in a neutral voice.
- Drop the per-task baseline table: the numbers couldn't have come from
  inference.py (it never selected tasks, and two listed tasks weren't in
  it). Re-run the fixed script to publish real results.
- CONTRIBUTING: real setup, the CI checks, and a task-authoring guide.
- LICENSE: Copyright (c) 2026, Rajdeep Chatale.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rajdeepchatale
rajdeepchatale merged commit d98381a into main Sep 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant