Fix grader exploits, sandbox agent SQL, add tests/CI, and rewrite docs - #1
Merged
Merged
Conversation
Grading and safety: - Make the episode database read-only with an SQLite authorizer. Writes were penalised but still executed, so DELETE-then-empty-SELECT scored 0.97 and DROP TABLE could break every later step. - Reject destructive queries without executing them. - Cap agent queries at 2s and 10,000 rows so a runaway recursive CTE can't block a server worker. - Compare result rows as multisets so join fan-out duplicates count against the query, and compare numbers by value (INTEGER == REAL). - End the episode when the result matches exactly (new is_correct flag) instead of at a 0.95 score that four reference queries couldn't reach. - Award the COALESCE style bonus only when the task needs it. - Keep penalty diagnostics when a query also fails to execute; detect comma-joined tables; add NOT_ALLOWED and RESOURCE_LIMIT diagnostics. Environment and baseline: - reset() accepts seed/episode_id/task_id and rejects unknown task ids instead of silently serving a random task. - inference.py now passes task_id (every run previously got a random task logged under the wrong name) and covers all 14 tasks. - State the "active employees" filter in company_medium_2 and define "items returned" in ecommerce_hard_1; both were hidden requirements. Tooling: - Add a pytest suite (71 tests) and a GitHub Actions workflow running ruff, pytest, and openenv validate on Python 3.10 and 3.12. - Commit uv.lock (openenv validate requires it); declare openai; add project metadata; bump to 0.2.0 since scores changed. - Remove the duplicate server/Dockerfile and the unused server/requirements.txt (it named the wrong package). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- README: name the project SQL Query and document episodes, scoring, diagnostics, sandboxing, quickstart, the baseline script, and known limitations in a neutral voice. - Drop the per-task baseline table: the numbers couldn't have come from inference.py (it never selected tasks, and two listed tasks weren't in it). Re-run the fixed script to publish real results. - CONTRIBUTING: real setup, the CI checks, and a task-authoring guide. - LICENSE: Copyright (c) 2026, Rajdeep Chatale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repo-wide cleanup. Running the original code turned up several grading and safety bugs, which are fixed and covered by tests here. The docs have been rewritten to match what the code actually does.
Bugs fixed
DELETE FROM employeesthen a wrong query scored 0.97 and ended the episodeDROP TABLEbroke the episodestep()raisedis_correct)COALESCEadded pointsRESOURCE_LIMITdiagnostictask_idinference.pyran random tasks logged under the wrong namecompany_medium_2filtered active employees without saying soTooling
openenv validateon Python 3.10 and 3.12uv.lock, whichopenenv validatefailed without. Declaredopenai, added project metadata, bumped to 0.2.0.server/Dockerfileand the unusedserver/requirements.txtDocs
inference.pycould not have produced those numbers. Re-run the fixed script to publish real results.Verification
ruff check .,pytest(71 passed) andopenenv validate .pass locally and in a clean Python 3.10 checkoutreset(task_id=...)works,DELETEis blocked, the reference query ends the episode as correctdocker build, because Docker wasn't running locallyBehaviour changes
Scores aren't comparable with runs made before this change, because the grader now penalises duplicate rows, no longer rewards irrelevant
COALESCE, and ends episodes on correctness.🤖 Generated with Claude Code