Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ OPENROUTER_API_KEY=

# --- RIMAPI (RimWorld mod) ---
RIMAPI_URL=http://localhost:8765
# Optional override for RimWorld's Saves folder (native load staging).
# Default: OS AppData / Application Support / ~/.config/unity3d/.../Saves
# RLE_RIMWORLD_SAVES=
# AppSprout runs: point these at the compiled fork checkout (not Workshop).
# Summaries record rimapi_dll_path, rimapi_dll_sha256, and rimapi_fork_commit.
# RIMAPI_DLL_PATH=../RIMAPI/1.6/Assemblies/RIMAPI.dll
Expand Down
13 changes: 7 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Four things must be set up before RLE can run against a live game:
1. **RimWorld** — Steam install at `C:\Steam\steamapps\common\RimWorld\` (or wherever Steam is)
2. **Harmony + RIMAPI mods** — Subscribe on Steam Workshop, then **enable both** in the in-game Mods menu. Load order: Harmony → Core → Royalty → RIMAPI. RIMAPI exposes REST API on `:8765` + SSE events.
3. **LLM provider** — [LM Studio](https://lmstudio.ai/) (local, port 1234) or [OpenRouter](https://openrouter.ai/) (cloud)
4. **Save file** — `rle_crashlanded_v1` save must exist in RimWorld's save folder (`C:\Users\<you>\AppData\LocalLow\Ludeon Studios\RimWorld by Ludeon Studios\Saves\`). The scenario auto-loads it.
4. **Save file** — `rle_crashlanded_v1` is pinned in `docker/saves/` (YAML `save_sha256`). Native runs stage that file into RimWorld AppData Saves (`C:\Users\<you>\AppData\LocalLow\Ludeon Studios\RimWorld by Ludeon Studios\Saves\`) when the live hash diverges, then `POST /game/load` by name. Docker entrypoint already symlinks `/opt/saves`.

### RIMAPI mod setup (critical)

Expand Down Expand Up @@ -272,11 +272,12 @@ Tick-specific priorities injected into all Felix agents (other harnesses get no
### Save Loading + Item Setup

`run_scenario.py` automatically:
1. Loads the scenario's save file (`rle_crashlanded_v1` seeds a built `SimpleResearchBench` at (128,136) and queues `Smithing`; the other five saves are derived from this base)
2. Polls until game is ready (colonist_count > 0)
3. Unforbids all starting items (via `POST /api/v1/things/set-forbidden`)
4. Runs any `setup_commands` declared in the scenario YAML (spawn_pawn, spawn_item, change_weather, drop_pod)
5. Unpauses game at speed 3 (if `--no-pause`)
1. Native only: `ensure_live_save()` copies `docker/saves/<name>.rws` into RimWorld AppData Saves when the live SHA ≠ YAML `save_sha256` (fail closed). Docker already symlinks `/opt/saves`. Summaries record `live_save_sha256` + `live_save_copied`.
2. Loads the scenario's save file (`rle_crashlanded_v1` seeds a built `SimpleResearchBench` at (128,136) and queues `Smithing`; the other five saves are derived from this base)
3. Polls until game is ready (colonist_count > 0)
4. Unforbids all starting items (via `POST /api/v1/things/set-forbidden`)
5. Runs any `setup_commands` declared in the scenario YAML (spawn_pawn, spawn_item, change_weather, drop_pod)
6. Unpauses game at speed 3 (if `--no-pause`)

### Regenerating scenario saves

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ You need four things set up:
1. **RimWorld** (Steam) with **Harmony** and **[RIMAPI](https://github.com/IlyaChichkov/RIMAPI)** mods subscribed and **enabled** in the Mods menu. Load order: Harmony → Core → (DLCs) → RIMAPI.
2. **LLM provider** — [LM Studio](https://lmstudio.ai/) (local, free) or [OpenRouter](https://openrouter.ai/) (cloud)
3. **Python 3.14+** with [uv](https://docs.astral.sh/uv/)
4. **Save file** — `rle_crashlanded_v1` in RimWorld's save folder (the scenario auto-loads it). The Crashlanded seed includes a built `SimpleResearchBench` so research can leave the 7/31 starting floor.
4. **Save file** — `rle_crashlanded_v1` is pinned in `docker/saves/` (YAML `save_sha256`). Native runs copy it into RimWorld's AppData Saves when the live hash diverges, then auto-load by name. The Crashlanded seed includes a built `SimpleResearchBench` so research can leave the 7/31 starting floor.

> **RIMAPI note:** The Workshop version may not have our contributed endpoints yet. See [CLAUDE.md](CLAUDE.md) for instructions on building and deploying our fork DLL. AppSprout runs set `RIMAPI_DLL_PATH` and `RIMAPI_FORK_PATH` to the compiled checkout; summaries record that path, the DLL SHA, and the fork commit. Workshop is not source of truth.

Expand Down
6 changes: 5 additions & 1 deletion saves/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,8 @@ gunzip -k rle_crashlanded_v1.rws.gz
cp rle_crashlanded_v1.rws ~/.config/unity3d/Ludeon\ Studios/RimWorld\ by\ Ludeon\ Studios/Saves/
```

`run_scenario.py` auto-loads the save by name — just make sure the `.rws` file exists in the saves folder.
`run_scenario.py` auto-loads the save by name. Native (non-docker) runs
stage `docker/saves/<name>.rws` into RimWorld's AppData Saves folder when
the live file's SHA-256 does not match the scenario YAML pin, then fail
closed if the copy cannot satisfy the pin. Docker entrypoint already
symlinks `/opt/saves` into the container Saves folder.
24 changes: 2 additions & 22 deletions scripts/create_scenario_saves.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,37 +21,17 @@

import argparse
import asyncio
import os
import shutil
import sys
from pathlib import Path
from typing import Any

from rle.rimapi.client import RimAPIClient
from rle.scenarios.loader import default_rimworld_saves_dir

DOCKER_SAVES_DIR = Path(__file__).resolve().parent.parent / "docker" / "saves"
BASE_SAVE = "rle_crashlanded_v1"


def _default_rimworld_save_dir() -> Path:
"""Return RimWorld's default save directory for the current OS."""
if sys.platform == "win32":
user_profile = Path(os.environ.get("USERPROFILE", ""))
return (
user_profile / "AppData" / "LocalLow" / "Ludeon Studios"
/ "RimWorld by Ludeon Studios" / "Saves"
)
if sys.platform == "darwin":
return (
Path.home() / "Library" / "Application Support"
/ "RimWorld" / "Saves"
)
# Linux/other
return (
Path.home() / ".config" / "unity3d" / "Ludeon Studios"
/ "RimWorld by Ludeon Studios" / "Saves"
)

# Colony center (from the base save)
COLONY_X, COLONY_Z = 132, 137

Expand Down Expand Up @@ -576,7 +556,7 @@ def main() -> None:

save_dir = (
Path(args.save_dir) if args.save_dir
else _default_rimworld_save_dir()
else default_rimworld_saves_dir()
)
if not save_dir.exists():
print(
Expand Down
77 changes: 63 additions & 14 deletions scripts/run_benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
from rle.orchestration.save_loader import load_save_and_settle
from rle.rimapi.client import RimAPIClient
from rle.scenarios.evaluator import ScenarioEvaluator
from rle.scenarios.loader import list_scenarios
from rle.scenarios.loader import LiveSavePinError, LiveSaveStatus, list_scenarios
from rle.scenarios.schema import ScenarioConfig
from rle.scoring.composite import CompositeScorer
from rle.scoring.delta import PairedResult, print_paired_leaderboard
Expand Down Expand Up @@ -67,16 +67,44 @@ class RunError(RuntimeError):
"""A harness could not be constructed for this run."""


async def _load_save(client: RimAPIClient, config: RLEConfig, scenario: ScenarioConfig) -> bool:
"""Load + settle the scenario save. Returns False when the run must be skipped."""
async def _load_save(
client: RimAPIClient,
config: RLEConfig,
scenario: ScenarioConfig,
*,
stage_live: bool = False,
) -> tuple[bool, LiveSaveStatus | None]:
"""Load + settle the scenario save.

Returns ``(ok, live_save)``. ``ok`` is False when the run must be skipped
for a transient load failure. ``LiveSavePinError`` is not caught — a
native pin miss fails closed instead of loading a stale AppData file.
"""
if not scenario.save_name:
return True
return True, None
try:
await load_save_and_settle(client, config.rimapi_url, scenario.save_name)
loaded = await load_save_and_settle(
client, config.rimapi_url, scenario.save_name,
save_sha256=scenario.save_sha256,
stage_live=stage_live,
)
except LiveSavePinError:
raise
except Exception as e:
logger.warning("Could not load save %s: %s", scenario.save_name, e)
return False
return True
return False, None
if loaded.live_save is not None:
logger.info(
"live_save_sha256=%s copied=%s",
loaded.live_save.live_save_sha256,
loaded.live_save.copied,
)
print(
f" live save {scenario.save_name}: "
f"sha256={loaded.live_save.live_save_sha256} "
f"copied={loaded.live_save.copied}",
)
return True, loaded.live_save


def _harness_failed(event_log: EventLog | None, start_index: int) -> bool:
Expand Down Expand Up @@ -287,9 +315,17 @@ async def _run_ablation( # noqa: PLR0913
for scenario in scenarios:
for run_id in range(num_runs):
run_label = f" (run {run_id + 1}/{num_runs})" if num_runs > 1 else ""
if not use_mock_rimapi and not await _load_save(client, config, scenario):
print(f" SKIP {scenario.name}{run_label} ({tag}): save load failed")
continue
if not use_mock_rimapi:
loaded_ok, _live = await _load_save(
client, config, scenario,
stage_live=not args.docker,
)
if not loaded_ok:
print(
f" SKIP {scenario.name}{run_label} "
f"({tag}): save load failed",
)
continue

print(f" {scenario.name}{run_label} ({tag})...")
options = {**harness_options, "exclude_agent": exclude}
Expand Down Expand Up @@ -477,9 +513,15 @@ async def main(args: argparse.Namespace) -> None: # noqa: PLR0912, PLR0915
for run_id in range(num_runs):
run_label = f" (run {run_id + 1}/{num_runs})" if num_runs > 1 else ""

if not use_mock_rimapi and not await _load_save(client, config, scenario):
print(f" SKIP {scenario.name}{run_label}: save load failed")
continue
live_save: LiveSaveStatus | None = None
if not use_mock_rimapi:
loaded_ok, live_save = await _load_save(
client, config, scenario,
stage_live=not args.docker,
)
if not loaded_ok:
print(f" SKIP {scenario.name}{run_label}: save load failed")
continue

print(
f"\nRunning: {scenario.name} ({scenario.difficulty}) "
Expand All @@ -499,6 +541,9 @@ async def main(args: argparse.Namespace) -> None: # noqa: PLR0912, PLR0915
except RunError as exc:
exit_with_harness_error(exc)
return
if live_save is not None:
result["live_save_sha256"] = live_save.live_save_sha256
result["live_save_copied"] = live_save.copied
results.append(result)
if paired:
paired.agent_scores.append(result["score"])
Expand All @@ -512,7 +557,11 @@ async def main(args: argparse.Namespace) -> None: # noqa: PLR0912, PLR0915

# Baseline run (reload same save, unmanaged colony)
if paired is not None:
if not await _load_save(client, config, scenario):
reload_ok, _live = await _load_save(
client, config, scenario,
stage_live=not args.docker,
)
if not reload_ok:
logger.warning("Could not reload save for baseline")
print(f" baseline{run_label}...")
baseline = await _run_scenario(
Expand Down
37 changes: 31 additions & 6 deletions scripts/run_scenario.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
from rle.rimapi.client import RimAPIClient
from rle.rimapi.sse_client import RimAPISSEClient
from rle.scenarios.evaluator import ScenarioEvaluator
from rle.scenarios.loader import list_scenarios, load_scenario
from rle.scenarios.loader import LiveSavePinError, list_scenarios, load_scenario
from rle.scoring.composite import CompositeScorer
from rle.scoring.recorder import TimeSeriesRecorder
from rle.tracking.cost_tracker import create_cost_tracker, fetch_billed_costs
Expand Down Expand Up @@ -83,10 +83,17 @@ def _build_run_summary( # noqa: PLR0913
cost_snapshot_dict: dict[str, object],
event_summary_dict: dict[str, object] | None,
billed_cost_dict: dict[str, object] | None = None,
live_save_sha256: str | None = None,
live_save_copied: bool | None = None,
) -> dict[str, object]:
"""Compose the per-scenario summary JSON (metadata + config + result)."""
summary: dict[str, object] = {
**collect_metadata(random_seed=args.seed, harness_describe=harness_describe),
**collect_metadata(
random_seed=args.seed,
harness_describe=harness_describe,
live_save_sha256=live_save_sha256,
live_save_copied=live_save_copied,
),
"scenario": scenario_name,
"scenario_save_name": scenario_save_name,
"harness": harness_name,
Expand Down Expand Up @@ -210,17 +217,33 @@ async def main(args: argparse.Namespace) -> None:
sse = RimAPISSEClient(config.rimapi_url)
sse_task = asyncio.create_task(sse.listen())

live_save_sha256: str | None = None
live_save_copied: bool | None = None
async with RimAPIClient(config.rimapi_url) as client:
# Load the scenario's save file for a consistent starting state
# Load the scenario's save file for a consistent starting state.
# Native path: stage docker/saves → AppData when the live hash ≠ pin
# (Docker entrypoint already symlinks /opt/saves; this CLI is native).
if scenario.save_name:
print(f"Loading save: {scenario.save_name}")
try:
unforbid_count = await load_save_and_settle(
loaded = await load_save_and_settle(
client, config.rimapi_url, scenario.save_name,
save_sha256=scenario.save_sha256,
stage_live=True,
)
if unforbid_count:
print(f"Unforbid {unforbid_count} items.")
if loaded.live_save is not None:
live_save_sha256 = loaded.live_save.live_save_sha256
live_save_copied = loaded.live_save.copied
print(
f"Live save sha256={live_save_sha256} "
f"copied={live_save_copied}",
)
if loaded.unforbid_count:
print(f"Unforbid {loaded.unforbid_count} items.")
print("Save loaded, game ready.")
except LiveSavePinError as e:
print(f"ERROR: Refusing to load save '{scenario.save_name}': {e}")
raise SystemExit(1) from e
except Exception as e:
print(f"Warning: Could not load save '{scenario.save_name}': {e}")
print("Continuing with current game state...")
Expand Down Expand Up @@ -353,6 +376,8 @@ async def main(args: argparse.Namespace) -> None:
billed_cost_dict=(
billed_report.model_dump() if billed_report else None
),
live_save_sha256=live_save_sha256,
live_save_copied=live_save_copied,
)
summary_path = output_dir / f"{scenario_path.stem}_summary.json"
summary_path.write_text(json.dumps(summary, indent=2, default=str))
Expand Down
39 changes: 33 additions & 6 deletions src/rle/orchestration/save_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@

import asyncio
import logging
from dataclasses import dataclass

from rle.docker import wait_for_rimapi
from rle.rimapi.client import RimAPIClient
from rle.scenarios.loader import LiveSaveStatus, ensure_live_save

logger = logging.getLogger(__name__)

Expand All @@ -25,20 +27,45 @@
MAX_POLLS = 30


@dataclass(frozen=True)
class LoadSettleResult:
"""Outcome of ``load_save_and_settle`` (unforbid count + optional staging)."""

unforbid_count: int
live_save: LiveSaveStatus | None = None


async def load_save_and_settle(
client: RimAPIClient,
rimapi_url: str,
save_name: str,
*,
unforbid_items: bool = True,
rimapi_timeout_s: float = 30.0,
) -> int:
save_sha256: str | None = None,
stage_live: bool = False,
) -> LoadSettleResult:
"""Load ``save_name`` and block until the colony is stable.

Returns the number of starting items unforbidden (0 when disabled).
Raises whatever ``load_game`` / ``wait_for_rimapi`` raise so callers can
decide whether to skip the run.
When ``stage_live`` is True and ``save_sha256`` is set (native path),
copy ``docker/saves/<name>.rws`` into RimWorld AppData Saves if the live
file does not already match the pin. Docker skips this — the entrypoint
already symlinks ``/opt/saves``. Staging failures raise
``LiveSavePinError`` (fail closed) before ``game/load``.

Returns ``LoadSettleResult`` (unforbid count + live-save status).
Raises whatever ``load_game`` / ``wait_for_rimapi`` / staging raise so
callers can decide whether to skip the run.
"""
live_save: LiveSaveStatus | None = None
if stage_live and save_sha256:
live_save = ensure_live_save(save_name, save_sha256)
logger.info(
"live_save_sha256=%s copied=%s path=%s",
live_save.live_save_sha256,
live_save.copied,
live_save.live_path,
)
await client.load_game(save_name)
await wait_for_rimapi(rimapi_url, timeout=rimapi_timeout_s)
stable_count = 0
Expand All @@ -60,6 +87,6 @@ async def load_save_and_settle(
else:
logger.warning("Save %s never reported a stable population; continuing", save_name)
if not unforbid_items:
return 0
return LoadSettleResult(unforbid_count=0, live_save=live_save)
count = await client.unforbid_all_items()
return int(count or 0)
return LoadSettleResult(unforbid_count=int(count or 0), live_save=live_save)
Loading
Loading