Infra: fix APPO learner-thread leak, macOS Ray runtime, parameterized experiment launcher

[Ramdam17]

What this is

Infrastructure + tooling extracted from the weekend experiment campaign, on top of feat/fresh-water-rework. No method change: the ES, the mechanism, the smoothing, the fitness and debug.py are all untouched. Merges cleanly into the current tip (e19593e).

Contents

Commit	What	Why
f117cfc	macOS Ray runtime fixes (runtime.py, _worker_hooks.py)	Makes the run survive on a laptop: configurable local_mode (hardcoded True breaks on Ray 2.53 when launched from inside the editable repo), BLAS/torch thread caps propagated to workers (uncapped → 1083 threads / 16 cores → whole-machine freeze), per-task setproctitle silenced (sync XPC round-trip to launchservicesd on every task → UI freeze at 90% idle CPU), and no more repo/.venv upload to workers.
3114cfd + b46ccf9	APPO _LearnerThread leak fix (policy_actor.py)	See below — this is the one that matters for your runs.
e634f72	exp_run.py, CLI-parameterized clone of debug.py	Lets us sweep --population / --optimize-params / --outer-iters / --seed without editing debug.py between runs. Defaults replicate your config exactly.
7a63803	Revert of our plot_es_population pop>1 guard	Your es_population.py rework (e19593e) supports arbitrary population sizes properly, so our workaround would now discard the diversity your new plots show.

The _LearnerThread leak (likely cause of the crashed/slow runs)

Two stacked RLlib issues in Ray 2.53, both only when num_learners=0 (local learner):

1. PolicyActor.reset() rebuilds the Algorithm every generation without stopping the previous one → its background threads stay alive.
2. Even with algo.stop(), one thread survives per generation: Algorithm.stop() → LearnerGroup.shutdown() only terminates remote learner backends. The local ImpalaLearner's _LearnerThread keeps running, and it is stuck inside CircularBuffer.sample() whose wait loop (while len(self)==0: time.sleep(0.0001), appo/utils.py:104) spins at 10 kHz without re-checking stopped.

Each zombie thread burns GIL at 10k wakeups/s. Measured on a 1000-generation run: train-iteration time 0.24 s → 6.5 s over 121 generations (linear growth → quadratic total time, unreachable end), 537 threads in the actor vs ~44 baseline, 527% CPU of pure lock contention.

Fix (_stop_algo()): flag thread.stopped, push one dummy entry into thread._in_queue to unblock the wait (step() re-checks stopped immediately after the dequeue), join(5 s), then algo.stop(). Used by both reset() and stop().

Verified: in-actor thread census flat at 2 over 10 generations (was +1/gen); full run flat at 12.0–12.7 s/gen through gen 60 (was 17 → 37 s/gen and 96 threads at the same point); 4 consecutive multi-hour runs completed with exit=0.

Weekend results (all 6 runs on your wandb, project bilevel)

Best objective, grid {population} × {mechanism dimensionality}:

	1-D (min_demand_frac)	2-D (+fixed_quota)
pop=1, seed 42 (1000 gens)	0.2393	0.2424
pop=1, seed 7 (1000 gens)	0.2329	0.2314
pop=16, seed 42 (300 gens)	0.2500	0.2500

• Robust: population=16 beats every pop=1 run in both dimensionalities (gap ~0.01–0.02, well above the ~0.01 seed-to-seed spread). Implementation is just num_envs_per_env_runner=16; your advantage-baseline branch switches to fitness whitening automatically at N>1.
• Not robust: the "2-D beats 1-D" signal at seed 42 reverses at seed 7 — at pop=1 the dimensionality effect is within seed noise. Not a finding.
• Open question: both pop=16 cells stop at exactly 0.2500, reached by ~generation 50 and never exceeded. That looks like a structural cap of the objective (fitness normalization / sustainability weighting) rather than an optimum — worth a look.

Notes

• .gitignore gains .wandb_nadine.env (local API-key file, never committed).
• Our earlier PR Debug + perf: run end-to-end without freezing, correct ES, linear-time #30 is left as-is, per Rémy — it predates your latest commits and conflicts on the method files.

🤖 Generated with Claude Code