perf: env_step sub-stage timing panel + optional numba TBB layer (+23% throughput on many-core)

[wlgys8]

Summary

排查 g1-wbt-dance 训练 collector 卡点（env_step 29ms 中 transition 占 9.8ms）过程中落地的两类改进：

1. Console Timing 面板：env_step 子阶段树

• TorchEnv.step 接入 Perf 打点（apply_action / physics / transition / reset），scope 命名与 ArrayEnv 一致
• MotrixSim sim_data.execute() 用 active_perf_scope("read") 记录原生批量读取，自动挂到调用方阶段下（transition / reset），无需修改任何具体环境
• async collector 启用被包装 env 的 Perf，以点分路径上报子阶段；worker 重建嵌套树（标量 total 与子节点任意到达顺序均可合并，修复了先标量后子节点时的 learner 崩溃）

面板效果：按 2 切换 Timing 页后可直接看到 physics / transition(read) / evaluate / reset(backend_apply…) 的耗时与占比。

2. 多核机器的 OpenMP 唤醒风暴修复（可选 TBB 层）

根因链：evaluate 每次 wall 8.4ms 但 CPU 只有 ~30 个有效线程 —— OMP_WAIT_POLICY=PASSIVE 下 192 个 OMP 线程每次 kernel 调用都要唤醒，barrier 长尾支配 wall time；OMP ACTIVE 自旋更糟（与 physics 线程池抢核）。

• motrix-env-core 新增 pinned tbb extra（tbb==2021.13.0），workspace 根暴露 --extra tbb
• preload_tbb() 以完整路径 RTLD_GLOBAL 预加载 <venv>/lib/libtbb.so.12（PyPI wheel 不含 Python 模块，numba 裸名 CDLL 找不到库），ManagerEnv.__init__ 在首个并行 kernel 前自动调用；未安装 extra 时静默回退默认层

测量数据

场景	evaluate	吞吐
OMP 192 线程（修复前）	8.4 ms	132.5k steps/s
OMP 限 64 线程	1.4 ms	168.8k steps/s
TBB 192 线程（本 PR）	1.0–1.3 ms	163k steps/s (+23%)
本地 32 线程 OMP vs TBB	1.49 vs 1.52 ms	打平（小核数无唤醒问题）

Test plan

• motrix_rl 全量测试（console 嵌套树渲染、collector 点分路径上报、worker 嵌套回归测试）
• motrix_env_core 全量测试 + TorchEnv / perf 模块测试
• 服务器（192 核 epyc-9004）端到端训练验证：自动启用 TBB、evaluate 1.16ms、零环境变量
• 本地 32 核 OMP/TBB 对照确认小核数无回退

[Copilot]

Copilot review overview

🟡 Changes recommended

There are a few confirmed correctness/documentation issues (Markdown inline-code formatting and ETA/duration formatting edge cases) plus a small performance improvement opportunity (cache preload_tbb()), which should be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 2

Open (3)

• Fix sub-minute duration formatting to match docs · New
• Cache TBB preload to avoid repeated rescans · New
• Split nested inline code to avoid broken Markdown · New

What changed in this PR

This PR adds richer, nested performance timing instrumentation across env stepping and async FastSAC training, and introduces an optional Intel TBB threading layer preload to avoid OpenMP wakeup overhead on many-core hosts.

Changes:

• Add nested env_step sub-stage timing (apply_action/physics/transition/reset + MotrixSim read scope) and merge dotted-path timing into a stable tree in the async worker/panel.
• Add optional tbb extra + installer flag to preload libtbb.so.12 (RTLD_GLOBAL) before numba parallel kernels run, improving throughput on many-core machines.
• Extend tests to cover dotted-path timing flattening, tree rebuild order-independence, and collector reporting of env_step substages.

File	Description
wiki/​design/​fastsac-async-heterogeneous-trainer.md	Update design doc to describe dotted-path nested timing metrics.
uv.lock	Add tbb package and expose workspace tbb extra wiring.
pyproject.toml	Add workspace-level tbb extra that pulls in motrix-env-core[tbb].
motrix_env_core/​pyproject.toml	Add tbb optional dependency extra (pinned).
install.sh	Add --tbb flag to install optional TBB extra.
motrix_env_core/​src/​motrix_env_core/​numba/​threading.py	Add preload_tbb() helper to preload TBB from <venv>/lib.
motrix_env_core/​src/​motrix_env_core/​numba/​manager/​env.py	Call preload_tbb() early in ManagerEnv init (before parallel kernels).
motrix_env_core/​src/​motrix_env_core/​perf.py	Add stage_mean_ms() to flatten nested scopes into dotted timing paths.
motrix_env_motrixsim/​src/​motrix_env_motrixsim/​torch_env.py	Add Perf scopes to TorchEnv.step() with ArrayEnv-consistent stage names.
motrix_env_motrixsim/​src/​motrix_env_motrixsim/​sim_data.py	Wrap simulator read execution with active_perf_scope("read").
motrix_rl/​src/​motrix_rl/​fastsac/​async_impl/​collector.py	Report env Perf sub-stage timings as dotted paths under env_step.*.
motrix_rl/​src/​motrix_rl/​fastsac/​async_impl/​worker.py	Rebuild nested timing trees from dotted-path keys; anchor elapsed time after first ingest.
motrix_rl/​src/​motrix_rl/​console.py	Add ETA + elapsed/remaining text rendering in console panels.
motrix_rl/​tests/​test_fastsac_collector.py	Add tests for collector dotted-path env_step sub-stage timing reporting.
motrix_rl/​tests/​test_fastsac_learner.py	Add tests for timing-path nesting merge behavior (scalar vs children order).
motrix_rl/​tests/​test_console.py	Update panel rendering tests for timing subtree + run-progress time text.
motrix_env_core/​tests/​test_perf.py	Add tests for dotted-path flattening in stage_mean_ms().
motrix_env_core/​tests/​test_numba_threading.py	Add smoke test for preload_tbb() returning bool without raising.
bench/​bench_learner_update.py	Add standalone learner-update profiling script.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The learner timing “anchor” resets last_log_time but not last_log_step, which can inflate the first logged throughput window after anchoring.

Review effort: Lite
Findings: None

Resolved since last review (3)

• Fix sub-minute duration formatting to match docs
• Split nested inline code to avoid broken Markdown
• Cache TBB preload to avoid repeated rescans

Previously missed (1)

In code that hasn't changed since last review

Reset last_log_step when anchoring the resumed start time

motrix_rl/​src/​motrix_rl/​fastsac/​async_impl/​worker.py:482

The start-time anchoring logic resets last_log_time but leaves last_log_step at the original resume_step, so the first logged steps_per_second window can be inflated (numerator counts steps since resume, denominator counts time since the first ingested batch). Reset last_log_step when anchoring so the first window rate matches the anchored time base.