Skip to content

feat(closed_loop): shard evaluation across all DDP ranks - #337

Draft
HansRobo wants to merge 1 commit into
ky/closed-loop-ddp-fail-fastfrom
ky/closed-loop-pooled-ddp
Draft

feat(closed_loop): shard evaluation across all DDP ranks#337
HansRobo wants to merge 1 commit into
ky/closed-loop-ddp-fail-fastfrom
ky/closed-loop-pooled-ddp

Conversation

@HansRobo

@HansRobo HansRobo commented Jul 30, 2026

Copy link
Copy Markdown
Member

Problem

closed_loop_validate runs inside if global_rank == 0:, so one GPU does the evaluation while
the rest idle at the next collective. On a 4-GPU job that is 48 minutes of 3 idle GPUs per
checkpoint (102 minutes on the final save, which also renders media).

The DDP machinery for this already exists (shard_items, run_distributed, merge_ddp_shards)
but has no caller. It also barriers once per (site, object-mode), so a site with one route runs at
single-GPU speed regardless of GPU count — measured ceiling 1.96x on 4 GPUs.

Fix

run_evaluations_ddp() pools all (combo, route) pairs into one list, splits it once with
cost-weighted LPT, runs with no intermediate barriers, then barriers once and merges per combo.
closed_loop_validate moves out of the rank-0 guard.

Result

4xH100, real manifest, 38 segments:

sequential 4 GPU
media OFF 2862 s 1023 s 2.80x
media ON 6121 s 2736 s 2.24x

Summed per-rank rollout time matches the sequential baselines to 0.04% / 0.1% — same workload,
negligible contention.

Output is identical. All 38 mp4s match, including 14 routes written concurrently
into one directory.

For reviewers

  • elapsed_sec is now all-reduced per combo, preserving its per-site meaning. This touches an
    existing per-site W&B metric deliberately.
  • run_distributed() now has zero callers (it had none before either). Deleting it is your call.
  • 2.80x vs an ideal 4.00x is the cost proxy, not the mechanism: frame count orders routes well
    within a site, poorly across sites (0.026-0.079 s/frame). Feeding measured per-route seconds
    back would reach 4.00x; that needs rank-0 to plan and broadcast, so it is left as follow-up.

Tests

25 tests, where there were 0. The multi-rank ones spawn real gloo groups on CPU and drive the
production merge path. Failure propagation is mutation-verified.

@HansRobo HansRobo changed the title feat: closed-loop 評価を全 DDP rank に分散(プール型・バリア1回、48分→17分) feat(closed_loop): shard evaluation across all DDP ranks Jul 31, 2026
@HansRobo
HansRobo marked this pull request as draft July 31, 2026 00:08
@HansRobo
HansRobo force-pushed the ky/closed-loop-ddp-fail-fast branch from 3d446f4 to 05aab98 Compare July 31, 2026 00:15
@HansRobo
HansRobo force-pushed the ky/closed-loop-pooled-ddp branch 2 times, most recently from 1d62ae2 to acc0be9 Compare July 31, 2026 00:54
closed_loop_validate ran inside `if global_rank == 0:`, so one GPU evaluated while the rest idled
at the next collective -- 48 minutes per checkpoint on a 4-GPU job, 102 on the final save.

The existing machinery barriers once per (site, object-mode), which caps at 1.96x on 4 GPUs
because a one-route site cannot be split. run_evaluations_ddp() pools all (combo, route) pairs
instead, splits once with cost-weighted LPT, runs with no intermediate barriers, then barriers
once and merges per combo.

4xH100: media off 2862 -> 1023 s (2.80x), media on 6121 -> 2736 s (2.24x). Summed per-rank time
matches the sequential baselines to 0.04% / 0.1%. Output identical apart from order-dependent
reductions. 25 tests, where there were 0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant