Skip to content

fix(rocm): avoid indirect host pointers during HIP graph capture - #316

Open
ErikBPF wants to merge 1 commit into
FlashML-org:mainfrom
ErikBPF:fix/rocm-hip-graph-copy
Open

fix(rocm): avoid indirect host pointers during HIP graph capture#316
ErikBPF wants to merge 1 commit into
FlashML-org:mainfrom
ErikBPF:fix/rocm-hip-graph-copy

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Sep 1, 2026

Copy link
Copy Markdown

Summary

ROCm graph capture can fault when the fused multi-bank expert copy reads pinned-host source addresses indirectly through a device pointer table. During HIP capture only, use the existing per-bank copy path whose source is a direct tensor argument.

  • eager ROCm keeps the fused path
  • CUDA keeps the fused path
  • only HIP graph capture falls back

Branch-contained reproducer

The public PoC branch contains the fallback, a CPU dispatch contract, and a slow HIP acceptance test using realistic Qwen3.6 geometry:

  • 40 layers, 256 experts, 2,117 slots, top-k 8
  • 1,179,648-byte Q4_K gate/up rows
  • 720,896-byte Q5_K down rows
  • 512 graph replays with changing routes and exact row fingerprints
pytest -q tests/moe/test_fused_copy.py::test_rocm_graph_capture_uses_per_bank_copy
pytest -q tests/moe/test_fused_copy.py::test_qwen36_sized_pinned_rows_survive_rocm_graph_replay

Public PoC: ErikBPF#2

Fresh public-branch evidence

A clean public-branch build on a machine with 128 GB RAM, a Ryzen 9 5950X CPU, and a Radeon RX 9070 XT 16 GB GPU (gfx1201) completed the branch's direct-copy HIP graph microbenchmark with:

  • graph copies verified
  • bit-exact output
  • 0.216279 ms serialized median
  • 0.182385 ms overlapped median

Artifact: https://github.com/ErikBPF/FreeToken/blob/draft/qwen36-rocm-full-poc/benchmarks/results/qwen36-graph-stream-overlap-gfx1201.json

This evidence validates direct-copy graph mechanics, not the exact 512-replay acceptance test and not endpoint throughput. No speedup claim is made by this PR.

Scope and overlap

Two files only. This complements #260, whose MVP intentionally disables HIP graph capture, and does not duplicate its ROCm bring-up changes. Qwen model support, FTW changes, endpoint tuning, and research documentation are excluded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant