Skip to content

Add RIKYU support for SALMON - #75

Open
william-dawson wants to merge 12 commits into
RIKEN-RCCS:developfrom
william-dawson:add-salmon-rikyu
Open

Add RIKYU support for SALMON#75
william-dawson wants to merge 12 commits into
RIKEN-RCCS:developfrom
william-dawson:add-salmon-rikyu

Conversation

@william-dawson

@william-dawson william-dawson commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Adds RIKYU (GB200 NVL4, 4 GPUs/node) as a target system for the SALMON benchmark. [code:salmon]

1. Pinned to the optimized SALMON branch. For RIKYU only, build.sh builds FugakuNEXT-v4 on william-dawson/SALMON2 (open upstream as SALMON-TDDFT/SALMON2#1276) instead of the v.2.2.2 tag, which is 14+ months stale and predates the OpenACC tuning, the Ewald loop-tripcount fix and the multi-node CUDA-context fix. Every other system stays on v.2.2.2.

Built with nvhpc-hpcx-cuda13/26.5, -DUSE_OPENACC -DUSE_GEMM -DUSE_NCCL and -cudalib=cublas,cusolver,nccl. USE_CUDA is deliberately not used — SALMON's hand-written CUDA kernels measured slower than the OpenACC path here.

The Fujitsu MPI topology patch is now applied only on Fugaku/FugakuCN, since no other system compiles that path.

2. Added RIKYU directions. build.sh, run.sh and list.csv entries for RIKYU at 1-4 nodes (4 GPUs/node), including per-rank GPU binding via a wrapper.sh. Depends on #74 for the RIKYU system/queue config.

--mca fcoll individual is required, not a tuning knob: the restart read does one MPI_File_read_all per rank into managed memory, and OMPIO's two-phase collective path deadlocks whenever the nproc_rgrid product exceeds 2.

UCX_IB_GPU_DIRECT_RDMA=no was needed on nvhpc 26.3 to stop a multi-node hang. Ablation-tested on 26.5 at 4, 8, 16 and 32 GPUs: no longer needed, and neither is UCX_TLS=^cma.

3. Both decompositions are benchmarked. At 4 ranks/node RIKYU runs the RT loop twice and emits two results, …-orbital and …-mixed:

  • orbitalnproc_ob = n_ranks, nproc_rgrid = 1,1,1, process_allocation='orbital_sequential'
  • mixednproc_ob = nodes, nproc_rgrid = 2,2,1, process_allocation='grid_sequential', which keeps the grid split inside a node (NVLink) and spreads orbitals across nodes; the default allocation would stride icomm_r across nodes.

Orbital is currently faster at every node count measured, but mixed is what a problem with less orbital parallelism has to fall back on, so both are worth tracking.

The FOM is SALMON's own rt iterations timer, not wall clock. The one-time restart read is ~40-150 s depending on whether Lustre is cold, which swamps and reorders the comparison when two layouts run in one job. Production runs do very many steps, so the RT loop is the part being benchmarked.

4. The benchmark starts from a pre-folded restart, so the ground state is out of scope. run.sh previously computed the GS as part of the benchmark. It now reads a k-point-folded restart prepared offline and runs TDDFT only. Restarts are pre-staged per system:

system path
RIKYU /data1/rkp00012/CX_input/SALMON/3x3x3-folded/
RC_DGXSP /lvs0/dne1/rccs-nghpcadu/CX_input/SALMON_2x2x2_folded/
Fugaku /home/ra000009/data/u10035/CX_input_fugaku/SALMON_3x3x3_folded/

Fugaku's list.csv row moves from 1 node to 24 nodes as a consequence. SALMON's default restart reader (method_wf_distributor='single') does one MPI_File_read_all per rank spanning its whole orbital slice, and Fujitsu MPI's MPI-IO throws MPI_ERR_ARG on per-rank reads above ~2 GB. 24 ranks (81 orbitals/rank, ~1.5 GB/rank) is the smallest node count validated for the 3x3x3 case; going lower reintroduces the failure. This is a real vendor-specific MPI-IO limit, reproduced independently of restart provenance and of physical memory.

william-dawson and others added 9 commits August 26, 2026 09:02
Build: nvhpc/26.3 + OpenACC + cc100 (B200/Blackwell), with
FORTRAN_COMPILER_HAS_MPI_VERSION3=OFF to work around the
MPI_Iallreduce hang at >1 rank.

Run: per-rank GPU binding via wrapper.sh (CUDA_VISIBLE_DEVICES),
UCX_IB_GPU_DIRECT_RDMA=no for 2+ nodes, and runtime patching of
the &parallel section to nproc_ob=N (orbital-only decomposition,
no real-space grid split) which is the verified GPU-optimal layout.

Depends on RIKEN-RCCS#74 for RIKYU system/queue config (gpu partition,
account=rkp00012).

💘 Generated with Crush

Signed-off-by: William Dawson <william.dawson@riken.jp>
Assisted-by: glm-5.2 via Crush <crush@charm.land>
Two changes to the RIKYU SALMON benchmark, validated end-to-end on real
hardware (job 28922, 4 GPUs, exit 0, FOM 337.2s, energy matching the
known-good 3x3x3 reference to 6 sig figs at step 10):

1. GS is now offline prep, not part of the benchmark. run.sh reads a
   python-folded 3x3x3 SiO2 restart pre-staged at
   /data1/rkp00012/CX_input/SALMON/3x3x3-folded/ (restart/ symlinked
   in, never copied -- it's ~35GB) and runs TDDFT only. This replaces
   the old Si-1-1-1 GS+TDDFT flow for RIKYU; other systems are
   unaffected (uses_prestaged_restart() gates the new path). See
   subwg2-benchmarks' README.md/salmon-benchmarking for how the restart
   was produced (`benchgen salmon create --python-fold`) and
   regenerated if the source system size changes.

2. build.sh switches RIKYU from nvhpc/26.3 + MPI3-off to native
   nvhpc/26.5 + MPI3 on, plus a new patch
   (patches/nvhpc265-ewald-reduction.patch, cherry-picked from
   william-dawson/SALMON2 commit d8442058) that works around a
   26.5-only OpenACC reduction-codegen bug in the Ewald ion-ion energy
   term. The patch is self-gating (checks __NVCOMPILER_MAJOR__/MINOR__
   at compile time) so it's applied unconditionally on every GPU build
   (RIKYU, RC_GH200, RC_DGXSP) -- a no-op on compilers it doesn't
   affect. This combination is bit-exact and was already confirmed
   faster than the old 26.3+MPI3-off recipe on Rikyu's domain-
   decomposition case; this PR is the first real validation on the
   orbital-decomposition layout RIKYU's list.csv actually uses.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: William Dawson <william.dawson@riken.jp>
Extends the pre-staged-restart pattern (added for RIKYU) to RC_DGXSP.
run.sh now reads a python-folded 2x2x2 SiO2 restart staged at
/lvs0/dne1/rccs-nghpcadu/CX_input/SALMON_2x2x2_folded/ and runs TDDFT
only, skipping GS. build.sh needed no change here -- the nvhpc/26.5
Ewald patch added for RIKYU already applies unconditionally to every
GPU build including RC_DGXSP's (a no-op there, since it still builds
on 26.3, unaffected by the bug).

The restart was produced by running SALMON's own GS(k) stage for real
(18-atom primitive cell, 2x2x2 k-mesh, 8 ranks on one ng-dgx-m1 GPU) via
the existing patched 26.5 binary already on that filesystem, then
folding it locally with `benchgen salmon fold-restart --primitive-grid
29,49,31 --repetitions 2,2,2`.

Validated end-to-end on real hardware (job 286864, ng-dgx-m1, 1 GPU):
clean build, FOM 722.2s, correct electron count (768), stable energy
across all 300 RT steps, clean exit ("end SALMON").

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: William Dawson <william.dawson@riken.jp>
Per instruction: extend the pre-staged-restart pattern to Fugaku and
fix list.csv's rank count (24x1, not 1x4 -- Fujitsu MPI-IO errors above
~2GB/rank on restart read).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: William Dawson <william.dawson@riken.jp>
…_CUDA

v.2.2.2 (BenchKit's global VERSION_TAG) is over a year stale and was never
actually what RIKYU's prior GPU-decomposition numbers were measured
against -- those came from a hand-patched checkout ~100 commits ahead
(develop-2.0.0@9b93a8c4 + PR #1276's OpenACC tuning), never landed here.

Also, -DUSE_CUDA=ON was actively wrong for RIKYU: it selects SALMON's old
hand-written CUDA kernels (zpseudo.cu, stencil_current.cu), which measured
net *slower* than plain OpenACC at this problem size (pseudo-pt alone is
3.1-3.5x slower under CUDA), and stencil_current.cu's host-side wrapper has
a real, deterministic device-buffer bug: it sizes its idx/idy/idz buffers
by each rank's LOCAL grid extent but indexes them with the RAW/global grid
coordinate. That only happens to fit when a rank owns index 1 on that axis
(single GPU, or orbital-only decomposition) -- any real-space
(nproc_rgrid>1) decomposition puts a non-first rank out of bounds and
reproduces a deterministic CUDA_ERROR_ILLEGAL_ADDRESS in calc_current.
Reproduced on every axis and Po x Pg combination tried; root-caused by
reading stencil_current.cu directly.

Fix: pin RIKYU specifically (not the other systems, which stay on stock
v.2.2.2 pending their own re-validation) to a new tag,
v.2.2.2-rikyu-optimized-265 on william-dawson/SALMON2 -- PR #1276's
stencil/current/pseudo-pt OpenACC optimizations (batched cuBLAS GEMM for
pseudo-pt via -DUSE_GEMM, not USE_CUDA) cherry-picked cleanly onto
develop-2.0.0@9b93a8c4, combined with the nvhpc/26.5 Ewald-reduction fix
already used here. This sidesteps the crash structurally (the buggy CUDA
kernels aren't even compiled) while also being the actually-faster path.

Verified on real hardware: build succeeds, and the three configs that
previously crashed (y-axis and z-axis real-space decomposition, and a
mixed Po x Pg case) all run correctly with matching energies -- see the
subwg2-benchmarks salmon-gpu-optimization-ideas skill (Open item 5) for
the full trail and numbers.

Signed-off-by: William Dawson <william.dawson@riken.jp>
native MPI3 on nvhpc/26.5 (the previous pin) is faster for single-node
and domain-decomposition cases, but genuinely hangs 8-GPU (2+ node)
orbital decomposition: it dies right after init_ps with a UCC inter-node
protocol error (cannot find remote protocol for: UCC_UCP_CONTEXT
inter-node cfg#N | tag_send from cuda-managed/GPU0), then spins at ~99%
CPU with zero further output.

Confirmed this is not the previously-documented version-mixing artifact
(that combo mixed a 26.3 binary with a 26.5 runtime; this hang reproduces
on a clean, matched, native 26.5 build) and not a GPU-binding bug
(verified correct per-rank CUDA_VISIBLE_DEVICES assignment via an
instrumented wrapper.sh, no collisions). It's specific to native MPI3 at
2+ nodes on 26.5, which had never actually been exercised at multi-node
scale on this project before.

nvhpc/26.3 + FORTRAN_COMPILER_HAS_MPI_VERSION3=OFF is the one
combination proven correct and hang-free at every scale BenchKit
actually needs: 1/2/4/8-GPU all bit-exact on this pinned source (8-GPU
orbital: 50.13s, matching -164703.595116277 eV), and 16/32-GPU proven on
an earlier source revision (see rikyu-32gpu-scaling skill). The
nvhpc/26.5 Ewald-reduction patch already applied here is a no-op on 26.3
(its version gate only fires for nvhpc>=26.5), so nothing needs to
change about how it's applied.

list.csv only exercises orbital decomposition, so the domain-
decomposition speed edge nvhpc/26.5+MPI3-on had is not something
BenchKit's own benchmark rows would ever have used anyway -- this trades
it for actually working at the node counts list.csv covers.

Signed-off-by: William Dawson <william.dawson@riken.jp>
FugakuNEXT-v1 (also open upstream as SALMON-TDDFT/SALMON2#1276) is the
single consolidated branch replacing the ad-hoc chain this pin has gone
through (v.2.2.2-rikyu-optimized-265 -> the nvhpc/26.3+MPI3-off
workaround -> this): develop-2.0.0@9b93a8c4 plus four commits, one per
concern -- the OpenACC tuning, the cmake wiring for it, the nvhpc/26.5
Ewald-reduction-bug guard, and (new) a real fix for the 2+ node UCC hang
that previously forced the 26.3+MPI3-off downgrade.

That fix (acc_init(acc_device_nvidia) before MPI_Init_thread, so UCC's
CUDA-aware protocol probe during MPI_Init's team bootstrap doesn't run
contextless and cache a broken config) means the downgrade is no longer
needed: back to nvhpc/26.5, native MPI3 ON, no
FORTRAN_COMPILER_HAS_MPI_VERSION3 override. This is the actually-tested
combination now -- orbital decomposition 1-8 nodes and domain
decomposition 1-2 nodes, all bit-exact, all previously-hanging
configurations included.

Signed-off-by: William Dawson <william.dawson@riken.jp>
Pin to FugakuNEXT-v2, which carries every fix RIKYU needs, and build it
unpatched -- the Ewald workaround is in the branch, so patching on top of
it was fragile (the reverse-apply check broke as soon as the branch
reworded a comment). Non-RIKYU systems still patch stock v.2.2.2, and
that helper now detects the fix by its macro rather than by patch
context.

Switch to nvhpc-hpcx-cuda13/26.5 (HPC-X 2.50 / OpenMPI 5), the stack all
of this was measured on, and add -DUSE_NCCL: MPI_Allreduce does its
reduction arithmetic on the host even given a device pointer, so the
pseudo-pt reduce left NVLink. NCCL takes it from 18.2s to 1.7s at 4 GPUs.

Run with mixed decomposition -- grid split 2x2x1 inside a node, orbitals
across nodes, process_allocation='grid_sequential' so each icomm_r is one
node's 4 ranks. Only the grid split reduces calculating-curr, which does
not scale under orbital decomposition. rt iterations vs pure orbital:
8 GPUs 56.2 -> 48.0 s, 16 GPUs 44.6 -> 31.0 s, energies bit-exact.

Add --mca fcoll individual, which is required: the restart read goes into
managed memory and OMPIO's two-phase collective I/O redistributes it
through cuMemcpyAsync, deadlocking whenever the nproc_rgrid product
exceeds 2. Drop UCX_IB_GPU_DIRECT_RDMA=no, which was a 26.3 fix and is
not needed on 26.5 (ablation-tested at 4, 8, 16 and 32 GPUs).

Verified with the real build.sh/run.sh on RIKYU: 1 node FOM 133.18 s,
2 nodes FOM 97.73 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: William Dawson <william.dawson@riken.jp>
…tions

Benchmark orbital and mixed decomposition rather than picking one. They
trade places depending on the problem and on which kernels have been
optimised -- orbital is currently faster at every node count, but mixed is
what a problem with less orbital parallelism falls back on -- so a
benchmark should track both. Each emits its own result line, Exp
...-orbital and ...-mixed.

Take the FOM from SALMON's rt-iterations timer instead of wall clock. The
one-time restart read is 40-150 s depending on whether Lustre is cold,
which both swamps the RT loop and, with two runs in one job, makes the
second look far faster than the first purely from page cache. Production
runs do very many steps, so the RT loop is the quantity of interest.
fom-version becomes rt_iterations_s_folded_restart.

Also bump the pin to FugakuNEXT-v3, which computes the nonlocal current
density with a batched GEMM: 29.5 -> 3.6 s on that kernel and 44.6 -> 18.7 s
on rt iterations at 16 GPUs orbital.

Verified on RIKYU, 1 node: orbital 81.84 s, mixed 80.59 s (v2 binary),
matching standalone measurements of the same layouts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: William Dawson <william.dawson@riken.jp>
@william-dawson
william-dawson marked this pull request as ready for review August 26, 2026 00:04
william-dawson and others added 2 commits August 26, 2026 09:42
The comments pointed at skills and documents in a separate, private
repository that a reader of this repo cannot open, and explained
root-cause history rather than what the code does. Rewrite them to be
shorter and self-contained: state the reason a workaround exists, not the
trail that found it.

No functional change.

Signed-off-by: William Dawson <william.dawson@riken.jp>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PEzS2ybD8VEnQa9Sq5qYUM
The nvhpc/26.5 Ewald problem was misdiagnosed as a compiler bug. It is the
documented 26.5 change to evaluating an OpenACC loop's tripcount on the
device: three loops in total_energy.f90 took their bound from a
derived-type member that is not device-resident, so they ran zero times
and their reductions returned the identity. FugakuNEXT-v4 fixes it at the
source by hoisting the bound into a local scalar; the loops stay on the
GPU and 26.5 matches 26.3 bit-for-bit.

Bump VERSION_TAG to FugakuNEXT-v4 and remove the patch, its apply function
and both call sites. Removal is required rather than tidy: the function
detected "already patched" by grepping for SALMON_EWALD_ACC, a macro v4
deletes, so it would have failed to match, then failed to apply against
the fixed source, and exited non-zero.

RC_GH200 and RC_DGXSP built stock v.2.2.2 with this patch applied. Both
use compilers below 26.5 (25.7 and 26.3), where the patch was already a
no-op, so dropping it changes nothing for them today -- but they will need
the source fix if either is moved to 26.5 or later.

Signed-off-by: William Dawson <william.dawson@riken.jp>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PEzS2ybD8VEnQa9Sq5qYUM
The patch chain had a RIKYU branch whose only job was to print that it was
not patching. Only Fugaku compiles the Fujitsu MPI path, so apply the patch
there and drop the special case.

Cut the comments back. The measurement history behind each flag does not
belong in a build script.

No functional change.

Signed-off-by: William Dawson <william.dawson@riken.jp>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PEzS2ybD8VEnQa9Sq5qYUM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant