Add RIKYU support for SALMON - #75
Open
william-dawson wants to merge 12 commits into
Open
Conversation
william-dawson
marked this pull request as draft
July 31, 2026 06:59
william-dawson
force-pushed
the
add-salmon-rikyu
branch
from
July 31, 2026 07:12
f410203 to
bda79ba
Compare
william-dawson
force-pushed
the
add-salmon-rikyu
branch
from
August 5, 2026 16:18
bda79ba to
bd1a3b9
Compare
william-dawson
force-pushed
the
add-salmon-rikyu
branch
from
August 17, 2026 09:05
a018adf to
6f778dc
Compare
Build: nvhpc/26.3 + OpenACC + cc100 (B200/Blackwell), with FORTRAN_COMPILER_HAS_MPI_VERSION3=OFF to work around the MPI_Iallreduce hang at >1 rank. Run: per-rank GPU binding via wrapper.sh (CUDA_VISIBLE_DEVICES), UCX_IB_GPU_DIRECT_RDMA=no for 2+ nodes, and runtime patching of the ¶llel section to nproc_ob=N (orbital-only decomposition, no real-space grid split) which is the verified GPU-optimal layout. Depends on RIKEN-RCCS#74 for RIKYU system/queue config (gpu partition, account=rkp00012). 💘 Generated with Crush Signed-off-by: William Dawson <william.dawson@riken.jp> Assisted-by: glm-5.2 via Crush <crush@charm.land>
Two changes to the RIKYU SALMON benchmark, validated end-to-end on real hardware (job 28922, 4 GPUs, exit 0, FOM 337.2s, energy matching the known-good 3x3x3 reference to 6 sig figs at step 10): 1. GS is now offline prep, not part of the benchmark. run.sh reads a python-folded 3x3x3 SiO2 restart pre-staged at /data1/rkp00012/CX_input/SALMON/3x3x3-folded/ (restart/ symlinked in, never copied -- it's ~35GB) and runs TDDFT only. This replaces the old Si-1-1-1 GS+TDDFT flow for RIKYU; other systems are unaffected (uses_prestaged_restart() gates the new path). See subwg2-benchmarks' README.md/salmon-benchmarking for how the restart was produced (`benchgen salmon create --python-fold`) and regenerated if the source system size changes. 2. build.sh switches RIKYU from nvhpc/26.3 + MPI3-off to native nvhpc/26.5 + MPI3 on, plus a new patch (patches/nvhpc265-ewald-reduction.patch, cherry-picked from william-dawson/SALMON2 commit d8442058) that works around a 26.5-only OpenACC reduction-codegen bug in the Ewald ion-ion energy term. The patch is self-gating (checks __NVCOMPILER_MAJOR__/MINOR__ at compile time) so it's applied unconditionally on every GPU build (RIKYU, RC_GH200, RC_DGXSP) -- a no-op on compilers it doesn't affect. This combination is bit-exact and was already confirmed faster than the old 26.3+MPI3-off recipe on Rikyu's domain- decomposition case; this PR is the first real validation on the orbital-decomposition layout RIKYU's list.csv actually uses. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: William Dawson <william.dawson@riken.jp>
Extends the pre-staged-restart pattern (added for RIKYU) to RC_DGXSP.
run.sh now reads a python-folded 2x2x2 SiO2 restart staged at
/lvs0/dne1/rccs-nghpcadu/CX_input/SALMON_2x2x2_folded/ and runs TDDFT
only, skipping GS. build.sh needed no change here -- the nvhpc/26.5
Ewald patch added for RIKYU already applies unconditionally to every
GPU build including RC_DGXSP's (a no-op there, since it still builds
on 26.3, unaffected by the bug).
The restart was produced by running SALMON's own GS(k) stage for real
(18-atom primitive cell, 2x2x2 k-mesh, 8 ranks on one ng-dgx-m1 GPU) via
the existing patched 26.5 binary already on that filesystem, then
folding it locally with `benchgen salmon fold-restart --primitive-grid
29,49,31 --repetitions 2,2,2`.
Validated end-to-end on real hardware (job 286864, ng-dgx-m1, 1 GPU):
clean build, FOM 722.2s, correct electron count (768), stable energy
across all 300 RT steps, clean exit ("end SALMON").
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: William Dawson <william.dawson@riken.jp>
Per instruction: extend the pre-staged-restart pattern to Fugaku and fix list.csv's rank count (24x1, not 1x4 -- Fujitsu MPI-IO errors above ~2GB/rank on restart read). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: William Dawson <william.dawson@riken.jp>
…_CUDA v.2.2.2 (BenchKit's global VERSION_TAG) is over a year stale and was never actually what RIKYU's prior GPU-decomposition numbers were measured against -- those came from a hand-patched checkout ~100 commits ahead (develop-2.0.0@9b93a8c4 + PR #1276's OpenACC tuning), never landed here. Also, -DUSE_CUDA=ON was actively wrong for RIKYU: it selects SALMON's old hand-written CUDA kernels (zpseudo.cu, stencil_current.cu), which measured net *slower* than plain OpenACC at this problem size (pseudo-pt alone is 3.1-3.5x slower under CUDA), and stencil_current.cu's host-side wrapper has a real, deterministic device-buffer bug: it sizes its idx/idy/idz buffers by each rank's LOCAL grid extent but indexes them with the RAW/global grid coordinate. That only happens to fit when a rank owns index 1 on that axis (single GPU, or orbital-only decomposition) -- any real-space (nproc_rgrid>1) decomposition puts a non-first rank out of bounds and reproduces a deterministic CUDA_ERROR_ILLEGAL_ADDRESS in calc_current. Reproduced on every axis and Po x Pg combination tried; root-caused by reading stencil_current.cu directly. Fix: pin RIKYU specifically (not the other systems, which stay on stock v.2.2.2 pending their own re-validation) to a new tag, v.2.2.2-rikyu-optimized-265 on william-dawson/SALMON2 -- PR #1276's stencil/current/pseudo-pt OpenACC optimizations (batched cuBLAS GEMM for pseudo-pt via -DUSE_GEMM, not USE_CUDA) cherry-picked cleanly onto develop-2.0.0@9b93a8c4, combined with the nvhpc/26.5 Ewald-reduction fix already used here. This sidesteps the crash structurally (the buggy CUDA kernels aren't even compiled) while also being the actually-faster path. Verified on real hardware: build succeeds, and the three configs that previously crashed (y-axis and z-axis real-space decomposition, and a mixed Po x Pg case) all run correctly with matching energies -- see the subwg2-benchmarks salmon-gpu-optimization-ideas skill (Open item 5) for the full trail and numbers. Signed-off-by: William Dawson <william.dawson@riken.jp>
native MPI3 on nvhpc/26.5 (the previous pin) is faster for single-node and domain-decomposition cases, but genuinely hangs 8-GPU (2+ node) orbital decomposition: it dies right after init_ps with a UCC inter-node protocol error (cannot find remote protocol for: UCC_UCP_CONTEXT inter-node cfg#N | tag_send from cuda-managed/GPU0), then spins at ~99% CPU with zero further output. Confirmed this is not the previously-documented version-mixing artifact (that combo mixed a 26.3 binary with a 26.5 runtime; this hang reproduces on a clean, matched, native 26.5 build) and not a GPU-binding bug (verified correct per-rank CUDA_VISIBLE_DEVICES assignment via an instrumented wrapper.sh, no collisions). It's specific to native MPI3 at 2+ nodes on 26.5, which had never actually been exercised at multi-node scale on this project before. nvhpc/26.3 + FORTRAN_COMPILER_HAS_MPI_VERSION3=OFF is the one combination proven correct and hang-free at every scale BenchKit actually needs: 1/2/4/8-GPU all bit-exact on this pinned source (8-GPU orbital: 50.13s, matching -164703.595116277 eV), and 16/32-GPU proven on an earlier source revision (see rikyu-32gpu-scaling skill). The nvhpc/26.5 Ewald-reduction patch already applied here is a no-op on 26.3 (its version gate only fires for nvhpc>=26.5), so nothing needs to change about how it's applied. list.csv only exercises orbital decomposition, so the domain- decomposition speed edge nvhpc/26.5+MPI3-on had is not something BenchKit's own benchmark rows would ever have used anyway -- this trades it for actually working at the node counts list.csv covers. Signed-off-by: William Dawson <william.dawson@riken.jp>
FugakuNEXT-v1 (also open upstream as SALMON-TDDFT/SALMON2#1276) is the single consolidated branch replacing the ad-hoc chain this pin has gone through (v.2.2.2-rikyu-optimized-265 -> the nvhpc/26.3+MPI3-off workaround -> this): develop-2.0.0@9b93a8c4 plus four commits, one per concern -- the OpenACC tuning, the cmake wiring for it, the nvhpc/26.5 Ewald-reduction-bug guard, and (new) a real fix for the 2+ node UCC hang that previously forced the 26.3+MPI3-off downgrade. That fix (acc_init(acc_device_nvidia) before MPI_Init_thread, so UCC's CUDA-aware protocol probe during MPI_Init's team bootstrap doesn't run contextless and cache a broken config) means the downgrade is no longer needed: back to nvhpc/26.5, native MPI3 ON, no FORTRAN_COMPILER_HAS_MPI_VERSION3 override. This is the actually-tested combination now -- orbital decomposition 1-8 nodes and domain decomposition 1-2 nodes, all bit-exact, all previously-hanging configurations included. Signed-off-by: William Dawson <william.dawson@riken.jp>
Pin to FugakuNEXT-v2, which carries every fix RIKYU needs, and build it unpatched -- the Ewald workaround is in the branch, so patching on top of it was fragile (the reverse-apply check broke as soon as the branch reworded a comment). Non-RIKYU systems still patch stock v.2.2.2, and that helper now detects the fix by its macro rather than by patch context. Switch to nvhpc-hpcx-cuda13/26.5 (HPC-X 2.50 / OpenMPI 5), the stack all of this was measured on, and add -DUSE_NCCL: MPI_Allreduce does its reduction arithmetic on the host even given a device pointer, so the pseudo-pt reduce left NVLink. NCCL takes it from 18.2s to 1.7s at 4 GPUs. Run with mixed decomposition -- grid split 2x2x1 inside a node, orbitals across nodes, process_allocation='grid_sequential' so each icomm_r is one node's 4 ranks. Only the grid split reduces calculating-curr, which does not scale under orbital decomposition. rt iterations vs pure orbital: 8 GPUs 56.2 -> 48.0 s, 16 GPUs 44.6 -> 31.0 s, energies bit-exact. Add --mca fcoll individual, which is required: the restart read goes into managed memory and OMPIO's two-phase collective I/O redistributes it through cuMemcpyAsync, deadlocking whenever the nproc_rgrid product exceeds 2. Drop UCX_IB_GPU_DIRECT_RDMA=no, which was a 26.3 fix and is not needed on 26.5 (ablation-tested at 4, 8, 16 and 32 GPUs). Verified with the real build.sh/run.sh on RIKYU: 1 node FOM 133.18 s, 2 nodes FOM 97.73 s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: William Dawson <william.dawson@riken.jp>
…tions Benchmark orbital and mixed decomposition rather than picking one. They trade places depending on the problem and on which kernels have been optimised -- orbital is currently faster at every node count, but mixed is what a problem with less orbital parallelism falls back on -- so a benchmark should track both. Each emits its own result line, Exp ...-orbital and ...-mixed. Take the FOM from SALMON's rt-iterations timer instead of wall clock. The one-time restart read is 40-150 s depending on whether Lustre is cold, which both swamps the RT loop and, with two runs in one job, makes the second look far faster than the first purely from page cache. Production runs do very many steps, so the RT loop is the quantity of interest. fom-version becomes rt_iterations_s_folded_restart. Also bump the pin to FugakuNEXT-v3, which computes the nonlocal current density with a batched GEMM: 29.5 -> 3.6 s on that kernel and 44.6 -> 18.7 s on rt iterations at 16 GPUs orbital. Verified on RIKYU, 1 node: orbital 81.84 s, mixed 80.59 s (v2 binary), matching standalone measurements of the same layouts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: William Dawson <william.dawson@riken.jp>
william-dawson
force-pushed
the
add-salmon-rikyu
branch
from
August 26, 2026 00:03
b131e8f to
16a1144
Compare
william-dawson
marked this pull request as ready for review
August 26, 2026 00:04
The comments pointed at skills and documents in a separate, private repository that a reader of this repo cannot open, and explained root-cause history rather than what the code does. Rewrite them to be shorter and self-contained: state the reason a workaround exists, not the trail that found it. No functional change. Signed-off-by: William Dawson <william.dawson@riken.jp> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEzS2ybD8VEnQa9Sq5qYUM
The nvhpc/26.5 Ewald problem was misdiagnosed as a compiler bug. It is the documented 26.5 change to evaluating an OpenACC loop's tripcount on the device: three loops in total_energy.f90 took their bound from a derived-type member that is not device-resident, so they ran zero times and their reductions returned the identity. FugakuNEXT-v4 fixes it at the source by hoisting the bound into a local scalar; the loops stay on the GPU and 26.5 matches 26.3 bit-for-bit. Bump VERSION_TAG to FugakuNEXT-v4 and remove the patch, its apply function and both call sites. Removal is required rather than tidy: the function detected "already patched" by grepping for SALMON_EWALD_ACC, a macro v4 deletes, so it would have failed to match, then failed to apply against the fixed source, and exited non-zero. RC_GH200 and RC_DGXSP built stock v.2.2.2 with this patch applied. Both use compilers below 26.5 (25.7 and 26.3), where the patch was already a no-op, so dropping it changes nothing for them today -- but they will need the source fix if either is moved to 26.5 or later. Signed-off-by: William Dawson <william.dawson@riken.jp> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEzS2ybD8VEnQa9Sq5qYUM
william-dawson
force-pushed
the
add-salmon-rikyu
branch
from
August 26, 2026 01:19
74f1305 to
9ec2925
Compare
The patch chain had a RIKYU branch whose only job was to print that it was not patching. Only Fugaku compiles the Fujitsu MPI path, so apply the patch there and drop the special case. Cut the comments back. The measurement history behind each flag does not belong in a build script. No functional change. Signed-off-by: William Dawson <william.dawson@riken.jp> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEzS2ybD8VEnQa9Sq5qYUM
william-dawson
force-pushed
the
add-salmon-rikyu
branch
from
August 26, 2026 01:25
cbfb921 to
06a9f03
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds RIKYU (GB200 NVL4, 4 GPUs/node) as a target system for the SALMON benchmark.
[code:salmon]1. Pinned to the optimized SALMON branch. For RIKYU only,
build.shbuildsFugakuNEXT-v4onwilliam-dawson/SALMON2(open upstream as SALMON-TDDFT/SALMON2#1276) instead of thev.2.2.2tag, which is 14+ months stale and predates the OpenACC tuning, the Ewald loop-tripcount fix and the multi-node CUDA-context fix. Every other system stays onv.2.2.2.Built with
nvhpc-hpcx-cuda13/26.5,-DUSE_OPENACC -DUSE_GEMM -DUSE_NCCLand-cudalib=cublas,cusolver,nccl.USE_CUDAis deliberately not used — SALMON's hand-written CUDA kernels measured slower than the OpenACC path here.The Fujitsu MPI topology patch is now applied only on
Fugaku/FugakuCN, since no other system compiles that path.2. Added RIKYU directions.
build.sh,run.shandlist.csventries for RIKYU at 1-4 nodes (4 GPUs/node), including per-rank GPU binding via awrapper.sh. Depends on #74 for the RIKYU system/queue config.--mca fcoll individualis required, not a tuning knob: the restart read does oneMPI_File_read_allper rank into managed memory, and OMPIO's two-phase collective path deadlocks whenever thenproc_rgridproduct exceeds 2.UCX_IB_GPU_DIRECT_RDMA=nowas needed on nvhpc 26.3 to stop a multi-node hang. Ablation-tested on 26.5 at 4, 8, 16 and 32 GPUs: no longer needed, and neither isUCX_TLS=^cma.3. Both decompositions are benchmarked. At 4 ranks/node RIKYU runs the RT loop twice and emits two results,
…-orbitaland…-mixed:nproc_ob = n_ranks,nproc_rgrid = 1,1,1,process_allocation='orbital_sequential'nproc_ob = nodes,nproc_rgrid = 2,2,1,process_allocation='grid_sequential', which keeps the grid split inside a node (NVLink) and spreads orbitals across nodes; the default allocation would strideicomm_racross nodes.Orbital is currently faster at every node count measured, but mixed is what a problem with less orbital parallelism has to fall back on, so both are worth tracking.
The FOM is SALMON's own
rt iterationstimer, not wall clock. The one-time restart read is ~40-150 s depending on whether Lustre is cold, which swamps and reorders the comparison when two layouts run in one job. Production runs do very many steps, so the RT loop is the part being benchmarked.4. The benchmark starts from a pre-folded restart, so the ground state is out of scope.
run.shpreviously computed the GS as part of the benchmark. It now reads a k-point-folded restart prepared offline and runs TDDFT only. Restarts are pre-staged per system:/data1/rkp00012/CX_input/SALMON/3x3x3-folded//lvs0/dne1/rccs-nghpcadu/CX_input/SALMON_2x2x2_folded//home/ra000009/data/u10035/CX_input_fugaku/SALMON_3x3x3_folded/Fugaku's
list.csvrow moves from 1 node to 24 nodes as a consequence. SALMON's default restart reader (method_wf_distributor='single') does oneMPI_File_read_allper rank spanning its whole orbital slice, and Fujitsu MPI's MPI-IO throwsMPI_ERR_ARGon per-rank reads above ~2 GB. 24 ranks (81 orbitals/rank, ~1.5 GB/rank) is the smallest node count validated for the 3x3x3 case; going lower reintroduces the failure. This is a real vendor-specific MPI-IO limit, reproduced independently of restart provenance and of physical memory.