Add new application: petsc-gmres

programs/petsc-gmres/FOM.awk

@@ -0,0 +1,31 @@
+#!/usr/bin/awk -f
+# Extract the per-routine detail behind the FOM from a PETSc -log_view file.
+# Column positions are -log_view's fixed layout: $4 time(max), $6 flop(max),
+# $21 total Mflop/s, $22 GPU Mflop/s. $4/$22 read "n/a" unless the run also
+# passed -log_view_gpu_time, so run.sh passes it on every GPU system.
+# Original by A. Suzuki, 25 Aug. 2026.
+BEGIN{
+   timeMatMult = 0.0; flopMatMult = 0.0; totalflopsMatMult = 0.0; gpuflopsMatMult = 0.0;
+   timeKSPSolve = 0.0; flopKSPSolve = 0.0; totalflopsKSPSolve = 0.0; gpuflopsKSPSolve = 0.0;
+   timeSFPack = 0.0;
+   timeSFUnpack = 0.0;
+}
+# space is mandatory to exclude MatMult{Add,Transpose}
+/^MatMult /{
+   timeMatMult = $4; flopMatMult = $6; totalflopsMatMult = $21; gpuflopsMatMult = $22;
+}
+/^KSPSolve/{
+   timeKSPSolve = $4; flopKSPSolve = $6; totalflopsKSPSolve = $21; gpuflopsKSPSolve = $22;
+}
+/^SFPack/{
+   timeSFPack = $4;
+}
+/^SFUnpack/{
+   timeSFUnpack = $4;
+}
+END{
+   printf("#routine     \ttime(sec) \tflop      flop/s(total) flop/s(GPU)\n");
+   printf("MatMult      \t%s \t%s \t%s \t%s\n", timeMatMult, flopMatMult, totalflopsMatMult, gpuflopsMatMult);
+   printf("KSPSolve     \t%s \t%s \t%s \t%s\n", timeKSPSolve, flopKSPSolve, totalflopsKSPSolve, gpuflopsKSPSolve);
+   printf("SFPack/Unpack\t%s\n", timeSFPack + timeSFUnpack);
+}

programs/petsc-gmres/README.md

@@ -0,0 +1,72 @@
+# petsc-gmres
+
+A PETSc KSP (GMRES + GAMG algebraic multigrid preconditioner) benchmark:
+load the `stokes2` sparse matrix (4,260,568 rows, 256,285,536 nonzeros —
+a Stokes flow saddle-point system on an ellipsoid mesh, generated via
+Gmsh/FreeFEM by Atsushi Suzuki), solve `Ax = b` for a known `x`, report
+the relative L2 error and GMRES-iterate wall-time (`FOM: ranks=<N>
+ksp_iter_time_s=<t>`).
+
+The FOM measures the GMRES **iterate only** — `KSPSetUp(ksp)` is called
+explicitly before the timed `KSPSolve`, so GAMG setup (coarsening + PtAP)
+is excluded. The iterate is the MatMult-bound SpMV loop that tracks memory
+bandwidth / Flop/s; setup is communication-bound and is read off
+`-log_view` (`PCSetUp_GAMG`, `MatMult`, ...) instead. (Previously the FOM
+included setup; it was narrowed after `-log_view` showed setup was ~70% of
+the timed `KSPSolve` on a 4-GPU run.)
+
+Source (`src/GMRES-PETSc.cpp`) is vendored directly in this directory
+rather than fetched at build time. PETSc itself is fetched via
+`bk_fetch_source` from its official repo, pinned to `v3.25.2`.
+
+Confirmed working on Rikyu (GB200 NVL4), Fugaku (A64FX), and R-CCS
+Cloud's DGX Spark (`RC_DGXSP`, GB10 Blackwell) — see this app's `build.sh`
+for the per-system recipe.
+
+## Correctness
+
+`relative L2 norm of the error` for `stokes2` lands near **0.23–0.26** at
+any rank count — higher than the ~0.03 typical for SPD matrices like the
+earlier `audikw_1`, because `stokes2` is a Stokes flow saddle-point system
+(velocity + pressure), which is indefinite and harder to solve. GMRES
+hits the default iteration limit rather than converging tightly, but the
+iteration count is stable across rank counts (193–232 on GPU), so the
+timing comparison is fair. A result far outside that band signals a real
+bug, not benchmark noise.
+
+## `-pc_gamg_square_graph 0`
+
+Required on every system, at any rank count above 1 (single-rank runs
+happen not to trigger it, but don't rely on that). Without it, GAMG's
+default aggressive-coarsening graph-squaring step blows up on the matrix's
+connectivity — a `CUSPARSE_STATUS_INSUFFICIENT_RESOURCES` crash on GPU,
+or a genuine multi-GB single-allocation PETSc "Out of memory" abort on
+CPU. Both `build.sh`'s configure line and `run.sh`'s launch command
+already account for everything needed except this flag, which is passed
+explicitly in `run.sh`.
+
+## Staging the data
+
+The matrix is generated by a FreeFEM script (`StokesPETSc0.edp`) that
+solves a Stokes flow on an ellipsoid mesh and writes the resulting matrix
+directly to PETSc binary format via `ObjectView(A, format="binary")` —
+the same format as PETSc's own `MatView`, so no separate conversion step
+(e.g. `mtx2petsc`) is needed. The `MatLoad` call in `GMRES-PETSc.cpp`
+reads it directly.
+
+Pre-staged locations (same convention as this repo's `ffb` and
+`LQCD_dw_solver`, which pre-stage their own — much larger — source
+archives at a fixed per-system path rather than fetching them at
+build/run time):
+
+| system | path |
+|---|---|
+| RIKYU | `/data1/rkp00015/benchkit-data/petsc-gmres/stokes2.dat` |
+| Fugaku | `/vol0500/data/ra250029/benchkit-data/petsc-gmres/stokes2.dat` — `/vol0002` (the volume the first attempt used) turned out to have a 0-byte hard quota limit for every group tried, `/vol0005` didn't; this app's `FJ` queue.csv template already declares `GFSCACHE` for `/vol0002:/vol0003:/vol0004:/vol0005`, so no per-app `-x` handling was needed once the right volume was found. Use `/vol0500`, not `/vol0005`, in an actual path — `/vol0005` is what a filesystem tool reports as the "resolved" location, but it isn't what resolves from a real compute-node job |
+| RC_DGXSP | `/home/users/william.dawson/benchkit-data/petsc-gmres/stokes2.dat` (this system has no separate group-storage tier, so — unlike RIKYU/Fugaku — this lives under `$HOME`) |
+
+To re-stage on a system: copy the `stokes2.dat` file (3,092,468,720 bytes)
+from an existing location above. To regenerate from scratch, run the
+FreeFEM script (`mpirun -np 8 FreeFem++-mpi StokesPETSc0.edp`) which writes
+`A.dat` in PETSc binary format — rename to `stokes2.dat`. Verify: 4,260,568
+× 4,260,568, nnz = 256,285,536.

programs/petsc-gmres/build.sh

@@ -0,0 +1,125 @@
+#!/bin/bash
+set -euo pipefail
+
+system="$1"
+
+# BenchKit invokes this as `bash programs/petsc-gmres/build.sh <system>`
+# from the repo root, not from inside this directory -- $PWD is the repo
+# root throughout (matching PETSC_DIR/ARTIFACT_DIR below, and bk_fetch_source's
+# own convention). src/GMRES-PETSc.cpp is *this script's own* source, so
+# anchor it to the script's location (APP_DIR) instead of assuming a
+# caller cwd -- found by actually running this through BenchKit's own
+# invocation convention rather than just replicating its commands by hand.
+APP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+PETSC_REPO="https://gitlab.com/petsc/petsc.git"
+PETSC_TAG="v3.25.2"
+PETSC_DIR="${PWD}/petsc"
+ARTIFACT_DIR="${PWD}/artifacts"
+
+source scripts/bk_functions.sh
+
+mkdir -p "${ARTIFACT_DIR}"
+bk_fetch_source "${PETSC_REPO}" "petsc" "${PETSC_TAG}"
+
+case "$system" in
+  RIKYU)
+    module load nvhpc-hpcx/26.3
+    export PETSC_ARCH=arch-rikyu-cuda
+    NVHPC=/shared/software/hpc_sdk/Linux_aarch64/26.3
+    NVPL="${NVHPC}/math_libs/nvpl/lib"
+    (
+      cd "${PETSC_DIR}"
+      # --with-fc=0: nvhpc-hpcx's Fortran wrapper doesn't support an F2018
+      # pointer-initialization feature PETSc's Fortran bindings need; this
+      # app is C++ only anyway. --with-cuda=1: run.sh launches this system
+      # with -mat_type aijcusparse, which a CPU-only PETSc does not
+      # register -- the two must agree. cuda-arch 100 is sm_100 (B200);
+      # hardcoded rather than probed with nvidia-smi (as RC_DGXSP does)
+      # because RIKYU is a cross-mode system whose build runner is a login
+      # node, so a GPU is not guaranteed to be visible at build time.
+      # Serial NVPL BLAS/LAPACK: this app runs flat MPI, no OpenMP.
+      ./configure \
+        --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 \
+        --with-debugging=0 \
+        --with-cuda=1 --with-cuda-dir="${NVHPC}/cuda/13.1" --with-cuda-arch=100 \
+        --with-blaslapack-lib="-L${NVPL} -Wl,-rpath,${NVPL} -lnvpl_lapack_lp64_seq -lnvpl_blas_lp64_seq" \
+        COPTFLAGS='-O3' CXXOPTFLAGS='-O3' CUDAOPTFLAGS='-O3'
+      make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" -j8 all
+    )
+    mpicxx -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
+      -I"${NVHPC}/cuda/13.1/include" \
+      "${APP_DIR}/src/GMRES-PETSc.cpp" -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      -Xlinker -rpath="${PETSC_DIR}/${PETSC_ARCH}/lib" \
+      -L"${PETSC_DIR}/${PETSC_ARCH}/lib" -lpetsc \
+      -L"${NVHPC}/cuda/13.1/lib64" -Xlinker -rpath="${NVHPC}/cuda/13.1/lib64" -lcudart
+    ;;
+  Fugaku)
+    # The LLVM cross-compiler (mpiclang++) measured faster than the
+    # Fujitsu compiler here, and Fugaku's Spack-provided PETSc hits the
+    # GAMG square-graph crash as a bare SIGKILL rather than a catchable
+    # PETSc error, so build our own.
+    module load lang/tcsds-1.2.43
+    module load LLVM/llvmorg-22.1.0
+    export PETSC_ARCH=arch-fugaku-llvm
+    TCSDS=/opt/FJSVxtclanga/tcsds-latest/lib64
+    ELF=/opt/FJSVxos/devkit/aarch64/rfs/usr/lib64/libelf.so
+    (
+      cd "${PETSC_DIR}"
+      # Login nodes are x86, compute nodes are A64FX/aarch64 -- a genuine
+      # cross-compile, hence --with-batch. BLAS/LAPACK is Fujitsu's own
+      # fjlapacksve (serial), not OpenBLAS -- OpenBLAS pulls in an
+      # unresolvable chain of Fortran-runtime symbols on this toolchain
+      # (RIKEN's own docs: "use of libraries provided with the Fujitsu
+      # compiler with the other compiler environments is not supported",
+      # but they document this exact non-Fujitsu-compiler linkage recipe).
+      ./configure \
+        --with-cc=mpiclang --with-cxx=mpiclang++ --with-fc=0 \
+        --with-debugging=0 --with-batch \
+        --with-blaslapack-lib="${TCSDS}/libfjlapacksve.so ${TCSDS}/libfj90i.so ${TCSDS}/libfj90f.so ${TCSDS}/libfjsrcinfo.so ${TCSDS}/libfjcrt.so ${ELF}" \
+        --PETSC_ARCH="${PETSC_ARCH}"
+      make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" all
+    )
+    # PETSCLIB is deliberately unquoted below: it's a space-separated list
+    # of separate -L/-Wl,-rpath/-l flags, not one path -- quoting it (an
+    # earlier version of this script did) passes the whole thing as a
+    # single malformed argument to the linker instead of word-splitting
+    # it into individual flags. Found the same way as the APP_DIR fix
+    # above: by actually running this build, not just reading it.
+    PETSCLIB="$(grep '^PETSC_WITH_EXTERNAL_LIB' "${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables" | cut -d= -f2-)"
+    # shellcheck disable=SC2086
+    mpiclang++ -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
+      "${APP_DIR}/src/GMRES-PETSc.cpp" -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      $PETSCLIB
+    ;;
+  RC_DGXSP)
+    # GPU build, 1 rank/GPU. -pc_gamg_square_graph 0 (set in run.sh)
+    # avoids a cuSPARSE crash on this matrix's connectivity during GAMG's
+    # aggressive-coarsening graph-squaring step.
+    source /etc/profile.d/modules.sh
+    module load system/ng-dgx nvhpc-hpcx
+    export PETSC_ARCH=arch-dgxsp-cuda
+    CUDA_ARCH=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1 | tr -d '.')
+    MATHLIBS=$(dirname "$(command -v nvcc)")/../../math_libs/*/lib64
+    (
+      cd "${PETSC_DIR}"
+      ./configure \
+        --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 \
+        --with-debugging=0 --with-cuda=1 --with-cuda-arch="${CUDA_ARCH}" \
+        LDFLAGS="-L${MATHLIBS} -Wl,-rpath,${MATHLIBS}" \
+        COPTFLAGS='-O3' CXXOPTFLAGS='-O3'
+      make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" -j8 all
+    )
+    CUDA_INC=$(dirname "$(command -v nvcc)")/../include
+    mpicxx -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
+      -I"${CUDA_INC}" \
+      "${APP_DIR}/src/GMRES-PETSc.cpp" -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      -Xlinker -rpath="${PETSC_DIR}/${PETSC_ARCH}/lib" \
+      -L"${PETSC_DIR}/${PETSC_ARCH}/lib" -lpetsc \
+      -L"${CUDA_INC}/../lib64" -Xlinker -rpath="${CUDA_INC}/../lib64" -lcudart
+    ;;
+  *)
+    echo "Unknown system: $system" >&2
+    exit 1
+    ;;
+esac

programs/petsc-gmres/list.csv

@@ -0,0 +1,7 @@
+system,enable,nodes,numproc_node,nthreads,elapse
+RIKYU,yes,1,1,1,0:30:00
+RIKYU,yes,1,2,1,0:30:00
+RIKYU,yes,1,4,1,0:30:00
+RIKYU,yes,2,4,1,0:30:00
+Fugaku,yes,1,48,1,0:15:00
+RC_DGXSP,yes,1,1,1,0:15:00