PARCEL SYCL v1.2 / User Guide User Guide API Reference Search Download 日本語

7 Running on GPUs and in Jobs

This chapter shows how to run a built program on GPUs and how to run it through the job scheduler of a system.

7.1 Running on GPUs

7.1.1 Finding the Target

An executable for GPUs is built by giving the kind of GPU (the target) to --acpp-targets. The program is the same as for the CPU. The target is determined by the GPU generation; find it on a compute node as follows.

nvidia-smi --query-gpu=name,compute_cap --format=csv    # NVIDIA: compute capability (e.g. 7.0) -> cuda:sm_70
rocminfo | grep gfx                                       # AMD: gfx number (e.g. gfx942) -> hip:gfx942
GPU Compute capability / gfx --acpp-targets
V100 7.0 cuda:sm_70
A100 8.0 cuda:sm_80
RTX A6000 8.6 cuda:sm_86
H100 (GH200) 9.0 cuda:sm_90
MI300A gfx942 hip:gfx942

Install AdaptiveCpp with the backend of the GPU to be used (hipsycl+cuda or hipsycl+rocm amdgpu_target=<gfx> with spack; Installing a SYCL Compiler). With the spack recipe parcel-sycl, specify +cuda cuda_arch=<n> or +rocm amdgpu_target=<gfx> (“Choosing a Configuration”).

7.1.2 CUDA Version and Driver

On NVIDIA GPUs, AdaptiveCpp builds and runs the kernels with CUDA. The CUDA used must be a version supported by the driver of the compute nodes or older. The newest version supported by the driver is shown at the upper right of the output of nvidia-smi (CUDA Version). For example, the driver of s86 GPU supports up to CUDA 12.8, and CUDA 12.6 was used in the tests. Give the location of the CUDA used with the environment variable ACPP_CUDA_PATH, with the same value at build and run time (“Settings for use” of each environment; the spack recipe sets it at spack load).

7.1.3 One GPU per Rank

PARCEL uses one GPU per rank (“GPUs, MPI Ranks and Data” in “Basic Structure”). If a node has several GPUs, start as many ranks as GPUs and assign a different GPU to each rank. By default all ranks may select the same GPU, so place gpu_bind.sh (“Mapping One Rank to One Device”) between mpirun and the program.

mpicxx -O3 --acpp-targets=cuda:sm_80 -march=native -I/usr/include/openblas -o run prog.cpp -lopenblas
export OMP_NUM_THREADS=9            # threads per rank (roughly cores per node / GPUs per node)
mpirun -n 4 ./gpu_bind.sh ./run

Even on GPUs, each rank uses CPU threads for host-side processing. Examples of the number of threads per rank for each environment are given in the table of “GPUs, MPI Ranks and Data”. How MIG instances and APUs (MI300A) are handled is described in the same section.

7.1.4 Checking the Device Used

Even if the GPU assignment is wrong, or the GPU backend cannot be loaded, PARCEL keeps running on the CPU OpenMP backend and returns correct numbers. It only becomes slow, so this error cannot be noticed from the numerical results. Print the name of the device used by each rank to check.

// C++: print the name of the device of the queue
std::cout << "rank " << comm.mpi_rank() << ": "
          << q.get_device().get_info<sycl::info::device::name>() << std::endl;

If the name is that of a CPU rather than of a GPU (for example Tesla V100-SXM2-32GB), the GPU is not used. Check that --acpp-targets includes the GPU, that AdaptiveCpp was installed with the GPU backend, the location (ACPP_CUDA_PATH) and version of CUDA (“CUDA Version and Driver”), and that no more ranks than GPUs of the node are placed. On AMD GPUs, do not set HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES at the same time.

7.1.5 Recommended Flags

7.2 Running in Jobs

The compute nodes of a system are used through its job scheduler. The test environments of this manual used PBS (s86, Miyabi-C, Miyabi-G, PS-A, PS-B) and Fujitsu’s job management system (PJM; Odyssey, Aquarius). A job script does the following.

  1. Specify the resources (nodes, ranks, threads per rank, GPUs, time). The syntax, queue names and accounting groups differ between systems; check the user guide of the system.
  2. Apply the same “Settings for use” as for the build (modules, spack environment, LD_LIBRARY_PATH, ACPP_CUDA_PATH). Jobs may not inherit the settings of the login shell.
  3. Set OMP_NUM_THREADS to the number of threads per rank.
  4. Do not set OMP_PROC_BIND. With the LLVM OpenMP runtime, setting it gathers all threads onto one core and makes the run much slower (beginning of Installing a SYCL Compiler). Bind ranks to cores with the binding options of mpirun.
  5. On GPUs, place gpu_bind.sh between mpirun and the program (“Running on GPUs”).

Minimal examples follow. Replace <...> for your system.

#!/bin/bash
# PBS: 1 node, 4 GPUs, 4 ranks x 12 threads
#PBS -q <queue>
#PBS -l select=1:ncpus=48:mpiprocs=4:ngpus=4
#PBS -l walltime=01:00:00
#PBS -W group_list=<group>
#PBS -j oe
cd ${PBS_O_WORKDIR}
# Settings for use (section of each environment, or . <prefix>/setup.sh and spack load)
export OMP_NUM_THREADS=12
unset OMP_PROC_BIND
mpirun -n 4 ./gpu_bind.sh ./run
#!/bin/bash
# PBS: CPU of 1 node (2 sockets x 20 cores), 8 ranks x 5 threads
#PBS -q <queue>
#PBS -l select=1:ncpus=40:mpiprocs=8:ompthreads=5
#PBS -l walltime=01:00:00
#PBS -j oe
cd ${PBS_O_WORKDIR}
# Settings for use
export OMP_NUM_THREADS=5
unset OMP_PROC_BIND
mpirun -n 8 --map-by ppr:4:socket:PE=5 --bind-to core ./run    # Open MPI syntax
#!/bin/bash
# Fujitsu job management system (PJM): 1 A64FX node, 4 ranks x 12 threads (one rank per CMG)
#PJM -L "rscgrp=<resource group>"
#PJM -L "node=1"
#PJM -L "elapse=1:00:00"
#PJM -g <group>
#PJM --mpi "proc=4"
#PJM --omp thread=12
#PJM -j
cd ${PJM_O_WORKDIR}
# Settings for use
export OMP_WAIT_POLICY=ACTIVE    # recommended on A64FX ("Optimizations for A64FX")
export KMP_BLOCKTIME=infinite
mpiexec -n 4 ./run