PARCEL SYCL v1.2 / User Guide User Guide API Reference Search Download 日本語

16 Glossary

This chapter lists the names of computers and devices, software, PARCEL-specific terms, and basic terms of iterative methods used in this manual. Device specifications are those of the configurations used in the v1.2 tests. The environment setup and build procedure for each system are given in the chapters Installing a SYCL Compiler and Installation.

16.1 Systems

The ten systems used for the v1.2 tests. This manual refers to them by these names.

Name System Node configuration Setup
s86 CPU SGI8600 CPU node Intel Xeon Gold 6242R (20 cores) ×2 s86
s86 GPU SGI8600 GPU node Intel Xeon Gold 6248R (24 cores) ×2 + NVIDIA V100-SXM2-32GB ×4 s86
Odyssey Odyssey of Wisteria/BDEC-01 (Wisteria-Odyssey) Fujitsu A64FX (48 cores) ×1 Odyssey
Aquarius Aquarius of Wisteria/BDEC-01 (Wisteria-Aquarius) Intel Xeon Platinum 8360Y (36 cores) ×2 + NVIDIA A100-SXM4-40GB ×8 Aquarius
Miyabi-C Miyabi CPU node Intel Xeon CPU Max 9480 (56 cores, HBM2e) ×2 Miyabi-C
Miyabi-G Miyabi GPU node NVIDIA GH200 ×1 (Grace 72 cores, H100 96GB) Miyabi-G
PS-A Plasma Simulator Subsystem A Intel Xeon 6980P (128 cores) ×2 PS-A
PS-B Plasma Simulator Subsystem B AMD Instinct MI300A ×4 (each = 24 cores + GPU) PS-B
Mac Mac with macOS 15 (installed with Homebrew) Apple M4 (10 cores: 4 performance, 6 efficiency) Mac
Linux Workstation Linux on Windows WSL2 (Ubuntu 24.04, Rocky Linux 9) Intel Xeon Gold 6226R (16 cores) ×2 + NVIDIA RTX A6000 ×2 Linux Workstation

“s86” alone means both s86 CPU and s86 GPU.

16.2 Processors and GPUs

V100
NVIDIA GPU (Volta). Installed in s86 GPU. AdaptiveCpp target: cuda:sm_70.
A100
NVIDIA GPU (Ampere). Installed in Aquarius. Target: cuda:sm_80. Supports MIG.
RTX A6000
NVIDIA workstation GPU (Ampere, 48 GB memory). Installed in the Linux Workstation. Target: cuda:sm_86.
H100
NVIDIA GPU (Hopper). In Miyabi-G it is the GPU part of the GH200. Target: cuda:sm_90. Supports MIG.
GH200
NVIDIA Grace Hopper Superchip: a Grace CPU and an H100 GPU in one module. One Miyabi-G node is one GH200.
Grace
NVIDIA Arm CPU (72 Neoverse V2 cores). The CPU part of the GH200.
MI300A
AMD Instinct MI300A. An APU with CPU cores and a GPU in one package sharing HBM3. Installed in PS-B. AdaptiveCpp target: hip:gfx942. See “APU (AMD Instinct MI300A)” for how PARCEL uses it.
A64FX
Fujitsu Arm CPU with SVE. Its 48 cores are grouped into four CMGs of 12 cores, each with its own HBM2. Installed in Odyssey. Target: omp.library-only (used with the Fujitsu compiler).
Xeon
Intel x86_64 CPU. The v1.2 tests used Xeon Gold 6242R / 6248R (s86), Xeon Platinum 8360Y (Aquarius), Xeon CPU Max 9480 (Miyabi-C; with in-package HBM2e) and Xeon 6980P (PS-A). Target: omp (OpenMP backend).

16.3 Components and Resource Units

APU
Accelerated Processing Unit: a processor with a CPU and a GPU in one package sharing memory. In this manual it means the MI300A.
MIG
Multi-Instance GPU: a feature that partitions an NVIDIA GPU (A100, H100, etc.) into instances with their own SMs and memory, each usable as an independent GPU. The size of an instance is given by its profile name (e.g. 2g.24gb: <n>g is the amount of compute, the rest is the memory size). See “MIG (NVIDIA Multi-Instance GPU)” for how PARCEL uses it.
SM
Streaming Multiprocessor: a group of compute units in an NVIDIA GPU. The parallelism of a GPU is determined by its number of SMs.
CMG
Core Memory Group: a unit of 12 cores of the A64FX and their HBM2. It corresponds to one NUMA node.
HBM
High Bandwidth Memory: high-bandwidth memory stacked in the same package as the processor. Used by GPUs, the A64FX (HBM2), Xeon CPU Max (HBM2e) and the MI300A (HBM3).
NUMA node
A group of cores that share nearby memory. Access to the memory of another NUMA node is slower, so the threads of one rank are kept within one NUMA node (GPUs, MPI Ranks and Data).
vnode
A virtual node of PBS (the job scheduler). On PS-B, resources are allocated as 1 NUMA node = 1 vnode = 1 APU.

16.4 Software

SYCL
A standard of the Khronos Group for writing parallel code for devices such as CPUs and GPUs in C++. PARCEL is written in SYCL.
AdaptiveCpp
A SYCL implementation (formerly hipSYCL / Open SYCL). The compiler driver is acpp, and --acpp-targets selects the generated code (omp, cuda:sm_XX, hip:gfxXXX, etc.). The v1.2 tests used version 24.10.
Backend
The mechanism AdaptiveCpp uses to run kernels. The OpenMP backend (CPU, omp), the CUDA backend (NVIDIA GPUs) and the HIP backend (AMD GPUs) were used.
CUDA / HIP and ROCm
Development and runtime environments for NVIDIA and AMD GPUs, respectively. AdaptiveCpp uses them to generate and run GPU code.
MPI
A standard for inter-process communication. PARCEL uses it for the halo exchange between ranks and for reducing inner products. The tests used Open MPI, Intel MPI and Fujitsu MPI.
spack
A package manager for HPC. AdaptiveCpp and LLVM were installed with spack on the v1.2 test systems. spack/ in the distribution package contains the PARCEL recipes and a script that installs everything from scratch.
PBS
A job scheduler, used on s86, Miyabi and the Plasma Simulator (Wisteria/BDEC-01 uses Fujitsu’s job management system).
PJM (Fujitsu job management system)
Fujitsu’s job scheduler (job management of Fujitsu Technical Computing Suite), used on Wisteria/BDEC-01 (Odyssey, Aquarius). Resources are written in the job script with #PJM, and jobs are submitted with pjsub (“Running in Jobs”).
OpenMP
A standard for writing thread parallelism in C, C++ and Fortran. The CPU backend of AdaptiveCpp runs kernels with OpenMP threads. The number of threads is given with the environment variable OMP_NUM_THREADS.
USM
Unified Shared Memory: the mechanism of SYCL 2020 to allocate and access device memory through pointers. PARCEL places its vectors and matrices in USM device memory.
BLAS, LAPACK, LAPACKE
BLAS and LAPACK are the standard interfaces (in Fortran) of libraries for basic vector and matrix operations and for dense matrix computations such as linear systems and eigenvalue problems. LAPACKE is the interface for calling LAPACK from C (header lapacke.h). OpenBLAS and others provide implementations. PARCEL uses LAPACKE in ks_arnoldi for eigenvalues of small dense matrices (“Installing LAPACKE”).
MPI compiler wrapper
Commands such as mpicxx, mpicc and mpif90 that call the underlying compiler with the MPI header and library options added. The compiler called can be changed with an environment variable (OMPI_CXX for Open MPI, MPICH_CXX for the MPICH family, I_MPI_CXX for Intel MPI); PARCEL makes them call acpp.
PIE
Position Independent Executable: an executable that runs regardless of the address at which it is loaded. acpp (clang) links as PIE, so objects linked together with it must be compiled with -fPIE (or -fPIC).
RPATH, LD_LIBRARY_PATH
Where shared libraries (.so) are searched at run time. RPATH is embedded in the executable at link time with -Wl,-rpath,<dir>; LD_LIBRARY_PATH is an environment variable given at run time. Without either, libraries outside the standard locations (such as the libomp.so of LLVM) are not found and the program does not start.
module (environment modules)
A mechanism to switch versions of software installed on a system with module load <name> (Environment Modules, Lmod). It sets PATH, LD_LIBRARY_PATH and so on. Available modules are listed with module avail. It is different from the Fortran module (parcel_sycl_mod).
venv
A Python virtual environment, created with python3 -m venv <dir>. Packages installed in it do not affect other Python environments. The examples install the Python package parcel in one (“Python Package”).

16.5 PARCEL Terms

Rank
An MPI process. PARCEL assigns one device and a contiguous block of rows of the coefficient matrix to each rank.
Device
The resource a rank computes on: one GPU, one MIG instance, the GPU part of an APU, or a group of CPU cores (OpenMP backend) (GPUs, MPI Ranks and Data).
Halo (ghost), halo exchange
The vector elements owned by other ranks that a sparse matrix-vector product needs (the halo, also called ghost), and the communication that receives them.
CRS
Compressed Row Storage: a sparse-matrix format with three arrays, row pointers (rowptr), column indices and values. Matrices are passed to PARCEL in this format.
In-order queue
A SYCL queue that executes kernels in the order they were submitted. PARCEL requires it.
Double-double (dd)
A precision equivalent to quadruple precision that represents a value as the sum of two double-precision numbers [12,13]. Type parcel_doubledouble (Precision Configurations).
FP16
Half precision (16-bit floating point). Type parcel_fp16.
Collective operation, collective communication
An operation that all ranks of the communicator must call. It includes MPI collective communication such as MPI_Allreduce (summing dot products). set_crsMat() and solver() of PARCEL are collective, so they hang if only some ranks call them. Call them on all ranks with the same settings.

16.6 Computational Terms

SpMV
Sparse matrix-vector product y=Axy = Ax. One of the most time-consuming operations in the iterations of Krylov subspace methods.
axpy
Linear combination of vectors y←αx+yy \leftarrow \alpha x + y (the BLAS name). In PARCEL the name also covers the form that scales yy (y←αx+βyy \leftarrow \alpha x + \beta y).
gemv
Matrix-vector product y←αVc+βyy \leftarrow \alpha Vc + \beta y (the BLAS name). PARCEL uses it to add a linear combination of the columns (several vectors) of a tall matrix VV to yy.
work item, work group
Units of execution of a SYCL kernel. One work item handles one index (one matrix row by default in PARCEL). A group of work items is a work group; work items in a work group can synchronize (barrier).
NX
Number of grid points along one side of the 3-D grid used by the verification sample and the tests. The number of unknowns (matrix rows) is NX3NX^{3}.
Manufactured solution
A verification method that builds the right-hand side b=Axb = Ax from a chosen solution xx, solves it, and measures the error against the known solution. Used by the verification sample.
Lucky breakdown
In GMRES-type methods, a new basis vector becoming zero (rank deficiency of the basis) while the basis is built. It means the Krylov subspace contains the exact solution, which is obtained from the basis up to that point. Despite the name, it is not an error.
Moment method
In the Cholesky QR of pcagmres, a method that builds the Gram matrix with fewer inner products, using the fact that for a symmetric matrix the inner products of the basis vectors depend only on the moments μk=v0TAkv0\mu_{k} = v_{0}^{T}A^{k}v_{0} (“Cholesky QR Method by the Moment Method”).
Krylov subspace method
A general name for iterative methods that seek an approximate solution in the subspace obtained by repeatedly multiplying a vector bb by the matrix AA (the Krylov subspace span{b,Ab,A2b,…}\mathrm{span}\{b, Ab, A^{2}b, \ldots\}). Examples are the conjugate gradient method (pcg), BiCGStab (pbicgstab), GMRES (pgmres) and the Arnoldi method (ks_arnoldi). The matrix is used only through matrix-vector products, so they suit large sparse matrices.
Preconditioning
To speed up convergence of an iterative method, prepare a matrix KK (the preconditioner) that is close to the coefficient matrix AA and easy to invert, and solve Kz=rKz = r inside the iteration (approximately applying A−1A^{-1}). The closer KK is to AA, the fewer the iterations, but the cost per application grows (the “Preconditioners” chapter).
Jacobi, block Jacobi, ILU(0)
Kinds of preconditioners. Point Jacobi takes only the diagonal of AA as KK; the Jacobi preconditioner of PARCEL performs Jacobi iterations the number of times given by set_jacobi(). Block Jacobi divides the rows of each rank into blocks and takes only the matrix inside the blocks as KK. PARCEL solves inside the blocks with ILU(0) (incomplete LU factorization that creates no values outside the original nonzero positions) (the “Preconditioners” chapter).
Restart
In GMRES and similar methods, rebuilding the basis from the current approximate solution when the number of basis vectors reaches a fixed number (the restart period). It limits the memory of the basis and the cost of orthogonalization, but may slow convergence. In the Krylov-Schur method, it refers to shrinking and rebuilding the subspace.
Relative residual
The norm of the residual r=b−Axr = b - Ax divided by the norm of the right-hand side, ∥r∥/∥b∥\|r\|/\|b\|. The linear solvers of PARCEL regard the iteration as converged when it falls below the tolerance (set_tolerance()). There are the residual updated by the recurrence and the true residual recomputed from b−Axb - Ax (“Recurrence Residual and True Residual”).
Symmetric positive definite
A matrix with A=ATA = A^{T} and xTAx>0x^{T}Ax > 0 for every nonzero vector xx. All its eigenvalues are positive real numbers. The conjugate gradient method (pcg) assumes this property. Typical examples are the difference matrices of diffusion and Poisson equations (with the appropriate sign).
Ritz value, Ritz pair
An eigenvalue of the matrix restricted to a subspace (Ritz value), paired with the corresponding vector in the original space (Ritz vector). The Arnoldi and Krylov-Schur methods use Ritz values as approximations of the eigenvalues of the original matrix.
Shift-invert
A method that computes eigenvalues near a shift σ\sigma by computing the eigenvalues of largest magnitude of (A−σI)−1(A - \sigma I)^{-1}. An eigenvalue λ\lambda of AA is mapped to 1/(λ−σ)1/(\lambda - \sigma), so eigenvalues near σ\sigma come to the side of largest magnitude. Each step requires solving (A−σI)z=v(A - \sigma I)z = v (with the internal pgmres in PARCEL).