16 Glossary
This chapter lists the names of computers and devices, software, PARCEL-specific terms, and basic terms of iterative methods used in this manual. Device specifications are those of the configurations used in the v1.2 tests. The environment setup and build procedure for each system are given in the chapters Installing a SYCL Compiler and Installation.
16.1 Systems
The ten systems used for the v1.2 tests. This manual refers to them by these names.
| Name | System | Node configuration | Setup |
|---|---|---|---|
| s86 CPU | SGI8600 CPU node | Intel Xeon Gold 6242R (20 cores) ×2 | s86 |
| s86 GPU | SGI8600 GPU node | Intel Xeon Gold 6248R (24 cores) ×2 + NVIDIA V100-SXM2-32GB ×4 | s86 |
| Odyssey | Odyssey of Wisteria/BDEC-01 (Wisteria-Odyssey) | Fujitsu A64FX (48 cores) ×1 | Odyssey |
| Aquarius | Aquarius of Wisteria/BDEC-01 (Wisteria-Aquarius) | Intel Xeon Platinum 8360Y (36 cores) ×2 + NVIDIA A100-SXM4-40GB ×8 | Aquarius |
| Miyabi-C | Miyabi CPU node | Intel Xeon CPU Max 9480 (56 cores, HBM2e) ×2 | Miyabi-C |
| Miyabi-G | Miyabi GPU node | NVIDIA GH200 ×1 (Grace 72 cores, H100 96GB) | Miyabi-G |
| PS-A | Plasma Simulator Subsystem A | Intel Xeon 6980P (128 cores) ×2 | PS-A |
| PS-B | Plasma Simulator Subsystem B | AMD Instinct MI300A ×4 (each = 24 cores + GPU) | PS-B |
| Mac | Mac with macOS 15 (installed with Homebrew) | Apple M4 (10 cores: 4 performance, 6 efficiency) | Mac |
| Linux Workstation | Linux on Windows WSL2 (Ubuntu 24.04, Rocky Linux 9) | Intel Xeon Gold 6226R (16 cores) ×2 + NVIDIA RTX A6000 ×2 | Linux Workstation |
“s86” alone means both s86 CPU and s86 GPU.
16.2 Processors and GPUs
- V100
-
NVIDIA GPU (Volta). Installed in s86 GPU. AdaptiveCpp target:
cuda:sm_70. - A100
-
NVIDIA GPU (Ampere). Installed in Aquarius. Target:
cuda:sm_80. Supports MIG. - RTX A6000
-
NVIDIA workstation GPU (Ampere, 48 GB memory). Installed in the Linux Workstation. Target:
cuda:sm_86. - H100
-
NVIDIA GPU (Hopper). In Miyabi-G it is the GPU part of the GH200. Target:
cuda:sm_90. Supports MIG. - GH200
- NVIDIA Grace Hopper Superchip: a Grace CPU and an H100 GPU in one module. One Miyabi-G node is one GH200.
- Grace
- NVIDIA Arm CPU (72 Neoverse V2 cores). The CPU part of the GH200.
- MI300A
-
AMD Instinct MI300A. An APU with CPU cores and a GPU in one package sharing HBM3. Installed in PS-B. AdaptiveCpp target:
hip:gfx942. See “APU (AMD Instinct MI300A)” for how PARCEL uses it. - A64FX
-
Fujitsu Arm CPU with SVE. Its 48 cores are grouped into four CMGs of 12 cores, each with its own HBM2. Installed in Odyssey. Target:
omp.library-only(used with the Fujitsu compiler). - Xeon
-
Intel x86_64 CPU. The v1.2 tests used Xeon Gold 6242R / 6248R (s86), Xeon Platinum 8360Y (Aquarius), Xeon CPU Max 9480 (Miyabi-C; with in-package HBM2e) and Xeon 6980P (PS-A). Target:
omp(OpenMP backend).
16.3 Components and Resource Units
- APU
- Accelerated Processing Unit: a processor with a CPU and a GPU in one package sharing memory. In this manual it means the MI300A.
- MIG
-
Multi-Instance GPU: a feature that partitions an NVIDIA GPU (A100, H100, etc.) into instances with their own SMs and memory, each usable as an independent GPU. The size of an instance is given by its profile name (e.g.
2g.24gb:<n>gis the amount of compute, the rest is the memory size). See “MIG (NVIDIA Multi-Instance GPU)” for how PARCEL uses it. - SM
- Streaming Multiprocessor: a group of compute units in an NVIDIA GPU. The parallelism of a GPU is determined by its number of SMs.
- CMG
- Core Memory Group: a unit of 12 cores of the A64FX and their HBM2. It corresponds to one NUMA node.
- HBM
- High Bandwidth Memory: high-bandwidth memory stacked in the same package as the processor. Used by GPUs, the A64FX (HBM2), Xeon CPU Max (HBM2e) and the MI300A (HBM3).
- NUMA node
- A group of cores that share nearby memory. Access to the memory of another NUMA node is slower, so the threads of one rank are kept within one NUMA node (GPUs, MPI Ranks and Data).
- vnode
- A virtual node of PBS (the job scheduler). On PS-B, resources are allocated as 1 NUMA node = 1 vnode = 1 APU.
16.4 Software
- SYCL
- A standard of the Khronos Group for writing parallel code for devices such as CPUs and GPUs in C++. PARCEL is written in SYCL.
- AdaptiveCpp
-
A SYCL implementation (formerly hipSYCL / Open SYCL). The compiler driver is
acpp, and--acpp-targetsselects the generated code (omp,cuda:sm_XX,hip:gfxXXX, etc.). The v1.2 tests used version 24.10. - Backend
-
The mechanism AdaptiveCpp uses to run kernels. The OpenMP backend (CPU,
omp), the CUDA backend (NVIDIA GPUs) and the HIP backend (AMD GPUs) were used. - CUDA / HIP and ROCm
- Development and runtime environments for NVIDIA and AMD GPUs, respectively. AdaptiveCpp uses them to generate and run GPU code.
- MPI
- A standard for inter-process communication. PARCEL uses it for the halo exchange between ranks and for reducing inner products. The tests used Open MPI, Intel MPI and Fujitsu MPI.
- spack
-
A package manager for HPC. AdaptiveCpp and LLVM were installed with spack on the v1.2 test systems.
spack/in the distribution package contains the PARCEL recipes and a script that installs everything from scratch. - PBS
- A job scheduler, used on s86, Miyabi and the Plasma Simulator (Wisteria/BDEC-01 uses Fujitsu’s job management system).
- PJM (Fujitsu job management system)
-
Fujitsu’s job scheduler (job management of Fujitsu Technical Computing Suite), used on Wisteria/BDEC-01 (Odyssey, Aquarius). Resources are written in the job script with
#PJM, and jobs are submitted withpjsub(“Running in Jobs”). - OpenMP
-
A standard for writing thread parallelism in C, C++ and Fortran. The CPU backend of AdaptiveCpp runs kernels with OpenMP threads. The number of threads is given with the environment variable
OMP_NUM_THREADS. - USM
- Unified Shared Memory: the mechanism of SYCL 2020 to allocate and access device memory through pointers. PARCEL places its vectors and matrices in USM device memory.
- BLAS, LAPACK, LAPACKE
-
BLAS and LAPACK are the standard interfaces (in Fortran) of libraries for basic vector and matrix operations and for dense matrix computations such as linear systems and eigenvalue problems. LAPACKE is the interface for calling LAPACK from C (header
lapacke.h). OpenBLAS and others provide implementations. PARCEL uses LAPACKE inks_arnoldifor eigenvalues of small dense matrices (“Installing LAPACKE”). - MPI compiler wrapper
-
Commands such as
mpicxx,mpiccandmpif90that call the underlying compiler with the MPI header and library options added. The compiler called can be changed with an environment variable (OMPI_CXXfor Open MPI,MPICH_CXXfor the MPICH family,I_MPI_CXXfor Intel MPI); PARCEL makes them callacpp. - PIE
-
Position Independent Executable: an executable that runs regardless of the address at which it is loaded.
acpp(clang) links as PIE, so objects linked together with it must be compiled with-fPIE(or-fPIC). - RPATH, LD_LIBRARY_PATH
-
Where shared libraries (
.so) are searched at run time. RPATH is embedded in the executable at link time with-Wl,-rpath,<dir>;LD_LIBRARY_PATHis an environment variable given at run time. Without either, libraries outside the standard locations (such as thelibomp.soof LLVM) are not found and the program does not start. - module (environment modules)
-
A mechanism to switch versions of software installed on a system with
module load <name>(Environment Modules, Lmod). It setsPATH,LD_LIBRARY_PATHand so on. Available modules are listed withmodule avail. It is different from the Fortran module (parcel_sycl_mod). - venv
-
A Python virtual environment, created with
python3 -m venv <dir>. Packages installed in it do not affect other Python environments. The examples install the Python packageparcelin one (“Python Package”).
16.5 PARCEL Terms
- Rank
- An MPI process. PARCEL assigns one device and a contiguous block of rows of the coefficient matrix to each rank.
- Device
- The resource a rank computes on: one GPU, one MIG instance, the GPU part of an APU, or a group of CPU cores (OpenMP backend) (GPUs, MPI Ranks and Data).
- Halo (ghost), halo exchange
- The vector elements owned by other ranks that a sparse matrix-vector product needs (the halo, also called ghost), and the communication that receives them.
- CRS
-
Compressed Row Storage: a sparse-matrix format with three arrays, row pointers (
rowptr), column indices and values. Matrices are passed to PARCEL in this format. - In-order queue
- A SYCL queue that executes kernels in the order they were submitted. PARCEL requires it.
- Double-double (dd)
-
A precision equivalent to quadruple precision that represents a value as the sum of two double-precision numbers [12,13]. Type
parcel_doubledouble(Precision Configurations). - FP16
-
Half precision (16-bit floating point). Type
parcel_fp16. - Collective operation, collective communication
-
An operation that all ranks of the communicator must call. It includes MPI collective communication such as
MPI_Allreduce(summing dot products).set_crsMat()andsolver()of PARCEL are collective, so they hang if only some ranks call them. Call them on all ranks with the same settings.
16.6 Computational Terms
- SpMV
- Sparse matrix-vector product . One of the most time-consuming operations in the iterations of Krylov subspace methods.
- axpy
- Linear combination of vectors (the BLAS name). In PARCEL the name also covers the form that scales ().
- gemv
- Matrix-vector product (the BLAS name). PARCEL uses it to add a linear combination of the columns (several vectors) of a tall matrix to .
- work item, work group
- Units of execution of a SYCL kernel. One work item handles one index (one matrix row by default in PARCEL). A group of work items is a work group; work items in a work group can synchronize (barrier).
- NX
- Number of grid points along one side of the 3-D grid used by the verification sample and the tests. The number of unknowns (matrix rows) is .
- Manufactured solution
- A verification method that builds the right-hand side from a chosen solution , solves it, and measures the error against the known solution. Used by the verification sample.
- Lucky breakdown
- In GMRES-type methods, a new basis vector becoming zero (rank deficiency of the basis) while the basis is built. It means the Krylov subspace contains the exact solution, which is obtained from the basis up to that point. Despite the name, it is not an error.
- Moment method
-
In the Cholesky QR of
pcagmres, a method that builds the Gram matrix with fewer inner products, using the fact that for a symmetric matrix the inner products of the basis vectors depend only on the moments (“Cholesky QR Method by the Moment Method”). - Krylov subspace method
-
A general name for iterative methods that seek an approximate solution in the subspace obtained by repeatedly multiplying a vector by the matrix (the Krylov subspace ). Examples are the conjugate gradient method (
pcg), BiCGStab (pbicgstab), GMRES (pgmres) and the Arnoldi method (ks_arnoldi). The matrix is used only through matrix-vector products, so they suit large sparse matrices. - Preconditioning
- To speed up convergence of an iterative method, prepare a matrix (the preconditioner) that is close to the coefficient matrix and easy to invert, and solve inside the iteration (approximately applying ). The closer is to , the fewer the iterations, but the cost per application grows (the “Preconditioners” chapter).
- Jacobi, block Jacobi, ILU(0)
-
Kinds of preconditioners. Point Jacobi takes only the diagonal of as ; the Jacobi preconditioner of PARCEL performs Jacobi iterations the number of times given by
set_jacobi(). Block Jacobi divides the rows of each rank into blocks and takes only the matrix inside the blocks as . PARCEL solves inside the blocks with ILU(0) (incomplete LU factorization that creates no values outside the original nonzero positions) (the “Preconditioners” chapter). - Restart
- In GMRES and similar methods, rebuilding the basis from the current approximate solution when the number of basis vectors reaches a fixed number (the restart period). It limits the memory of the basis and the cost of orthogonalization, but may slow convergence. In the Krylov-Schur method, it refers to shrinking and rebuilding the subspace.
- Relative residual
-
The norm of the residual divided by the norm of the right-hand side, . The linear solvers of PARCEL regard the iteration as converged when it falls below the tolerance (
set_tolerance()). There are the residual updated by the recurrence and the true residual recomputed from (“Recurrence Residual and True Residual”). - Symmetric positive definite
-
A matrix with and for every nonzero vector . All its eigenvalues are positive real numbers. The conjugate gradient method (
pcg) assumes this property. Typical examples are the difference matrices of diffusion and Poisson equations (with the appropriate sign). - Ritz value, Ritz pair
- An eigenvalue of the matrix restricted to a subspace (Ritz value), paired with the corresponding vector in the original space (Ritz vector). The Arnoldi and Krylov-Schur methods use Ritz values as approximations of the eigenvalues of the original matrix.
- Shift-invert
-
A method that computes eigenvalues near a shift by computing the eigenvalues of largest magnitude of . An eigenvalue of is mapped to , so eigenvalues near come to the side of largest magnitude. Each step requires solving (with the internal
pgmresin PARCEL).