PARCEL SYCL v1.2 / User Guide User Guide API Reference Search Download 日本語

13 Optimization

PARCEL has two kinds of optimizations that increase speed without changing the computation (the algorithm and the number of iterations): optimizations that change the loop structure of the kernels to suit the environment (blocking, ladder splitting, decomposing gemv, optimizations for A64FX), and optimization options that make the solvers faster by using more memory (“Choosing Between Memory and Speed”). Both are selected with macros at build time or environment variables at run time.

The effects in the following sections were measured with the code of the distribution package (the verification sample): executables that differ only in the optimization were run alternately in the same job, the number of iterations was fixed, and the time per iteration was compared. The problem is the same 3D Poisson equation as in the performance tests (NX=512) on 4 devices (4 sockets on CPUs, 4 nodes on Odyssey, 4 GPUs; the placement of ranks and threads is the same as in the measurement conditions of the performance chapter). The solvers are mainly pcg (100 iterations) and pgmres (mm = 60, 60 iterations; 120 iterations for blocking on s86 CPU and ladder splitting on PS-B) with the Jacobi preconditioner. The time per iteration is the time of the iteration part excluding initialization (the main loop) ÷ the number of iterations (the number of restarts for ks_arnoldi), averaged over 2 to 5 rounds. The times of the sections in the text and tables (dot products, SpMV, preconditioner, Gram-Schmidt, etc.) are also per-iteration times within the main loop. The performance ratio in each table is the time without the optimization ÷ the time with it; above 1 means that the optimization makes it faster. The main-effect column shows the sections with large differences as “without → with”.

The default form of the computational kernels (dot products, axpy, SpMV, preconditioners, gemv, etc.) is a flat loop in which one work item handles one row, which suits GPUs. PARCEL has optimizations that change the loop structure to suit environments such as CPUs and A64FX, selected by macros at build time. The macros are listed in “Optimization” in the Installation chapter, and the compile commands per environment in Recommended Options. The defaults per environment are in the next section, and how each optimization works and its effects in the sections that follow.

13.1 Defaults per Environment

The optimizations enabled by the recommended build (Recommended Options; the same as kernel=auto of the spack recipe parcel-sycl) are shown per environment. “—” means not used in that environment.

Environment Blocking Ladder splitting (SpMV / preconditioner) gemv decomposition For A64FX Register limit on V100 Memory saving G / H
s86 CPU, Miyabi-C, PS-A, Mac, Linux Workstation (CPU) Reductions only No / no No — — On / on
Odyssey All kernels Yes / no Yes A, C, D, E, F (B at run time) — On / off
s86 GPU (V100) No No / no No — Specified On / on
Miyabi-G (GH200), Aquarius (A100), Linux Workstation (RTX A6000) No No / no No — — On / on
PS-B (MI300A) No Yes / yes No — — On / on

13.2 Blocking

_BLOCKING_LOOP_=1 replaces all kernels except SpMV (dot products, axpy, preconditioners, etc.) with blocked versions, in which one thread handles a contiguous range of rows. A reduction only sums the partial sums of the threads at the end, so no barrier within a work group is needed. _BLOCKING_LOOP_DOT_=1 replaces only the reductions (dot, dddot, max) with blocked versions.

Flat (default): 1 row = 1 work item N work items (mapped to GPU threads) Reductions: tree reduction with barriers Blocked: 1 block = 1 work item = 1 thread block 0 s₀ block 1 s₁ block 2 s₂ block 3 s₃ Σ Sum the partial sums (no barrier) nblock = threads per rank (4 in the figure)
Figure: Flat loop and blocked loop. With blocking, one thread handles a contiguous range of rows, and reductions use partial sums.

The effect is concentrated in the reductions, because the flat reductions are tree reductions with barriers within work groups, and the fixed cost of a barrier is large on CPUs. The effect of blocking only the reductions (_BLOCKING_LOOP_DOT_=1), recommended for CPUs other than A64FX, is shown below.

Environment Solver Without blocking [ms] With blocking [ms] Performance ratio Main effect
s86 CPU pcg 172.4 130.5 1.32 dot 56.6 ms → 15.0 ms
s86 CPU pgmres 975.0 422.3 2.31 Gram-Schmidt 850.3 ms → 313.1 ms
Miyabi-C pcg 40.3 28.4 1.42 dot 15.0 ms → 4.0 ms
Miyabi-C pgmres 220.7 73.4 3.01 Gram-Schmidt 193.0 ms → 49.5 ms
PS-A pcg 24.0 22.7 1.06 dot 4.7 ms → 4.1 ms
PS-A pgmres 65.5 49.0 1.34 Gram-Schmidt 46.8 ms → 30.6 ms

The effect of blocking all kernels (_BLOCKING_LOOP_=1), recommended for A64FX, is shown below.

Environment Solver Without blocking [ms] With blocking [ms] Performance ratio Main effect
Odyssey pcg 1603.0 40.0 40.1 dot 1546.2 ms → 7.7 ms
Odyssey pgmres 16437.4 82.0 200 Gram-Schmidt 15848.4 ms → 48.8 ms

The number of blocks is determined at run time: the environment variable PARCEL_NBLOCK if set, otherwise omp_get_max_threads() (the number of threads per rank).

With blocked reductions, the number of partial sums depends on the run configuration (number of ranks × number of threads), and so does the order of the reduction. This does not affect results in double precision, but when comparing low-precision configurations across environments, use the same setting of this flag and the same run configuration.

13.3 Ladder Splitting

The iteration count of the inner loop of CRS SpMV is given by the number of nonzeros of each row, so the compiler does not know it and cannot unroll or prefetch. PARCEL_SPMV_LADDER=1 splits the inner loop into fixed-length stages of 4, 2 and 1 elements so that, for any number of nonzeros, all of them go into fixed-length bodies (4+2+1 for 7 nonzeros). The number of nonzeros may differ per row, and no extra memory is needed. PARCEL_PRECON_LADDER applies the same splitting to the iteration loops of the Jacobi preconditioner (PRECON_JACOBI).

Default: 7 iterations of 1 element (count unknown until run time) 1 2 3 4 5 6 7 Ladder: 4 → 2 → 1 1 2 3 4 4-element body 5 6 2 elements 7 1 element Example with 7 nonzeros per row. Each stage is a loop of fixed length, so the compiler can unroll and prefetch it. The number of nonzeros may differ per row.
Figure: Ladder splitting of the inner loop of SpMV. The nonzeros of a row are processed in fixed-length stages of 4, 2 and 1 elements.

The effect depends on the environment, and it can be slower. The time per iteration with and without ladder splitting is shown for PS-B (MI300A) and Odyssey (A64FX), where ladder splitting is recommended.

Environment Solver Without ladder splitting [ms] With ladder splitting [ms] Performance ratio Main effect
PS-B (MI300A) pcg 11.62 6.95 1.67 SpMV 4.9 ms → 2.6 ms
Preconditioner 4.7 ms → 2.4 ms
PS-B (MI300A) pgmres 24.81 19.88 1.25 SpMV 5.1 ms → 2.9 ms
Preconditioner 4.9 ms → 2.4 ms
Odyssey (A64FX) pcg 49.35 40.01 1.23 SpMV 23.6 ms → 14.3 ms
Preconditioner 15.2 ms → 15.3 ms
Odyssey (A64FX) pgmres 91.41 82.01 1.11 SpMV 24.2 ms → 14.8 ms
Preconditioner 15.3 ms → 15.4 ms

13.4 Decomposing gemv

gemv (y←y+Vc=y+∑jcjvjy \leftarrow y + Vc = y + \sum_{j} c_j v_j, where VV is a matrix of M columns and vjv_j its jj-th column) reads M columns at the same time and reads and writes yy only once. PARCEL_GEMV_AS_AXPY=1 decomposes it into M axpy operations (y←y+cjvjy \leftarrow y + c_j v_j).

gemv (default): y ← y + Σ cj vj at once v₀ v₁ v₂ v₃ y Reads M columns at once; reads/writes y once Decomposed into axpy: y ← y + cj vj, M times 1 v₀ y 2 v₁ y 3 v₂ y 4 v₃ y Reads one column at a time; reads/writes y M times
Figure: gemv and its decomposition into axpy (M = 4). The decomposition reads more data, but streams only one column at a time.

The decomposition reads and writes yy M times, which increases data movement, but only one column is streamed at a time. On A64FX, gemv streaming M columns at once can be slow (when the column spacing is a power of 2, the M columns fall into the same cache sets), so gemv is decomposed. On x86 CPUs and GPUs gemv is not decomposed.

In builds for CPUs (_BLOCKING_LOOP_ or _BLOCKING_LOOP_DOT_), the undecomposed gemv is also the row-block version that reads one column at a time within each block of rows (PARCEL_CPU_ROWBLOCK; default 1 on CPUs), and the column spacing of the basis is padded (PARCEL_BASIS_PAD; default 8 elements on CPUs). The cache conflicts of the M columns therefore rarely occur even without the decomposition, and the decomposition speeds up the gemv section of ks_arnoldi by only about 1.1 to 1.2 times (1.11 times for NX=128 and 1.20 times for NX=256 on one node of Odyssey).

On Odyssey, the time without the decomposition (-DPARCEL_GEMV_AS_AXPY=0) was compared with the time with it (the default). gemv is used heavily by ks_arnoldi and pcagmres (ks_arnoldi with m=40m = 40 for 10 restarts; pcagmres with s=8s = 8, t=4t = 4 and the Jacobi preconditioner for 64 iterations).

Environment Solver Not decomposed [ms] Decomposed [ms] Performance ratio Main effect
Odyssey ks_arnoldi (per restart) 2760.9 2645.7 1.04 gemv 888.1 ms → 738.9 ms
Odyssey pcagmres (per iteration) 65.9 65.6 1.00 gemv 0.55 ms → 0.45 ms

The default is the value of _BLOCKING_LOOP_, so it becomes 1 in builds for A64FX without being specified. gemv accumulates first and multiplies by the coefficient once, whereas the decomposed version multiplies each term by the coefficient before adding, so the rounding order changes.

13.5 Optimizations for A64FX (A to F)

Builds for A64FX (_BLOCKING_LOOP_=1) have the following six optimizations. B to F do not change the computation (the expression and the order of summation for each element), so the results are bit-identical to those without them (checked on Odyssey: the residual history agrees over 200 iterations for C and D, and until convergence (306 iterations) for E and F). A fuses multiply-adds into FMA instructions, so the rounding changes. In builds for x86 CPUs and GPUs, the code of C to F is not used and the generated code does not change.

What it does How to enable
A Fuses multiply-adds (a×b+ca \times b + c) into FMA instructions. The clang mode of the Fujitsu compiler does not fuse them by default -ffp-contract=fast at build time (Recommended Options)
B Keeps the OpenMP threads spinning while they wait between kernels, which reduces the fixed cost of launching and waiting for kernels At run time, OMP_WAIT_POLICY=ACTIVE and KMP_BLOCKTIME=infinite (recommended; not the default)
C Rewrites the axpy that takes coefficients by value (axpy_value / axpby_value) and the axpy-decomposed gemv (Decomposing gemv): splits the vector into contiguous ranges, one per thread, copies the pointers and coefficients to local (restrict) variables, and writes with std::fma the fusion fma(β,y,αx)\mathrm{fma}(\beta, y, \alpha x) that the Fujitsu compiler chooses for the original expression βy+αx\beta y + \alpha x (the path for β=1\beta = 1 skips the multiplication). Without it, depending on the surrounding code, the other form fma(α,x,βy)\mathrm{fma}(\alpha, x, \beta y) can be chosen and the rounding changes Default (PARCEL_A64FX_AXPY_C; disable with -DPARCEL_A64FX_AXPY_C=0)
D Fuses the kernels of one iteration of pcg (pstep = 1, the default): the updates of xx and pp (x←x+αpx \leftarrow x + \alpha p and p←s+βpp \leftarrow s + \beta p) into one kernel, and the update of rr and the dot product ⟨r,r⟩\langle r, r \rangle into one kernel. xx and pp are read and written once each, and the fixed cost of one kernel launch is saved Default (PARCEL_CG_FUSED; disable with -DPARCEL_CG_FUSED=0)
E In the preparation of the right-hand side of the Jacobi preconditioner (the kernel that forms DA−1bD_A^{-1} b), selects a version without the division when there is no vector scaling (the divisor is 1, the default for double and single precision). The SVE division on A64FX is not pipelined and took about 30% of the preconditioner section Default (PARCEL_JACOBI_NODIV_UNIT; disable with -DPARCEL_JACOBI_NODIV_UNIT=0)
F When the preconditioner matrix is single precision (precon_dataF etc.), processes two rows alternately in the Jacobi sweep so that the waits of the two dependent multiply-add chains overlap (the summation order within each row is unchanged). Not used for a double-precision preconditioner matrix (the compiler already overlaps rows, and it becomes slower) Default (PARCEL_JACOBI_ROW2; disable with -DPARCEL_JACOBI_ROW2=0)

The effect of A to F was measured on 4 nodes of Odyssey (16 ranks × 12 threads; 8.4 M rows per rank; pcg with the Jacobi preconditioner for 100 iterations and pgmres (m=60m = 60) for 60 iterations, time per iteration [ms], median of 3 rounds; the value in parentheses is the performance ratio, the time without optimization ÷ the time of that version). “Without optimization” is a build without any of A to F (without -ffp-contract=fast, with -DPARCEL_A64FX_AXPY_C=0 -DPARCEL_CG_FUSED=0 -DPARCEL_JACOBI_NODIV_UNIT=0 -DPARCEL_JACOBI_ROW2=0), run without B; A+B+C+D is the build with only E and F removed, and A+B+C+D+E+F is the recommended build, both run with B. All are the default builds that store the values of the Jacobi preconditioner.

Precision Solver Without optimization [ms] A+B+C+D [ms] (performance ratio) A+B+C+D+E+F [ms] (performance ratio)
Double pcg 46.64 44.68 (1.04) 39.97 (1.17)
Double pgmres 90.32 86.85 (1.04) 82.03 (1.10)
precon_dataF pcg 50.41 48.10 (1.05) 41.06 (1.23)
precon_dataF pgmres 93.69 90.08 (1.04) 83.21 (1.13)

Notes:

13.6 Register Limit on V100

On V100 (cuda:sm_70), -Xcuda-ptxas -maxrregcount=255 is added to the build. It raises the per-thread register limit of ptxas (the CUDA assembler); with the default allocation, the loads of the matrix and vectors in kernels such as the Jacobi preconditioner are split and serialized, which is slower. No spills occur and the results do not change. The flag is not used on GH200 and A100.

The time per iteration with and without the flag, for the recommended build (without ladder splitting) on s86 GPU, is shown below.

Environment Solver Without the flag [ms] With the flag [ms] Performance ratio Main effect
s86 GPU (V100) pcg 45.53 25.17 1.81 SpMV 18.0 ms → 8.7 ms
Preconditioner 20.8 ms → 9.9 ms
s86 GPU (V100) pgmres 94.63 74.29 1.27 SpMV 18.7 ms → 9.3 ms
Preconditioner 21.2 ms → 10.1 ms

13.7 Choosing Between Memory and Speed

The default settings of v1.2 enable two settings that reduce memory (only G in builds for A64FX; see below).

If memory is available, the following optimization options make the solvers faster by using more memory. Neither changes the algorithm or the number of iterations.

Optimization option How to specify Additional memory (per rank) What becomes faster Effect on results
Without H -DPARCEL_JACOBI_ONTHEFLY=0 at build time Values of BB: number of nonzeros of AA × size of the preconditioner matrix precision (about 7 vectors for a 7-point stencil in double precision) Application of the Jacobi preconditioner; effective on GPUs (almost no change on CPUs) Bit-identical with -ffp-contract=off (rounding may change in builds that allow fused multiply-add)
Without G PARCEL_PSTEP=4 at run time (or -DPARCEL_PCG_NO_XHIST=0 at build time) History of search directions of pcg: 3 vectors with halo Solution update of pcg (reads and writes of xx drop to 1/4) The solution is accumulated differently, so the results differ in the last digits (same number of iterations)

The time per iteration of pcg with memory-saving settings G and H switched on and off in the recommended build of each environment is shown below (the conditions are the same as at the beginning of the chapter). “G+H” means both G and H enabled, “G” only G enabled (without H), “H” only H enabled (without G), and “off” both disabled. The performance ratio is the time of off ÷ the time of G+H; below 1 means that disabling both is faster.

Environment Solver G+H [ms] G [ms] H [ms] off [ms] Performance ratio (off/G+H)
s86 CPU pcg 130.5 129.3 128.6 126.3 0.97
Miyabi-C pcg 28.38 27.94 27.93 26.41 0.93
PS-A pcg 22.69 22.31 22.16 21.93 0.97
Odyssey pcg 44.32 40.01 44.07 39.59 0.89
s86 GPU (V100) pcg 25.17 24.69 24.65 24.04 0.95
Miyabi-G (GH200) pcg 4.72 4.60 4.55 4.55 0.97
Aquarius (A100) pcg 12.91 12.61 12.47 12.36 0.96
PS-B (MI300A) pcg 6.67 6.48 6.59 6.35 0.95

The memory per device read in the same measurement is shown below, in the same order as the time table. The memory ratio is the memory of off ÷ the memory of G+H; above 1 means that disabling both uses more memory.

Environment Solver G+H [GiB] G [GiB] H [GiB] off [GiB] Memory ratio (off/G+H)
s86 CPU pcg 9.08 9.61 9.08 10.40 1.15
Miyabi-C pcg 8.11 8.64 8.10 9.43 1.16
PS-A pcg 10.31 11.10 10.26 11.91 1.15
Odyssey pcg 5.68 7.42 6.47 8.22 1.45
s86 GPU (V100) pcg 5.88 7.62 6.64 8.39 1.43
Miyabi-G (GH200) pcg 5.57 7.32 6.33 8.08 1.45
Aquarius (A100) pcg 5.99 7.74 6.75 8.50 1.42
PS-B (MI300A) pcg 5.57 7.32 6.33 8.08 1.45