13 Optimization
PARCEL has two kinds of optimizations that increase speed without changing the computation (the algorithm and the number of iterations): optimizations that change the loop structure of the kernels to suit the environment (blocking, ladder splitting, decomposing gemv, optimizations for A64FX), and optimization options that make the solvers faster by using more memory (“Choosing Between Memory and Speed”). Both are selected with macros at build time or environment variables at run time.
The effects in the following sections were measured with the code of the distribution package (the verification sample): executables that differ only in the optimization were run alternately in the same job, the number of iterations was fixed, and the time per iteration was compared. The problem is the same 3D Poisson equation as in the performance tests (NX=512) on 4 devices (4 sockets on CPUs, 4 nodes on Odyssey, 4 GPUs; the placement of ranks and threads is the same as in the measurement conditions of the performance chapter). The solvers are mainly pcg (100 iterations) and pgmres ( = 60, 60 iterations; 120 iterations for blocking on s86 CPU and ladder splitting on PS-B) with the Jacobi preconditioner. The time per iteration is the time of the iteration part excluding initialization (the main loop) ÷ the number of iterations (the number of restarts for ks_arnoldi), averaged over 2 to 5 rounds. The times of the sections in the text and tables (dot products, SpMV, preconditioner, Gram-Schmidt, etc.) are also per-iteration times within the main loop. The performance ratio in each table is the time without the optimization ÷ the time with it; above 1 means that the optimization makes it faster. The main-effect column shows the sections with large differences as “without → with”.
The default form of the computational kernels (dot products, axpy, SpMV, preconditioners, gemv, etc.) is a flat loop in which one work item handles one row, which suits GPUs. PARCEL has optimizations that change the loop structure to suit environments such as CPUs and A64FX, selected by macros at build time. The macros are listed in “Optimization” in the Installation chapter, and the compile commands per environment in Recommended Options. The defaults per environment are in the next section, and how each optimization works and its effects in the sections that follow.
13.1 Defaults per Environment
The optimizations enabled by the recommended build (Recommended Options; the same as kernel=auto of the spack recipe parcel-sycl) are shown per environment. “—” means not used in that environment.
| Environment | Blocking | Ladder splitting (SpMV / preconditioner) | gemv decomposition | For A64FX | Register limit on V100 | Memory saving G / H |
|---|---|---|---|---|---|---|
| s86 CPU, Miyabi-C, PS-A, Mac, Linux Workstation (CPU) | Reductions only | No / no | No | — | — | On / on |
| Odyssey | All kernels | Yes / no | Yes | A, C, D, E, F (B at run time) | — | On / off |
| s86 GPU (V100) | No | No / no | No | — | Specified | On / on |
| Miyabi-G (GH200), Aquarius (A100), Linux Workstation (RTX A6000) | No | No / no | No | — | — | On / on |
| PS-B (MI300A) | No | Yes / yes | No | — | — | On / on |
- “Reductions only” in the Blocking column is
_BLOCKING_LOOP_DOT_=1, and “All kernels” is_BLOCKING_LOOP_=1(Blocking). “For A64FX” refers to Optimizations for A64FX (A to F), and “Memory saving G / H” to the settings G and H of “Choosing Between Memory and Speed” (separate from the optimizations A to F for A64FX). - For Linux Workstation, the blocking is reductions only on the CPU (Xeon Gold 6226R) and none on the GPU (RTX A6000). Both are the same as the defaults for environments not in the table (next item).
- In environments not in the table, CPUs are the same as s86 CPU, NVIDIA GPUs as Miyabi-G, and AMD GPUs as PS-B (
kernel=autoof spack). - The defaults of the ladder splitting of the preconditioner, the gemv decomposition, C, D and F of the optimizations for A64FX, and memory-saving setting H follow from the blocking and ladder macros (the default of
PARCEL_PRECON_LADDERis the value ofPARCEL_SPMV_LADDER; those ofPARCEL_GEMV_AS_AXPY,PARCEL_CG_FUSEDandPARCEL_JACOBI_ROW2are the value of_BLOCKING_LOOP_; that ofPARCEL_JACOBI_ONTHEFLYis 0 if_BLOCKING_LOOP_is 1). E (PARCEL_JACOBI_NODIV_UNIT) defaults to 1 and is effective only in_BLOCKING_LOOP_builds. To change them individually, specify the macros in “Optimization” in the Installation chapter. - A of the optimizations for A64FX (
-ffp-contract=fast) is given on the command line, and B (OMP_WAIT_POLICY=ACTIVEandKMP_BLOCKTIME=infinite) by run-time environment variables (Optimizations for A64FX). - The register limit on V100 is
-Xcuda-ptxas -maxrregcount=255(Register Limit on V100). - Memory-saving setting G can be changed at run time with
PARCEL_PSTEP, and H at build time withPARCEL_JACOBI_ONTHEFLY(“Choosing Between Memory and Speed”). “Off” means that the values of the Jacobi preconditioner are stored (using memory for speed).
13.2 Blocking
_BLOCKING_LOOP_=1 replaces all kernels except SpMV (dot products, axpy, preconditioners, etc.) with blocked versions, in which one thread handles a contiguous range of rows. A reduction only sums the partial sums of the threads at the end, so no barrier within a work group is needed. _BLOCKING_LOOP_DOT_=1 replaces only the reductions (dot, dddot, max) with blocked versions.
The effect is concentrated in the reductions, because the flat reductions are tree reductions with barriers within work groups, and the fixed cost of a barrier is large on CPUs. The effect of blocking only the reductions (_BLOCKING_LOOP_DOT_=1), recommended for CPUs other than A64FX, is shown below.
| Environment | Solver | Without blocking [ms] | With blocking [ms] | Performance ratio | Main effect |
|---|---|---|---|---|---|
| s86 CPU | pcg |
172.4 | 130.5 | 1.32 | dot 56.6 ms → 15.0 ms |
| s86 CPU | pgmres |
975.0 | 422.3 | 2.31 | Gram-Schmidt 850.3 ms → 313.1 ms |
| Miyabi-C | pcg |
40.3 | 28.4 | 1.42 | dot 15.0 ms → 4.0 ms |
| Miyabi-C | pgmres |
220.7 | 73.4 | 3.01 | Gram-Schmidt 193.0 ms → 49.5 ms |
| PS-A | pcg |
24.0 | 22.7 | 1.06 | dot 4.7 ms → 4.1 ms |
| PS-A | pgmres |
65.5 | 49.0 | 1.34 | Gram-Schmidt 46.8 ms → 30.6 ms |
The effect of blocking all kernels (_BLOCKING_LOOP_=1), recommended for A64FX, is shown below.
| Environment | Solver | Without blocking [ms] | With blocking [ms] | Performance ratio | Main effect |
|---|---|---|---|---|---|
| Odyssey | pcg |
1603.0 | 40.0 | 40.1 | dot 1546.2 ms → 7.7 ms |
| Odyssey | pgmres |
16437.4 | 82.0 | 200 | Gram-Schmidt 15848.4 ms → 48.8 ms |
- On CPUs other than A64FX, blocking only the reductions is enough. On s86 CPU, blocking all kernels (
_BLOCKING_LOOP_=1) gives 134.5 ms per iteration forpcg(1.03 times that with the reductions only) and 421.8 ms forpgmres(1.00 times), no different from the reductions only. - On A64FX (
omp.library-onlywith the Fujitsu compiler), the dot products are extremely slow without blocking and the solvers are not practical. Always specify blocking of all kernels. “With blocking” on Odyssey is the recommended build (including A to F of the optimizations for A64FX). - On macOS (Apple M4), blocking the reductions is also required. Measured on one machine with one process without MPI, 4 threads and NX=128, the version without blocking is 33.5 times slower than blocking the reductions for
pcg(325.1 ms → 9.70 ms; dot 315.7 ms → 1.13 ms) and 112 times forpgmres(, 120 iterations; 3311 ms → 29.6 ms, Gram-Schmidt 3193 ms → 21.2 ms). Blocking all kernels gives 0.96 times forpcg, 1.00 times forpgmresand 1.04 times forks_arnoldiper iteration of the reductions only (0.97 to 0.99 times with 10 threads), a small difference, and uses more memory because it stores the values of the Jacobi preconditioner (352 MiB → 464 MiB allocated by PARCEL forpcgwith NX=128), so the reductions only are used. - Do not use it on GPUs (the flat form suits GPUs).
The number of blocks is determined at run time: the environment variable PARCEL_NBLOCK if set, otherwise omp_get_max_threads() (the number of threads per rank).
With blocked reductions, the number of partial sums depends on the run configuration (number of ranks × number of threads), and so does the order of the reduction. This does not affect results in double precision, but when comparing low-precision configurations across environments, use the same setting of this flag and the same run configuration.
13.3 Ladder Splitting
The iteration count of the inner loop of CRS SpMV is given by the number of nonzeros of each row, so the compiler does not know it and cannot unroll or prefetch. PARCEL_SPMV_LADDER=1 splits the inner loop into fixed-length stages of 4, 2 and 1 elements so that, for any number of nonzeros, all of them go into fixed-length bodies (4+2+1 for 7 nonzeros). The number of nonzeros may differ per row, and no extra memory is needed. PARCEL_PRECON_LADDER applies the same splitting to the iteration loops of the Jacobi preconditioner (PRECON_JACOBI).
The effect depends on the environment, and it can be slower. The time per iteration with and without ladder splitting is shown for PS-B (MI300A) and Odyssey (A64FX), where ladder splitting is recommended.
| Environment | Solver | Without ladder splitting [ms] | With ladder splitting [ms] | Performance ratio | Main effect |
|---|---|---|---|---|---|
| PS-B (MI300A) | pcg |
11.62 | 6.95 | 1.67 | SpMV 4.9 ms → 2.6 ms Preconditioner 4.7 ms → 2.4 ms |
| PS-B (MI300A) | pgmres |
24.81 | 19.88 | 1.25 | SpMV 5.1 ms → 2.9 ms Preconditioner 4.9 ms → 2.4 ms |
| Odyssey (A64FX) | pcg |
49.35 | 40.01 | 1.23 | SpMV 23.6 ms → 14.3 ms Preconditioner 15.2 ms → 15.3 ms |
| Odyssey (A64FX) | pgmres |
91.41 | 82.01 | 1.11 | SpMV 24.2 ms → 14.8 ms Preconditioner 15.3 ms → 15.4 ms |
- PS-B compares with and without ladder splitting of both SpMV and the preconditioner (
-DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=1), and Odyssey compares with and without ladder splitting of SpMV with the preconditioner not split.pgmreson PS-B was run for 120 iterations (2 cycles). - On MI300A, SpMV and the preconditioner both become about twice as fast, the largest effect. Use it on AMD GPUs.
- On A64FX, ladder splitting of SpMV is effective and that of the preconditioner is slower, so
PARCEL_PRECON_LADDER=0is set explicitly. Adding ladder splitting of the preconditioner (-DPARCEL_PRECON_LADDER=1) to the recommended build slows one iteration from 39.99 ms to 43.76 ms (0.91 times) forpcgand from 82.43 ms to 85.65 ms (0.96 times) forpgmres(preconditioner section 15.2 ms → 18.9 ms; measured in a separate run under the same conditions). - Ladder splitting is not used on NVIDIA GPUs. Under the same conditions, one iteration of
pcgbecomes slower with ladder splitting on V100 (25.1 ms → 29.6 ms), and there is no difference on GH200 and A100 (4.70 ms → 4.68 ms, 13.15 ms → 13.13 ms).
13.4 Decomposing gemv
gemv (, where is a matrix of M columns and its -th column) reads M columns at the same time and reads and writes only once. PARCEL_GEMV_AS_AXPY=1 decomposes it into M axpy operations ().
The decomposition reads and writes M times, which increases data movement, but only one column is streamed at a time. On A64FX, gemv streaming M columns at once can be slow (when the column spacing is a power of 2, the M columns fall into the same cache sets), so gemv is decomposed. On x86 CPUs and GPUs gemv is not decomposed.
In builds for CPUs (_BLOCKING_LOOP_ or _BLOCKING_LOOP_DOT_), the undecomposed gemv is also the row-block version that reads one column at a time within each block of rows (PARCEL_CPU_ROWBLOCK; default 1 on CPUs), and the column spacing of the basis is padded (PARCEL_BASIS_PAD; default 8 elements on CPUs). The cache conflicts of the M columns therefore rarely occur even without the decomposition, and the decomposition speeds up the gemv section of ks_arnoldi by only about 1.1 to 1.2 times (1.11 times for NX=128 and 1.20 times for NX=256 on one node of Odyssey).
On Odyssey, the time without the decomposition (-DPARCEL_GEMV_AS_AXPY=0) was compared with the time with it (the default). gemv is used heavily by ks_arnoldi and pcagmres (ks_arnoldi with for 10 restarts; pcagmres with , and the Jacobi preconditioner for 64 iterations).
| Environment | Solver | Not decomposed [ms] | Decomposed [ms] | Performance ratio | Main effect |
|---|---|---|---|---|---|
| Odyssey | ks_arnoldi (per restart) |
2760.9 | 2645.7 | 1.04 | gemv 888.1 ms → 738.9 ms |
| Odyssey | pcagmres (per iteration) |
65.9 | 65.6 | 1.00 | gemv 0.55 ms → 0.45 ms |
- The block orthogonalization of
ks_arnoldi(the default CGS2) performs the projections onto the basis of the subspace with gemv. The gemv section is about 30% of a restart (738.9 ms of 2645.7 ms), and the decomposition makes it 1.20 times faster, so one restart becomes 1.04 times faster. - In
pcagmresthe decomposition also makes the gemv section 1.22 times faster, but gemv is less than 1% of an iteration (0.45 ms of 65.6 ms), so one iteration does not change. - For
pcgandpgmresthe gemv section is less than 1% of an iteration, and the difference is within 1% (1.00 times for bothpcgandpgmres).
The default is the value of _BLOCKING_LOOP_, so it becomes 1 in builds for A64FX without being specified. gemv accumulates first and multiplies by the coefficient once, whereas the decomposed version multiplies each term by the coefficient before adding, so the rounding order changes.
13.5 Optimizations for A64FX (A to F)
Builds for A64FX (_BLOCKING_LOOP_=1) have the following six optimizations. B to F do not change the computation (the expression and the order of summation for each element), so the results are bit-identical to those without them (checked on Odyssey: the residual history agrees over 200 iterations for C and D, and until convergence (306 iterations) for E and F). A fuses multiply-adds into FMA instructions, so the rounding changes. In builds for x86 CPUs and GPUs, the code of C to F is not used and the generated code does not change.
| What it does | How to enable | |
|---|---|---|
| A | Fuses multiply-adds () into FMA instructions. The clang mode of the Fujitsu compiler does not fuse them by default | -ffp-contract=fast at build time (Recommended Options) |
| B | Keeps the OpenMP threads spinning while they wait between kernels, which reduces the fixed cost of launching and waiting for kernels | At run time, OMP_WAIT_POLICY=ACTIVE and KMP_BLOCKTIME=infinite (recommended; not the default) |
| C | Rewrites the axpy that takes coefficients by value (axpy_value / axpby_value) and the axpy-decomposed gemv (Decomposing gemv): splits the vector into contiguous ranges, one per thread, copies the pointers and coefficients to local (restrict) variables, and writes with std::fma the fusion that the Fujitsu compiler chooses for the original expression (the path for skips the multiplication). Without it, depending on the surrounding code, the other form can be chosen and the rounding changes |
Default (PARCEL_A64FX_AXPY_C; disable with -DPARCEL_A64FX_AXPY_C=0) |
| D | Fuses the kernels of one iteration of pcg (pstep = 1, the default): the updates of and ( and ) into one kernel, and the update of and the dot product into one kernel. and are read and written once each, and the fixed cost of one kernel launch is saved |
Default (PARCEL_CG_FUSED; disable with -DPARCEL_CG_FUSED=0) |
| E | In the preparation of the right-hand side of the Jacobi preconditioner (the kernel that forms ), selects a version without the division when there is no vector scaling (the divisor is 1, the default for double and single precision). The SVE division on A64FX is not pipelined and took about 30% of the preconditioner section | Default (PARCEL_JACOBI_NODIV_UNIT; disable with -DPARCEL_JACOBI_NODIV_UNIT=0) |
| F | When the preconditioner matrix is single precision (precon_dataF etc.), processes two rows alternately in the Jacobi sweep so that the waits of the two dependent multiply-add chains overlap (the summation order within each row is unchanged). Not used for a double-precision preconditioner matrix (the compiler already overlaps rows, and it becomes slower) |
Default (PARCEL_JACOBI_ROW2; disable with -DPARCEL_JACOBI_ROW2=0) |
The effect of A to F was measured on 4 nodes of Odyssey (16 ranks × 12 threads; 8.4 M rows per rank; pcg with the Jacobi preconditioner for 100 iterations and pgmres () for 60 iterations, time per iteration [ms], median of 3 rounds; the value in parentheses is the performance ratio, the time without optimization ÷ the time of that version). “Without optimization” is a build without any of A to F (without -ffp-contract=fast, with -DPARCEL_A64FX_AXPY_C=0 -DPARCEL_CG_FUSED=0 -DPARCEL_JACOBI_NODIV_UNIT=0 -DPARCEL_JACOBI_ROW2=0), run without B; A+B+C+D is the build with only E and F removed, and A+B+C+D+E+F is the recommended build, both run with B. All are the default builds that store the values of the Jacobi preconditioner.
| Precision | Solver | Without optimization [ms] | A+B+C+D [ms] (performance ratio) | A+B+C+D+E+F [ms] (performance ratio) |
|---|---|---|---|---|
| Double | pcg |
46.64 | 44.68 (1.04) | 39.97 (1.17) |
| Double | pgmres |
90.32 | 86.85 (1.04) | 82.03 (1.10) |
precon_dataF |
pcg |
50.41 | 48.10 (1.05) | 41.06 (1.23) |
precon_dataF |
pgmres |
93.69 | 90.08 (1.04) | 83.21 (1.13) |
- A+B+C+D mainly shortens SpMV (15.87 ms → 14.28 ms, the FMA of A) and the preconditioner (20.96 ms → 19.83 ms) (
pcg, double precision). Inpcg, D merges the updates of and into one kernel and removes the gemv section (1.27 ms) and part of the axpy section (3.02 ms → 2.11 ms), but the kernel that fuses the update of with the dot product increases the dot-product section from 4.96 ms to 7.82 ms. Whether D pays off depends on the number of rows per rank: in this configuration (8.4 M rows), removing only D is about 3% faster, while with about 2 M rows or fewer D is 1 to 10% faster. The Gram-Schmidt section ofpgmres(about 49 ms) is the same in all versions. - E and F speed up the preconditioner section: 19.83 ms → 15.09 ms in double precision (only E applies) and 23.56 ms → 16.49 ms with
precon_dataF(E and F). The residual histories are the same with and without E and F. - The preconditioner section consists of the preparation of the right-hand side (where E applies), one sweep (where F applies), and the halo exchange.
- The preconditioner section of
precon_dataFis 1.09 times that of double precision (both storing the values of ; 1.19 times without E and F). The preconditioner on A64FX is limited by instruction execution rather than memory bandwidth, and a single-precision preconditioner matrix needs more instructions for type conversion.
Notes:
- With B, the threads keep using their cores while waiting. This is fine with the recommended placement (one rank per CMG, 12 threads), but counterproductive when processes compete for cores (several threads per core, or other processes on the same node).
- E removes the division only when the divisor (the maximum of the vector scaling) is 1 (, so the results are the same). Configurations with scaling enabled (such as half-precision preconditioners) divide as before. F applies only to the version that stores the Jacobi values; it is not used with on-the-fly computation (
PARCEL_JACOBI_ONTHEFLY=1) or ladder splitting (PARCEL_PRECON_LADDER=1).PARCEL_JACOBI_ROW2=1can be specified only with_BLOCKING_LOOP_=1andPARCEL_PRECON_LADDER=0; otherwise the compilation stops with#error. - The fusion of the update of and the dot product in D uses the same kernel as the CPU MGS (
axpy_dot_value).PARCEL_CG_FUSEDcan be 1 only in builds with_BLOCKING_LOOP_andPARCEL_GS_FUSED; specifying 1 otherwise stops the compilation with#error. The kernels are not fused for type combinations whose coefficients cannot be passed by value (those including double-double) and withRESIDUAL_REPLACE.
13.6 Register Limit on V100
On V100 (cuda:sm_70), -Xcuda-ptxas -maxrregcount=255 is added to the build. It raises the per-thread register limit of ptxas (the CUDA assembler); with the default allocation, the loads of the matrix and vectors in kernels such as the Jacobi preconditioner are split and serialized, which is slower. No spills occur and the results do not change. The flag is not used on GH200 and A100.
The time per iteration with and without the flag, for the recommended build (without ladder splitting) on s86 GPU, is shown below.
| Environment | Solver | Without the flag [ms] | With the flag [ms] | Performance ratio | Main effect |
|---|---|---|---|---|---|
| s86 GPU (V100) | pcg |
45.53 | 25.17 | 1.81 | SpMV 18.0 ms → 8.7 ms Preconditioner 20.8 ms → 9.9 ms |
| s86 GPU (V100) | pgmres |
94.63 | 74.29 | 1.27 | SpMV 18.7 ms → 9.3 ms Preconditioner 21.2 ms → 10.1 ms |
- The SpMV and Jacobi preconditioner kernels both become about twice as fast. In
pgmresthe Gram-Schmidt section (about 53 ms) does not change with the flag, so the performance ratio per iteration is smaller than forpcg.
13.7 Choosing Between Memory and Speed
The default settings of v1.2 enable two settings that reduce memory (only G in builds for A64FX; see below).
- G:
pcgkeeps no history of search directions and updates the solution at every iteration (default ofPARCEL_PSTEPis 1). - H: The Jacobi preconditioner (
PRECON_JACOBI) does not keep the values of and computes them from the values of and at every application (Jacobi preconditioner).
If memory is available, the following optimization options make the solvers faster by using more memory. Neither changes the algorithm or the number of iterations.
| Optimization option | How to specify | Additional memory (per rank) | What becomes faster | Effect on results |
|---|---|---|---|---|
| Without H | -DPARCEL_JACOBI_ONTHEFLY=0 at build time |
Values of : number of nonzeros of × size of the preconditioner matrix precision (about 7 vectors for a 7-point stencil in double precision) | Application of the Jacobi preconditioner; effective on GPUs (almost no change on CPUs) | Bit-identical with -ffp-contract=off (rounding may change in builds that allow fused multiply-add) |
| Without G | PARCEL_PSTEP=4 at run time (or -DPARCEL_PCG_NO_XHIST=0 at build time) |
History of search directions of pcg: 3 vectors with halo |
Solution update of pcg (reads and writes of drop to 1/4) |
The solution is accumulated differently, so the results differ in the last digits (same number of iterations) |
- G affects only
pcg(including when the inner linear solver of an eigenvalue solver ispcg). H affects all solvers that use the Jacobi preconditioner (including the default preconditioner of the inner linear solvers of the eigenvalue solvers). - In the combinations excluded from H (matrix scaling enabled, half-precision preconditioners (scaling is enabled by default), double-double coefficient matrix, preconditioner matrix or preconditioner computation precision), the values of are stored even with the default. A configuration in which only the solver computation precision is double-double (
calcDD) is included in H. - In builds for A64FX (
_BLOCKING_LOOP_or_BLOCKING_LOOP_SPMV_is 1; Recommended Options), H is disabled by default (the default ofPARCEL_JACOBI_ONTHEFLYis 0). On A64FX the on-the-fly computation is not hidden: on 4 nodes of Odyssey (NX=512,pcg+ Jacobi preconditioner), “Without H” makes the preconditioner section 0.77 times and one iteration 0.90 times, so speed is chosen at the cost of memory (per rank, the number of nonzeros of × the size of the preconditioner matrix precision; 8 bytes per nonzero in double precision). To enable H, specify-DPARCEL_JACOBI_ONTHEFLY=1explicitly.
The time per iteration of pcg with memory-saving settings G and H switched on and off in the recommended build of each environment is shown below (the conditions are the same as at the beginning of the chapter). “G+H” means both G and H enabled, “G” only G enabled (without H), “H” only H enabled (without G), and “off” both disabled. The performance ratio is the time of off ÷ the time of G+H; below 1 means that disabling both is faster.
| Environment | Solver | G+H [ms] | G [ms] | H [ms] | off [ms] | Performance ratio (off/G+H) |
|---|---|---|---|---|---|---|
| s86 CPU | pcg |
130.5 | 129.3 | 128.6 | 126.3 | 0.97 |
| Miyabi-C | pcg |
28.38 | 27.94 | 27.93 | 26.41 | 0.93 |
| PS-A | pcg |
22.69 | 22.31 | 22.16 | 21.93 | 0.97 |
| Odyssey | pcg |
44.32 | 40.01 | 44.07 | 39.59 | 0.89 |
| s86 GPU (V100) | pcg |
25.17 | 24.69 | 24.65 | 24.04 | 0.95 |
| Miyabi-G (GH200) | pcg |
4.72 | 4.60 | 4.55 | 4.55 | 0.97 |
| Aquarius (A100) | pcg |
12.91 | 12.61 | 12.47 | 12.36 | 0.96 |
| PS-B (MI300A) | pcg |
6.67 | 6.48 | 6.59 | 6.35 | 0.95 |
- The default is the G+H column except on Odyssey, and the G column (H disabled) on Odyssey (the build for A64FX). On A64FX, enabling H makes the preconditioner section slower (15.3 ms → 19.9 ms).
- Removing G (
PARCEL_PSTEP=4) about halves the solution update ofpcg(the gemv section) (7.7 ms → 3.7 ms on s86 CPU, 1.10 ms → 0.55 ms on V100). - Removing H makes the Jacobi preconditioner section faster (52.6 ms → 50.2 ms on s86 CPU, 9.85 ms → 9.39 ms on V100). Disabling both (off) makes one iteration 0.89 to 0.97 times that of G+H.
The memory per device read in the same measurement is shown below, in the same order as the time table. The memory ratio is the memory of off ÷ the memory of G+H; above 1 means that disabling both uses more memory.
| Environment | Solver | G+H [GiB] | G [GiB] | H [GiB] | off [GiB] | Memory ratio (off/G+H) |
|---|---|---|---|---|---|---|
| s86 CPU | pcg |
9.08 | 9.61 | 9.08 | 10.40 | 1.15 |
| Miyabi-C | pcg |
8.11 | 8.64 | 8.10 | 9.43 | 1.16 |
| PS-A | pcg |
10.31 | 11.10 | 10.26 | 11.91 | 1.15 |
| Odyssey | pcg |
5.68 | 7.42 | 6.47 | 8.22 | 1.45 |
| s86 GPU (V100) | pcg |
5.88 | 7.62 | 6.64 | 8.39 | 1.43 |
| Miyabi-G (GH200) | pcg |
5.57 | 7.32 | 6.33 | 8.08 | 1.45 |
| Aquarius (A100) | pcg |
5.99 | 7.74 | 6.75 | 8.50 | 1.42 |
| PS-B (MI300A) | pcg |
5.57 | 7.32 | 6.33 | 8.08 | 1.45 |
- The default is the G+H column except on Odyssey, and the G column (H disabled) on Odyssey (the build for A64FX).
- The measurement differs by environment. CPUs (s86 CPU, Miyabi-C, PS-A) show host memory (the sum of the peak resident memory over all ranks divided by the 4 sockets); s86 GPU and Aquarius show device memory (the peak of one GPU read with
nvidia-smi); Miyabi-G and PS-B show the peak of device memory allocated by PARCEL (maximum over GPUs). On Odyssey the host resident memory cannot be measured correctly this way, so the peak of memory allocated by PARCEL (the sum over all ranks divided by the 4 nodes) is shown. The memory allocated by PARCEL is about 0.4 GiB smaller than thenvidia-smivalue (5.56 GiB vs 5.99 GiB with G+H on Aquarius). - The memory increase is, when H is removed, the number of nonzeros of × the size of the preconditioner matrix precision per rank, and when G is removed, 3 vectors with halo; it is proportional to the local number of rows per rank (half with twice the number of devices). On CPUs the host memory does not change when G is removed because the peak is reached when the matrix is generated.
- Since the time hardly changes, normally keep the defaults (the memory-saving settings). Specify the options when the GPU has spare memory and even a small speedup matters.
- With mixed precision that lowers the preconditioner matrix precision (
precon_dataFetc.), keeping H reads the values of in double precision at every application, so lowering the preconditioner data to single precision has little effect. To make mixed precision faster, combine it with “Without H” (also on NVIDIA GPUs; the additional memory is the number of nonzeros of × the size of the preconditioner matrix precision, 4 bytes in single precision; see “Mixed Precision” for the effect. Builds for A64FX are “Without H” by default, but mixed precision does not make them faster). The mixed-precision performance test of the distribution package (tests/perf/gen_perf_prec.py) measures this combination (Measuring Your Own Environment).