14 Performance
This chapter shows the results of v1.2 measured on 8 test environments (CPUs: s86 CPU, PS-A, Miyabi-C, Odyssey; GPUs: s86 GPU, Miyabi-G, Aquarius, PS-B): the time to solve 3D problems with NX=512 to convergence, the effect of mixed precision that lowers only the preconditioner data precision, a comparison with PETSc / SLEPc, a comparison with v1.0, and the memory usage. All values were measured with the bundled verification sample tests/verify/parcel_krylov_sample_verify.cpp and do not guarantee the performance for your problem. v1.2 was measured with the default settings and the Recommended Options. The conditions are the same as those of the performance tests in the distribution package (tests/perf/); how to measure your own environment under the same conditions is described in “Measuring Your Own Environment”.
14.1 Measurement Conditions
All charts in this chapter were measured under the following conditions.
| Item | Condition |
|---|---|
| Problem | 3D 7-point stencil, NX=512 (n = 134,217,728 unknowns). pcg, pbicgstab and the eigenvalue solvers use the Poisson equation (matrix poisson of the verification sample; symmetric); pgmres and pcagmres, which are for nonsymmetric matrices, use the convection-diffusion equation (convdiff, convection coefficient c = 0.05; nonsymmetric). Right-hand side b = 1, initial guess x = 0 (PARCEL_VERIFY=0), double precision |
| Linear solvers | Two variants: no preconditioner and the Jacobi preconditioner (PARCEL_PRECON=4, 1 iteration). Tolerance 1.0e-8 (relative residual), maximum 10000 iterations. The parameters are the library defaults (PARCEL_LIB_DEFAULTS=1); only the restart length = 60 of pgmres (default 30) and of pcagmres (default ) are specified. pcagmres with formed (EXPLICIT, QR_MGS) and not formed (IMPLICIT, QR_CHOLESKYQR) |
| Eigenvalue solvers | No preconditioner, tolerance 1.0e-8. power: maximum 20000 iterations. inverse_power: maximum 200 outer iterations; the inner linear solver is pcg with the Jacobi preconditioner (PARCEL_LINSOLVE_SOLVER=pcg PARCEL_LINSOLVE_PRECON=4) with tolerance 1.0e-10 and maximum 5000 iterations (with the default pgmres (restart 30), the inner solves of the Poisson equation reach the limit and become slow). ks_arnoldi: = 40, = 10, = 1, maximum 200 restarts (default), the orthogonalization is the default of the build for each environment: MGS2 (PARCEL_KS_ORTHO_DEFAULT=mgs) on PS-A and CGS2 on the other 7 environments (the same in the performance tests of the distribution package tests/perf) |
| Devices | 4 / 8 / 16. “1 device” is one CPU socket (one A64FX = one node on Odyssey) or one GPU (one GH200 or MI300A) |
| Repetitions | 3 runs each; the time is the fastest |
| Timed region | The value <solver name> time = ... (total solver wall time) printed by each solver (wall-clock time of rank 0). It includes the setup of the preconditioner and the computation of the initial residual, but not set_crsMat() or the matrix generation of the sample |
| Build | Flags of the Recommended Options (no ladder splitting on NVIDIA GPUs, with -Xcuda-ptxas -maxrregcount=255 only on V100; ladder splitting on MI300A, blocking of all kernels and ladder splitting of SpMV only plus the optimizations for A64FX A to F (B by run-time environment variables) on A64FX, blocking of the reductions on the other CPUs). Built from the bundled code in environments installed with the spack procedure of the distribution package |
| Default settings | The library defaults (the values of of the Jacobi preconditioner are not stored; the build for A64FX stores them; PARCEL_PSTEP 1). Only precon_dataF of the mixed-precision test was built with -DPARCEL_JACOBI_ONTHEFLY=0 |
Ranks and threads per device in each environment (with the PETSc ranks used in the comparison with PETSc / SLEPc). PETSc on CPUs uses all cores of a socket (of a node on Odyssey) with one rank per core (flat MPI). PETSc on GPUs uses one rank per device as PARCEL does, and each rank is given the same CPU cores as in PARCEL (cores of the socket of its GPU).
| Environment | 1 device | PARCEL ranks × threads (per device) | Ranks on 4 / 8 / 16 devices | PETSc ranks (per device) |
|---|---|---|---|---|
| s86 CPU | 1 socket of Xeon Gold 6242R | 4 × 5 | 16 / 32 / 64 | 20 |
| PS-A | 1 socket of Xeon 6980P | 9 × 14 | 36 / 72 / 144 | 128 |
| Miyabi-C | 1 socket of Xeon CPU Max 9480 | 4 × 14 | 16 / 32 / 64 | 56 |
| Odyssey | 1 A64FX (1 node) | 4 × 12 | 16 / 32 / 64 | 48 |
| s86 GPU | 1 V100 | 1 × 12 | 4 / 8 / 16 | 1 |
| Miyabi-G | 1 GH200 | 1 × 72 | 4 / 8 / 16 | 1 |
| Aquarius | 1 A100 | 1 × 9 | 4 / 8 / 16 | 1 |
| PS-B | 1 MI300A | 1 × 6 | 4 / 8 / 16 | 1 |
On PS-A, 3 ranks (14 threads each) are placed on each NUMA domain to use all 128 cores of a socket, so the number of ranks differs from the other CPU environments. On Odyssey, one rank is placed on each CMG. On 4 devices of Odyssey, the configurations that do not fit in the memory of one node (pgmres, pcagmres and ks_arnoldi of PARCEL, and pgmres of PETSc) were spread over 8 nodes with the same number of ranks (2 ranks per node). ks_arnoldi of PETSc / SLEPc stayed on 4 nodes, so the comparison of ks_arnoldi on 4 devices of Odyssey uses different numbers of nodes. In addition, pgmres of PETSc spread over 8 nodes (24 ranks per node) was measured with the ranks pinned to two CMGs of each node, i.e., the same number of CMGs as PARCEL (2 ranks per node). With the default placement of Fujitsu MPI, the PETSc ranks are spread over all four CMGs of a node and use twice the memory bandwidth of PARCEL (the ratio PETSc time ÷ PARCEL time would be 0.71 instead of 1.04). The memory of PETSc was measured with the default placement.
14.2 Time to Convergence
All configurations converged. The iteration counts were as follows; those of the linear solvers are the values at the convergence check.
pcg: 1225 iterations without a preconditioner and 611 with the Jacobi preconditioner, identical on all 8 environments and numbers of devices.pbicgstab: 776–1040 iterations without a preconditioner and 397–466 with the Jacobi preconditioner, varying with the environment and the number of devices.pgmres: 1920 without a preconditioner and 1140 with the Jacobi preconditioner;pcagmresEXPLICIT: 1984 and 1152; identical on all environments and numbers of devices.pcagmresIMPLICIT: 1920–1984 and 1152–1216, varying with the environment and the number of devices.powerconverged in 15002–15004 iterations.inverse_powerconverged in 9 outer iterations. The innerpcgtook 3816 iterations in total over the 9 solves (same on all environments and numbers of devices).ks_arnoldiconverged in 122 restarts.
The time [s] on 16 devices is shown for each environment, separately for the linear and the eigenvalue solvers (fastest of 3 runs; pgmres and pcagmres solve the convection-diffusion equation, the others the Poisson equation). The solvers are listed for each environment, with two bars for the linear solvers (no preconditioner and the Jacobi preconditioner). The horizontal axis is the time on a logarithmic scale, common to all environments. The values at the ends of the bars are in seconds.
- Among the linear solvers,
pcgwith the Jacobi preconditioner is the fastest on all 8 environments.pgmresandpcagmres, which are for nonsymmetric matrices, solve the convection-diffusion equation, so their problem differs from that ofpcgandpbicgstab. - The Jacobi preconditioner reduces the iterations to 50–60% and is 1.1 to 1.5 times as fast as no preconditioner in most configurations. For
pbicgstabon PS-A, no preconditioner is slightly faster. pcagmresIMPLICIT is faster than EXPLICIT on all 8 environments and faster thanpgmreson 7 environments except Odyssey (1.2 to 1.5 times with the Jacobi preconditioner). EXPLICIT is faster thanpgmreson some environments and slower on others.- Among the eigenvalue solvers,
inverse_power(innerpcgwith the Jacobi preconditioner) is the fastest on all 8 environments. Because the solvers compute different eigenvalues (powerandks_arnoldithe largest in magnitude,inverse_powerthe smallest), their speed depends on the eigenvalue distribution of the problem. For this problem,powerneeded more than 15000 iterations.
14.3 Mixed Precision (precon_dataF)
The time of the configuration that lowers only the data precision of the preconditioner to single (precon_dataF) is compared with double precision for pcg with the Jacobi preconditioner (Poisson equation, 16 devices). The time is split into the rest of the solver (left) and the preconditioner (its setup and application; right), stacked. The horizontal axis is the ratio to the total time in double precision of each environment, on a linear scale common to all environments. The full length of a bar is the ratio of its total time to double precision, and the lengths of the left and right parts show the shares of the time. After each bar, the total time [s], the ratio to double precision (in parentheses; below 1 means precon_dataF is faster) and the preconditioner time are given. The iteration counts are the same as in double precision (611) in all configurations.
precon_dataF was built with the setting that stores the values of of the Jacobi preconditioner (without on-the-fly computation; “without H”, -DPARCEL_JACOBI_ONTHEFLY=0) and compared with double precision built with the library defaults. With the default (values of not stored), every application reads the double-precision values of even if the preconditioner matrix is single precision, so lowering the precision has little effect. Builds for A64FX store the values of by default, so on Odyssey double precision also stores them.
Conditions: NX=512, time to convergence (fastest of 3 runs), 16 devices in the chart. precon_dataF is without on-the-fly computation (-DPARCEL_JACOBI_ONTHEFLY=0) and double precision uses the library defaults. The iteration counts are the same as in double precision in all configurations (pcg 611, pgmres on the convection-diffusion equation 1140). The other conditions are those of “Measurement Conditions”. The ranges of the ratios below cover all configurations of pcg and pgmres on 4, 8 and 16 devices.
- On the 7 environments other than Odyssey,
precon_dataFis faster. The ratio of the total time to double precision is 0.86–0.95 forpcgand 0.91–0.98 forpgmres. The preconditioner time becomes 0.63–0.84 times, and the rest hardly changes. - The preconditioner takes 30–40% of the total, which bounds the reduction of the total. For
pgmres, whose orthogonalization takes much time, the ratio of the totals is close to 1. - On Odyssey (A64FX), mixed precision brings almost no benefit: the ratio of the total time is 0.99–1.03 and that of the preconditioner time 1.03–1.10. The preconditioner on A64FX is bound by instruction execution, not by memory bandwidth, and a single-precision preconditioner matrix needs additional type-conversion instructions (E and F of the optimizations for A64FX).
- On the 7 environments, PARCEL allocates 0.61–0.62 GiB more memory per device than the double-precision default on 4 devices (0.14–0.15 GiB on 16 devices), for the single-precision values of (4 bytes per nonzero of ). On Odyssey, double precision also stores , in double precision, so
precon_dataFuses 1.13 GiB less per device on 4 devices (0.29 GiB on 16 devices).
To make a solver faster with mixed precision, build precon_dataF without on-the-fly computation (-DPARCEL_JACOBI_ONTHEFLY=0). The same applies to all NVIDIA GPUs including V100, without ladder splitting (Recommended Options). In exchange, the memory per GPU grows by about 0.6–0.9 GiB (solving pcg with NX=512 on 4 V100 GPUs, the peak memory allocated by PARCEL per GPU is 6,340 MiB, 0.62 GiB more than the 5,703 MiB of the double-precision default and 0.87 GiB more than the 5,445 MiB of precon_dataF with on-the-fly computation).
14.4 Comparison with PETSc / SLEPc and v1.0
For pcg (Poisson equation) and pgmres ( = 60, convection-diffusion equation) with the Jacobi preconditioner and ks_arnoldi (Poisson equation) without a preconditioner, the time and memory of v1.2 are compared with PETSc 3.22 / SLEPc 3.22 and the public v1.0 (16 devices). Bar lengths are ratios to v1.2 in each environment, with the v1.2 bars (dashed line) as the reference. The scale is common to the six charts, and bars with a ratio above 3 are drawn shortened (double slash). Time charts are shown in blue tones and memory charts in green tones. The value in parentheses is the ratio to v1.2; above 1 means v1.2 is faster (uses less memory). v1.0 has no eigenvalue solvers, so ks_arnoldi is compared only with SLEPc.
- PETSc / SLEPc: solved with the same matrices and conditions.
pgmresuses right preconditioning and modified Gram-Schmidt; the Jacobi preconditioner is PETSc’sPCKSP(inner Richardson iteration +PCJACOBI), which applies the same operator as PARCEL’sPRECON_JACOBI; convergence is checked on the unpreconditioned residual norm (KSP_NORM_UNPRECONDITIONED) with relative residual 1.0e-8.ks_arnoldiis compared with the Krylov-Schur method of SLEPc (ncv 40, 10 kept, nev 1, maximum 200 restarts). PETSc was run with one rank per core on CPUs (flat MPI; 5–14 times as many ranks as PARCEL) and one rank per GPU on GPUs (cuSPARSE / cuBLAS on NVIDIA GPUs, HIP on PS-B). The iteration counts werepcg611 (same for both),ks_arnoldi122 restarts (same), andpgmres1140 for PARCEL and 1132–1133 for PETSc (PARCEL checks convergence ofpgmreson the true residual at the start of a restart, PETSc on the residual estimate at every iteration). - v1.0: solved with the library of the public v1.0 under the same conditions and the same numbers of ranks. The iteration counts are the same as those of v1.2. On the 4 GPU environments, v1.0 was built so that it does not allocate the work array of SpMV (1,792 bytes per row, not used by the GPU SpMV); the source is the same, with macros added that skip this array. With the original settings, the work array is about 60 GB per GPU on 4 devices, and v1.0 cannot run on 4 and 8 devices of s86 GPU (V100 32 GB) for lack of GPU memory. The 4 CPU environments were measured with the original settings (the SpMV of the A64FX version on Odyssey does use the array).
- Memory: measured with a fixed number of iterations (60 for
pcgandpgmres, i.e. one restart cycle ofpgmres; one cycle of = 40 forks_arnoldi; maximum of 3 runs). The values are per device (total over all devices ÷ number of devices): host memory on CPUs (sum of the peak resident memoryVmHWMof the ranks;PetscMemoryGetMaximumUsagefor PETSc) and device memory on GPUs (peak read every 100 ms withnvidia-smi, VRAM ofamd-smion PS-B). On Odyssey, the values are the increase of the memory used by the whole node (the maximum increase of the used memory in/proc/meminfoof each node since before the start, sampled every 0.5 s, summed over the nodes), because programs built with the Fujitsu compiler place the heap in huge pages (HugeTLB) and the memory allocated by PARCEL does not appear inVmHWM(the verification sample also printsHugetlbPages; see the output example in Measuring Your Own Environment). These values include fluctuations of the OS and other processes, so they are measured differently from the other CPU environments.
- Time: v1.2 is faster than v1.0 on all 8 environments (1.02 to 1.56 times for
pcg, 1.14 to 2.17 times forpgmres). Relative to PETSc,pcgis faster or comparable on 7 environments (s86 CPU 1.76 times, Odyssey 1.19 times, Miyabi-C 1.14 times, Aquarius 1.13 times, etc.) and the same on PS-A (1.00).pgmresis faster or comparable on s86 CPU, Odyssey and the GPUs, and PETSc is faster on PS-A and Miyabi-C.ks_arnoldiis faster than SLEPc on the 4 GPU environments, s86 CPU and Odyssey, and SLEPc is faster on PS-A and Miyabi-C. - Memory: on GPUs, v1.2 uses 0.55–0.65 times the memory of v1.0 for
pcgand 0.83–0.85 times forpgmres, and 0.97–1.11 times that of PETSc / SLEPc, almost the same. On CPUs, v1.2 uses less because PETSc runs many ranks with flat MPI (PETSc uses 1.1–5.1 times as much as v1.2; 1.1–1.5 times on Odyssey). The CPU comparison includes the difference of the parallelization (flat MPI vs. MPI + threads) and of the number of ranks. - By default v1.2 does not store the values of of the Jacobi preconditioner and computes them at every application to save memory. Settings that use memory for speed are described in “Choosing Between Memory and Speed”.
14.5 Measuring Your Own Environment
To measure under the same conditions as this chapter, build the verification sample with the Recommended Options (verify_double) and run it without verification (PARCEL_VERIFY=0). The right-hand side is then b = 1, and each rank generates only its own rows. The run-time options are described in Run-Time Options of the Verification Sample.
# pgmres (m = 60, Jacobi preconditioner)
mpirun -n 4 ./verify_double PARCEL_NX=512 PARCEL_SOLVER=pgmres PARCEL_PRECON=4 \
PARCEL_GMRES_RESTART=60 PARCEL_VERIFY=0 PARCEL_TOL=1.0e-8 PARCEL_MAX_ITER=10000
# ks_arnoldi (m = 40, 10 kept, nev = 1)
mpirun -n 4 ./verify_double PARCEL_NX=512 PARCEL_SOLVER=ks_arnoldi PARCEL_PRECON=0 \
PARCEL_KS_M=40 PARCEL_KS_K_KEEP=10 PARCEL_KS_NEV=1 PARCEL_KS_RESTART=default \
PARCEL_VERIFY=0 PARCEL_TOL=1.0e-8Place the ranks as in Measurement Conditions; on CPUs, bind the ranks to sockets with the mpirun binding options (e.g. --map-by ppr:<ranks per socket>:socket:PE=<threads> --bind-to core). Do not set OMP_PROC_BIND on s86 CPU and PS-A (see “Installing a SYCL Compiler”). To see the memory usage as well, add PARCEL_MEM_REPORT=1.
At the end of the solver, rank 0 prints the timings in the following form (pcg; the values are from the fastest of the 3 runs on s86 GPU with 4 devices above; the [MEM] and [MEMG] lines at the end appear only with PARCEL_MEM_REPORT). The same content is also written to parcel_result.txt (changed with PARCEL_OUTPUT). pstep=1 of loop gemv shows that the default pcg updates the solution at every iteration (pstep=4 with “Without G” of “Choosing Between Memory and Speed”).
finish : residual true norm || b - Ax || = 9.5999802982e-09
-----------------
pcg time = 1.5333772000e+01 [sec] (total solver wall time)
init time = 4.3617000000e-02 [sec] (before main loop)
init precon = 1.0722000000e-02 [sec]
init convert = 3.3840000000e-03 [sec]
init other = 2.9511000000e-02 [sec] (vec alloc/zero, init SpMV/dot/precon)
main loop time = 1.5277877000e+01 [sec] (sum over all CG iterations)
loop dot = 2.6397240000e+00 [sec] calls = 1832
dot comm = 9.7834400000e-01 [sec] (MPI_Allreduce)
dot kernel = 1.6613800000e+00 [sec] (局所内積)
loop axpy = 4.2147000000e-02 [sec] calls = 1222
loop gemv = 6.7014200000e-01 [sec] calls = 611 (pstep=1 本まとめた x の更新)
loop SpMV = 5.9033560000e+00 [sec] calls = 611
SpMV comm = 1.5370920000e+00 [sec] (ハロ交換)
SpMV kernel = 4.3662640000e+00 [sec] (局所行列ベクトル積)
loop precon = 5.9604370000e+00 [sec] calls = 610
loop other = 6.2071000000e-02 [sec] (kernel launch/sync, cout per iter)
post-loop time = 1.2278000000e-02 [sec] (final r_true compute, finish print)
converged_reason = 1 (CONVERGED_RTOL) , its = 611 , residual = 9.599865e-09 , true_residual = 9.599980e-09
[MEM] rank 0 solver 後 peak RSS = 5068 MiB
[MEM] rank 0 solver 後 device memory used = 3900 MiB
[MEM] rank 0 solver 後 PARCEL device peak = 5679 MiB
[MEMG] nranks=4 host_mib=5068,5083,5085,5068 dev_mib=3900,3910,3910,3900 parcel_devpeak_mib=5679,5703,5703,5679
- The first line,
<solver name> time, is the value in the charts of this chapter (total time).init timeis the setup before the iterations (including the preconditioner setup),main loop timeis the whole iteration, andpost-loop timeis the final true-residual computation. loop dotand the like are the times and call counts of each part in the iterations.dot commandSpMV commare the times of the communication between ranks (including the halo exchange).- The times are measured by rank 0 with the host clock, so the time of a kernel that returns before completion may be counted in the next part. By default the device is not synchronized around the timing of the inner products, so
loop dotincludes the time of the preceding kernel (e.g. the axpy of the modified Gram-Schmidt inpgmres). To get an accurate breakdown, addPARCEL_TIMER_SYNC=1(the total solver time becomes slightly longer). - The
[MEM]lines are the peak resident host memory of rank 0, the GPU usage after the solver (nvidia-smi; NVIDIA GPUs only), and the peak of the device memory allocated by PARCEL (including the work areas of the solver). The[MEMG]line lists the same values for all ranks. The current verification sample also prints, after the peak resident memory, the amount of huge pages (HugeTLB) of each rank (HugetlbPagesin/proc/self/status; the value at the time of the report, not a peak), andhost_hugetlb_mib=at the end of[MEMG]. On A64FX (Fujitsu compiler), the heap is placed in huge pages and is not counted in the peak resident memory, so read the host memory as the sum of the two. It is 0 in environments that do not use huge pages. - Check the convergence status (
converged_reason) and the iteration count (its) at the end. Measure the same configuration 3 times or more and use the minimum or the median.
The performance tests in the distribution package (tests/perf/gen_perf.py, and gen_perf_prec.py for mixed precision) generate job scripts that perform the measurements of this chapter together (usage in tests/perf/README.md). The conditions are the same as in this chapter. One job measures one solver × one number of devices; each configuration is run 3 times, and the summary (tests/perf/parse_perf.py) takes the fastest run. precon_dataF of the mixed-precision test is built with -DPARCEL_JACOBI_ONTHEFLY=0 into bin/<environment>/perf_prec/ by tests/perf/sub_build_prec_<environment>.sh (the double-precision reference and precon_dataF of the regression tests keep the defaults).