PARCEL SYCL v1.2 / User Guide User Guide API Reference Search Download 日本語

14 Performance

This chapter shows the results of v1.2 measured on 8 test environments (CPUs: s86 CPU, PS-A, Miyabi-C, Odyssey; GPUs: s86 GPU, Miyabi-G, Aquarius, PS-B): the time to solve 3D problems with NX=512 to convergence, the effect of mixed precision that lowers only the preconditioner data precision, a comparison with PETSc / SLEPc, a comparison with v1.0, and the memory usage. All values were measured with the bundled verification sample tests/verify/parcel_krylov_sample_verify.cpp and do not guarantee the performance for your problem. v1.2 was measured with the default settings and the Recommended Options. The conditions are the same as those of the performance tests in the distribution package (tests/perf/); how to measure your own environment under the same conditions is described in “Measuring Your Own Environment”.


14.1 Measurement Conditions

All charts in this chapter were measured under the following conditions.

Item Condition
Problem 3D 7-point stencil, NX=512 (n = 134,217,728 unknowns). pcg, pbicgstab and the eigenvalue solvers use the Poisson equation (matrix poisson of the verification sample; symmetric); pgmres and pcagmres, which are for nonsymmetric matrices, use the convection-diffusion equation (convdiff, convection coefficient c = 0.05; nonsymmetric). Right-hand side b = 1, initial guess x = 0 (PARCEL_VERIFY=0), double precision
Linear solvers Two variants: no preconditioner and the Jacobi preconditioner (PARCEL_PRECON=4, 1 iteration). Tolerance 1.0e-8 (relative residual), maximum 10000 iterations. The parameters are the library defaults (PARCEL_LIB_DEFAULTS=1); only the restart length mm = 60 of pgmres (default 30) and (s,t)=(8,8)(s, t) = (8, 8) of pcagmres (default (8,4)(8, 4)) are specified. pcagmres with QQ formed (EXPLICIT, QR_MGS) and not formed (IMPLICIT, QR_CHOLESKYQR)
Eigenvalue solvers No preconditioner, tolerance 1.0e-8. power: maximum 20000 iterations. inverse_power: maximum 200 outer iterations; the inner linear solver is pcg with the Jacobi preconditioner (PARCEL_LINSOLVE_SOLVER=pcg PARCEL_LINSOLVE_PRECON=4) with tolerance 1.0e-10 and maximum 5000 iterations (with the default pgmres (restart 30), the inner solves of the Poisson equation reach the limit and become slow). ks_arnoldi: mm = 40, kkeepk_{keep} = 10, nevnev = 1, maximum 200 restarts (default), the orthogonalization is the default of the build for each environment: MGS2 (PARCEL_KS_ORTHO_DEFAULT=mgs) on PS-A and CGS2 on the other 7 environments (the same in the performance tests of the distribution package tests/perf)
Devices 4 / 8 / 16. “1 device” is one CPU socket (one A64FX = one node on Odyssey) or one GPU (one GH200 or MI300A)
Repetitions 3 runs each; the time is the fastest
Timed region The value <solver name> time = ... (total solver wall time) printed by each solver (wall-clock time of rank 0). It includes the setup of the preconditioner and the computation of the initial residual, but not set_crsMat() or the matrix generation of the sample
Build Flags of the Recommended Options (no ladder splitting on NVIDIA GPUs, with -Xcuda-ptxas -maxrregcount=255 only on V100; ladder splitting on MI300A, blocking of all kernels and ladder splitting of SpMV only plus the optimizations for A64FX A to F (B by run-time environment variables) on A64FX, blocking of the reductions on the other CPUs). Built from the bundled code in environments installed with the spack procedure of the distribution package
Default settings The library defaults (the values of BB of the Jacobi preconditioner are not stored; the build for A64FX stores them; PARCEL_PSTEP 1). Only precon_dataF of the mixed-precision test was built with -DPARCEL_JACOBI_ONTHEFLY=0

Ranks and threads per device in each environment (with the PETSc ranks used in the comparison with PETSc / SLEPc). PETSc on CPUs uses all cores of a socket (of a node on Odyssey) with one rank per core (flat MPI). PETSc on GPUs uses one rank per device as PARCEL does, and each rank is given the same CPU cores as in PARCEL (cores of the socket of its GPU).

Environment 1 device PARCEL ranks × threads (per device) Ranks on 4 / 8 / 16 devices PETSc ranks (per device)
s86 CPU 1 socket of Xeon Gold 6242R 4 × 5 16 / 32 / 64 20
PS-A 1 socket of Xeon 6980P 9 × 14 36 / 72 / 144 128
Miyabi-C 1 socket of Xeon CPU Max 9480 4 × 14 16 / 32 / 64 56
Odyssey 1 A64FX (1 node) 4 × 12 16 / 32 / 64 48
s86 GPU 1 V100 1 × 12 4 / 8 / 16 1
Miyabi-G 1 GH200 1 × 72 4 / 8 / 16 1
Aquarius 1 A100 1 × 9 4 / 8 / 16 1
PS-B 1 MI300A 1 × 6 4 / 8 / 16 1

On PS-A, 3 ranks (14 threads each) are placed on each NUMA domain to use all 128 cores of a socket, so the number of ranks differs from the other CPU environments. On Odyssey, one rank is placed on each CMG. On 4 devices of Odyssey, the configurations that do not fit in the memory of one node (pgmres, pcagmres and ks_arnoldi of PARCEL, and pgmres of PETSc) were spread over 8 nodes with the same number of ranks (2 ranks per node). ks_arnoldi of PETSc / SLEPc stayed on 4 nodes, so the comparison of ks_arnoldi on 4 devices of Odyssey uses different numbers of nodes. In addition, pgmres of PETSc spread over 8 nodes (24 ranks per node) was measured with the ranks pinned to two CMGs of each node, i.e., the same number of CMGs as PARCEL (2 ranks per node). With the default placement of Fujitsu MPI, the PETSc ranks are spread over all four CMGs of a node and use twice the memory bandwidth of PARCEL (the ratio PETSc time ÷ PARCEL time would be 0.71 instead of 1.04). The memory of PETSc was measured with the default placement.


14.2 Time to Convergence

All configurations converged. The iteration counts were as follows; those of the linear solvers are the values at the convergence check.

The time [s] on 16 devices is shown for each environment, separately for the linear and the eigenvalue solvers (fastest of 3 runs; pgmres and pcagmres solve the convection-diffusion equation, the others the Poisson equation). The solvers are listed for each environment, with two bars for the linear solvers (no preconditioner and the Jacobi preconditioner). The horizontal axis is the time on a logarithmic scale, common to all environments. The values at the ends of the bars are in seconds.

Time of the linear solvers [s] (16 devices) 1 1 10 10 100 100 1000 1000 Time [s] (logarithmic scale) s86 CPU (Xeon Gold 6242R) pcg 30.8 24.2 pbicgstab 45.7 34.2 pgmres 183 126 pcagmres (explicit) 136 96.5 pcagmres (implicit) 110 83.3 Odyssey (A64FX) pcg 12.1 9.64 pbicgstab 13.6 11.5 pgmres 48.8 35.7 pcagmres (explicit) 63.9 44.1 pcagmres (implicit) 47.7 36.9 Miyabi-C (Xeon CPU Max 9480) pcg 8.59 6.82 pbicgstab 9.30 8.65 pgmres 34.4 26.1 pcagmres (explicit) 39.1 27.9 pcagmres (implicit) 30.1 22.4 PS-A (Xeon 6980P) pcg 6.55 5.73 pbicgstab 7.44 7.79 pgmres 21.6 17.7 pcagmres (explicit) 26.6 18.7 pcagmres (implicit) 16.5 14.3 s86 GPU (V100) pcg 8.26 7.18 pbicgstab 11.7 9.12 pgmres 39.4 29.0 pcagmres (explicit) 36.6 27.2 pcagmres (implicit) 27.0 23.2 Miyabi-G (GH200) pcg 1.77 1.34 pbicgstab 2.11 1.53 pgmres 11.6 7.69 pcagmres (explicit) 12.1 7.91 pcagmres (implicit) 8.66 6.37 Aquarius (A100) pcg 4.68 3.92 pbicgstab 6.33 4.98 pgmres 24.3 17.5 pcagmres (explicit) 22.4 16.1 pcagmres (implicit) 17.7 13.4 PS-B (MI300A) pcg 2.81 2.11 pbicgstab 3.39 2.78 pgmres 14.9 10.2 pcagmres (explicit) 17.8 12.4 pcagmres (implicit) 12.3 8.65 no preconditioner Jacobi preconditioner
Time of the eigenvalue solvers [s] (16 devices) 1 1 10 10 100 100 1000 1000 Time [s] (logarithmic scale) s86 CPU (Xeon Gold 6242R) power 307 inverse_power 154 ks_arnoldi 435 Odyssey (A64FX) power 123 inverse_power 61.7 ks_arnoldi 101 Miyabi-C (Xeon CPU Max 9480) power 79.2 inverse_power 43.5 ks_arnoldi 91.1 PS-A (Xeon 6980P) power 64.0 inverse_power 35.8 ks_arnoldi 52.7 s86 GPU (V100) power 89.1 inverse_power 45.4 ks_arnoldi 90.1 Miyabi-G (GH200) power 17.4 inverse_power 8.49 ks_arnoldi 29.6 Aquarius (A100) power 49.1 inverse_power 25.1 ks_arnoldi 56.5 PS-B (MI300A) power 29.3 inverse_power 14.4 ks_arnoldi 38.4


14.3 Mixed Precision (precon_dataF)

The time of the configuration that lowers only the data precision of the preconditioner to single (precon_dataF) is compared with double precision for pcg with the Jacobi preconditioner (Poisson equation, 16 devices). The time is split into the rest of the solver (left) and the preconditioner (its setup and application; right), stacked. The horizontal axis is the ratio to the total time in double precision of each environment, on a linear scale common to all environments. The full length of a bar is the ratio of its total time to double precision, and the lengths of the left and right parts show the shares of the time. After each bar, the total time [s], the ratio to double precision (in parentheses; below 1 means precon_dataF is faster) and the preconditioner time are given. The iteration counts are the same as in double precision (611) in all configurations.

precon_dataF was built with the setting that stores the values of BB of the Jacobi preconditioner (without on-the-fly computation; “without H”, -DPARCEL_JACOBI_ONTHEFLY=0) and compared with double precision built with the library defaults. With the default (values of BB not stored), every application reads the double-precision values of AA even if the preconditioner matrix is single precision, so lowering the precision has little effect. Builds for A64FX store the values of BB by default, so on Odyssey double precision also stores them.

pcg (Jacobi preconditioner): time by part (16 devices; total time in double = 1) 0 0 0.25 0.25 0.5 0.5 0.75 0.75 1 1 Ratio to the total time in double precision s86 CPU (Xeon Gold 6242R) Double 24.2 s, Preconditioner 9.10 Mixed 21.2 s (0.87), Preconditioner 6.09 Odyssey (A64FX) Double 9.64 s, Preconditioner 3.26 Mixed 9.57 s (0.99), Preconditioner 3.39 Miyabi-C (Xeon CPU Max 9480) Double 6.82 s, Preconditioner 2.21 Mixed 6.09 s (0.89), Preconditioner 1.78 PS-A (Xeon 6980P) Double 5.73 s, Preconditioner 1.63 Mixed 4.97 s (0.87), Preconditioner 1.13 s86 GPU (V100) Double 7.18 s, Preconditioner 2.67 Mixed 6.81 s (0.95), Preconditioner 2.18 Miyabi-G (GH200) Double 1.34 s, Preconditioner 0.467 Mixed 1.22 s (0.91), Preconditioner 0.351 Aquarius (A100) Double 3.92 s, Preconditioner 1.29 Mixed 3.52 s (0.90), Preconditioner 1.03 PS-B (MI300A) Double 2.16 s, Preconditioner 0.635 Mixed 2.02 s (0.93), Preconditioner 0.472 Solver (other than preconditioner) Preconditioner

Conditions: NX=512, time to convergence (fastest of 3 runs), 16 devices in the chart. precon_dataF is without on-the-fly computation (-DPARCEL_JACOBI_ONTHEFLY=0) and double precision uses the library defaults. The iteration counts are the same as in double precision in all configurations (pcg 611, pgmres on the convection-diffusion equation 1140). The other conditions are those of “Measurement Conditions”. The ranges of the ratios below cover all configurations of pcg and pgmres on 4, 8 and 16 devices.

To make a solver faster with mixed precision, build precon_dataF without on-the-fly computation (-DPARCEL_JACOBI_ONTHEFLY=0). The same applies to all NVIDIA GPUs including V100, without ladder splitting (Recommended Options). In exchange, the memory per GPU grows by about 0.6–0.9 GiB (solving pcg with NX=512 on 4 V100 GPUs, the peak memory allocated by PARCEL per GPU is 6,340 MiB, 0.62 GiB more than the 5,703 MiB of the double-precision default and 0.87 GiB more than the 5,445 MiB of precon_dataF with on-the-fly computation).


14.4 Comparison with PETSc / SLEPc and v1.0

For pcg (Poisson equation) and pgmres (mm = 60, convection-diffusion equation) with the Jacobi preconditioner and ks_arnoldi (Poisson equation) without a preconditioner, the time and memory of v1.2 are compared with PETSc 3.22 / SLEPc 3.22 and the public v1.0 (16 devices). Bar lengths are ratios to v1.2 in each environment, with the v1.2 bars (dashed line) as the reference. The scale is common to the six charts, and bars with a ratio above 3 are drawn shortened (double slash). Time charts are shown in blue tones and memory charts in green tones. The value in parentheses is the ratio to v1.2; above 1 means v1.2 is faster (uses less memory). v1.0 has no eigenvalue solvers, so ks_arnoldi is compared only with SLEPc.

pcg: time [s] (16 devices; ratio to v1.2 in parentheses) s86 CPU (Xeon Gold 6242R) 24.2 s 30.3 s (1.25) 42.6 s (1.76) Odyssey (A64FX) 9.64 s 15.0 s (1.55) 11.5 s (1.19) Miyabi-C (Xeon CPU Max 9480) 6.82 s 7.32 s (1.07) 7.77 s (1.14) PS-A (Xeon 6980P) 5.73 s 6.62 s (1.15) 5.76 s (1.00) s86 GPU (V100) 7.18 s 9.47 s (1.32) 7.67 s (1.07) Miyabi-G (GH200) 1.34 s 1.47 s (1.09) 1.35 s (1.01) Aquarius (A100) 3.92 s 4.02 s (1.02) 4.45 s (1.13) PS-B (MI300A) 2.11 s 3.29 s (1.56) 2.13 s (1.01) v1.2 v1.0 PETSc 3.22
pcg: memory per device [GiB] (16 devices; ratio to v1.2 in parentheses) s86 CPU (Xeon Gold 6242R) 2.89 GiB 3.41 GiB (1.18) 6.20 GiB (2.14) Odyssey (A64FX) 2.97 GiB 3.93 GiB (1.32) 4.55 GiB (1.53) Miyabi-C (Xeon CPU Max 9480) 2.37 GiB 2.93 GiB (1.23) 9.39 GiB (3.95) PS-A (Xeon 6980P) 4.74 GiB 4.65 GiB (0.98) 23.9 GiB (5.05) s86 GPU (V100) 1.73 GiB 2.80 GiB (1.62) 1.56 GiB (0.90) Miyabi-G (GH200) 1.97 GiB 3.05 GiB (1.55) 1.82 GiB (0.92) Aquarius (A100) 1.84 GiB 2.92 GiB (1.58) 1.69 GiB (0.92) PS-B (MI300A) 1.46 GiB 2.67 GiB (1.83) 1.50 GiB (1.03) v1.2 v1.0 PETSc 3.22
pgmres: time [s] (16 devices; ratio to v1.2 in parentheses) s86 CPU (Xeon Gold 6242R) 126 s 273 s (2.17) 160 s (1.27) Odyssey (A64FX) 35.7 s 70.2 s (1.97) 35.3 s (0.99) Miyabi-C (Xeon CPU Max 9480) 26.1 s 44.1 s (1.69) 23.9 s (0.92) PS-A (Xeon 6980P) 17.7 s 34.3 s (1.94) 15.6 s (0.88) s86 GPU (V100) 29.0 s 37.2 s (1.28) 29.2 s (1.01) Miyabi-G (GH200) 7.69 s 10.7 s (1.39) 7.43 s (0.97) Aquarius (A100) 17.5 s 19.9 s (1.14) 18.3 s (1.05) PS-B (MI300A) 10.2 s 20.9 s (2.05) 10.4 s (1.02) v1.2 v1.0 PETSc 3.22
pgmres: memory per device [GiB] (16 devices; ratio to v1.2 in parentheses) s86 CPU (Xeon Gold 6242R) 7.27 GiB 8.01 GiB (1.10) 10.0 GiB (1.38) Odyssey (A64FX) 7.53 GiB 8.59 GiB (1.14) 8.33 GiB (1.11) Miyabi-C (Xeon CPU Max 9480) 6.74 GiB 7.52 GiB (1.12) 13.2 GiB (1.96) PS-A (Xeon 6980P) 10.5 GiB 10.4 GiB (0.99) 28.0 GiB (2.67) s86 GPU (V100) 5.63 GiB 6.71 GiB (1.19) 5.31 GiB (0.94) Miyabi-G (GH200) 5.88 GiB 6.95 GiB (1.18) 5.41 GiB (0.92) Aquarius (A100) 5.75 GiB 6.82 GiB (1.19) 5.45 GiB (0.95) PS-B (MI300A) 5.50 GiB 6.58 GiB (1.20) 5.22 GiB (0.95) v1.2 v1.0 PETSc 3.22
ks_arnoldi: time [s] (16 devices; ratio to v1.2 in parentheses) s86 CPU (Xeon Gold 6242R) 435 s 557 s (1.28) Odyssey (A64FX) 101 s 111 s (1.09) Miyabi-C (Xeon CPU Max 9480) 91.1 s 78.0 s (0.86) PS-A (Xeon 6980P) 52.7 s 45.0 s (0.85) s86 GPU (V100) 90.1 s 110 s (1.23) Miyabi-G (GH200) 29.6 s 34.6 s (1.17) Aquarius (A100) 56.5 s 73.5 s (1.30) PS-B (MI300A) 38.4 s 55.4 s (1.44) v1.2 SLEPc 3.22
ks_arnoldi: memory per device [GiB] (16 devices; ratio to v1.2 in parentheses) s86 CPU (Xeon Gold 6242R) 5.72 GiB 8.53 GiB (1.49) Odyssey (A64FX) 5.92 GiB 7.04 GiB (1.19) Miyabi-C (Xeon CPU Max 9480) 5.18 GiB 11.6 GiB (2.24) PS-A (Xeon 6980P) 8.55 GiB 21.9 GiB (2.56) s86 GPU (V100) 4.17 GiB 3.84 GiB (0.92) Miyabi-G (GH200) 4.42 GiB 4.13 GiB (0.94) Aquarius (A100) 4.29 GiB 3.95 GiB (0.92) PS-B (MI300A) 4.08 GiB 3.82 GiB (0.94) v1.2 SLEPc 3.22


14.5 Measuring Your Own Environment

To measure under the same conditions as this chapter, build the verification sample with the Recommended Options (verify_double) and run it without verification (PARCEL_VERIFY=0). The right-hand side is then b = 1, and each rank generates only its own rows. The run-time options are described in Run-Time Options of the Verification Sample.

# pgmres (m = 60, Jacobi preconditioner)
mpirun -n 4 ./verify_double PARCEL_NX=512 PARCEL_SOLVER=pgmres PARCEL_PRECON=4 \
  PARCEL_GMRES_RESTART=60 PARCEL_VERIFY=0 PARCEL_TOL=1.0e-8 PARCEL_MAX_ITER=10000
# ks_arnoldi (m = 40, 10 kept, nev = 1)
mpirun -n 4 ./verify_double PARCEL_NX=512 PARCEL_SOLVER=ks_arnoldi PARCEL_PRECON=0 \
  PARCEL_KS_M=40 PARCEL_KS_K_KEEP=10 PARCEL_KS_NEV=1 PARCEL_KS_RESTART=default \
  PARCEL_VERIFY=0 PARCEL_TOL=1.0e-8

Place the ranks as in Measurement Conditions; on CPUs, bind the ranks to sockets with the mpirun binding options (e.g. --map-by ppr:<ranks per socket>:socket:PE=<threads> --bind-to core). Do not set OMP_PROC_BIND on s86 CPU and PS-A (see “Installing a SYCL Compiler”). To see the memory usage as well, add PARCEL_MEM_REPORT=1.

At the end of the solver, rank 0 prints the timings in the following form (pcg; the values are from the fastest of the 3 runs on s86 GPU with 4 devices above; the [MEM] and [MEMG] lines at the end appear only with PARCEL_MEM_REPORT). The same content is also written to parcel_result.txt (changed with PARCEL_OUTPUT). pstep=1 of loop gemv shows that the default pcg updates the solution at every iteration (pstep=4 with “Without G” of “Choosing Between Memory and Speed”).

 finish : residual true norm  || b - Ax || = 9.5999802982e-09

 -----------------
pcg time              = 1.5333772000e+01 [sec]  (total solver wall time)
  init time           = 4.3617000000e-02 [sec]  (before main loop)
    init precon       = 1.0722000000e-02 [sec]
    init convert      = 3.3840000000e-03 [sec]
    init other        = 2.9511000000e-02 [sec]  (vec alloc/zero, init SpMV/dot/precon)
  main loop time      = 1.5277877000e+01 [sec]  (sum over all CG iterations)
    loop dot          = 2.6397240000e+00 [sec]  calls = 1832
      dot comm        = 9.7834400000e-01 [sec]  (MPI_Allreduce)
      dot kernel      = 1.6613800000e+00 [sec]  (局所内積)
    loop axpy         = 4.2147000000e-02 [sec]  calls = 1222
    loop gemv         = 6.7014200000e-01 [sec]  calls = 611  (pstep=1 本まとめた x の更新)
    loop SpMV         = 5.9033560000e+00 [sec]  calls = 611
      SpMV comm       = 1.5370920000e+00 [sec]  (ハロ交換)
      SpMV kernel     = 4.3662640000e+00 [sec]  (局所行列ベクトル積)
    loop precon       = 5.9604370000e+00 [sec]  calls = 610
    loop other        = 6.2071000000e-02 [sec]  (kernel launch/sync, cout per iter)
  post-loop time      = 1.2278000000e-02 [sec]  (final r_true compute, finish print)
 converged_reason = 1 (CONVERGED_RTOL) , its = 611 , residual = 9.599865e-09 , true_residual = 9.599980e-09
[MEM] rank 0 solver 後 peak RSS = 5068 MiB
[MEM] rank 0 solver 後 device memory used = 3900 MiB
[MEM] rank 0 solver 後 PARCEL device peak = 5679 MiB
[MEMG] nranks=4 host_mib=5068,5083,5085,5068 dev_mib=3900,3910,3910,3900 parcel_devpeak_mib=5679,5703,5703,5679

The performance tests in the distribution package (tests/perf/gen_perf.py, and gen_perf_prec.py for mixed precision) generate job scripts that perform the measurements of this chapter together (usage in tests/perf/README.md). The conditions are the same as in this chapter. One job measures one solver × one number of devices; each configuration is run 3 times, and the summary (tests/perf/parse_perf.py) takes the fastest run. precon_dataF of the mixed-precision test is built with -DPARCEL_JACOBI_ONTHEFLY=0 into bin/<environment>/perf_prec/ by tests/perf/sub_build_prec_<environment>.sh (the double-precision reference and precon_dataF of the regression tests keep the defaults).