PARCEL SYCL v1.2 / User Guide User Guide API Reference Search Download 日本語

12 Numerical Accuracy and Reproducibility

This chapter shows how to check the results returned by the solvers, and how the results change with the precision specification, the degree of parallelism and the execution environment. How to specify precision is shown in “Precision Configurations”.


12.1 Checking the Convergence Status

solver() returns no exception or error even if it does not converge. Before using the result, check it with is_converged() (or get_converged_reason() > 0). The value is the same on all ranks.

Convergence status Main cause Example remedies
CONVERGED_RTOL (1) The tolerance was satisfied For the linear solvers, also check the true residual (next section)
DIVERGED_ITS (-1) The maximum number of iterations (for ks_arnoldi, the maximum number of restarts) was reached Add a preconditioner, increase the restart length of pgmres, check that the tolerance is reachable with the precision configuration
DIVERGED_NANORINF (-2) The residual or the eigenvalue estimate became NaN / Inf Review low-precision configurations (below), check that the matrix and the right-hand side contain no non-finite values
DIVERGED_BREAKDOWN (-3) Breakdown of the algorithm (division by zero, etc.) For pbicgstab, switch to pgmres; raise the precision of the dot products

The conditions under which each status occurs for each solver are given in “Convergence Status” in the API Reference; use parcel::krylov::converged_reason_name() to turn the value into a string. For inverse_power and shift-invert of the eigenvalue solvers, check the number of times the inner linear solver did not converge with get_linsolve_failures() (it is not reflected in the outer convergence status).


12.2 Recurrence Residual and True Residual

The residual used in the convergence check of the linear solvers is one of the following two kinds, depending on the solver.

All linear solvers compute and print the true relative residual ∥b−Ax∥/∥b∥\|b - Ax\| / \|b\| at termination (finish : residual true norm) and return it with get_true_residual(). To check the accuracy of the solution with pcg and pbicgstab, look at get_true_residual(). The difference between the recurrence residual and the true residual grows when the vectors are held at low precision. This is why the guideline for the tolerance of dataF16 is 1e-2 (“Precision Configurations”).

When built with -DRESIDUAL_REPLACE, pcg and pbicgstab also compute and print the true residual in every iteration (for diagnosis; one more SpMV per iteration). pcg updates the solution every PARCEL_PSTEP (default 1) iterations. With 2 or more, xx keeps its previous value in the iterations in between, so keep the default of 1 to see the true residual of every iteration (Preconditioned Conjugate Gradient Method).

A small residual does not mean that the error of the solution is equally small. The relative error of the solution is roughly bounded by the product of the condition number of AA and the relative residual, so for a matrix with a large condition number, the error of the solution is larger than the relative residual.

The convergence check of the eigenvalue solvers is the relative change of the eigenvalue estimate for power and inverse_power, and an estimate of the Ritz residual for ks_arnoldi; neither is the true residual ∥Ax−λx∥\|Ax - \lambda x\| itself. get_true_residual() of the eigenvalue solvers returns NaN. To check the accuracy of an eigenpair, compute the residual from the returned eigenvector.


12.3 Differences by Degree of Parallelism and Execution Environment

When run with the same execution environment, the same build, the same numbers of ranks and threads, and the same precision configuration, the results (number of iterations and solution) are the same every time. Changing the number of ranks, the number of threads or the hardware changes the last digits for the following reasons.

Factor What changes
Order of the partial sums of dot products and norms (within a rank) On CPUs with _BLOCKING_LOOP_DOT_=1, the order is determined by the number of blocks (the number of threads by default); on GPUs, by the size of the work-group
Reduction between ranks The algorithm of MPI_Allreduce (varies with the MPI implementation, the number of ranks and the message length)
Row partitioning The contents of each rank’s partial sums change with the number of ranks
Fusion of multiply-add (FMA) Differs by compiler and device. AdaptiveCpp adds -ffp-contract=fast for all targets (with omp.library-only it is not added for the host compiler, so on A64FX it is specified explicitly in the recommended options), and the CUDA backend fuses multiplications and additions into FMA

In double precision, these differences often do not appear in the number of iterations. In the measurements solving the Poisson problem with NX=512 and the Jacobi preconditioner (“Performance” chapter), the number of iterations of pcg (611 iterations) agreed across all 8 environments and all numbers of devices. pgmres for the convection-diffusion equation (mm = 60, 1140 iterations) and the variant of pcagmres that forms QQ (1152 iterations) also agreed. However, the solutions do not necessarily agree to the last digit. In the same measurements, the number of iterations of pbicgstab (397-466 iterations) and of pcagmres without forming QQ (1152-1216 iterations) varied with the environment and the number of devices.

On the other hand, the following do not depend on the number of ranks.

For the block Jacobi preconditioner, the blocks do not cross ranks, so the action of the preconditioner itself changes with the number of ranks and the number of blocks nbnb. The default number of blocks is the number of threads per rank on CPUs and the value giving 2 rows per block on GPUs, so the number of iterations changes with the number of ranks, threads and hardware even in double precision. To compare, fix the number of blocks with set_blockjacobi() or use the Jacobi preconditioner.

In configurations with lowered precision, the differences in the last digit of double precision are amplified by rounding to low precision, and the number of iterations varies with the environment. Agreement of the number of iterations between environments is not guaranteed. Reaching the tolerance and the accuracy of the solution are preserved; what changes is the number of iterations needed. The configuration that lowers only the data precision of the preconditioner to single (precon_dataF) had the same number of iterations as double precision in the NX=512 measurements above (pcg 611, pgmres for the convection-diffusion equation 1140).

To compare low-precision configurations between environments, align the build flags (such as _BLOCKING_LOOP_DOT_) and the run configuration (ranks × threads), and leave a margin in the maximum number of iterations. If you want even the number of iterations to agree across environments, use double precision (or a configuration that lowers only the data precision of the preconditioner).

Even with the same source and the same flags, the results may change when the executable is rebuilt. AdaptiveCpp adds -ffp-contract=fast for all targets, so the compiler decides at each place whether to fuse multiplications and additions into FMA, and that choice can change from build to build. On PS-B (MI300A, hip:gfx942), rebuilt executables differed by a few iterations, mainly in single-precision configurations (the same executable agrees bit for bit however many times it is run). This has not occurred on s86. To make the results agree even after rebuilding, compile with -ffp-contract=off. The results of two executables built with -ffp-contract=off agreed (the effect on performance has not been measured). The macro PARCEL_FP_CONTRACT_OFF enables #pragma clang fp contract(off) in the headers, but it is effective only together with -ffp-contract=fast-honor-pragmas, so normally use -ffp-contract=off.


12.4 Known Limits at Low Precision

The following restrictions are not defects of the implementation but limits of the precision itself (the list of FP32 and FP16 configurations is in “Precision Configurations”). In configurations including FP16, there is a lower limit (a residual floor) on the relative residual that the precision can reach; giving a smaller tolerance leads to stagnation or breakdown with NaN. In the bundled regression tests (NX=8), the tolerances of the following combinations are relaxed. Use them as a guideline for choosing the tolerance for your own problem.

Precision configuration Solver Preconditioner Tolerance in the regression tests
calcF16 pbicgstab Point Jacobi, block Jacobi 3e-2
calcF16 pcg Point Jacobi 1e-2
calcF16, dataF16 pcg Block Jacobi 1e-1 (only to check that it runs)
calcF16 pcagmres (variant that forms QQ) All 4 1e-2
precon_calcF16, precon_allF16 pcg, pgmres, pcagmres Point Jacobi 1e-5
precon_calcF16, precon_allF16 power Block Jacobi, Jacobi 1e-5


12.5 double-double

double-double (parcel_doubledouble) [12,13] represents one value by two double-precision numbers, the high part (hi) and the low part (lo), and extracts the rounding error of each operation with error-free transformations (TwoSum for addition and TwoProd for multiplication [14]) and keeps it in lo. The eigenvalues returned by the eigenvalue solvers are computed in double precision on the host, so they stay at double-precision accuracy even if dd is specified.

The error-free transformation of the product (TwoProd) is built with fma (p=fma(a,b,0)p = \mathrm{fma}(a,b,0), e=fma(a,b,−p)e = \mathrm{fma}(a,b,-p)). The result of fma is not fused any further, so the value does not change even if the compiler contracts expressions. In addition, within the dd operations on the host, float_control(precise) and fp contract(off) forbid rewriting and contraction of expressions. With -DPARCEL_DD_NO_FMA, Dekker’s splitting [12] is used. It is meant for CPUs without FMA instructions; do not use it on GPUs (CUDA), where multiplications and additions are fused into FMA and Dekker’s splitting breaks (the dd product falls to double-precision accuracy).


12.6 Do Not Use -ffast-math

Building with -ffast-math or -ffinite-math-only makes parcel_sycl.hpp stop the build with #error (detected by __FAST_MATH__ and __FINITE_MATH_ONLY__). There are two reasons.

To build anyway, knowing this, add -DPARCEL_ALLOW_FAST_MATH. The accuracy of dd and the detection of NaN are then not guaranteed. The build flags for PS-B use only -fno-math-errno -fno-trapping-math, which do not change values (“Compile Options” in “Installation”). Do not use -fno-exceptions either, since it is incompatible with std::invalid_argument of the argument checks.