1 Common Items
This reference presents the PARCEL API in separate chapters for each language. Items that do not depend on the language (solvers, input matrix, meaning and defaults of the settings, constants, meaning of the results, convergence states, handling under MPI) are collected in this chapter and in the next chapter, “Convergence by Solver”. The chapters for each language give the names and syntax in that language, how errors are reported, and examples. Read this chapter, the next one, and the chapters of the language you use. Switch the language with the tabs at the top of each page.
| Language | Chapters |
|---|---|
| C++ | From PARCEL SYCL Classes to parcel::krylov::ks_arnoldi |
| C | C API |
| Fortran | Fortran API |
| Python | Python API |
The beginning of each language’s chapters has a table that maps the settings, constants and results of this chapter to the names in that language.
1.1 Format of the Entries
Each function or type entry in the language chapters consists of the following fields. Fields that do not apply are omitted.
| Field | Contents |
|---|---|
| Description | What the function does |
| Synopsis | The declaration in that language |
| Arguments | Name, direction (In / Out / In/Out), type, description, and default value. The default value is the value the library uses when nothing is specified |
| Requirements | Valid range of the arguments and the behavior when they are violated. If the library does not check them, “not checked” is stated |
| Return Value | Meaning of the return value |
| MPI | Whether the operation is collective, and whether all ranks must pass the same value |
| Notes | Remarks on usage |
| See Also | Related functions and chapters |
| Example | A call example |
1.2 Solvers
PARCEL has the following seven solvers. The solver names are common to all languages. The algorithms are described in the “Linear Solvers” and “Eigenvalue Solvers” chapters of the User Guide, and how to choose among them in “Choosing a Solver”.
| Solver | Method | Problem |
|---|---|---|
| pcg | Preconditioned conjugate gradient method | with and the preconditioner matrix symmetric positive definite |
| pbicgstab | Preconditioned stabilized biconjugate gradient method | (nonsymmetric matrices) |
| pgmres | Preconditioned generalized minimal residual method (right preconditioning, with restarts) | (nonsymmetric matrices) |
| pcagmres | Preconditioned communication-avoiding generalized minimal residual method (CA-GMRES). It generates basis vectors at once and orthogonalizes them by QR factorization | (nonsymmetric matrices) |
| power | Power method | One eigenpair of with the largest magnitude |
| inverse_power | Inverse power method. At each iteration an inner linear solver solves | One eigenpair of with the smallest magnitude |
| ks_arnoldi | Arnoldi process with Krylov-Schur restarts. Also handles nonsymmetric matrices and complex conjugate eigenvalue pairs | Several eigenpairs of on the side of the largest magnitude; in shift-invert mode, the eigenpairs closest to the shift |
The eigenvalue solvers handle the standard eigenvalue problem . When the spectrum of the preconditioned matrix is selected, they compute the eigenvalues of instead of . Solvers can be excluded from the build (“Selecting the Solvers to Build” in the User Guide). What happens when an excluded solver is called is described in each language’s chapter.
1.3 Symbols and Terms
This reference uses the following symbols and terms.
| Symbol / term | Meaning |
|---|---|
| Number of rows owned by each rank | |
| Sum of over all ranks (number of rows of the matrix) | |
| Convergence tolerance | |
| Maximum number of iterations | |
| Restart length of pgmres; maximum dimension of the Krylov subspace of ks_arnoldi | |
| , | Number of vectors generated in one basis generation of pcagmres, and the number of basis generations before a restart (setting) |
| , | Number of eigenpairs that ks_arnoldi keeps after a restart (and returns at the end), and number of eigenpairs checked for convergence (setting) |
| Maximum number of restarts of ks_arnoldi | |
| , | Preconditioner matrix (built from according to the preconditioner settings) and the preconditioner matrix of the inner linear solver of the eigenvalue solvers |
| shift-invert | Method for finding the eigenvalues near a shift . It computes the eigenvalues of largest magnitude of and maps them back by |
| Operator to which ks_arnoldi applies the Arnoldi process: , (when the spectrum of the preconditioned matrix is computed), and in shift-invert mode or | |
| Ritz pair | Approximate eigenvalue and approximate eigenvector of computed within the Krylov subspace |
| True residual | Relative residual recomputed from the solution . In contrast, the residual updated by the recurrence of the iteration is called the recurrence residual |
| Lucky breakdown | The Krylov subspace becomes an invariant subspace and the next basis vector cannot be built. The solution (eigenpair) lies in that subspace, so this by itself is not a failure |
| rank 0 | The rank with number 0 in the communicator. It writes the progress and the results to standard output |
1.4 Input Matrix
The coefficient matrix is given in CRS format. When solving in parallel with MPI, each rank owns a block of contiguous rows ( rows from the global row number ) and passes only those rows. The format is explained in the data structure section of the User Guide.
- The row pointer has length and gives positions from the start of the owned rows. It is nondecreasing, and the difference between its last and first entries equals the number of nonzeros of the owned rows.
- Column indices are global indices of the whole matrix.
- Indices are 0-based. The C and Fortran APIs can also use 1-based indices.
- The integers for the number of rows, the number of nonzeros, the row pointer and the column indices are passed as 64-bit integers (in C++ the integer type can also be made 32-bit at build time).
The library checks the following conditions; if they are not satisfied, all ranks report the same error (how errors are reported is described in each language’s chapter).
- Each rank owns at least one row (a rank without rows is not supported). The number may differ between ranks.
- The row blocks of all ranks together cover without overlaps or gaps. They need not be in rank order.
- The row pointer satisfies the conditions above, and the column indices lie in ( for 1-based indices).
- The number of nonzeros of one rank, and the sum of its number of owned rows and its number of halo elements, fit in the range of the library’s internal local index ( or less with the default 32 bits). If they exceed it, increase the number of ranks (in C++ the local index can also be made 64-bit at build time).
The following conditions are not checked; if they are not satisfied, wrong results may be returned silently.
- Store both the upper and the lower triangle even for a symmetric matrix (there is no format that stores only one triangle).
- Store the diagonal element in every row (even if its value is 0). The Jacobi-type preconditioners use the diagonal elements.
- Sort the column indices of each row in ascending order, and do not put the same column index twice in a row.
In a build without MPI, one process owns the whole matrix (all rows).
1.5 Settings
The behavior of the solvers is determined by the following settings. Settings that are not specified take the defaults in the table. The names in each language are given in the mapping table of each language’s chapters. Values outside the accepted values are rejected (an error is reported). Give the same values on all ranks.
| Setting | Used by | Meaning | Accepted values | Default |
|---|---|---|---|---|
| Convergence tolerance | All solvers | Threshold for the relative residual in the linear solvers, for the relative change of the eigenvalue in power and inverse_power, and for the relative residual of the Ritz pairs in ks_arnoldi (in shift-invert mode it is checked on before mapping back to ) | Finite positive number | 1.0e-8 |
| Maximum number of iterations | Linear solvers, power, inverse_power | Upper limit of the iterations: the total number of inner iterations in pgmres and pcagmres, the number of outer iterations in power and inverse_power. ks_arnoldi does not use it (it uses the maximum number of restarts) | 1 or greater. Need not be a multiple of () in pgmres and pcagmres | 10000 |
| Preconditioner type | All solvers | Type of the preconditioner matrix (constants). In the linear solvers it is the preconditioner of the iteration; in the eigenvalue solvers it is used when the spectrum of the preconditioned matrix is computed | None, point Jacobi, block Jacobi, Jacobi | None |
| Number of Jacobi iterations | When the preconditioner is Jacobi | Number of Jacobi iterations | 1 or greater | 1 |
| Number of block Jacobi blocks | When the preconditioner is block Jacobi | Number of blocks per rank, equal to the degree of parallelism of the application. Finer blocks make the preconditioner weaker | 1 or greater and at most the number of owned rows (the upper bound is not checked) | Automatic. On CPUs, the environment variable PARCEL_NBLOCK, otherwise the number of OpenMP threads, otherwise the value given at build time. On GPUs, 2 rows per block |
| Scaling of the preconditioner matrix and vectors | When the preconditioner is block Jacobi or Jacobi | Whether to divide the matrix and residual by their maximum to bring them into the range of half precision (specified separately for the matrix and the vectors) | On, off, automatic | Automatic (on for block Jacobi; otherwise on only when the arithmetic precision of the preconditioner or the precision of the preconditioner matrix is half precision) |
| Restart length | pgmres | Memory for basis vectors is allocated | 1 or greater | 30 |
| Orthogonalization method | pgmres | Orthogonalization in the Arnoldi process (constants) | Classical Gram-Schmidt, modified Gram-Schmidt | Modified Gram-Schmidt |
| Basis generation size , | pcagmres | Number of vectors generated at once and number of basis generations before a restart. The restart length is | Both 1 or greater | , |
| QR factorization method | pcagmres | QR factorization of the basis (constants, combination rules) | Classical Gram-Schmidt, modified Gram-Schmidt, Cholesky QR, Cholesky QR by the moment method | Modified Gram-Schmidt (Cholesky QR when is not formed) |
| Q mode | pcagmres | Whether the orthonormal basis is formed explicitly (constants, combination rules) | AUTO, EXPLICIT, IMPLICIT | AUTO |
| Spectrum of the preconditioned matrix | Eigenvalue solvers | False: compute the eigenvalues of ; the outer eigenvalue iteration does not use the preconditioner. True: compute the eigenvalues of (notes) | True, false | False |
| Type of the inner linear solver | inverse_power | Inner linear solver (constants). pcg is only for symmetric (positive or negative definite) (not checked). When the spectrum of the preconditioned matrix is computed, pgmres is used regardless of the type | pgmres, pcg | pgmres |
| Tolerance of the inner linear solver | inverse_power, shift-invert of ks_arnoldi | Threshold for the relative residual of the inner linear solver | Finite positive number | 1.0e-10 |
| Maximum iterations of the inner linear solver | Same as above | Upper limit of the iterations of the inner linear solver (for pgmres, the total number of inner iterations; need not be a multiple of the restart length) | 1 or greater | inverse_power 5000, ks_arnoldi 2000 |
| Restart length of the inner linear solver | Same as above | Restart length of the inner pgmres. Not used when the inner solver is pcg | 1 or greater | 30 |
| Preconditioner of the inner linear solver | Same as above | Type of the preconditioner of the inner linear solver, number of Jacobi iterations, number of block Jacobi blocks, scaling (notes) | Same as for the preconditioner type, iterations, blocks and scaling | Jacobi (1 iteration); blocks and scaling automatic |
| Shift-invert | ks_arnoldi | When enabled, computes the eigenvalues closest to . The value is the shift | Any real number. must not coincide with an eigenvalue (not checked) | Disabled |
| Size of the Krylov subspace , , | ks_arnoldi | Maximum dimension of the subspace, number of eigenpairs kept after a restart (and returned at the end), number of eigenpairs checked for convergence | , , | |
| Maximum number of restarts | ks_arnoldi | Upper limit on the number of restarts | 1 or greater | 200 |
| Orthogonalization method (ks_arnoldi) | ks_arnoldi | Orthogonalization of the Arnoldi process (constants). Either method is applied twice (classical Gram-Schmidt twice = CGS2, modified Gram-Schmidt twice = MGS2). The accuracy of the eigenvalues is equivalent; only the speed differs | Classical Gram-Schmidt, modified Gram-Schmidt | Classical Gram-Schmidt (CGS2). Can be changed to modified Gram-Schmidt when building the library |
| Initial vector | ks_arnoldi | Initial vector of the Arnoldi process. power and inverse_power always use the default vector | Any vector. If its norm is 0 or non-finite, it is replaced by the default vector (rank 0 reports this) | Vector built from the global row numbers (independent of the row partitioning) |
Some settings cannot be specified in some languages, or are expressed differently from the table (for example, “use the default” expressed as 0). These are shown in the mapping table of each language’s chapters.
1.5.1 Notes on the Preconditioners
- The block Jacobi preconditioner is slow on GPUs; on GPUs the Jacobi preconditioner is recommended. The types of preconditioners are described in the “Preconditioners” chapter of the User Guide.
- The Jacobi preconditioner shares the row pointers and column indices of the coefficient matrix without copying them, and by default it does not keep the values of either but computes them from the values of at every application (the default of builds for A64FX stores the values). The preconditioner is built inside the solve and used only there, so do not destroy or modify while the solve is running. Building with
-DPARCEL_JACOBI_ONTHEFLY=0stores the values of , which uses more memory and makes the preconditioner faster on GPUs (“Choosing Between Memory and Speed” in the User Guide). With matrix scaling, and when the coefficient matrix, the preconditioner matrix or the preconditioner computation precision is double-double, the values are stored even with the default. - When the number of blocks and the scaling are automatic, the values actually used are printed if the environment variable
PARCEL_PRECON_VERBOSEis set. - Scaling is not used with the point Jacobi preconditioner or without preconditioning.
1.5.2 Notes on the Eigenvalue Solvers
- When the spectrum of the preconditioned matrix is not computed (default), the preconditioner settings are not used by the outer eigenvalue iteration. Specifying only the preconditioner does not make the eigenvalues of converge faster. To speed up the inner linear solver of inverse_power and shift-invert, set the preconditioner of the inner linear solver.
- When the spectrum of the preconditioned matrix is computed, the inner linear solver (inverse_power, shift-invert of ks_arnoldi) becomes a right-preconditioned pgmres that solves and returns , and the preconditioner is itself (the settings of the preconditioner of the inner linear solver are not used).
- The settings of the inner linear solver are used by inverse_power and by the shift-invert mode of ks_arnoldi, and not by power or ks_arnoldi without a shift. The orthogonalization of the inner pgmres is fixed to the modified Gram-Schmidt method.
- The outer iteration continues even if the inner linear solver does not converge. The number of such cases is obtained as a result. It is not reflected in the outer convergence state.
- In the shift-invert mode of ks_arnoldi without the spectrum of the preconditioned matrix, the inner pgmres solves , and is built from . How is given is described in each language’s chapter.
- The error of the inner linear solver becomes the error of the eigenvalue estimate as it is, so the inner linear solver must converge more tightly than the outer .
1.5.3 Q Modes and QR Factorization of pcagmres
The Q modes (constants) are as follows.
| Mode | Behavior |
|---|---|
| AUTO | is not formed only when the QR factorization is Cholesky QR and ; otherwise is formed |
| EXPLICIT | is always formed (even for Cholesky QR with ) |
| IMPLICIT | is not formed for any . Only is obtained from the Cholesky factorization of the Gram matrix of the basis , and the solution is updated with . The QR factorization must be Cholesky QR or Cholesky QR by the moment method |
The combination of the Q mode and the QR factorization is checked at the entry of the solve, independently of the order in which the settings are given, and the following cases are rejected (the same judgment on all ranks).
- IMPLICIT with classical or modified Gram-Schmidt explicitly specified (if no QR factorization is specified, Cholesky QR is used).
- Cholesky QR by the moment method combined with AUTO or EXPLICIT.
- Cholesky QR by the moment method with a preconditioner other than none, or with that is not an even number of 2 or more.
Cholesky QR by the moment method (moment method) is only for symmetric . This is not checked; with a nonsymmetric a wrong is used and wrong results are returned. AUTO and EXPLICIT perform the same computation except for Cholesky QR with . IMPLICIT computes in a different order, so its results differ within rounding errors even with the same , and QR factorization. With IMPLICIT, the orthogonality error of the Cholesky QR method is proportional to the square of the condition number of the basis, so a larger can worsen convergence. With FP16 data or arithmetic precision, do not use the Cholesky QR family (IMPLICIT and the moment method): the result is NaN or no convergence regardless of and (not checked). The methods are described in the “Linear Solvers” chapter and the “QR Factorization” chapter of the User Guide.
1.6 Constants
The values of the settings are expressed with the following constants. The values are the same in all languages; the names in each language are given in that language’s chapters.
Preconditioner type:
| Constant | Value | Preconditioner |
|---|---|---|
| NONE | 0 | No preconditioning |
| POINTJACOBI | 1 | Point Jacobi preconditioner |
| BLOCKJACOBI | 2 | Block Jacobi preconditioner |
| JACOBI | 4 | Jacobi preconditioner |
Orthogonalization and QR factorization method:
| Constant | Value | Method | Solvers |
|---|---|---|---|
| CGS | 0 | Classical Gram-Schmidt method | pgmres, pcagmres (when is formed), ks_arnoldi (applied twice) |
| MGS | 1 | Modified Gram-Schmidt method | pgmres, pcagmres (when is formed), ks_arnoldi (applied twice) |
| CHOLESKYQR | 2 | Cholesky QR method | pcagmres |
| CAGMRES_CHOLESKYQR | 202 | Cholesky QR method by the moment method (symmetric matrices without preconditioning only) | pcagmres (IMPLICIT only) |
Type of the inner linear solver:
| Constant | Value | Solver |
|---|---|---|
| PGMRES | 0 | Right-preconditioned pgmres |
| PCG | 1 | pcg |
Q mode of pcagmres:
| Constant | Value | Behavior |
|---|---|---|
| AUTO | 0 | See above |
| EXPLICIT | 1 | Same as above |
| IMPLICIT | 2 | Same as above |
The constants of the convergence states are given in the section below.
1.7 Results
Besides the solution (the eigenvalues and eigenvectors for the eigenvalue solvers), a solve returns the following values. They are the same on all ranks. The details of their meaning differ between the solvers and are given in the convergence status section and in the next chapter.
| Result | Meaning |
|---|---|
| Convergence state | One of the convergence state constants. Positive means converged, negative means failure, and 0 means that no solve has been run |
| Iteration count | How iterations are counted differs between the solvers (table below) |
| Residual | Relative value compared with the convergence tolerance (table below). NaN if it has not been computed |
| True residual | computed at the end of a linear solver, returned without additional computation. NaN for the eigenvalue solvers. The residual of pcg and pbicgstab is the recurrence residual, which can drift from the true residual through rounding errors; to check the accuracy of the solution, look at the true residual |
| Number of converged eigenpairs | Number of eigenpairs among the ones that satisfied the tolerance in the last convergence check of ks_arnoldi (NaN is not counted; if true residuals were checked, for example in an invariant subspace, that number). on convergence. 0 for the other solvers |
| Number of inner solver failures | Number of times the inner linear solver ended without satisfying its tolerance in inverse_power and the shift-invert mode of ks_arnoldi. 0 for power and ks_arnoldi without a shift. Not reflected in the outer convergence state. If it is not 0, review the maximum iterations and the preconditioner of the inner linear solver |
1.7.1 Output of Eigenvalues and Eigenvectors
- power and inverse_power return one eigenvalue and one eigenvector. The eigenvalue is real.
- ks_arnoldi returns eigenpairs in descending order of (closest to first in shift-invert mode). Convergence is guaranteed only for the first pairs.
- An eigenvalue is given as a pair of real and imaginary parts. For a complex conjugate pair, the corresponding eigenvectors are stored consecutively as two vectors, the real part and the imaginary part.
- If the run ends with an effective dimension below , the missing eigenpairs are returned as 0 (rank 0 reports this). Eigenvalues for which in the inverse transformation of shift-invert are returned as .
- The layout of the arrays in each language is given in that language’s chapters.
1.8 Convergence Status
The convergence state returned by a solve is one of the following constants (following PETSc’s KSPConvergedReason: positive means converged, negative means failure).
| Constant | Value | Meaning |
|---|---|---|
| CONVERGED_RTOL | 1 | The convergence tolerance was satisfied |
| ITERATING | 0 | Not run (before a solve, or not solved because of an argument error) |
| DIVERGED_ITS | -1 | The maximum number of iterations (for ks_arnoldi, the maximum number of restarts) was reached |
| DIVERGED_NANORINF | -2 | NaN / Inf appeared in the residual or eigenvalue estimate used for the convergence check |
| DIVERGED_BREAKDOWN | -3 | Breakdown of the algorithm (table below) |
The cases in which each state occurs are shown for each solver. The details are given in the next chapter.
| Solver | CONVERGED_RTOL | DIVERGED_NANORINF | DIVERGED_BREAKDOWN |
|---|---|---|---|
| pcg | Relative recurrence residual . ( without iterating, iteration count 0). Relative initial residual ( left at the initial guess without iterating, iteration count 0) | Stops as soon as the relative recurrence residual becomes non-finite | Does not occur (division by zero is not detected; if the result is non-finite, NANORINF) |
| pbicgstab | Same as pcg. Stopped at with (lucky breakdown) | Same as pcg | is 0 or non-finite, , , or with |
| pgmres | True relative residual at the start of a restart , or true relative residual at termination when the iteration limit was reached. | Stops as soon as the true relative residual or of the Arnoldi process becomes non-finite | Lucky breakdown with no column usable for the update ( singular at the first column) |
| pcagmres | Same as pgmres | Stops as soon as the true relative residual becomes non-finite | No column usable for the update after a lucky breakdown (rank deficiency of , ) |
| power, inverse_power | Relative change of the eigenvalue | Stops as soon as the eigenvalue estimate, its relative change, or an inner product used for the check becomes non-finite (inverse_power: also when the inner linear solver ended with NANORINF) | power: ; inverse_power: the solution of the inner linear solver is 0 |
| ks_arnoldi | Ritz residual estimates of the target pairs (with an invariant subspace or random directions, also confirmed by the true residual) | Stops as soon as a value of the Arnoldi process, a Ritz value, or a residual estimate becomes non-finite | An invariant subspace that cannot be confirmed by the true residual, an effective dimension below , or a LAPACK failure |
DIVERGED_ITS means that none of the above occurred and the maximum number of iterations (for ks_arnoldi, the maximum number of restarts) was reached. The algorithms are described in the “Linear Solvers” chapter and the “Eigenvalue Solvers” chapter of the User Guide.
1.8.1 Meaning of the Iteration Count and Residuals
The meanings of the iteration count, the residual and the true residual for each solver are as follows.
| Solver | Iteration count | Residual | True residual |
|---|---|---|---|
| pcg, pbicgstab | Number of iterations | Relative recurrence residual ( when pbicgstab stops at ) | True relative residual computed at termination |
| pgmres | Total number of inner iterations (number of restarts × ; the maximum number of iterations if the limit was reached) | True relative residual computed at the start of each restart. If the iteration limit was reached, the value after the last update (= the true residual) | Same as left |
| pcagmres | Total number of basis vectors (number of cycles × ; the maximum number of iterations if the limit was reached) | Same as pgmres | Same as left |
| power, inverse_power | Number of outer iterations | Relative change of the eigenvalue . NaN if stopped on a non-finite value | NaN |
| ks_arnoldi | Number of restarts performed | Maximum of the Ritz residual estimates of the target pairs (the maximum of the true residuals if those were checked) | NaN |
1.8.2 Common Handling in the Linear Solvers
In the linear solvers (pcg, pbicgstab, pgmres, pcagmres), returns without iterating, with the state CONVERGED_RTOL (iteration count 0; residual and true residual 0). Rank 0 writes the line ||b|| = 0 : x = 0.
If the relative initial residual of the initial guess is below (including the case where is already the solution), every linear solver returns CONVERGED_RTOL without iterating (iteration count 0) and leaves at the initial guess. The residual and the true residual are this relative initial residual.
1.8.3 Standard Output
Rank 0 writes one line of the following form to standard output at the end of a solve (non-finite values are printed as n/a). inverse_power appends , linsolve_failures = ..., and ks_arnoldi appends , ks_nconv = <number converged>/nev (plus linsolve_failures in shift-invert mode).
Example: converged_reason = 1 (CONVERGED_RTOL) , its = 27 , residual = 4.200000e-10 , true_residual = 4.211375e-10
In addition, rank 0 writes the progress of the iteration and a breakdown of the computing time. The linear solvers write the true relative residual at termination on the line finish : residual true norm. pcagmres does not write this line when it returns at the check before the iteration. The format of the standard output is not part of the API. To obtain the results from a program, use the result interface of each language.
1.9 Settings and MPI
- The settings are used when a solve is run. Give the same values on all ranks. Values outside the accepted values are rejected, and all ranks report the same error.
- A solve is a collective operation (
MPI_Barrier,MPI_Allreducefor inner products, and halo communication in SpMV). All ranks of the communicator with which the matrix is given must call it with the same solver and the same settings. - At the entry of a solve, the conditions on the input matrix, the size of the subspace of ks_arnoldi (including ), and the combination of the Q mode and the QR factorization of pcagmres are checked. In every case, all ranks report the same error before any collective communication.
- A solve does not report an error even if it does not converge. Check the convergence state before using the result. The convergence state is reset at the entry of a solve and set at its exit.
- If device memory cannot be allocated, or the size of the matrix exceeds the range of the library’s internal integers, an error occurs. These can happen on only some ranks, so the handling differs between the languages (see each language’s chapter).
- The dense LAPACK computations (ks_arnoldi) are done on rank 0, and the results are distributed to all ranks. All values used for decisions are aggregated over all ranks, so all ranks take the same branch.
1.10 Precision
PARCEL can choose the precision in which data are stored (data precision) and the precision of the arithmetic (arithmetic precision) separately for each operation (SpMV, preconditioner, vector operations, inner products) and each piece of data (matrix, vectors, preconditioner matrix, etc.). The types are half precision (FP16), single precision, double precision, long double, and double-double (about quadruple precision). How the precision is chosen differs between the languages.
| Language | Choosing the precision |
|---|---|
| C++ | Template arguments for each operation and each piece of data (PARCEL Data Types) |
| C, Fortran | Fixed by macros when the wrapper is compiled; cannot be changed at run time (all double precision by default). The input and output arrays are double precision |
| Python | Double precision only |
With configurations that use half precision, there is a lower limit on the relative residual that the precision can reach; with a tolerance below it, the iteration stagnates or breaks down with NaN. The details of the restrictions are given in “Specifying the Precision” in the “Usage” chapter and in “Numerical Accuracy and Reproducibility” of the User Guide.