5 Usage
This chapter describes how to use the SYCL version of PARCEL from C++, C, Fortran, and Python. The description for each language is shown in tabs (the selected language is kept across pages and is shared with the language selection of the API Reference). The tab of each language is written so that it can be used by reading that tab alone. The sections for C++, C, and Fortran consist of the same items (setting the matrix, setting the vectors, setting the preconditioner, calling a linear solver, calling an eigenvalue solver, specifying precision, MPI decomposition, compiling and running, passing your own matrix), so corresponding items can be compared across languages. How to pass the matrix of your own problem to PARCEL (building CRS from triplets, reading a Matrix Market file, distributing to the ranks, choosing the partition) is in the last subsection, “Passing Your Own Matrix”, of the section for each language.
PARCEL itself is a C++ header library. From C and Fortran, the same 7 solvers as in C++ can be selected at run time and called through the bundled C API (capi/parcel_sycl_c.h, implemented in capi/wrapper_parcel_sycl.cpp) and Fortran module (capi/parcel_sycl_mod.f90). The differences between the languages are summarized below. Python calls the library through a native layer separate from the C / Fortran API; its differences are summarized in the Python section.
| Item | C++ | C / Fortran |
|---|---|---|
| Linear solvers | pcg, pbicgstab, pgmres, pcagmres | Same as C++ (parcel_solve_linear) |
| Eigenvalue solvers | power, inverse_power, ks_arnoldi | Same as C++ (parcel_solve_eigen) |
| Specifying the solver, preconditioner, convergence tolerance, and iteration count | At run time (member functions of the solver class) | At run time (values in the structure parcel_options) |
| Precision (mixed precision) | Template arguments | Compile-time macros of the wrapper. Input/output arrays are always double |
| Input arrays of the matrix | set_crsMat() overwrites the column indices and values |
Not overwritten (copied inside the wrapper) |
| Index base | 0 | 0 or 1 (index_base) |
| Layout of multiple vectors (eigenvectors) | One vector after another (the j-th vector of EigenVec starts at j*n_local) |
Likewise one vector after another (column-major). Matches the layout of the Fortran array eigvec(n_local, k); no transposition is needed |
| MPI row decomposition | Arbitrary (nstart, nend per rank). set_crsMat() checks the coverage of the row blocks |
Arbitrary (nstart, n_local per rank). The coverage of the row blocks is checked |
| Integers such as row and column indices | PARCEL_INT (64-bit by default; 32-bit with -DPARCEL_INT32). The rank-local indices inside the library are PARCEL_LINT (32-bit by default) |
Always 64-bit (parcel_int in C, integer(parcel_int_kind) in Fortran) |
| Reuse of matrices and vectors | Objects can be created once and used repeatedly | A SYCL queue is created and the matrix is registered on every call |
| Retrieving the convergence status (whether converged, number of iterations, residual) | Query functions of the solver class (is_converged(), get_converged_reason(), get_iterations(), etc.) |
The structure parcel_result passed as the last argument (NULL allowed in C, optional in Fortran). The return value is only an error code for argument checks and the like |
set_precon_space_output() and set_shift() of pgmres (features used internally by the eigenvalue solvers; available from C++) and the scaling setting of the inner linear solver (set_linsolve_scaling_matrix/vector) cannot be specified from C/Fortran (the latter is determined automatically).
5.1 C++
5.1.1 Overview
Include Include/parcel_sycl.hpp and make the calls in the following order.
- Create an in-order
sycl::queueand aparcel::comm(MPI communicator). Create the queue with thein_orderproperty, as insycl::queue q{sycl::default_selector_v, sycl::property::queue::in_order{}};(the device is selected in the same way as withsycl::queue q;). The reason is given in the Quick Start. The check can be removed with-DPARCEL_REQUIRE_IN_ORDER=0, but this is not recommended. - Register a matrix in CRS format to a
parcel::matrixwithset_crsMat(). - Create
parcel::vectorobjects and copy values from host arrays withsetValue(). - Create a solver class (
parcel::krylov::pcg, etc.) and set the preconditioner, convergence tolerance, and maximum number of iterations. - Call
solver<...>()and copy the result back to the host array withgetValue().
In a build with MPI, MPI_Init() and parcel::comm are required even when running with a single process (in a build without MPI, MPI_Init() is not needed and the communicator is created with parcel::comm comm(&q);; see “Build without MPI”). Linear solvers and eigenvalue solvers can share the same matrix object; the only difference in the call is the meaning of the second and third arguments of solver().
Below is a complete program that registers a row-partitioned 1D Laplacian (tridiagonal matrix), solves with pcg, and then computes eigenvalues of the same matrix with ks_arnoldi. A similar example is in examples/example_pcg_ks.cpp.
#include <cmath>
#include <iostream>
#include <vector>
#include "./Include/parcel_sycl.hpp"
int main(int argc, char *argv[]) {
MPI_Init(&argc, &argv);
// SYCL queue: create it in-order (assumed by PARCEL)
sycl::queue q{sycl::default_selector_v, sycl::property::queue::in_order{}};
parcel::comm comm(MPI_COMM_WORLD,&q);
int myrank = comm.mpi_rank();
int nprocess = comm.mpi_size();
// ---- MPI decomposition: assign contiguous rows to each rank ----
const PARCEL_INT n_global = 1000;
PARCEL_INT nper = n_global / nprocess;
PARCEL_INT nstart = nper*myrank;
PARCEL_INT nend = nper*(myrank+1)-1;
if(myrank == nprocess-1){ nend = n_global-1; } // the last rank takes the remainder
PARCEL_INT n_local = nend - nstart + 1; // number of locally owned rows
// ---- Setting the matrix: CRS of the locally owned rows (row pointer: local, 0-based; column indices: global, 0-based) ----
std::vector<PARCEL_INT> rowptr(n_local+1), col;
std::vector<double> val;
rowptr[0] = 0;
for(PARCEL_INT i = nstart; i <= nend; i++){
if(i > 0) { col.push_back(i-1); val.push_back(-1.0); }
col.push_back(i); val.push_back(2.0);
if(i < n_global-1){ col.push_back(i+1); val.push_back(-1.0); }
rowptr[i-nstart+1] = (PARCEL_INT)col.size();
}
parcel::matrix<parcel_double> A(&q);
A.set_crsMat<double>(&comm, nstart, nend, n_local, col.size(),
rowptr.data(), col.data(), val.data());
// Note: at this point col and val have been rewritten to local indices
// ---- Setting the vectors ----
std::vector<double> x_h(n_local, 0.0), b_h(n_local, 1.0);
parcel::vector<parcel_double> x(n_local,&q), b(n_local,&q);
x.setValue(x_h.data());
b.setValue(b_h.data());
// ---- Linear solver (pcg) ----
parcel::krylov::pcg<
parcel_double, parcel_double, parcel_double, parcel_double // Tspmv,Tprecon,Taxpy,Tdot
> cg;
cg.set_tolerance(1.0e-9);
cg.set_max_iteration(10000);
cg.set_precon_algo(parcel::PRECON_JACOBI); // setting the preconditioner
cg.set_jacobi(1);
q.wait();
cg.solver<
parcel_double, parcel_double, parcel_double, // Tmat,Tvecx,Tvecb
parcel_double, parcel_double, parcel_double // Tvec,Tprecon_mat,Tprecon_vec
>(A, x, b);
x.getValue(x_h.data()); q.wait();
if(!cg.is_converged()){ // solver() does not return an error even if not converged
if(myrank == 0)
std::cout << "pcg: " << parcel::krylov::converged_reason_name(cg.get_converged_reason())
<< " after " << cg.get_iterations() << " iterations" << std::endl;
MPI_Abort(MPI_COMM_WORLD, 1);
}
// ---- Eigenvalue solver (ks_arnoldi): nev eigenvalues of largest magnitude ----
const PARCEL_INT m = 30, k_keep = 10, nev = 4;
parcel::krylov::ks_arnoldi<
parcel_double, parcel_double, parcel_double, parcel_double
> ks;
ks.set_ks_subspace(m, k_keep, nev);
ks.set_ks_max_restart(50);
ks.set_tolerance(1.0e-9);
ks.set_precon_spectrum(false); // default. If true, gives the eigenvalues of A K^-1 with the K of set_precon_algo()
std::vector<double> eval_h(2*k_keep, 0.0); // real and imaginary parts alternate
std::vector<double> evec_h(n_local*k_keep, 0.0); // the first n_local elements are the initial vector
for(PARCEL_INT i = 0; i < n_local; i++){
double g = (double)(nstart + i);
evec_h[i] = 1.0 + 0.5*std::sin(0.317*g + 0.7) + 0.3*std::cos(0.181*g - 1.3);
}
parcel::vector<parcel_double> eval(2*k_keep,&q), evec(n_local*k_keep,&q);
eval.setValue(eval_h.data());
evec.setValue(evec_h.data());
q.wait();
ks.solver<parcel_double, parcel_double, parcel_double>(A, eval, evec);
eval.getValue(eval_h.data()); evec.getValue(evec_h.data()); q.wait();
if(myrank == 0){
std::cout << "ks_arnoldi: " << parcel::krylov::converged_reason_name(ks.get_converged_reason())
<< ", restarts = " << ks.get_iterations()
<< ", converged pairs = " << ks.get_ks_nconv() << "/" << nev << std::endl;
for(int j = 0; j < nev; j++)
std::cout << "lambda_" << j << " = " << eval_h[2*j] << " + i " << eval_h[2*j+1] << std::endl;
}
MPI_Finalize();
return 0;
}5.1.2 Setting the Matrix
Create a parcel::matrix<T> (T is the storage precision) and register the CRS matrix of the locally owned rows with set_crsMat<Ttype>(comm, nstart, nend, n, nnz, rowptr, crsCol, val). Ttype is the type of the input array val; the values are converted to T on registration.
rowptr: length n+1. Local row pointer in which the first locally owned row starts at 0.crsCol: length nnz. Global 0-based column indices.n,nnz: the number of locally owned rows (= nend−nstart+1) and the number of nonzero elements in the locally owned rows.
set_crsMat() overwrites crsCol and val in place (it converts the column indices to local indices and, in each row, reorders the entries so that columns owned by the rank come first and halo columns come after). rowptr is not modified. If a second matrix (e.g., for shift-invert) is created from the same arrays, the column indices are converted twice and the halo exchange breaks in MPI runs. When creating multiple matrices, copy the arrays before the first call to set_crsMat().
set_crsMat() checks the following 4 points before registration. The results of all ranks are gathered by collective communication, and if any check fails, all ranks throw the same std::invalid_argument (the message names the offending ranks and the reason). The checks are made both in the normal version and in the version that takes the high and low parts of double-double as separate arrays.
n == nend - nstart + 1, and at least one row (a rank with 0 rows is not allowed).rowptr[0] == 0,rowptris non-decreasing, androwptr[n] == nnz.- The row blocks of all ranks cover without gaps or overlaps (they need not be in rank order).
- The column indices lie in .
The cost is one pass over the nonzeros plus a few collective communications. The checks can be removed with -DPARCEL_NO_CRS_CHECK=1.
In addition, the input CRS must satisfy the following conditions. They are not checked, so violating them can silently give wrong results. The same applies to the C, Fortran and Python APIs.
- Even for a symmetric matrix, store both the upper and lower triangles (there is no format that stores only one side).
- Store the diagonal element in every row (even if its value is 0). The Jacobi-type preconditioners use the diagonal elements; a row without its diagonal element makes the preconditioner wrong.
- Sort the column indices in each row in ascending order, and do not put the same column index twice in a row.
5.1.3 Setting the Vectors
parcel::vector<T> v(n, &q) allocates a vector of length n on the device; setValue(ptr) copies from a host array and getValue(ptr) copies back to the host. The length is the number of locally owned rows. For linear solvers, x in solver(A, x, b) is the initial guess (input) and the solution (output), and b is the right-hand side. For eigenvalue solvers, the second argument holds the eigenvalues and the third the eigenvectors; their sizes differ by solver (see “Calling an Eigenvalue Solver”). The size can be changed with resize(n, &q).
5.1.4 Setting the Preconditioner
The preconditioner is set at run time with member functions of the solver object. This is common to all solvers (including the eigenvalue solvers).
| Function | Description |
|---|---|
set_precon_algo(algo) |
parcel::PRECON_NONE (0), PRECON_POINTJACOBI (1), PRECON_BLOCKJACOBI (2), PRECON_JACOBI (4) |
set_jacobi(iter) |
Number of iterations of the Jacobi preconditioner (default 1) |
set_blockjacobi(nblock) |
Number of blocks of the block Jacobi preconditioner. If not called, it is determined automatically from the device (CPU: number of threads per rank; GPU: 2 rows per block) |
set_scaling_matrix(flag) |
Scaling of the preconditioner matrix. If not called, automatic (enabled for block Jacobi and half precision, disabled otherwise) |
set_scaling_vector(flag) |
Scaling of the preconditioner vector. Default as above |
How to choose a preconditioner on GPUs is shown in “Preconditioners” in “Choosing a Solver”. For the eigenvalue solvers, these settings are used only when set_precon_spectrum(true) is specified, in which case all three solvers return the eigenvalues of instead of . By default (set_precon_spectrum(false)), the preconditioner settings are not used and the eigenvalues of are returned. The preconditioner of the inner linear solver of inverse_power and of the shift-invert of ks_arnoldi is set with separate functions set_linsolve_precon_algo(), set_linsolve_blockjacobi(), set_linsolve_jacobi() (default: Jacobi preconditioner with 1 iteration).
5.1.5 Calling a Linear Solver
All four classes are template<Tspmv, Tprecon, Taxpy, Tdot> and have set_tolerance(tol) (convergence tolerance of the relative residual, default 1.0e-8) and set_max_iteration(itr) (default 10000). The number of template arguments of solver() differs by solver. The template arguments of the class, and those of solver() other than Tmat, Tvecx, Tvecb (which are deduced from the actual arguments), default to parcel_double and can be omitted (e.g., parcel::krylov::pcg<> s; s.solver(A, x, b);).
| Class | Solver-specific settings | Template arguments of solver<...>(A, x, b) |
|---|---|---|
parcel::krylov::pcg |
None | Tmat, Tvecx, Tvecb, Tvec, Tprecon_mat, Tprecon_vec |
parcel::krylov::pbicgstab |
None | Same as above |
parcel::krylov::pgmres |
set_restart(m) (default 30), set_qr_algo(algo) (QR_CGS / QR_MGS; default QR_MGS) |
Same as above + TGS, THessenberg |
parcel::krylov::pcagmres |
set_restart(s,t) (default 8, 4), set_explicit_q(mode) (CAGMRES_Q_AUTO / CAGMRES_Q_EXPLICIT / CAGMRES_Q_IMPLICIT; default CAGMRES_Q_AUTO), set_qr_algo(algo) (QR_CGS / QR_MGS / QR_CHOLESKYQR / QR_CAGMRES_CHOLESKYQR; default QR_MGS. With CAGMRES_Q_IMPLICIT, QR_CHOLESKYQR or QR_CAGMRES_CHOLESKYQR, and QR_CHOLESKYQR if not specified) |
Same as above + Tbasis, TcalcQR, THessenberg [, Todd_basis, Teven_basis] |
parcel::krylov::pgmres<parcel_double,parcel_double,parcel_double,parcel_double> gm;
gm.set_restart(8);
gm.set_qr_algo(parcel::qr::QR_MGS);
gm.set_tolerance(1.0e-9);
gm.set_max_iteration(10000);
gm.set_precon_algo(parcel::PRECON_JACOBI);
gm.solver<parcel_double,parcel_double,parcel_double,parcel_double,
parcel_double,parcel_double,
parcel_double,parcel_double // TGS, THessenberg
>(A, x, b);
if(!gm.is_converged() && myrank == 0){
std::printf("pgmres: %s, its = %d, residual = %.3e, true residual = %.3e\n",
parcel::krylov::converged_reason_name(gm.get_converged_reason()),
(int)gm.get_iterations(), gm.get_residual(), gm.get_true_residual());
}solver() throws no exception and returns no error even if it does not converge, so check is_converged() (or get_converged_reason() > 0) before using the result. The value is the same on all ranks. The meanings of get_iterations(), get_residual(), and get_true_residual() differ by solver (“Convergence Status Constants and Standard Output” in the API Reference). The iteration history and the computation times are printed to the standard output of rank 0.
When an out-of-range value is given to a setter function (set_tolerance(0.0), set_precon_algo(3), set_restart(0), etc.), or when a check at the entry of solver() (an in-order queue, the subspace size of ks_arnoldi, the combination of the way is formed and the QR factorization of pcagmres, etc.) is violated, std::invalid_argument is thrown. The list of checked values is given in “Handling of Settings and Exceptions” in the API Reference.
If device memory cannot be allocated, the library throws util_sycl::device_bad_alloc (derived from std::bad_alloc; what() gives the requested number of bytes and the place of the allocation). The allocation often fails on only some of the ranks, while the other ranks wait in the next collective communication. Do not catch the exception and continue; stop the whole job, e.g., with MPI_Abort().
5.1.6 Calling an Eigenvalue Solver
Eigenvalue solvers are called in the form solver(A, EigenValue, EigenVec). The template arguments of the class are the same as for the linear solvers, <Tspmv, Tprecon, Taxpy, Tdot>.
| Class | What is computed | Size of EigenValue | Size of EigenVec | Template arguments of solver |
|---|---|---|---|---|
parcel::krylov::power |
One real eigenvalue of largest magnitude | 1 or more (stored at the beginning) | n | Tmat, Teigen_val, Teigen_vec, Tvec, Tprecon_mat, Tprecon_vec |
parcel::krylov::inverse_power |
One real eigenvalue of smallest magnitude | 1 or more (stored at the beginning) | n | Same as above |
parcel::krylov::ks_arnoldi |
nev eigenvalues of largest magnitude (or those near σ) | 2·k_keep (real and imaginary parts alternate) | n·nev or more (min(k_keep, size/n) vectors are written; on input, the first n elements are the initial vector) | Tmat, Teigen_val, Teigen_vec [, Tprecon_mat, Tprecon_vec] |
power and inverse_power generate the initial vector internally, so the input values of EigenVec are not used. The convergence criterion (set_tolerance) is the relative change of the eigenvalue for power / inverse_power, and the Ritz residual for ks_arnoldi. When eigenvalues form a complex conjugate pair, the corresponding eigenvector is stored as two vectors, its real part and imaginary part.
// Power method (for the inverse power method, just replace power with inverse_power)
parcel::krylov::power<parcel_double,parcel_double,parcel_double,parcel_double> pw;
pw.set_tolerance(1.0e-9);
pw.set_max_iteration(2000);
pw.set_precon_spectrum(false); // default: eigenvalues of A. The preconditioner settings are not used in the outer iteration
parcel::vector<parcel_double> lam(1,&q), v(n_local,&q);
pw.solver<parcel_double,parcel_double,parcel_double,
parcel_double,parcel_double,parcel_double>(A, lam, v);
double lam_h; lam.getValue(&lam_h); q.wait();
if(!pw.is_converged()){ /* not converged (DIVERGED_ITS, etc.). pw.get_residual() is the relative change of the eigenvalue */ }The solver-specific settings are as follows.
- Common to the 3 solvers : with
set_precon_spectrum(true), the eigenvalues of are returned ( is built from byset_precon_algo()and related functions). By default (false), the eigenvalues of are returned and is not used in the outer eigenvalue iteration. - inverse_power : the inner linear solver is right-preconditioned pgmres by default, controlled by
set_linsolve_tolerance(tol)(default 1.0e-10),set_linsolve_max_iteration(itr)(default 5000),set_linsolve_restart(m)(default 30),set_linsolve_precon_algo(algo)(defaultparcel::PRECON_JACOBIwith 1 iteration),set_linsolve_blockjacobi(nblock), andset_linsolve_jacobi(iter). It can be switched to pcg withset_linsolve_algo(parcel::krylov::LINSOLVE_PCG)(when is symmetric definite; withset_precon_spectrum(true), pgmres is always used). - ks_arnoldi :
set_ks_subspace(m, k_keep, nev)(default 30, 10, 4;std::invalid_argumentunless ),set_ks_max_restart(R)(default 200), orthogonalizationset_qr_algo(algo)(parcel::qr::QR_CGSfor CGS2 (default),QR_MGSfor MGS2; “Default Orthogonalization of ks_arnoldi”). Eigenvalues near σ are computed by shift-invert. Callset_shift(σ)and pass as the fourth argument (the preconditioner of the inner linear solver is built from this matrix). If the fourth argument is omitted, the problem is solved from and the shift σ (then , andset_linsolve_precon_algo()is not used). The inner pgmres is controlled by the sameset_linsolve_*functions as inverse_power (the default maximum number of iterations is 2000) (the deprecated aliasesset_shift_linsolve_tolerance,set_shift_linsolve_max_iteration,set_shift_linsolve_restartcan also be used). When combined withset_precon_spectrum(true), the eigenvalues of near σ are returned and the fourth argument is not needed.
// shift-invert: register A - σI from separate arrays (because set_crsMat overwrites its arrays)
// copy the originals (col, val) before calling set_crsMat for A
std::vector<PARCEL_INT> col0 = col;
std::vector<double> val0 = val;
// ... A.set_crsMat(..., col.data(), val.data()) ... (col and val are overwritten)
std::vector<PARCEL_INT> col_s = col0;
std::vector<double> val_s = val0;
const double sigma = -13.0;
for(PARCEL_INT i = nstart; i <= nend; i++)
for(PARCEL_INT p = rowptr[i-nstart]; p < rowptr[i-nstart+1]; p++)
if(col_s[p] == i){ val_s[p] -= sigma; break; }
parcel::matrix<parcel_double> As(&q);
As.set_crsMat<double>(&comm, nstart, nend, n_local, col_s.size(),
rowptr.data(), col_s.data(), val_s.data());
ks.set_shift(sigma);
ks.solver<parcel_double,parcel_double,parcel_double>(A, eval, evec, &As);
if(!ks.is_converged() || ks.get_linsolve_failures() > 0){
/* outer iteration not converged, or inner pgmres not converged (revise set_linsolve_*) */
}For the eigenvalue solvers, too, check convergence with is_converged() and related functions. get_ks_nconv() of ks_arnoldi is the number of eigenpairs (out of nev) that satisfy the tolerance. How to check the inner linear solver of inverse_power and shift-invert (get_linsolve_failures()) is shown in “Checking the Convergence Status”.
5.1.7 Specifying Precision
Precision is specified with template arguments. The types are parcel_fp16, parcel_float, parcel_double, and parcel_doubledouble.
- Data precision : T of
parcel::matrix<T>,parcel::vector<T>, andsolver<Tmat, Tvecx, Tvecb, Tvec, ...>. - Arithmetic precision :
Tspmv,Taxpy,Tdotof the class. - Preconditioner :
Tpreconof the class (arithmetic precision) andTprecon_mat,Tprecon_vecofsolver(data precision).
The template arguments can be omitted, and an omitted precision becomes parcel_double. This applies to parcel::matrix<>, parcel::vector<>, the 4 arguments of the class, and the data precisions of solver() that cannot be deduced from the actual arguments (Tvec, Tprecon_mat, Tprecon_vec, TGS and THessenberg of pgmres, and Tbasis, TcalcQR, THessenberg of pcagmres) (for ks_arnoldi, Tprecon_mat and Tprecon_vec default to Teigen_vec). For example, parcel::krylov::pcg<> s; s.solver(A, x, b); solves everything in double precision, and the eigenvalue solvers can also be written as s.solver(A, ev, evec). The meaning of explicitly given arguments does not depend on whether others are omitted.
The verification sample (tests/verify/parcel_krylov_sample_verify.cpp) and the C / Fortran wrapper give these through the macros parcel_solver_data_type, parcel_solver_calc_type, parcel_solver_calc_precon_type, parcel_solver_data_precon_mat_type, parcel_solver_data_precon_vec_type (all default to parcel_double), which can be changed with -D. The restrictions at low precision and with double-double are given in “Known Limits at Low Precision” and “double-double”, and the reason -ffast-math cannot be used in “Do Not Use -ffast-math”.
5.1.8 MPI Decomposition
The matrix is partitioned into row blocks. Each rank owns contiguous rows nstart to nend (0-based, inclusive) and holds the CRS of those rows and vectors of length nend−nstart+1. The number of rows may differ between ranks (in the example above the last rank takes the remainder). The row blocks need not be in rank order, but together they must cover without gaps or overlaps, and every rank must own at least one row. set_crsMat() checks this (“Setting the Matrix”). The halo exchange tables are built automatically by set_crsMat() from the column indices (for how this works, see “GPUs, MPI Ranks and Data” in the “Basic Structure” chapter). Communication uses the MPI communicator passed to parcel::comm; communicators other than MPI_COMM_WORLD can also be specified.
5.1.9 Compiling and Running
Compile as described in the “Installation” chapter. By default all solvers are included, so lapacke.h is required even when only the linear solvers are used; specify -I/usr/include/openblas and -lopenblas (or the corresponding LAPACKE path and library). This is not needed if ks_arnoldi is excluded with -DPARCEL_ENABLE_KS_ARNOLDI=0 (“Selecting the Solvers to Build”). Solver selection macros such as -Dpcg_solver select the default solver of the verification sample and are not needed for programs that use the classes directly as above.
export MPICXX_CXX=acpp OMPI_CXX=acpp
mpicxx -O3 --acpp-targets=cuda:sm_70 -I/usr/include/openblas -o run main.cpp -lopenblas
mpirun -n 4 ./runIn a CPU build of AdaptiveCpp (--acpp-targets=omp), if you write your own nd_range kernel that uses item.barrier() in the same source file that includes PARCEL, exceptions thrown through std::function<void()> (such as std::invalid_argument from PARCEL’s setter functions) do not reach catch and the program stops with std::terminate. In your own kernels, use ::util_sycl::group_barrier(item) instead of item.barrier().
5.1.10 Run-Time Options of the Verification Sample
tests/verify/parcel_krylov_sample_verify.cpp is a sample that verifies the correctness of the solution by comparing it with a manufactured solution or with a reference solution from a dense direct solver (LAPACK dgesv). So that all solvers and preconditioners can be tried with a single build, the settings can be changed at run time with the environment variables below. The same settings can also be given as command-line arguments KEY=VALUE (mpirun -n 4 ./run PARCEL_NX=256 PARCEL_SOLVER=pcg). Use these in MPI environments where environment variables are not forwarded to multiple nodes. If both are given, the environment variable takes precedence.
| Option | Default | Description |
|---|---|---|
PARCEL_SOLVER |
-D<name>_solver at compile time; pcg if not specified |
pcg, pbicgstab, pgmres, pcagmres, power, inverse_power, ks_arnoldi. Specifying a solver excluded from the build terminates abnormally. cholqr_pcagmres is not accepted; the program prints the replacement setting (PARCEL_CAGMRES_Q=implicit) and terminates abnormally |
PARCEL_CAGMRES_Q |
auto |
How pcagmres forms the orthonormal basis (set_explicit_q()): auto, explicit, implicit. With implicit, the QR factorization is QR_CHOLESKYQR (CAGMRES_QR_ALGO is not used). Other values terminate abnormally |
PARCEL_CHOLQR_MOMENT |
0 | 1 makes the QR factorization the moment method (QR_CAGMRES_CHOLESKYQR). Usable only with PARCEL_CAGMRES_Q=implicit; solving with pcagmres otherwise terminates abnormally |
PARCEL_NX |
NX (240) |
Grid size per side. The number of unknowns is NX³ |
PARCEL_MATRIX |
poisson (convdiff with -DMATRIX_CONVDIFF) |
poisson (3D 7-point, symmetric), convdiff (convection-diffusion, nonsymmetric; convdiff2 is the same), unstruct (nonsymmetric matrix with 4 to 10 nonzeros per row) |
PARCEL_CONVDIFF_C |
CONVDIFF_C (0.05) |
Convection coefficient of the convection-diffusion matrix |
PARCEL_VERIFY |
1 | 0 skips the verification (for performance measurement; the right-hand side is b=1) |
PARCEL_TOL |
VERIFY_TOLERANCE (1.0e-9) |
Convergence tolerance passed to set_tolerance() |
PARCEL_MAX_ITER |
MAX_ITERATION (10000) |
Maximum number of iterations passed to set_max_iteration(). The default is the same as the library default |
PARCEL_PRECON |
parcel_solver_precon_algo (0) |
Preconditioner. 0: none, 1: point Jacobi, 2: block Jacobi, 4: Jacobi |
PARCEL_SCALING_MATRIX |
Automatic | 0/1. Explicitly specifies scaling of the preconditioner matrix (effective only with preconditioning) |
PARCEL_SCALING_VECTOR |
Automatic | 0/1. Explicitly specifies scaling of the preconditioner vector (same as above) |
PARCEL_BJ_BLOCK |
Automatic | Number of blocks of block Jacobi. BLOCKJACOBI_BLOCK is used only in builds with -DBLOCKJACOBI_BLOCK_EXPLICIT |
PARCEL_PRECON_SYMCHECK |
0 | If nonzero, checks the symmetry of the preconditioner |
PARCEL_KS_M |
KS_M (30) |
Maximum subspace dimension of ks_arnoldi |
PARCEL_KS_K_KEEP |
KS_K_KEEP (10) |
Number of Ritz pairs kept at each restart of ks_arnoldi |
PARCEL_KS_NEV |
KS_NEV (4) |
Number of eigenvalues subject to the convergence check in ks_arnoldi |
PARCEL_KS_RESTART |
KS_MAX_RESTART (50) |
Maximum number of restarts of ks_arnoldi. default does not call set_ks_max_restart() and uses the library default (200) |
PARCEL_KS_QR |
unset (library default) | Orthogonalization of ks_arnoldi. cgs calls set_qr_algo(QR_CGS) (CGS2), mgs calls set_qr_algo(QR_MGS) (MGS2). When unset, set_qr_algo() is not called and the build-time default (“Default Orthogonalization of ks_arnoldi”) is used. The method used is printed in the ks_qr_algo = ... line. Other strings terminate abnormally |
PARCEL_GMRES_RESTART |
GMRES_RESTART (8) |
Restart length of pgmres. default does not call set_restart() and uses the library default (30) |
PARCEL_CAGMRES_S / PARCEL_CAGMRES_T |
CAGMRES_RESTART_S (8) / CAGMRES_RESTART_T (1) |
and of pcagmres. PARCEL_CAGMRES_S=default does not call set_restart() and uses the library default (8, 4) (PARCEL_CAGMRES_T is then ignored) |
PARCEL_PRECON_SPECTRUM |
0 | 1 makes the eigenvalue solvers (power, inverse_power, ks_arnoldi) compute the eigenvalues of (set_precon_spectrum(true)) |
PARCEL_LINSOLVE_PRECON |
Library default (4) | Preconditioner of the inner linear solver of inverse_power and of the shift-invert of ks_arnoldi (set_linsolve_precon_algo()). Values are the same as PARCEL_PRECON. If not specified, the setter is not called and the library default (Jacobi preconditioner) is used. Not used with PARCEL_PRECON_SPECTRUM=1 |
PARCEL_LINSOLVE_SOLVER |
pgmres |
Inner linear solver of inverse_power. pgmres or pcg (set_linsolve_algo()) |
PARCEL_KS_SHIFT |
Disabled | Giving σ enables the shift-invert of ks_arnoldi at run time (only set_shift(σ); no shift matrix is passed). With PARCEL_PRECON_SPECTRUM=1 it computes the eigenvalues of near σ, otherwise those of ; in the latter case the inner pgmres has no preconditioner () and PARCEL_LINSOLVE_PRECON has no effect. Ignored in builds with -DKS_SHIFT_INVERT |
PARCEL_KS_SIGMA |
KS_SIGMA (0.0) |
Effective only in builds with -DKS_SHIFT_INVERT. σ of |
PARCEL_INNER_TOL |
INVPOW_LINSOLVE_TOL (1.0e-10) |
Convergence tolerance of the inner solver of inverse_power and of the inner pgmres of the shift-invert of ks_arnoldi |
PARCEL_INNER_MAX |
INVPOW_LINSOLVE_MAX (5000) |
Maximum number of iterations of the same |
PARCEL_PRECISION |
Precision at compile time | Effective only in builds with -DPARCEL_RUNTIME_PRECISION. double, calcF16, dataF16, precon_calcF16, calcF, dataF, precon_calcF, calcDD, dataDD |
PARCEL_LIB_DEFAULTS |
0 | 1: do not call the solver parameter setters (set_tolerance, set_max_iteration, set_restart, set_qr_algo, set_explicit_q, set_ks_subspace, set_ks_max_restart, set_linsolve_* of the inner solver, set_jacobi) with the sample’s values, and solve with the library defaults. Values given explicitly by environment variables are used as is, and the values actually used are printed in a [LIB_DEFAULTS] line. Used by the bundled performance tests |
PARCEL_OUTPUT |
parcel_result.txt |
File to which the standard output of rank 0 is duplicated. none disables writing |
PARCEL_MEM_REPORT |
Unset | If set, after the solver each rank prints in [MEM] lines its peak resident host memory (and the amount in huge pages (HugeTLB), HugetlbPages), the GPU device memory usage (read with nvidia-smi; NVIDIA GPUs only) and the peak of the device memory allocated by PARCEL, and rank 0 writes the values of all ranks in a [MEMG] line and to mem_gather.txt (determined by the setting on rank 0). It also enables the recording in the library (table below) |
The following solver settings are given at compile time with -D<macro>=<value> (defaults in tests/verify/parcel_default.hpp). Some differ from the library defaults. The restart lengths can be overridden at run time with PARCEL_GMRES_RESTART, PARCEL_CAGMRES_S and PARCEL_CAGMRES_T, and giving default as the value uses the library default (see the table above).
| Macro | Sample default | Library default | Description |
|---|---|---|---|
GMRES_RESTART |
8 | 30 | Restart length of pgmres (set_restart()) |
GMRES_QR_ALGO |
parcel::qr::QR_MGS |
QR_MGS |
Orthogonalization method of pgmres (set_qr_algo()) |
CAGMRES_RESTART_S |
8 | 8 | of pcagmres (set_restart(s, t)) |
CAGMRES_RESTART_T |
1 | 4 | of pcagmres |
CAGMRES_QR_ALGO |
parcel::qr::QR_MGS |
QR_MGS |
QR factorization method of pcagmres (set_qr_algo()). Not used with PARCEL_CAGMRES_Q=implicit |
The defaults of the sample are values used in the regression tests; GMRES_RESTART (8), CAGMRES_RESTART_T (1), KS_MAX_RESTART of ks_arnoldi (50) and others differ from the library defaults (30, 4, 200). In your own program, start from the library defaults without calling the setter functions, and change only what you need. To run the sample with the library defaults, add PARCEL_LIB_DEFAULTS=1.
The following environment variables are read by the library itself (Include/), so they are effective not only for the sample but for all languages (including via the C and Fortran wrappers).
| Environment variable | Default | Description |
|---|---|---|
PARCEL_NBLOCK |
Number of OpenMP threads | Number of partitions of the CPU kernels (also used for the automatic number of block Jacobi blocks) |
PARCEL_PSTEP |
1 (4 when built with -DPARCEL_PCG_NO_XHIST=0) |
Number of iterations over which pcg updates the solution vector at once. With the default 1, a single search direction is kept and the solution is updated at every iteration. With 4, a history of 4 search directions is kept, the reads and writes of the solution drop to 1/4 and pcg becomes slightly faster, at the cost of the memory of 3 vectors (optimization option “without G”; “Choosing Between Memory and Speed”). With 2 or more, the true residual is meaningless in between. The rounding changes because the solution is accumulated differently (same number of iterations) |
PARCEL_PRECON_VERBOSE |
Unset | If set, prints whether the automatically determined preconditioner scaling is enabled |
PARCEL_TIMER_SYNC |
Unset (0) | With 1, the device is synchronized before and after the timing of inner products and of the axpy that takes its coefficient by value. By default it is not, so the time of the preceding kernel is included in the inner-product time. Use it to get an accurate time breakdown (the total solver time becomes slightly longer) |
PARCEL_MEM_REPORT |
Unset | If set, the library counts the current amount and the peak of the device memory that PARCEL allocates with USM (read with util_sycl: / device_mem_peak_bytes(); allocations of the SYCL runtime itself are not included; with the CPU backend it is the part of the host memory used by the PARCEL arrays) |
5.1.11 Passing Your Own Matrix
The matrix passed to PARCEL is the CRS of the rows owned by each rank (the format is in “Data Structures” in “Basic Structure”). This section shows how to bring the matrix of your own problem into this form.
5.1.11.1 Building CRS from Triplets (COO)
Finite element codes and others often assemble a matrix as (row, column, value) triplets (COO format). The following function builds CRS from the triplets of the owned rows. It sorts the triplets by row and column and sums the values of repeated (row, column) pairs (accumulation of element contributions). The result has ascending column indices without duplicates in each row, as PARCEL requires.
#include <algorithm>
#include <vector>
#include "./Include/parcel_sycl.hpp" // PARCEL_INT
struct Triplet { PARCEL_INT row, col; double val; }; // row and column are global, 0-based
// Build CRS (rowptr, col, val) from the triplets t of the owned rows [nstart, nstart + n_local)
void coo_to_crs(PARCEL_INT nstart, PARCEL_INT n_local, std::vector<Triplet> t,
std::vector<PARCEL_INT>& rowptr, std::vector<PARCEL_INT>& col,
std::vector<double>& val)
{
std::sort(t.begin(), t.end(), [](const Triplet& a, const Triplet& b) {
return a.row != b.row ? a.row < b.row : a.col < b.col; });
rowptr.assign(n_local + 1, 0);
col.clear();
val.clear();
for (size_t k = 0; k < t.size(); ++k) {
if (k > 0 && t[k].row == t[k-1].row && t[k].col == t[k-1].col) {
val.back() += t[k].val; // sum repeated (row, column) pairs
continue;
}
col.push_back(t[k].col);
val.push_back(t[k].val);
rowptr[t[k].row - nstart + 1]++; // count the nonzeros of each row
}
for (PARCEL_INT i = 0; i < n_local; ++i) rowptr[i + 1] += rowptr[i];
}- Pass only the triplets of the owned rows (
nstart <= row < nstart + n_local). - If some rows may lack a diagonal entry, first add (i, i, 0.0) to every row (the diagonal is stored even when its value is 0).
- Even for a symmetric matrix, include the entries of both the upper and lower triangles.
5.1.11.2 Reading a Matrix Market File
The Matrix Market format (extension .mtx) is a text format for exchanging sparse matrices. The main points for reading it:
- A file in coordinate form (
coordinate) starts with a header such as%%MatrixMarket matrix coordinate real general, followed by comment lines starting with%, a line “rows columns nonzeros”, and then one “row column value” (1-based) per line. - If the header says
symmetric, the file contains only one triangle (usually the lower one). Add each off-diagonal entry also at (column, row) with the same value to obtain both triangles (with the sign reversed forskew-symmetric). - The
pattern(no values) andcomplexformats cannot be handled by PARCEL (PARCEL solves real matrices). - Subtract 1 from the row and column to make them 0-based, keep only the triplets of the owned rows, and convert them to CRS with
coo_to_crsabove.
5.1.11.3 Distributing a Global Matrix to the Ranks
With MPI, each rank passes only the CRS of its own rows to set_crsMat(). There are two ways to prepare it.
- Each rank builds only its own rows: if the matrix can be generated from formulas, as with finite differences, each rank generates only the range of its own rows. No rank holds the global matrix, so memory is smallest. The bundled examples (
examples/quick_start.cpp,examples/example_pcg_ks.cpp) take this form. - Take the owned rows from a global matrix: if a global CRS (
grp,gcol,gval, 0-based) is at hand, for example read from a file, the part of rows nstart to nend is a contiguous range of the arrays (fromgrp[nstart]up to just beforegrp[nend + 1]). The local row pointers aregrp[nstart + i] - grp[nstart], and the column indices are passed as global ones. Either every rank reads the same file and keeps only its rows, or one rank reads it and distributes it withMPI_Scatterv.
set_crsMat() overwrites the column-index and value arrays passed to it (“Setting the Matrix”), so copy the part taken from the global arrays into separate arrays before passing it.
// Take rows [nstart, nend] from a global CRS (0-based), copy them into separate arrays and register them
const PARCEL_INT off = grp[nstart], n_local = nend - nstart + 1;
std::vector<PARCEL_INT> rowptr(n_local + 1);
for (PARCEL_INT i = 0; i <= n_local; ++i) rowptr[i] = grp[nstart + i] - off;
std::vector<PARCEL_INT> col(gcol.begin() + off, gcol.begin() + grp[nend + 1]);
std::vector<double> val(gval.begin() + off, gval.begin() + grp[nend + 1]);
A.set_crsMat<double>(&comm, nstart, nend, n_local, col.size(),
rowptr.data(), col.data(), val.data());5.1.11.4 Choosing the Partition
The rows of each rank need only be contiguous, and the number of rows may differ between ranks (“MPI Decomposition”). Decide the partition considering the following.
- Most of the time of an iteration is spent in the SpMV and the preconditioner, whose cost is proportional to the number of nonzeros. In collective operations such as dot products all ranks wait for the slowest one, so for matrices whose number of nonzeros per row varies widely, balance the number of nonzeros rather than the number of rows. With the global row pointer
grp, let the first row of rank () be the first row at which the cumulative number of nonzeros reaches (leave at least one row to each rank). - For matrices with nearly the same number of nonzeros per row (such as finite-difference grids), dividing the rows evenly is enough.
- The amount of halo communication is determined by the number of nonzeros that refer to columns of other ranks. For grid problems, numbering nearby grid points with nearby row numbers (a careful choice of numbering) reduces the halo.
- The memory of each rank is also roughly proportional to the number of nonzeros of its rows. If GPU memory is insufficient, increase the number of ranks (GPUs) to reduce the rows per rank.
5.2 C
5.2.1 Overview
From C, include capi/parcel_sycl_c.h and call the functions below. They are implemented in capi/wrapper_parcel_sycl.cpp, which calls the C++ library internally; all 7 solvers can be selected at run time.
| Function | Description |
|---|---|
parcel_options_init(&opt) |
Fills the option structure parcel_options with default values |
parcel_solve_linear(solver, &opt, comm, ...) |
Solves the linear system (pcg, pbicgstab, pgmres, pcagmres) |
parcel_solve_eigen(solver, &opt, comm, ...) |
Solves the eigenvalue problem ( when precon_spectrum=1) (power, inverse_power, ks_arnoldi) |
parcel_eigen_count(solver, &opt) |
Number k of eigenpairs returned by the eigenvalue solver |
parcel_solver_from_name(name) |
Returns PARCEL_SOLVER_* for a solver name ("pcg", "ks_arnoldi", etc.); -1 if the name is unknown |
parcel_options_size(), parcel_result_size() |
Return sizeof(parcel_options) and sizeof(parcel_result) (for checking against the Fortran derived types) |
run_parcel_sycl_c(...) |
Legacy API (compatibility with v1.0; pcg only) |
The solver is specified with constants such as PARCEL_SOLVER_PCG. The specifications of the constants, the structures, and the arguments and return values of each function are given in “C API” in the API Reference. Row and column indices, row counts and numbers of nonzeros are of the 64-bit type parcel_int.
The calling procedure is as follows.
- In a build with MPI, call
MPI_Init()first (required even with one process; not needed in a build without MPI). - Fill in the default values with
parcel_options opt; parcel_options_init(&opt);and change only the fields you need. The structure must not be used without initialization. - Prepare the CRS arrays and vectors for the rows owned by each rank.
- Call
parcel_solve_linear()orparcel_solve_eigen()on all ranks of the communicator and check the return value. - Check whether the iteration converged using
converged_reasonof theparcel_resultpassed as the last argument.
Each call creates a SYCL queue (an in-order queue on the default device) and registers the matrix. The device is selected with environment variables of the SYCL implementation, such as ACPP_VISIBILITY_MASK. With opt.verbose = 1, rank 0 prints the device name, the solver, the preconditioner and the convergence tolerance.
Settings are given in the fields of the structure parcel_options (all fields and defaults: parcel_options in the Reference). The defaults set by parcel_options_init() are the library defaults. The functions check the arguments and report errors through return values (PARCEL_OK or a negative error code). The result of the argument check is the same on all ranks. If an error occurs inside the library (running out of device memory, etc.), all ranks are terminated with MPI_Abort in a build with MPI. The return value does not indicate whether the iteration converged, so check converged_reason (positive means converged) of the parcel_result passed as the last argument (return values and convergence status constants, parcel_result).
The bundled sample_c.c is a program that partitions the rows of a 3-D 7-point Poisson matrix (center -6, neighbors 1, rows) unevenly, calls all seven solvers (for pcagmres, in addition to the default method that forms , the method that does not form (cagmres_q_mode = PARCEL_CAGMRES_Q_IMPLICIT) in two variants, with CholeskyQR and with the moment method), ks_arnoldi with shift-invert and the legacy API, compares the results with the relative residual or the analytical eigenvalues, and prints PASS / FAIL. Parts of it are shown in the subsections below.
5.2.2 Setting the Matrix
Each rank passes a row block with contiguous global row indices [nstart, nstart + n_local - 1] in CRS format. The meanings of the arguments (n_local, nstart, n_global, nnz_local, rowptr, col, val) are given in parcel_solve_linear() in the Reference. nstart and col are numbered according to index_base, and col holds global column indices. Only differences from rowptr[0] are used for the row pointer.
The integers are of type parcel_int (defined as int64_t in parcel_sycl_c.h; 64 bits). They are 64 bits regardless of the library’s internal PARCEL_INT, and the row pointer and column index arrays must also be allocated as parcel_int (an int array cannot be passed). If the wrapper is built with -DPARCEL_INT32 (“Integer Type” in “Installation”) and a value does not fit in 32 bits, PARCEL_ERR_ARG is returned on all ranks before the arrays are allocated. The fields of parcel_options and parcel_result and the return value of parcel_eigen_count() are int. Build the wrapper, parcel_sycl_mod.f90 and your program with the same version of parcel_sycl_c.h. The wrapper copies the input arrays before registering them, so col and val are not modified. Another solver may be called subsequently with the same arrays. At the call, the wrapper checks that the row pointer is non-decreasing with rowptr[n_local] - rowptr[0] == nnz_local, and that the column indices and the row range lie within ; otherwise it returns PARCEL_ERR_ARG. In each row, the column indices must be in ascending order without duplicates, the diagonal element must be present (even if its value is 0), and both triangles must be stored even for a symmetric matrix (“Data Structures” in “Basic Structure”).
/* sample_c.c: build the CRS of global rows [rstart, rstart+nrow) with 0-based indices (only count if col == NULL) */
static parcel_int poisson_rows(parcel_int nx, parcel_int rstart, parcel_int nrow,
parcel_int *rowptr, parcel_int *col, double *val){
parcel_int nnz = 0;
rowptr[0] = 0;
for(parcel_int r = 0; r < nrow; r++){
const parcel_int g = rstart + r;
const parcel_int ix = g % nx, iy = (g / nx) % nx, iz = g / (nx * nx);
#define PUT(c, v) do{ if(col){ col[nnz] = (c); val[nnz] = (v); } nnz++; }while(0)
if(iz > 0) PUT(g - nx * nx, 1.0);
if(iy > 0) PUT(g - nx, 1.0);
if(ix > 0) PUT(g - 1, 1.0);
PUT(g, -6.0);
if(ix < nx - 1) PUT(g + 1, 1.0);
if(iy < nx - 1) PUT(g + nx, 1.0);
if(iz < nx - 1) PUT(g + nx * nx, 1.0);
#undef PUT
rowptr[r + 1] = nnz;
}
return nnz;
}5.2.3 Setting the Vectors
All vectors are the local parts for the locally owned rows and are passed as double arrays. Transfer to the device and copy-back are done inside the wrapper.
- Linear solvers:
b(lengthn_local) is the right-hand side;x(lengthn_local) is the initial guess on input and the solution on output. - Eigenvalue solvers: with k =
parcel_eigen_count(solver, &opt), allocate2*kelements foreigvalandn_local*kelements foreigvec. k is 1 for power / inverse_power andks_k_keepfor ks_arnoldi (of which only the firstks_nevare guaranteed to have converged).eigval: real and imaginary parts are stored alternately (eigval[2*j]is the real part of the j-th eigenvalue andeigval[2*j+1]its imaginary part). For power / inverse_power,eigval[0]holds the eigenvalue andeigval[1]holds 0.eigvec: column-major; then_localelements starting ateigvec + j*n_localare the locally owned part of the j-th eigenvector. For a complex-conjugate pair of eigenvalues, the corresponding eigenvectors are stored as two consecutive vectors, the real and imaginary parts.- Initial vector: power / inverse_power generate it internally. For ks_arnoldi, by default (
eigvec_seed=0) the wrapper generates it from the global row indices, so the result does not depend on how the rows are partitioned. Witheigvec_seed=1, the firstn_localelements ofeigvecare used as the initial vector.
5.2.4 Setting the Preconditioner
The preconditioner is set at run time through the fields of parcel_options.
| Field | Description (default) |
|---|---|
precon_algo |
Kind of preconditioner: PARCEL_PRECON_NONE (default), PARCEL_PRECON_POINTJACOBI, PARCEL_PRECON_BLOCKJACOBI, PARCEL_PRECON_JACOBI |
jacobi_iter |
Number of iterations of the Jacobi preconditioner (1); used only for the Jacobi preconditioner |
blockjacobi |
Number of blocks of the block Jacobi preconditioner (0 = automatic) |
scaling_matrix, scaling_vector |
Scaling of the preconditioner matrix and vector: -1 (default) automatic, 0 off, 1 on |
linsolve_precon_algo, linsolve_jacobi_iter, linsolve_blockjacobi |
Preconditioner of the internal linear solver of inverse_power and ks_arnoldi shift-invert (default: Jacobi preconditioner with 1 iteration) |
For the eigenvalue solvers, settings such as precon_algo are used only for in when precon_spectrum=1. The preconditioner of the internal linear solver is set with linsolve_precon_* (not used when precon_spectrum=1, where the internal solver is also preconditioned with ).
5.2.5 Calling a Linear Solver
Give one of PARCEL_SOLVER_PCG, _PBICGSTAB, _PGMRES, _PCAGMRES as solver of parcel_solve_linear(). The fields used in common are tolerance (tolerance, default 1.0e-8), max_iteration (maximum number of iterations, default 10000) and the preconditioner fields; pgmres also uses gmres_restart (default 30) and gmres_qr_algo, and pcagmres uses cagmres_s, cagmres_t (defaults 8, 4), cagmres_qr_algo and cagmres_q_mode (Reference). The iteration history and timings are printed to standard output of rank 0. The convergence status is in res.converged_reason, the number of iterations in res.iterations, and the residuals in res.residual and res.true_residual.
/* From sample_c.c: call the linear solvers with the block Jacobi preconditioner
(the last two are pcagmres without forming Q; they differ only in the choice of QR factorization) */
const int lin_solvers[6] = { PARCEL_SOLVER_PCG, PARCEL_SOLVER_PBICGSTAB,
PARCEL_SOLVER_PGMRES, PARCEL_SOLVER_PCAGMRES,
PARCEL_SOLVER_PCAGMRES, PARCEL_SOLVER_PCAGMRES };
for(int s = 0; s < 6; s++){
parcel_options opt;
parcel_options_init(&opt);
opt.tolerance = 1.0e-9;
opt.max_iteration = 10000;
opt.precon_algo = PARCEL_PRECON_BLOCKJACOBI;
opt.gmres_restart = 8; /* default 30 */
opt.gmres_qr_algo = PARCEL_QR_MGS;
opt.cagmres_s = 8;
opt.cagmres_t = 1; /* default 4 */
opt.cagmres_qr_algo = PARCEL_QR_MGS; /* same as the default */
if(s == 4){
/* without forming Q: CholeskyQR. t = 2 also exercises the orthogonalization between blocks */
opt.cagmres_t = 2;
opt.cagmres_q_mode = PARCEL_CAGMRES_Q_IMPLICIT;
opt.cagmres_qr_algo = PARCEL_QR_CHOLESKYQR;
}else if(s == 5){
/* moment method: A symmetric (holds here), no preconditioner, s even */
opt.precon_algo = PARCEL_PRECON_NONE;
opt.cagmres_t = 2;
opt.cagmres_q_mode = PARCEL_CAGMRES_Q_IMPLICIT;
opt.cagmres_qr_algo = PARCEL_QR_CAGMRES_CHOLESKYQR;
}
for(parcel_int i = 0; i < n_local; i++) x[i] = 0.0;
parcel_result res;
const int rc = parcel_solve_linear(lin_solvers[s], &opt, MPI_COMM_WORLD,
n_local, nstart, n_global, nnz_local,
rowptr, col, val, b, x, &res);
if(rc != PARCEL_OK){ /* argument error, etc. */ }
else if(res.converged_reason <= 0){ /* not converged (PARCEL_DIVERGED_*) */
printf("reason = %d, its = %d, residual = %.3e, true residual = %.3e\n",
res.converged_reason, res.iterations, res.residual, res.true_residual);
}
}The legacy API run_parcel_sycl_c(), the C API of v1.0, is also kept for compatibility (pcg only; the settings are determined by compile-time macros of the wrapper; Reference). For new programs, use parcel_solve_linear().
gmres_restart = 8 and cagmres_t = 1 in this example are the values of the verification sample, not the library defaults (30, 4). In your own program, start from the defaults set by parcel_options_init() and change only the fields you need.
5.2.6 Calling an Eigenvalue Solver
Give one of PARCEL_SOLVER_POWER, _INVERSE_POWER, _KS_ARNOLDI as solver of parcel_solve_eigen(). Obtain the number k of returned eigenpairs with parcel_eigen_count() (for ks_arnoldi, the subspace sizes ks_m, ks_k_keep, ks_nev default to 30, 10, 4, the maximum number of restarts ks_max_restart to 200, and the orthogonalization ks_qr_algo is CGS2 with PARCEL_QR_CGS and MGS2 with PARCEL_QR_MGS), and allocate eigval (2*k elements; real and imaginary parts alternating) and eigvec (n_local*k elements; column-major). The eigenvalues sought, the fields used, and the handling of shift-invert (shift_enable, shift_sigma; ks_arnoldi only) and precon_spectrum are given in parcel_solve_eigen() in the Reference. For shift-invert with precon_spectrum=0, the wrapper assembles , so the user does not need to prepare it.
/* From sample_c.c: ks_arnoldi with shift-invert (4 eigenvalues near σ = -0.15) */
parcel_options opt;
parcel_options_init(&opt);
opt.tolerance = 1.0e-8; /* Ritz residual */
opt.ks_m = 30;
opt.ks_k_keep = 10;
opt.ks_nev = 4;
opt.ks_max_restart = 100;
opt.shift_enable = 1; /* precon_spectrum=0: the wrapper assembles (A - σI) */
opt.shift_sigma = -0.15;
opt.linsolve_tolerance = 1.0e-11;
opt.linsolve_max_iteration = 5000; /* default (0) gives 2000 for ks_arnoldi */
opt.linsolve_restart = 30;
opt.linsolve_precon_algo = PARCEL_PRECON_BLOCKJACOBI;
const int k = parcel_eigen_count(PARCEL_SOLVER_KS_ARNOLDI, &opt); /* = ks_k_keep */
double *ev = (double *)malloc(sizeof(double) * 2 * k); /* Re, Im alternating */
double *v = (double *)malloc(sizeof(double) * (size_t)n_local * k); /* column-major */
parcel_result res;
const int rc = parcel_solve_eigen(PARCEL_SOLVER_KS_ARNOLDI, &opt, MPI_COMM_WORLD,
n_local, nstart, n_global, nnz_local,
rowptr, col, val, ev, v, &res);
if(rc == PARCEL_OK && res.converged_reason <= 0){
/* not converged: only res.ks_nconv eigenpairs satisfied the tolerance. Also check res.linsolve_failures */
}
for(int j = 0; j < opt.ks_nev; j++){
const double re = ev[2 * j], im = ev[2 * j + 1];
const double *vj = v + (size_t)j * n_local; /* j-th eigenvector (locally owned rows) */
/* ... */
}For power and inverse_power, it is sufficient to pass double ev[2] and v of length n_local.
5.2.7 Specifying Precision
The input/output arrays are fixed to double. The data precision and arithmetic precision inside PARCEL are determined by macros given when compiling capi/wrapper_parcel_sycl.cpp (defaults in capi/parcel_capi_default.hpp), and cannot be changed at run time or per call. The macro names are the same as for the verification sample, and all default to parcel_double.
| Macro | Purpose |
|---|---|
parcel_solver_data_type |
Data precision of the matrix and vectors |
parcel_solver_calc_type |
Arithmetic precision |
parcel_solver_calc_precon_type |
Arithmetic precision of the preconditioner |
parcel_solver_data_precon_mat_type |
Data precision of the preconditioner matrix |
parcel_solver_data_precon_vec_type |
Data precision of the preconditioner vector |
parcel_solver_calc_qr_type |
Precision of the QR factorization in pcagmres |
For example, to perform the arithmetic in single precision, add -Dparcel_solver_calc_type=parcel_float when compiling the wrapper. The usable combinations of precisions are in “Precision Configurations” in “Basic Structure”, and the restrictions at low precision in “Known Limits at Low Precision”.
5.2.8 MPI Decomposition
Pass an MPI communicator (MPI_Comm) as the third argument comm. Communicators other than MPI_COMM_WORLD can also be specified. Because the functions contain collective operations, call them on all ranks of the communicator with the same solver and options of the same content. The handling of the communicator in a build without MPI is described in “Build without MPI”.
The row block of each rank only needs to consist of contiguous rows, and the number of rows may differ between ranks. The row blocks need not be ordered by rank number, but together the row blocks of all ranks must cover without overlaps or gaps. The wrapper gathers (nstart, n_local) of every rank with MPI_Allgather to check this and, if the condition is not met, returns PARCEL_ERR_DISTRIBUTION on all ranks. n_local of each rank must be 1 or greater.
/* From sample_c.c: uneven partition that assigns rows to rank r with weight (r+1) */
const long long W = (long long)np * (np + 1) / 2;
long long acc = 0;
for(int r = 0; r < np; r++){
const int s = (int)((long long)n_global * acc / W);
acc += r + 1;
const int e = (int)((long long)n_global * acc / W);
displs[r] = s; counts[r] = e - s;
}
const parcel_int nstart = displs[myrank];
const parcel_int n_local = counts[myrank];5.2.9 Compiling and Running
Compile the wrapper capi/wrapper_parcel_sycl.cpp with a SYCL compiler and link it with the C main program using the SYCL compiler. The manual steps, with and without MPI, are in “C” of the Quick Start, and the flags for each environment in “C and Fortran Programs” in “Installation”. examples/build_examples.sh builds the samples in examples/ (sample_c, sample_f, etc.) with the same steps (outputs in examples/build/; the compilers, target and LAPACK are changed with environment variables).
export OMPI_CXX=acpp
bash examples/build_examples.sh # with MPI (without MPI: PARCEL_MPI=0 bash examples/build_examples.sh)
mpirun -n 3 examples/build/sample_c 16 # argument is NX (default 16)The iteration limit of the legacy API run_parcel_sycl_c is decided by the wrapper macro MAX_ITERATION (default 10000) and does not affect the new API. To change the internal precision, add the precision macros to the wrapper compilation (examples/build_examples.sh takes them from the environment variable WRAPPER_FLAGS). If the solvers to build are restricted with PARCEL_ENABLE_*, specifying an excluded solver returns PARCEL_ERR_NOT_BUILT.
sample_c prints the result of each solver on one line as PASS / FAIL (12 items in total: 7 solvers, the two variants of pcagmres without forming (pcagmres(implicit), pcagmres(impl,sym)), ks_arnoldi with shift-invert, the legacy API, and, first, the version check (version)), and returns exit code 1 if any item fails.
5.2.10 Passing Your Own Matrix
The matrix passed to PARCEL is the CRS of the rows owned by each rank (the format is in “Data Structures” in “Basic Structure”). This section shows how to bring the matrix of your own problem into this form.
5.2.10.1 Building CRS from Triplets (COO)
Finite element codes and others often assemble a matrix as (row, column, value) triplets (COO format). The following function builds CRS from the triplets of the owned rows. It sorts the triplets by row and column and sums the values of repeated (row, column) pairs. The result has ascending column indices without duplicates in each row, as PARCEL requires.
#include <stdlib.h>
#include "parcel_sycl_c.h" /* parcel_int */
typedef struct { parcel_int row, col; double val; } triplet; /* row and column are global, 0-based */
static int cmp_triplet(const void *pa, const void *pb){
const triplet *a = (const triplet *)pa, *b = (const triplet *)pb;
if(a->row != b->row) return a->row < b->row ? -1 : 1;
if(a->col != b->col) return a->col < b->col ? -1 : 1;
return 0;
}
/* Build CRS (rowptr, col, val) from the triplets t[0..nt) of the owned rows [nstart, nstart + n_local) and return the number of nonzeros.
Allocate n_local + 1 elements for rowptr and nt elements for col and val. t is sorted in place */
parcel_int coo_to_crs(parcel_int nstart, parcel_int n_local, triplet *t, size_t nt,
parcel_int *rowptr, parcel_int *col, double *val){
qsort(t, nt, sizeof(triplet), cmp_triplet);
for(parcel_int i = 0; i <= n_local; i++) rowptr[i] = 0;
parcel_int nnz = 0;
for(size_t k = 0; k < nt; k++){
if(k > 0 && t[k].row == t[k-1].row && t[k].col == t[k-1].col){
val[nnz - 1] += t[k].val; /* sum repeated (row, column) pairs */
continue;
}
col[nnz] = t[k].col;
val[nnz] = t[k].val;
nnz++;
rowptr[t[k].row - nstart + 1]++; /* count the nonzeros of each row */
}
for(parcel_int i = 0; i < n_local; i++) rowptr[i + 1] += rowptr[i];
return nnz;
}- Pass only the triplets of the owned rows (
nstart <= row < nstart + n_local), and pass the returned number of nonzeros asnnz_local. - If some rows may lack a diagonal entry, first add (i, i, 0.0) to every row (the diagonal is stored even when its value is 0).
- Even for a symmetric matrix, include the entries of both the upper and lower triangles.
5.2.10.2 Reading a Matrix Market File
The Matrix Market format (extension .mtx) is a text format for exchanging sparse matrices. The main points for reading it with fgets, sscanf, etc.:
- A file in coordinate form (
coordinate) starts with a header such as%%MatrixMarket matrix coordinate real general, followed by comment lines starting with%, a line “rows columns nonzeros”, and then one “row column value” (1-based) per line. - If the header says
symmetric, the file contains only one triangle (usually the lower one). Add each off-diagonal entry also at (column, row) with the same value to obtain both triangles (with the sign reversed forskew-symmetric). - The
pattern(no values) andcomplexformats cannot be handled by PARCEL (PARCEL solves real matrices). - Pass the row and column 1-based with
opt.index_base = 1, or subtract 1 to make them 0-based. Keep only the triplets of the owned rows and convert them to CRS withcoo_to_crsabove.
5.2.10.3 Distributing a Global Matrix to the Ranks
With MPI, each rank passes only the CRS of its own rows. There are two ways to prepare it.
- Each rank builds only its own rows: if the matrix can be generated from formulas, as with finite differences, each rank generates only the range of its own rows. No rank holds the global matrix, so memory is smallest. The bundled examples (
examples/quick_start_c.c,poisson_rowsinexamples/sample_c.c) take this form. - Take the owned rows from a global matrix: if a global CRS (
grp,gcol,gval, 0-based) is at hand, for example read from a file, the part of rows [nstart,nstart + n_local) is a contiguous range of the arrays. Only differences fromrowptr[0]are used and the wrapper does not modify the input arrays, so the middle of the global arrays can be passed as is. Either every rank reads the same file, or one rank reads it and distributes it withMPI_Scatterv.
/* Pass rows [nstart, nstart + n_local) taken from a global CRS */
parcel_int off = grp[nstart];
parcel_int nnz_local = grp[nstart + n_local] - off;
rc = parcel_solve_linear(PARCEL_SOLVER_PCG, &opt, MPI_COMM_WORLD, n_local, nstart, n_global,
nnz_local, &grp[nstart], &gcol[off], &gval[off], b, x, &res);5.2.10.4 Choosing the Partition
The rows of each rank need only be contiguous, and the number of rows may differ between ranks (“MPI Decomposition”). Decide the partition considering the following.
- Most of the time of an iteration is spent in the SpMV and the preconditioner, whose cost is proportional to the number of nonzeros. In collective operations such as dot products all ranks wait for the slowest one, so for matrices whose number of nonzeros per row varies widely, balance the number of nonzeros rather than the number of rows. With the global row pointer
grp, let the first row of rank () be the first row at which the cumulative number of nonzeros reaches (leave at least one row to each rank). - For matrices with nearly the same number of nonzeros per row (such as finite-difference grids), dividing the rows evenly is enough.
- The amount of halo communication is determined by the number of nonzeros that refer to columns of other ranks. For grid problems, numbering nearby grid points with nearby row numbers reduces the halo.
- The memory of each rank is also roughly proportional to the number of nonzeros of its rows. If GPU memory is insufficient, increase the number of ranks (GPUs) to reduce the rows per rank.
5.3 Fortran
5.3.1 Overview
From Fortran, use the module parcel_sycl_mod (capi/parcel_sycl_mod.f90) and call the procedures below. They are implemented by the wrapper capi/wrapper_parcel_sycl.cpp, which calls the C++ library; all 7 solvers can be selected at run time.
| Procedure | Description |
|---|---|
call parcel_options_init(opt) |
Fills the settings derived type parcel_options with the defaults |
rc = parcel_solve_linear(solver, opt, comm, ...) |
Solves the linear system (pcg, pbicgstab, pgmres, pcagmres) |
rc = parcel_solve_eigen(solver, opt, comm, ...) |
Solves the eigenvalue problem ( with precon_spectrum=1) (power, inverse_power, ks_arnoldi) |
k = parcel_eigen_count(solver, opt) |
Number k of eigenpairs returned by an eigenvalue solver |
parcel_options_check(), parcel_result_check() |
Whether the sizes of the derived types match the structures of the wrapper (logical) |
run_parcel_sycl_f(...) |
Legacy API (compatibility with v1.0; pcg only) |
The solver is specified with constants such as PARCEL_SOLVER_PCG. The constants, the fields of the derived types, and the arguments and return values of each procedure are specified in “Fortran API” in the API Reference. Row and column indices, numbers of rows and numbers of nonzeros are passed as 64-bit integers integer(parcel_int_kind).
The calling sequence is as follows.
- In a build with MPI, call
MPI_Initfirst (also when running with one process; not needed in a build without MPI). - Check with
parcel_options_check()andparcel_result_check()that the sizes of the derived types match the structures of the wrapper (if not, the versions of the wrapper and the module differ). - Fill the defaults with
call parcel_options_init(opt)and change only the needed fields, as inopt%tolerance. Do not use it uninitialized. - Prepare the CRS arrays and vectors of the rows owned by each rank.
- Call
parcel_solve_linearorparcel_solve_eigenon all ranks of the communicator and check that the return value isPARCEL_OK. - Check with
res%converged_reasonof theparcel_resultpassed as the last argument whether the iteration converged (positive means converged).resmay be omitted.
Each call creates a SYCL queue (an in-order queue on the default device) and registers the matrix. The device is selected with environment variables of the SYCL implementation such as ACPP_VISIBILITY_MASK. With opt%verbose = 1, rank 0 prints the device name, the solver, the preconditioner and the tolerance.
The defaults set by parcel_options_init are the library defaults (all fields and defaults: parcel_options in the Reference). The procedures check the arguments and report errors through return values (PARCEL_OK or a negative error code). The result of the argument check is the same on all ranks. If an error occurs inside the library (running out of device memory, etc.), all ranks are terminated with MPI_Abort in a build with MPI. The return value does not indicate whether the iteration converged, so check res%converged_reason.
! From sample_f.f90
program sample_f
use mpi
use iso_c_binding
use parcel_sycl_mod
implicit none
type(parcel_options) :: opt
type(parcel_result) :: res
integer :: rc, ierr
integer(parcel_int_kind) :: n_local, nstart, n_global, nnz_local ! 64 bits
integer(parcel_int_kind), allocatable :: rowptr(:), col(:)
! ...
call MPI_Init(ierr)
! Check the sizes of the Fortran types against the C structs
if(.not. parcel_options_check() .or. .not. parcel_result_check()) then
call MPI_Abort(MPI_COMM_WORLD, 2, ierr)
end if
! ...
call parcel_options_init(opt)
opt%index_base = 1 ! pass nstart / col 1-based
opt%tolerance = 1.0d-9
opt%max_iteration = 10000
opt%precon_algo = PARCEL_PRECON_BLOCKJACOBI
opt%gmres_restart = 8 ! default 30
rc = parcel_solve_linear(PARCEL_SOLVER_PGMRES, opt, MPI_COMM_WORLD, n_local, nstart, n_global, &
nnz_local, rowptr, col, val, b, x, res)
if(rc /= PARCEL_OK) then
! argument error, etc.
else if(res%converged_reason <= 0) then
! not converged (PARCEL_DIVERGED_*). Inspect res%iterations, res%residual, res%true_residual
end ifThe bundled sample_f.f90 partitions the 3D 7-point Poisson matrix (center -6, neighbors 1, rows) unevenly, calls all 7 solvers, ks_arnoldi with shift-invert and the legacy API, and prints PASS / FAIL by comparing the results with the relative residual or the analytic eigenvalues. Parts of it are shown in the following sections.
5.3.2 Setting the Matrix
Each rank passes a row block with contiguous global row numbers (n_local rows starting at nstart) in CRS format. The arguments are as follows (details: parcel_solve_linear in the Reference).
| Argument | Description |
|---|---|
n_local |
Number of rows owned by this rank (1 or more) |
nstart |
Global row number of the first owned row (numbered with index_base) |
n_global |
Number of rows of the whole matrix |
nnz_local |
Number of nonzeros of the owned rows |
rowptr(n_local+1) |
Row pointer; only differences from rowptr(1) are used |
col(nnz_local) |
Global column indices (numbered with index_base) |
val(nnz_local) |
Matrix elements |
The integer arguments and the integer arrays (rowptr, col) are integer(parcel_int_kind) (= integer(c_int64_t), 64 bits), and matrix elements are real(c_double) (real(8)). Passing a default integer variable for a value argument does not match the interface and causes a compile error, so declare the variable with integer(parcel_int_kind) or convert it with int(n, parcel_int_kind) when passing it. With opt%index_base = 1, nstart and the column indices col can be passed 1-based in the Fortran convention. Only differences from the first element of the row pointer rowptr are used, so it can remain 1-based with rowptr(1) = 1.
The wrapper copies the input arrays before registering them, so col and val are not modified, and another solver may be called subsequently with the same arrays. At the call, the wrapper checks that the row pointer is non-decreasing with rowptr(n_local+1) - rowptr(1) equal to nnz_local, and that the column indices and the row range lie within the matrix; otherwise it returns PARCEL_ERR_ARG. In each row, the column indices must be in ascending order without duplicates, the diagonal element must be present (even if its value is 0), and both triangles must be stored even for a symmetric matrix (“Data Structures” in “Basic Structure”). If the wrapper was built with -DPARCEL_INT32 (“Integer Type” in “Installation”) and a value does not fit in 32 bits, PARCEL_ERR_ARG is returned on all ranks.
! From sample_f.f90: build the CRS of global rows [rstart, rstart+nrow-1] (1-based) with 1-based indices
! rstart, nrow, r, g, ix, iy, iz, nnz, rowptr(:), col(:) are integer(parcel_int_kind)
nnz = 0
rowptr(1) = 1
do r = 1, nrow
g = rstart + r - 1 ! 1-based global row
ix = mod(g-1, nx); iy = mod((g-1)/nx, nx); iz = (g-1)/(nx*nx)
if(iz > 0) call put(g - nx*nx, 1.0d0)
if(iy > 0) call put(g - nx, 1.0d0)
if(ix > 0) call put(g - 1, 1.0d0)
call put(g, -6.0d0)
if(ix < nx - 1) call put(g + 1, 1.0d0)
if(iy < nx - 1) call put(g + nx, 1.0d0)
if(iz < nx - 1) call put(g + nx*nx, 1.0d0)
rowptr(r+1) = nnz + 1
end do5.3.3 Setting the Vectors
All vectors are the locally owned parts and are passed as real(8) arrays. Transfers to and from the device are done inside the wrapper.
- Linear solvers:
b(lengthn_local) is the right-hand side, andx(lengthn_local) is the initial guess on input and the solution on output. - Eigenvalue solvers: with k =
parcel_eigen_count(solver, opt), allocateeigval(2*k)andeigvec(n_local*k). k is 1 for power / inverse_power andopt%ks_k_keepfor ks_arnoldi (of which convergence is guaranteed only for the firstopt%ks_nev).- The real part of the j-th (1-based) eigenvalue is
eigval(2*j-1)and the imaginary part iseigval(2*j). Even for power / inverse_power,eigvalneeds 2 elements (the imaginary part is 0). - The owned part of the j-th eigenvector is
eigvec((j-1)*n_local+1 : j*n_local)(column j if declared aseigvec(n_local, k)). For a complex-conjugate pair of eigenvalues, the corresponding eigenvectors are stored as two consecutive vectors, the real and imaginary parts. - Initial vector: power / inverse_power generate it internally. For ks_arnoldi, by default (
opt%eigvec_seed = 0) the wrapper builds it from the global row numbers, so the results do not depend on the partitioning. Withopt%eigvec_seed = 1, the firstn_localelements ofeigvecare used as the initial vector.
- The real part of the j-th (1-based) eigenvalue is
5.3.4 Setting the Preconditioner
The preconditioner is set at run time through the fields of parcel_options.
| Field | Description (default) |
|---|---|
opt%precon_algo |
Kind of preconditioner: PARCEL_PRECON_NONE (default), PARCEL_PRECON_POINTJACOBI, PARCEL_PRECON_BLOCKJACOBI, PARCEL_PRECON_JACOBI |
opt%jacobi_iter |
Number of iterations of the Jacobi preconditioner (1); used only for the Jacobi preconditioner |
opt%blockjacobi |
Number of blocks of the block Jacobi preconditioner (0 = automatic) |
opt%scaling_matrix, opt%scaling_vector |
Scaling of the preconditioner matrix and vector: -1 (default) automatic, 0 off, 1 on |
opt%linsolve_precon_algo, opt%linsolve_jacobi_iter, opt%linsolve_blockjacobi |
Preconditioner of the internal linear solver of inverse_power and ks_arnoldi shift-invert (default: Jacobi preconditioner with 1 iteration) |
For the eigenvalue solvers, settings such as opt%precon_algo are used only for in when opt%precon_spectrum = 1. The preconditioner of the internal linear solver is set with opt%linsolve_precon_* (not used when opt%precon_spectrum = 1, where the internal solver is also preconditioned by ).
5.3.5 Calling a Linear Solver
parcel_solve_linear is a function returning an integer; give one of PARCEL_SOLVER_PCG, PARCEL_SOLVER_PBICGSTAB, PARCEL_SOLVER_PGMRES, PARCEL_SOLVER_PCAGMRES as solver. The fields used in common are opt%tolerance (tolerance, default 1.0e-8), opt%max_iteration (maximum number of iterations, default 10000) and the preconditioner fields; pgmres also uses opt%gmres_restart (default 30) and opt%gmres_qr_algo, and pcagmres uses opt%cagmres_s, opt%cagmres_t (defaults 8, 4), opt%cagmres_qr_algo and opt%cagmres_q_mode (Reference). x is the initial guess on input and the solution on output. The iteration history and timings are printed to standard output of rank 0. The convergence status is in res%converged_reason, the number of iterations in res%iterations, and the residuals in res%residual and res%true_residual.
opt%gmres_restart = 8 in the example above is the value of the verification sample, not the library default (30). In your own programs, start from the defaults set by parcel_options_init and change only the fields you need.
The legacy Fortran API of v1.0, run_parcel_sycl_f, is also kept for compatibility (pcg only). It is not included in the module, so the user writes the interface (Reference). Use parcel_solve_linear in new programs.
5.3.6 Calling an Eigenvalue Solver
Give one of PARCEL_SOLVER_POWER, PARCEL_SOLVER_INVERSE_POWER, PARCEL_SOLVER_KS_ARNOLDI as solver of parcel_solve_eigen. Get the number k of returned eigenpairs with parcel_eigen_count. ks_arnoldi uses opt%ks_m, opt%ks_k_keep, opt%ks_nev (maximum dimension of the subspace, number of Ritz pairs kept, number of wanted eigenvalues; defaults 30, 10, 4), opt%ks_max_restart (default 200) and the orthogonalization method opt%ks_qr_algo (PARCEL_QR_CGS for CGS2, PARCEL_QR_MGS for MGS2). Eigenvalues near a shift are computed with shift-invert (opt%shift_enable = 1, opt%shift_sigma; ks_arnoldi only). With shift-invert and opt%precon_spectrum = 0, the wrapper assembles , so the user does not need to prepare it. The internal linear solver is set with opt%linsolve_tolerance, opt%linsolve_max_iteration and opt%linsolve_restart (0 or less gives the solver default). Details of the fields are in parcel_solve_eigen in the Reference.
! From sample_f.f90: ks_arnoldi with shift-invert (4 eigenvalues near σ = -0.15)
call parcel_options_init(opt)
opt%index_base = 1
opt%tolerance = 1.0d-8
opt%ks_m = 30
opt%ks_k_keep = 10
opt%ks_nev = 4
opt%ks_max_restart = 100
opt%shift_enable = 1
opt%shift_sigma = -0.15d0
opt%linsolve_tolerance = 1.0d-11
opt%linsolve_max_iteration = 5000 ! default (0) gives 2000 for ks_arnoldi
opt%linsolve_restart = 30
opt%linsolve_precon_algo = PARCEL_PRECON_BLOCKJACOBI
k = parcel_eigen_count(PARCEL_SOLVER_KS_ARNOLDI, opt)
nev = opt%ks_nev
allocate(ev(2*k), v(n_local*k))
rc = parcel_solve_eigen(PARCEL_SOLVER_KS_ARNOLDI, opt, MPI_COMM_WORLD, n_local, nstart, n_global, &
nnz_local, rowptr, col, val, ev, v, res)
if(rc == PARCEL_OK .and. res%converged_reason <= 0) then
! not converged: only res%ks_nconv eigenpairs satisfied the tolerance. Also check res%linsolve_failures
end if
do j = 1, nev
! eigenvalue ev(2*j-1) + i ev(2*j), eigenvector v((j-1)*n_local+1 : j*n_local)
end doFor power and inverse_power, pass ev(2) and v of length n_local.
5.3.7 Specifying Precision
The input/output arrays are fixed to real(8) (real(c_double)). The data and arithmetic precisions inside PARCEL are determined by macros when compiling the wrapper capi/wrapper_parcel_sycl.cpp and cannot be changed at run time or per call. The defaults are in capi/parcel_capi_default.hpp and are all parcel_double (double precision).
| Macro | Purpose |
|---|---|
parcel_solver_data_type |
Data precision of the matrix and vectors |
parcel_solver_calc_type |
Arithmetic precision |
parcel_solver_calc_precon_type |
Arithmetic precision of the preconditioner |
parcel_solver_data_precon_mat_type |
Data precision of the preconditioner matrix |
parcel_solver_data_precon_vec_type |
Data precision of the preconditioner vector |
parcel_solver_calc_qr_type |
Precision of the QR factorization in pcagmres |
For example, to perform the arithmetic in single precision, add -Dparcel_solver_calc_type=parcel_float when compiling the wrapper. The usable combinations of precisions are in “Precision Configurations” in “Basic Structure”, and the restrictions at low precision in “Known Limits at Low Precision”.
5.3.8 MPI Decomposition
The communicator is passed directly as a Fortran integer handle (such as MPI_COMM_WORLD of the mpi module). Communicators other than MPI_COMM_WORLD can also be passed. The procedures include collective operations, so call them on all ranks of the communicator with the same solver and the same contents of opt. The handling of the communicator in a build without MPI is in “Build without MPI”.
The row block of each rank only needs to consist of contiguous rows, and the number of rows may differ between ranks. The row blocks need not be ordered by rank number, but together the row blocks of all ranks must cover all rows of the matrix without overlaps or gaps. The wrapper gathers (nstart, n_local) of every rank to check this and, if the condition is not met, returns PARCEL_ERR_DISTRIBUTION on all ranks. n_local of each rank must be 1 or greater. When index_base = 1, nstart is also given 1-based.
! From sample_f.f90: assign rows to rank r with weight (r+1)
w = int(nprocs, 8) * (nprocs + 1) / 2
acc = 0
do r = 0, nprocs - 1
displs(r) = int(n_global * acc / w)
acc = acc + r + 1
counts(r) = int(n_global * acc / w) - displs(r)
end do
n_local = counts(myrank) ! n_global, n_local, nstart are integer(parcel_int_kind)
nstart = displs(myrank) + 1 ! 1-based5.3.9 Compiling and Running
Compile the module capi/parcel_sycl_mod.f90 and the user program with the Fortran compiler (mpif90 with MPI; compile the module first), and link them, together with the object of the wrapper capi/wrapper_parcel_sycl.cpp compiled with the SYCL compiler, using the SYCL compiler. Add the Fortran runtime library (and, with MPI, the MPI Fortran libraries) when linking. The manual steps, with and without MPI, are in “Fortran” of the Quick Start, and the flags for each environment in “C and Fortran Programs” in “Installation”. examples/build_examples.sh also builds sample_f with the same steps.
export OMPI_CXX=acpp
bash examples/build_examples.sh # with MPI (without MPI: PARCEL_MPI=0 bash examples/build_examples.sh)
mpirun -n 3 examples/build/sample_f 16 # argument is NX (default 16)sample_f prints PASS / FAIL per line for 12 items in total (7 solvers, pcagmres twice more without forming , ks_arnoldi with shift-invert, the legacy API, and first the version check), passing the CRS 1-based (index_base = 1).
When compiling user programs with Fujitsu Fortran (frt, mpifrt), add -Kopenmp. On Odyssey, sample_f compiled without -Kopenmp made pcg and the other linear solvers diverge from the first iteration when run with several threads (OMP_NUM_THREADS=12) (with -Kopenmp, or with one thread, all 7 solvers passed; it also passed when compiled with gfortran).
5.3.10 Passing Your Own Matrix
The matrix passed to PARCEL is the CRS of the rows owned by each rank (the format is in “Data Structures” in “Basic Structure”). This section shows how to bring the matrix of your own problem into this form.
5.3.10.1 Building CRS from Triplets (COO)
Finite element codes and others often assemble a matrix as (row, column, value) triplets (COO format). The following subroutine builds a 1-based CRS (to be passed with opt%index_base = 1) from the triplets of the owned rows. Repeated (row, column) pairs are summed and the column indices of each row are sorted. The result has the form PARCEL requires.
! Build a 1-based CRS from the triplets ti(k), tj(k), tv(k) (k = 1..nt) of the owned rows nstart .. nstart+n_local-1 (1-based).
! Repeated (row, column) pairs are summed and the column indices of each row are sorted. nnz returns the number of nonzeros
subroutine coo_to_crs(nstart, n_local, nt, ti, tj, tv, rowptr, col, val, nnz)
use parcel_sycl_mod, only: parcel_int_kind
implicit none
integer(parcel_int_kind), intent(in) :: nstart, n_local, nt
integer(parcel_int_kind), intent(in) :: ti(nt), tj(nt)
real(8), intent(in) :: tv(nt)
integer(parcel_int_kind), intent(out) :: rowptr(n_local+1), col(nt), nnz
real(8), intent(out) :: val(nt)
integer(parcel_int_kind) :: k, r, p, q, jc, pos(n_local), first(n_local+1)
real(8) :: v
! determine the start of each row from the counts and place the triplets row by row
first = 0
do k = 1, nt
r = ti(k) - nstart + 1
first(r+1) = first(r+1) + 1
end do
first(1) = 1
do r = 1, n_local
first(r+1) = first(r+1) + first(r)
end do
pos = first(1:n_local)
do k = 1, nt
r = ti(k) - nstart + 1
col(pos(r)) = tj(k)
val(pos(r)) = tv(k)
pos(r) = pos(r) + 1
end do
! sort each row by column index (insertion sort), sum equal columns and compact
nnz = 0
do r = 1, n_local
do p = first(r) + 1, first(r+1) - 1
jc = col(p)
v = val(p)
q = p - 1
do while (q >= first(r))
if (col(q) <= jc) exit
col(q+1) = col(q)
val(q+1) = val(q)
q = q - 1
end do
col(q+1) = jc
val(q+1) = v
end do
rowptr(r) = nnz + 1
do p = first(r), first(r+1) - 1
if (nnz >= rowptr(r)) then
if (col(nnz) == col(p)) then
val(nnz) = val(nnz) + val(p)
cycle
end if
end if
nnz = nnz + 1
col(nnz) = col(p)
val(nnz) = val(p)
end do
end do
rowptr(n_local+1) = nnz + 1
end subroutine coo_to_crs- Pass only the triplets of the owned rows (
nstart <= ti(k) <= nstart + n_local - 1), and pass the returnednnzasnnz_local. - If some rows may lack a diagonal entry, first add (i, i, 0.0) to every row (the diagonal is stored even when its value is 0).
- Even for a symmetric matrix, include the entries of both the upper and lower triangles.
5.3.10.2 Reading a Matrix Market File
The Matrix Market format (extension .mtx) is a text format for exchanging sparse matrices. The main points for reading it with read statements:
- A file in coordinate form (
coordinate) starts with a header such as%%MatrixMarket matrix coordinate real general, followed by comment lines starting with%, a line “rows columns nonzeros”, and then one “row column value” (1-based) per line. - If the header says
symmetric, the file contains only one triangle (usually the lower one). Add each off-diagonal entry also at (column, row) with the same value to obtain both triangles (with the sign reversed forskew-symmetric). - The
pattern(no values) andcomplexformats cannot be handled by PARCEL (PARCEL solves real matrices). - The rows and columns are 1-based, so they can be used as they are with
opt%index_base = 1. Keep only the triplets of the owned rows and convert them to CRS withcoo_to_crsabove.
5.3.10.3 Distributing a Global Matrix to the Ranks
With MPI, each rank passes only the CRS of its own rows. There are two ways to prepare it.
- Each rank builds only its own rows: if the matrix can be generated from formulas, as with finite differences, each rank generates only the range of its own rows. No rank holds the global matrix, so memory is smallest. The bundled examples (
examples/quick_start_f.f90,examples/sample_f.f90) take this form. - Take the owned rows from a global matrix: if a global CRS (
grp,gcol,gval, 1-based) is at hand, for example read from a file, the part of rowsnstarttonstart + n_local - 1is a contiguous range of the arrays. Only differences from the first element of the row pointer are used and the wrapper does not modify the input arrays, so array elements can be passed as the start. Either every rank reads the same file, or one rank reads it and distributes it withMPI_Scatterv.
! Pass rows nstart .. nstart+n_local-1 taken from a global CRS (1-based) (opt%index_base = 1)
off = grp(nstart)
nnz_local = grp(nstart + n_local) - off
rc = parcel_solve_linear(PARCEL_SOLVER_PCG, opt, MPI_COMM_WORLD, n_local, nstart, n_global, &
nnz_local, grp(nstart), gcol(off), gval(off), b, x, res)5.3.10.4 Choosing the Partition
The rows of each rank need only be contiguous, and the number of rows may differ between ranks (“MPI Decomposition”). Decide the partition considering the following.
- Most of the time of an iteration is spent in the SpMV and the preconditioner, whose cost is proportional to the number of nonzeros. In collective operations such as dot products all ranks wait for the slowest one, so for matrices whose number of nonzeros per row varies widely, balance the number of nonzeros rather than the number of rows. With the global row pointer
grp, let the first row of rank () be the first row at which the cumulative number of nonzeros reaches (leave at least one row to each rank). - For matrices with nearly the same number of nonzeros per row (such as finite-difference grids), dividing the rows evenly is enough.
- The amount of halo communication is determined by the number of nonzeros that refer to columns of other ranks. For grid problems, numbering nearby grid points with nearby row numbers reduces the halo.
- The memory of each rank is also roughly proportional to the number of nonzeros of its rows. If GPU memory is insufficient, increase the number of ranks (GPUs) to reduce the rows per rank.
5.4 Python
5.4.1 Overview
From Python, import the package parcel (python/ in the distribution package). The package has its own native layer (libparcel_py.so) that calls the C++ library directly, and is independent of the C / Fortran API (capi/). All 7 solvers can be selected and called at run time. One call creates the SYCL queue, registers the matrix and solves, and returns the solution as a NumPy array. Only double precision (for both data and computation) is available.
The default version without MPI runs as a single process with python script.py and holds the whole matrix and vectors. With the version with MPI (Installation), start the script as mpirun -n 4 python script.py; all ranks make the same calls (collective operations), and each rank holds only the rows it owns (a contiguous row block). The same script runs with both (Running without MPI, Running with MPI). Installation is described in Python Package in “Installation”.
# Solve the 3-D Poisson equation with pcg + block Jacobi preconditioning (python poisson.py; with MPI: mpirun -n 4 python poisson.py)
import numpy as np
import scipy.sparse as sp
import parcel
nx = 32
T = sp.diags([-1.0, 2.0, -1.0], [-1, 0, 1], shape=(nx, nx))
I = sp.identity(nx)
G = (sp.kron(sp.kron(T, I), I) + sp.kron(sp.kron(I, T), I) + sp.kron(sp.kron(I, I), T)).tocsr()
b = G @ np.ones(G.shape[0]) # the exact solution is all ones
A = parcel.DistributedCSR.from_global(G) # take the rows owned by this rank
x, info = parcel.solve(A, b[A.local_slice()], method="pcg",
precond="block_jacobi", tol=1e-10)
if A.comm.rank == 0:
print(info.reason, info.iterations, info.true_residual)| Item | Python |
|---|---|
| Linear solvers | parcel.solve(A,b,method=...); method is "pcg", "pbicgstab", "pgmres", "pcagmres" |
| Eigenvalue solvers | parcel.eigs(A,method=...); method is "power", "inverse_power", "ks_arnoldi" |
| Specifying the solver, preconditioner, convergence tolerance, and iteration count | At run time (keyword arguments). Items not given take the library defaults |
| Precision | Double only |
| Input arrays of the matrix | Not overwritten (copied into int64 and float64 arrays) |
| Index base | 0 |
| Layout of eigenvectors | An array of shape (n_local, k); column j is the owned part of the j-th eigenvector |
| MPI row decomposition | Arbitrary (row_start, n_local per rank). The coverage of the row blocks is checked |
| Integers such as row and column indices | Passed as int64. Inside the library, PARCEL_INT and PARCEL_LINT |
| Reuse of matrices and vectors | A SYCL queue is created and the matrix is registered on every call |
| Retrieving the convergence status | The returned SolveInfo / EigInfo. Argument errors raise exceptions |
5.4.2 Setting the Matrix
A matrix is represented by parcel.DistributedCSR. Each rank holds the CRS of the rows it owns (global rows row_start to row_start + n_local - 1). The layout is the same as scipy.sparse.csr_matrix: indptr is the 0-based local row pointer and indices are the 0-based global column indices.
There are three ways to create it: the constructor from the three CRS arrays, from_scipy() from a scipy.sparse matrix of the owned rows only, and from_global(), which takes the owned rows from the whole matrix (for small problems and tests). The arrays passed are copied, so they may be modified afterwards. The arguments, the attributes (n_local, row_start, n_global, nnz, comm) and the checks are given in parcel.DistributedCSR in the Reference. If the matrix check fails, all ranks raise the same ValueError.
5.4.3 Setting the Vectors
Pass the right-hand side b and the initial value x0 as the owned part (a 1-D array of length A.n_local). The returned solution x is also the owned part. If you hold the whole vector, take the part with b[A.local_slice()]. To obtain the whole solution, gather it with, e.g., mpi4py’s allgather.
5.4.4 Setting the Preconditioner
The preconditioner is chosen with precond ("point_jacobi", "block_jacobi", "jacobi"; the default is no preconditioning), and its settings (iterations of the Jacobi preconditioner, blocks of the block Jacobi preconditioner, scaling) are given in the dictionary precond_options (parcel.solve() in the Reference). For the eigenvalue solvers, the preconditioner is used when precond_spectrum=True (computing the eigenvalues of ). The preconditioner of the inner linear solver is given with inner_precond and inner_precond_options.
5.4.5 Calling a Linear Solver
x, info = parcel.solve(A, b[A.local_slice()], method="pgmres", restart=60,
precond="jacobi", tol=1e-10)
if not info.converged:
print(info.reason, info.iterations, info.true_residual)Choose the solver with method ("pcg", "pbicgstab", "pgmres", "pcagmres") and give settings such as the convergence tolerance as keyword arguments. Items given as None take the library defaults. The list of arguments and the defaults are given in parcel.solve() in the Reference. Non-convergence does not raise an exception, so check converged of the returned info (SolveInfo).
5.4.6 Calling an Eigenvalue Solver
Choose "power", "inverse_power" or "ks_arnoldi" with method. values is a 1-D complex array, and vectors is an array of shape (A.n_local, count) whose column j is the owned part of the j-th eigenvector. The arguments (m, k_keep, nev of ks_arnoldi, sigma of shift-invert, inner_* of the inner linear solver, etc.) and the order of the returned values are given in parcel.eigs() in the Reference.
# ks_arnoldi: 4 eigenvalues of largest magnitude; shift-invert: 2 eigenvalues closest to 0
vals, vecs, info = parcel.eigs(A, method="ks_arnoldi", m=40, k_keep=10, nev=4, tol=1e-10)
vals, vecs, info = parcel.eigs(A, method="ks_arnoldi", m=20, k_keep=6, nev=2, sigma=0.0,
tol=1e-10, inner_tol=1e-12)5.4.7 Exceptions
Invalid arguments or settings and a failed matrix check raise ValueError, exceeding the range of the integer type raises OverflowError, a solver excluded from the build raises parcel.NotBuiltError, and other failures raise parcel.ParcelError (list: “Exceptions” in the Reference).
In the version without MPI, these exceptions can be caught in the calling process, and failures inside the library such as running out of device memory also raise parcel.ParcelError. The handling in the version with MPI (errors that raise the same exception on all ranks, and errors that terminate with MPI_Abort) is described in Running with MPI.
5.4.8 Specifying Precision
Both data and computation are fixed to double precision (parcel_double); mixed precision is not available. The input and output arrays are float64.
5.4.9 Running without MPI
Run in the same environment as the build (the AdaptiveCpp and LAPACK libraries must be found; spack load py-parcel-sycl... when installed with spack, or sourcing python/envs/mac_brew.sh on macOS). The number of threads is given by OMP_NUM_THREADS. The device is chosen by the default selector, so when several GPUs are visible, choose the GPU to use with CUDA_VISIBLE_DEVICES or the like.
export CUDA_VISIBLE_DEVICES=0
$HOME/venv_parcel/bin/python script.pyThe single process owns all rows (row_start = 0, n_local = n_global; create the matrix with from_global()). parcel.Comm() has rank 0 and size 1, and mpi4py communicators cannot be passed. All exceptions can be caught in the calling process (running out of device memory and the like raise parcel.ParcelError).
AdaptiveCpp prints warnings about unused backends on every call, so the package sets ACPP_DEBUG_LEVEL to 0 unless it is already set.
5.4.10 Running with MPI
Start with mpirun; all ranks make the same calls with the same arguments (collective operations). Give the number of threads with OMP_NUM_THREADS. To assign one GPU to each rank, put a wrapper such as gpu_bind.sh between mpirun and Python.
export OMP_NUM_THREADS=12
mpirun -n 4 ./gpu_bind.sh $HOME/venv_parcel_mpi/bin/python script.pyMPI is initialized at the first call. If mpi4py or another module has already initialized MPI, that is used, and the package calls MPI_Finalize at exit only when it initialized MPI itself.
A communicator is represented by parcel.Comm. parcel.Comm() is MPI_COMM_WORLD, and parcel.Comm(c) accepts an mpi4py communicator. The comm argument of the matrix accepts either a parcel.Comm or an mpi4py communicator; if omitted, MPI_COMM_WORLD is used. comm.rank, comm.size, and comm.allreduce(value, op) (sum, maximum or minimum over all ranks; op is "sum", "max", "min") are available.
Any decomposition into contiguous row blocks is allowed; it may be uneven and need not follow the order of the rank numbers. parcel.partition_rows(n_global, comm) returns (row_start, n_local) of an even decomposition (the remainder is given one row each to the first ranks).
row_start, n_local = parcel.partition_rows(n_global)
# make_rows(row_start, n_local) is a user function returning a scipy.sparse CSR matrix of only the owned rows
# (shape (n_local, n_global), global column indices)
A = parcel.DistributedCSR.from_scipy(make_rows(row_start, n_local), row_start, n_global)Errors detected by the library (invalid combinations of settings, the matrix check) raise the same exception on all ranks, so processing can continue after catching them. Python-side checks such as array lengths and value ranges are made on each rank, so if the arguments differ between ranks and only some ranks raise, the remaining ranks wait forever in a collective operation. Other errors (such as running out of device memory) can occur on only some ranks, so a message is printed and all ranks are terminated with MPI_Abort.
5.4.11 Passing Your Own Matrix
The matrix passed to PARCEL is the CRS of the rows owned by each rank (the format is in “Data Structures” in “Basic Structure”). This section shows how to bring the matrix of your own problem into this form.
5.4.11.1 Building CRS from Triplets (COO)
Finite element codes and others often assemble a matrix as (row, column, value) triplets (COO format). Converting a SciPy coo_matrix to CSR sums the values at the same position. Call sum_duplicates() and sort_indices() before passing it, to be safe.
import numpy as np
import scipy.sparse as sp
import parcel
# i, j, v: triplets (row and column are global, 0-based). n_global: number of rows of the matrix
G = sp.coo_matrix((v, (i, j)), shape=(n_global, n_global)).tocsr()
G.sum_duplicates(); G.sort_indices()
A = parcel.DistributedCSR.from_global(G) # take the rows owned by this rank (all rows without MPI)- If some rows may lack a diagonal entry, add zero diagonal entries (i, i, 0.0) to the triplets (the diagonal is stored even when its value is 0; do not call
eliminate_zeros(), which removes zero entries). - Even for a symmetric matrix, include the entries of both the upper and lower triangles.
5.4.11.2 Reading a Matrix Market File
The Matrix Market format (extension .mtx) is a text format for exchanging sparse matrices. It can be read with SciPy’s mmread. mmread also expands files in symmetric form (symmetric) into both triangles. The pattern (no values) and complex formats cannot be handled by PARCEL (PARCEL solves real matrices).
import scipy.io, scipy.sparse
import parcel
G = scipy.sparse.csr_matrix(scipy.io.mmread("matrix.mtx"))
G.sum_duplicates(); G.sort_indices()
A = parcel.DistributedCSR.from_global(G) # take the rows owned by this rank (all rows without MPI)5.4.11.3 Distributing a Global Matrix to the Ranks
With MPI, each rank holds only the CRS of its own rows. There are two ways to prepare it.
- Each rank builds only its own rows: if the matrix can be generated from formulas, as with finite differences, determine the owned rows with
parcel.partition_rows(), build only that range, and pass it withDistributedCSR.from_scipy()(make_rowsin “Running with MPI”). No rank holds the global matrix, so memory is smallest. - Take the owned rows from a global matrix: if every rank can hold the global matrix,
DistributedCSR.from_global()takes the owned rows with an even partition. Take the owned part of vectors such as the right-hand side withb[A.local_slice()].
5.4.11.4 Choosing the Partition
The rows of each rank need only be contiguous, and the number of rows may differ between ranks. from_global() and partition_rows() divide the rows evenly; for another partition, decide the owned rows (row_start, n_local) yourself and pass them with from_scipy(). Decide the partition considering the following.
- Most of the time of an iteration is spent in the SpMV and the preconditioner, whose cost is proportional to the number of nonzeros. In collective operations such as dot products all ranks wait for the slowest one, so for matrices whose number of nonzeros per row varies widely, balance the number of nonzeros rather than the number of rows. With the row pointer
G.indptrof the global matrixG, let the first row of rank () be the first row at which the cumulative number of nonzeros reaches (leave at least one row to each rank). - For matrices with nearly the same number of nonzeros per row (such as finite-difference grids), dividing the rows evenly is enough.
- The amount of halo communication is determined by the number of nonzeros that refer to columns of other ranks. For grid problems, numbering nearby grid points with nearby row numbers reduces the halo.
- The memory of each rank is also roughly proportional to the number of nonzeros of its rows. If GPU memory is insufficient, increase the number of ranks (GPUs) to reduce the rows per rank.