1 Quick Start
This chapter shows, with a minimal program that solves the linear system for the finite-difference matrix of the one-dimensional Laplacian (a tridiagonal matrix) with the conjugate gradient method, the steps from installing a SYCL compiler to building and running. The same problem is solved in four languages (C++, C, Fortran and Python), each without MPI (one process) and with MPI (several processes). Names of machines and devices (s86, V100, etc.) and abbreviations are collected in the Glossary.
1.1 Overview of the Program
The problem is as follows. The programs in all four languages solve it and print the same results.
- : problem size (global number of rows),
n_globalin the programs, - : row index, counted from 0: (Fortran array indices start at 1)
- : the coefficient matrix, tridiagonal with 2 on the diagonal and on the neighbors (columns and of row )
- : the right-hand side of length , all ones ()
- : the solution of length , initial value ; is the first element
- Method: conjugate gradient (
pcg) without preconditioning, double precision; convergence threshold (tolerance) on the relative residual = 1.0e-9, at most 10000 iterations
The right-hand side is symmetric under reversal () and has components only along eigenvectors of , so without rounding errors the conjugate gradient method reaches the exact solution () in exactly iterations. Every program prints the convergence status, the iteration count, the residuals and , computes the maximum over all components of the relative error against the exact solution, and prints check: ... PASS if it is at most 1e-6 (with MPI, the maximum over the rows owned by each rank is reduced across the ranks). If the solver does not converge or the error exceeds the limit, it prints FAIL and exits with status 1. The limit is much smaller than the worst-case estimate 4e-4 from the condition number (about 4.1e5) and the tolerance 1e-9, but the actual error is at the rounding level (0 to 1.3e-15), and a wrong solution gives an error larger by orders of magnitude, so the two are distinguished. To build and run the examples in all four languages and get a pass/fail summary, use bash examples/run_quick_checks.sh (Installation, “Checking After the Build”).
The flow is the same in every language.
- Distribute the rows over the ranks. Each rank owns a contiguous block of rows. Without MPI, one process owns all rows.
- Build the owned rows of the coefficient matrix in CRS format (row pointers, column indices, values). Column indices are global.
- Set the solver (threshold, maximum iterations) and solve. Settings not given take the library defaults.
- Check the convergence status.
The programs hardly differ between without and with MPI. With MPI, each rank builds only its own rows, and all ranks call the same function with the same settings (a collective operation). Without MPI, one process owns everything. Which version is built is selected by the build setting PARCEL_MPI (“Default of MPI” in “Installation”).
Requirements:
- A SYCL implementation (AdaptiveCpp; next section)
- LAPACKE (e.g., OpenBLAS; used by the eigenvalue solvers)
- For the version with MPI, an MPI library (Open MPI, etc.)
- For C / Fortran, C and Fortran compilers (gcc, gfortran, Fujitsu fcc, frt, etc.; “Environments Whose Standard Compiler Is Not GCC”); for Python, Python 3.8 or later and NumPy (the example also uses SciPy)
1.2 Installing a SYCL Compiler
PARCEL is a header-only library compiled with a SYCL compiler. v1.2 uses AdaptiveCpp.
Choose the installation route by environment as follows.
| Environment | Route | Notes |
|---|---|---|
| Test environments of this manual (s86, Odyssey, Aquarius, Miyabi-C, Miyabi-G, PS-A, PS-B) | install_stack.sh (from scratch), or AdaptiveCpp into an existing spack |
The environment definitions spack/stack/envs/<env>/ are included. The procedure for an existing spack is in each section of the “Installing a SYCL Compiler” chapter |
| Other Linux systems (RHEL family, Ubuntu family, Windows WSL2) | install_stack.sh with the generic template, or AdaptiveCpp into spack |
Confirmed on Ubuntu 24.04 and Rocky Linux 9 (both on WSL2). “Generic Linux” |
| macOS | Homebrew | “macOS (Homebrew)” |
| Python only | pip (pip install . in python/), or py-parcel-sycl of spack |
Install AdaptiveCpp and LAPACKE first by one of the routes above. “Python Package” |
The main routes are used as follows.
- From scratch (Linux):
spack/stack/install_stack.shin the distribution package installs, into a new spack environment separated from any existing spack, everything from fetching spack through LLVM, AdaptiveCpp, OpenBLAS, Python with NumPy and SciPy, to PARCEL itself and its C, Fortran and Python APIs. MPI, CUDA / ROCm and the compilers are taken from each machine. Definitions for the test environments (s86, Miyabi-C, Miyabi-G, PS-A, PS-B, Aquarius, Odyssey) are included (“Installing with spack” in “Installation”).
bash spack/stack/install_stack.sh s86 $HOME/parcel_stack 16 # <env> <prefix> [jobs]
. $HOME/parcel_stack/setup.sh # when using it (every time)
spack load parcel-sycl~mpi~cuda~rocm py-parcel-sycl~mpi~cuda~rocm # CPU version without MPI- macOS: spack is not used; prebuilt binaries are installed with Homebrew (“macOS (Homebrew)” in “Installing a SYCL Compiler”).
brew install adaptivecpp open-mpi openblas gcc numpy scipy- Into an existing spack: the procedure for each test environment is in the “Installing a SYCL Compiler” chapter, and that for other Linux systems in “Generic Linux”.
The examples below assume a Linux environment (s86, etc.) with CPU execution (the OpenMP backend of AdaptiveCpp) and LAPACKE in /usr/include/openblas, and are run in the top directory of the package. Environment variables and modules differ by environment, so first apply the “Settings for use” of each environment in the “Installing a SYCL Compiler” chapter. The flags for macOS + Homebrew are in “macOS (Homebrew)” in “Installation”. CPU executables use the LLVM OpenMP runtime (libomp.so), but acpp does not embed its location in the executable. The “Settings for use” of the test environments resolve this by adding the LLVM lib/ to LD_LIBRARY_PATH. In your own environment, add the LLVM lib/ to LD_LIBRARY_PATH at run time or add -Wl,-rpath,<LLVM lib> at link time (without either, the OS libomp is used or the program fails to start). Also give the location of the Fortran runtime library (libgfortran) with -L (the examples find it with gfortran -print-file-name). For NVIDIA GPUs, change the target, e.g. --acpp-targets=cuda:sm_70; the program itself does not change. On V100 (sm_70), adding -Xcuda-ptxas -maxrregcount=255 is also recommended (Recommended Options).
1.2.1 Environments Whose Standard Compiler Is Not GCC
The examples below compile the C and Fortran main programs with gcc and gfortran (or mpicc and mpif90 of Open MPI). The same procedure works where the standard compilers are not GCC. The combinations used in the test environments are shown below; the detailed flags for each environment are in “Recommended Options” in “Installation”.
| Standard compiler (test environment) | C++ (PARCEL itself) | C and Fortran main | MPI wrappers | Notes |
|---|---|---|---|---|
| GCC (most Linux systems such as s86) | acpp (clang of LLVM 18) |
gcc, gfortran | mpicxx, mpicc, mpif90 of Open MPI (OMPI_CXX=acpp) |
— |
| Fujitsu compilers (Odyssey) | acpp --acpp-targets=omp.library-only; Fujitsu FCC (clang mode, given by ACPP_CPU_CXX) compiles everything including the kernels |
Fujitsu fcc, frt | mpiFCC (OMPI_CXX=acpp), mpifcc, mpifrt |
Add -Kopenmp to fcc and frt (“Fortran” in “Usage”). The login and compute nodes have different architectures, so build on a compute node (“Odyssey”) |
| Intel oneAPI (PS-A) | acpp (clang of LLVM 18; used after loading the site modules intel and intelmpi) |
gcc, gfortran (mpicc, mpif90 of Intel MPI) |
mpicxx of Intel MPI (I_MPI_CXX=acpp) |
Get the MPI Fortran libraries with mpif90 -show (“PS-A”) |
| Apple clang (macOS) | acpp --acpp-targets=omp.library-only; Apple clang compiles everything |
clang (C), gfortran (Homebrew gcc) |
mpicxx, mpicc, mpif90 of Open MPI (Homebrew; OMPI_CXX=acpp) |
Add blocking of the reductions (-D_BLOCKING_LOOP_DOT_=1); -march=native is not needed (it makes no difference in speed); launch with mpirun --bind-to none (“macOS (Homebrew)” in “Installation”) |
- With
omp.library-only, AdaptiveCpp acts only as the SYCL runtime, and the host C++ compiler (Fujitsu FCC, Apple clang) compiles everything including the kernels. With Fujitsu FCC (A64FX) specify blocking of all kernels (-D_BLOCKING_LOOP_=1); with Apple clang specify blocking of the reductions only (-D_BLOCKING_LOOP_DOT_=1), as on other CPUs. Without blocking the dot products are extremely slow in both cases (“Blocking” in the “Optimization” chapter). - The variable that makes the MPI wrapper call
acppdepends on the MPI (OMPI_CXXfor Open MPI and Fujitsu MPI,I_MPI_CXXfor Intel MPI,MPICH_CXXfor MPICH-based MPIs).
The thread counts in the examples below match the macOS + Homebrew machine on which the outputs were obtained (Apple M4, 10 cores): 8 threads without MPI, and 4 ranks × 2 threads with MPI. On your machine, choose them so that ranks × threads does not exceed the number of cores. The outputs were obtained with these two settings, and the printed lines were the same. The residuals change in the last digits with the environment and the number of processes (for example, the true residual is 2.49e-14 with 4 ranks on s86 CPU), but the iteration count 500 and do not. The per-iteration output and the time breakdown are omitted.
1.3 C++
1.3.1 Program
The source is examples/quick_start.cpp. Without and with MPI differ only in the four #if PARCEL_MPI blocks.
#include <algorithm>
#include <cmath>
#include <cstdio>
#include <vector>
#include "../Include/parcel_sycl.hpp"
int main(int argc, char *argv[]) {
#if PARCEL_MPI
MPI_Init(&argc, &argv);
#else
(void)argc; (void)argv;
#endif
{
// default device (determined by --acpp-targets); use an in-order queue
// (PARCEL submits kernels in order without events; solvers throw if the queue is not in-order)
sycl::queue q{sycl::default_selector_v, sycl::property::queue::in_order{}};
#if PARCEL_MPI
parcel::comm comm(MPI_COMM_WORLD, &q);
#else
parcel::comm comm(&q); // without MPI: rank 0, one process
#endif
const int myrank = comm.mpi_rank(), nproc = comm.mpi_size();
// row distribution: each rank owns contiguous rows (the remainder goes to the last rank)
const PARCEL_INT n_global = 1000;
const PARCEL_INT nstart = (n_global / nproc) * myrank;
const PARCEL_INT nend = (myrank == nproc - 1) ? n_global - 1 : nstart + n_global / nproc - 1;
const PARCEL_INT n = nend - nstart + 1;
// CRS of the owned rows (row pointers local 0-based, column indices global 0-based)
std::vector<PARCEL_INT> rowptr(n + 1, 0), col;
std::vector<double> val;
for (PARCEL_INT i = nstart; i <= nend; i++) {
if (i > 0) { col.push_back(i - 1); val.push_back(-1.0); }
col.push_back(i); val.push_back( 2.0);
if (i < n_global - 1) { col.push_back(i + 1); val.push_back(-1.0); }
rowptr[i - nstart + 1] = (PARCEL_INT)col.size();
}
parcel::matrix<parcel_double> A(&q);
A.set_crsMat<double>(&comm, nstart, nend, n, col.size(), rowptr.data(), col.data(), val.data());
std::vector<double> x_h(n, 0.0), b_h(n, 1.0);
parcel::vector<parcel_double> x(n, &q), b(n, &q);
x.setValue(x_h.data());
b.setValue(b_h.data());
// conjugate gradient (all arithmetic in double)
parcel::krylov::pcg<parcel_double, parcel_double, parcel_double, parcel_double> cg;
cg.set_tolerance(1.0e-9); // relative residual ||r|| / ||b||
cg.set_max_iteration(10000);
q.wait();
cg.solver<parcel_double, parcel_double, parcel_double,
parcel_double, parcel_double, parcel_double>(A, x, b);
x.getValue(x_h.data());
q.wait();
// comparison with the exact solution. The matrix is the difference -x[i-1] + 2 x[i] - x[i+1] = 1 with zero values outside the boundary (Dirichlet), so
// the exact solution is x[i] = (i + 1) (n_global - i) / 2 (i is the global 0-based index; x[0] = n_global / 2, maximum about n_global^2 / 8 at the center).
// The maximum of the relative error |x[i] - exact| / exact over all components is computed over the owned rows of each rank and reduced over the ranks.
// limit 1e-6: the condition number is cot^2(pi / (2 (n_global + 1))) = about 4.1e5, and the worst-case estimate from tolerance 1e-9
// (relative error <= condition number x relative residual) is 4e-4. The actual error is far smaller: b = 1 is symmetric about the center, so the Krylov subspace
// is exhausted at dimension 500, pcg reaches an almost zero residual in 500 iterations, and the error is at the rounding level (0 to 1.3e-15 on s86 CPU and GPU, 1 and 2 ranks).
// A wrong solution (matrix, distribution or communication error) gives an error larger by orders of magnitude, so 1e-6 distinguishes them.
const double err_limit = 1.0e-6;
double err_local = 0.0;
for (PARCEL_INT k = 0; k < n; k++) {
const double i = (double)(nstart + k);
const double exact = 0.5 * (i + 1.0) * ((double)n_global - i);
double e = std::fabs(x_h[k] - exact) / exact;
if (!(e == e)) e = HUGE_VAL; // treat NaN as infinity (fail)
err_local = std::max(err_local, e);
}
double err_max = err_local;
#if PARCEL_MPI
MPI_Allreduce(&err_local, &err_max, 1, MPI_DOUBLE, MPI_MAX, MPI_COMM_WORLD);
#endif
const bool ok = cg.is_converged() && err_max <= err_limit;
// convergence status and check (same on all ranks)
if (myrank == 0) {
std::printf("reason = %d (%s), iterations = %d, residual = %.3e, true residual = %.3e\n",
(int)cg.get_converged_reason(),
parcel::krylov::converged_reason_name(cg.get_converged_reason()),
(int)cg.get_iterations(), cg.get_residual(), cg.get_true_residual());
std::printf("x[0] = %.6f (exact %.6f)\n", x_h[0], 0.5 * 1.0 * (double)n_global);
std::printf("check: max relative error = %.3e (limit %.1e) %s\n", err_max, err_limit, ok ? "PASS" : "FAIL");
std::fflush(stdout); // flush the output before MPI_Abort
}
#if PARCEL_MPI
if (!ok) { MPI_Abort(MPI_COMM_WORLD, 1); }
}
MPI_Finalize();
#else
if (!ok) { return 1; }
}
#endif
return 0;
}The SYCL queue must be an in-order queue (created with sycl::property::queue::in_order{}). PARCEL submits kernels to a single queue without sycl::event dependencies, and each kernel reads vectors written by the previous one, so it assumes execution in submission order. Passing a queue that is not in-order makes set_crsMat() and solver() of each solver throw std::invalid_argument.
The precision template arguments may be omitted; omitted precisions are double (parcel_double). The matrix, vectors and solver above can also be written as parcel::matrix<> A(&q);, parcel::vector<> x(n, &q);, parcel::krylov::pcg<> cg; and cg.solver(A, x, b); (“Precision Configurations”). When the threshold and the maximum number of iterations are not set, the defaults are 1.0e-8 and 10000.
1.3.2 Building and Running (Without MPI)
Compile directly with the SYCL compiler (acpp) with -DPARCEL_MPI=0 and run without mpirun.
acpp -O3 --acpp-targets=omp -march=native -DPARCEL_MPI=0 -I/usr/include/openblas \
-o quick_start examples/quick_start.cpp -lopenblas
OMP_NUM_THREADS=8 ./quick_start1.3.3 Building and Running (With MPI)
Set OMPI_CXX=acpp (Open MPI) so that the MPI wrapper (mpicxx) calls acpp, compile, and start with mpirun. PARCEL_MPI defaults to 1, so no option is needed.
export OMPI_CXX=acpp
mpicxx -O3 --acpp-targets=omp -march=native -I/usr/include/openblas \
-o quick_start examples/quick_start.cpp -lopenblas
OMP_NUM_THREADS=2 mpirun -n 4 ./quick_startWith the OpenMP backend, do not set the environment variable OMP_PROC_BIND. In some environments it gathers all threads onto one core and slows down significantly (the “Installing a SYCL Compiler” chapter).
1.3.4 Output
The first line is the convergence status printed by the library; the other three lines are printed by the program.
converged_reason = 1 (CONVERGED_RTOL) , its = 500 , residual = 1.161602e-15 , true_residual = 1.183245e-13
reason = 1 (CONVERGED_RTOL), iterations = 500, residual = 1.162e-15, true residual = 1.183e-13
x[0] = 500.000000 (exact 500.000000)
check: max relative error = 1.252e-15 (limit 1.0e-06) PASS
1.4 C
1.4.1 Program
From C, include capi/parcel_sycl_c.h and call parcel_solve_linear(). The settings go into the structure parcel_options (fill in the defaults with parcel_options_init() and change only what is needed), and the convergence status is returned in parcel_result. Integers such as row indices are the 64-bit parcel_int. Without MPI, the communicator argument (PARCEL_C_COMM) becomes int and its value is ignored. The source is examples/quick_start_c.c.
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include "parcel_sycl_c.h"
int main(int argc, char *argv[]) {
#if PARCEL_MPI
MPI_Init(&argc, &argv);
int myrank, nproc;
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
MPI_Comm_size(MPI_COMM_WORLD, &nproc);
PARCEL_C_COMM comm = MPI_COMM_WORLD;
#else
int myrank = 0, nproc = 1;
PARCEL_C_COMM comm = 0; /* unused without MPI */
#endif
/* row distribution: each rank owns contiguous rows (the remainder goes to the last rank) */
const parcel_int n_global = 1000;
const parcel_int nstart = (n_global / nproc) * myrank;
const parcel_int nend = (myrank == nproc - 1) ? n_global - 1 : nstart + n_global / nproc - 1;
const parcel_int n = nend - nstart + 1;
/* CRS of the owned rows (row pointers local 0-based, column indices global 0-based) */
parcel_int *rowptr = malloc((n + 1) * sizeof(parcel_int));
parcel_int *col = malloc(3 * n * sizeof(parcel_int));
double *val = malloc(3 * n * sizeof(double));
double *b = malloc(n * sizeof(double));
double *x = calloc(n, sizeof(double));
parcel_int nnz = 0;
rowptr[0] = 0;
for (parcel_int i = nstart; i <= nend; i++) {
if (i > 0) { col[nnz] = i - 1; val[nnz++] = -1.0; }
col[nnz] = i; val[nnz++] = 2.0;
if (i < n_global - 1) { col[nnz] = i + 1; val[nnz++] = -1.0; }
rowptr[i - nstart + 1] = nnz;
b[i - nstart] = 1.0;
}
/* conjugate gradient (no preconditioner, double) */
parcel_options opt;
parcel_options_init(&opt); /* fill in the defaults, then change what is needed */
opt.tolerance = 1.0e-9; /* relative residual ||r|| / ||b|| */
opt.max_iteration = 10000;
parcel_result res;
int rc = parcel_solve_linear(PARCEL_SOLVER_PCG, &opt, comm, n, nstart, n_global, nnz,
rowptr, col, val, b, x, &res);
/* comparison with the exact solution. The matrix is the difference -x[i-1] + 2 x[i] - x[i+1] = 1 with zero values outside the boundary (Dirichlet), so
* the exact solution is x[i] = (i + 1) (n_global - i) / 2 (i is the global 0-based index). The maximum relative error over all components is computed over the owned rows,
* and the maximum over the ranks is reduced. For the limit 1e-6 see quick_start.cpp (condition number about 4.1e5, tolerance 1e-9). */
const double err_limit = 1.0e-6;
double err_local = 0.0, err_max;
for (parcel_int k = 0; k < n; k++) {
const double i = (double)(nstart + k);
const double exact = 0.5 * (i + 1.0) * ((double)n_global - i);
double e = fabs(x[k] - exact) / exact;
if (!(e == e)) e = HUGE_VAL; /* treat NaN as infinity (fail) */
if (e > err_local) err_local = e;
}
err_max = err_local;
#if PARCEL_MPI
MPI_Allreduce(&err_local, &err_max, 1, MPI_DOUBLE, MPI_MAX, MPI_COMM_WORLD);
#endif
const int ok = (rc == PARCEL_OK && res.converged_reason > 0 && err_max <= err_limit);
if (myrank == 0) {
printf("rc = %d, reason = %d, iterations = %d, residual = %.3e, true residual = %.3e\n",
rc, res.converged_reason, res.iterations, res.residual, res.true_residual);
printf("x[0] = %.6f (exact %.6f)\n", x[0], 0.5 * (double)n_global);
printf("check: max relative error = %.3e (limit %.1e) %s\n", err_max, err_limit, ok ? "PASS" : "FAIL");
}
free(rowptr); free(col); free(val); free(b); free(x);
#if PARCEL_MPI
MPI_Finalize();
#endif
return ok ? 0 : 1;
}1.4.2 Building and Running (Without MPI)
Compile the implementation of the C API (the wrapper capi/wrapper_parcel_sycl.cpp) with the SYCL compiler, and link it with the C main program using the SYCL compiler. Give -DPARCEL_MPI=0 to both the wrapper and the main program (otherwise the types of the communicator argument do not match). acpp (clang) links as PIE, so add -fPIE to objects built with gcc.
acpp -O3 --acpp-targets=omp -march=native -DPARCEL_MPI=0 -I/usr/include/openblas \
-c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
gcc -O2 -fPIE -DPARCEL_MPI=0 -Icapi -c examples/quick_start_c.c -o quick_start_c.o
acpp --acpp-targets=omp -o quick_start_c quick_start_c.o wrapper_parcel_sycl.o -lopenblas
OMP_NUM_THREADS=8 ./quick_start_c1.4.3 Building and Running (With MPI)
Compile the wrapper with mpicxx (which calls acpp) and the main program with mpicc, and link with mpicxx.
export OMPI_CXX=acpp
mpicxx -O3 --acpp-targets=omp -march=native -I/usr/include/openblas \
-c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
mpicc -O2 -fPIE -Icapi -c examples/quick_start_c.c -o quick_start_c.o
mpicxx --acpp-targets=omp -o quick_start_c quick_start_c.o wrapper_parcel_sycl.o -lopenblas
OMP_NUM_THREADS=2 mpirun -n 4 ./quick_start_cWith the OpenMP backend, do not set the environment variable OMP_PROC_BIND. In some environments it gathers all threads onto one core and slows down significantly (the “Installing a SYCL Compiler” chapter).
1.4.4 Output
The first line is the convergence status printed by the library; the other three lines are printed by the program.
converged_reason = 1 (CONVERGED_RTOL) , its = 500 , residual = 1.161602e-15 , true_residual = 1.183245e-13
rc = 0, reason = 1, iterations = 500, residual = 1.162e-15, true residual = 1.183e-13
x[0] = 500.000000 (exact 500.000000)
check: max relative error = 1.252e-15 (limit 1.0e-06) PASS
1.5 Fortran
1.5.1 Program
From Fortran, use the module parcel_sycl_mod (capi/parcel_sycl_mod.f90) and call parcel_solve_linear(). The settings go into the derived type parcel_options (fill in the defaults with parcel_options_init and change only what is needed), and the convergence status is returned in parcel_result. This example sets index_base = 1 and passes the row pointers, column indices and the first owned row (nstart) 1-based, as usual in Fortran. Integers are integer(parcel_int_kind) (64 bits). Without MPI, use mpi is satisfied by a single-process substitute module (examples/sample_f_mpi_stub.f90), so the program does not change. The source is examples/quick_start_f.f90.
program quick_start_f
use mpi
use iso_c_binding
use parcel_sycl_mod
implicit none
integer(parcel_int_kind), parameter :: n_global = 1000
integer(parcel_int_kind) :: nstart, nend, n, nnz, i
integer(parcel_int_kind), allocatable :: rowptr(:), col(:)
real(c_double), allocatable :: val(:), b(:), x(:)
type(parcel_options) :: opt
type(parcel_result) :: res
integer :: myrank, nproc, ierr, rc
! limit of the comparison with the exact solution. See quick_start.cpp (condition number about 4.1e5, tolerance 1e-9)
real(c_double), parameter :: err_limit = 1.0d-6
real(c_double) :: xi, exact, e, err_local, err_max
logical :: ok
call MPI_Init(ierr)
call MPI_Comm_rank(MPI_COMM_WORLD, myrank, ierr)
call MPI_Comm_size(MPI_COMM_WORLD, nproc, ierr)
! row distribution: each rank owns contiguous rows (the remainder goes to the last rank); computed 0-based
nstart = (n_global / nproc) * myrank
if (myrank == nproc - 1) then
nend = n_global - 1
else
nend = nstart + n_global / nproc - 1
end if
n = nend - nstart + 1
! CRS of the owned rows (index_base = 1: row pointers, column indices and nstart are passed 1-based)
allocate(rowptr(n + 1), col(3 * n), val(3 * n), b(n), x(n))
nnz = 0
rowptr(1) = 1
do i = nstart, nend
if (i > 0) then
nnz = nnz + 1; col(nnz) = i; val(nnz) = -1.0d0 ! column i-1 (0-based) = i (1-based)
end if
nnz = nnz + 1; col(nnz) = i + 1; val(nnz) = 2.0d0
if (i < n_global - 1) then
nnz = nnz + 1; col(nnz) = i + 2; val(nnz) = -1.0d0
end if
rowptr(i - nstart + 2) = nnz + 1
end do
b = 1.0d0
x = 0.0d0
! conjugate gradient (no preconditioner, double)
call parcel_options_init(opt)
opt%index_base = 1
opt%tolerance = 1.0d-9
opt%max_iteration = 10000
rc = parcel_solve_linear(PARCEL_SOLVER_PCG, opt, MPI_COMM_WORLD, n, nstart + 1, n_global, nnz, &
rowptr, col, val, b, x, res)
! comparison with the exact solution. The matrix is the difference -x(i-1) + 2 x(i) - x(i+1) = 1 with zero values
! outside the boundary (Dirichlet), so the exact solution is x = (i + 1) (n_global - i) / 2 for the global 0-based
! index i. The maximum relative error over the owned rows is reduced over the ranks
err_local = 0.0d0
do i = nstart, nend
xi = dble(i)
exact = 0.5d0 * (xi + 1.0d0) * (dble(n_global) - xi)
e = abs(x(i - nstart + 1) - exact) / exact
if (e /= e) e = huge(e) ! treat NaN as the largest value (fail)
err_local = max(err_local, e)
end do
call MPI_Allreduce(err_local, err_max, 1, MPI_DOUBLE_PRECISION, MPI_MAX, MPI_COMM_WORLD, ierr)
ok = rc == PARCEL_OK .and. res%converged_reason > 0 .and. err_max <= err_limit
if (myrank == 0) then
write(*, '(a,i0,a,i0,a,i0,a,es10.3,a,es10.3)') 'rc = ', rc, ', reason = ', res%converged_reason, &
', iterations = ', res%iterations, ', residual = ', res%residual, ', true residual = ', res%true_residual
write(*, '(a,f12.6,a,f12.6,a)') 'x(1) = ', x(1), ' (exact ', 0.5d0 * dble(n_global), ')'
if (ok) then
write(*, '(a,es9.3,a,es7.1,a)') 'check: max relative error = ', err_max, ' (limit ', err_limit, ') PASS'
else
write(*, '(a,es9.3,a,es7.1,a)') 'check: max relative error = ', err_max, ' (limit ', err_limit, ') FAIL'
end if
end if
call MPI_Finalize(ierr)
if (.not. ok) stop 1
end program quick_start_f1.5.2 Building and Running (Without MPI)
Compile the implementation of the Fortran API (the wrapper capi/wrapper_parcel_sycl.cpp, written in C++) with the SYCL compiler and -DPARCEL_MPI=0, compile the substitute mpi module, the PARCEL module and the main program with gfortran in this order, and link them using the SYCL compiler. acpp (clang) links as PIE, so add -fPIE to objects built with gfortran. Add the Fortran runtime library (-lgfortran) when linking.
acpp -O3 --acpp-targets=omp -march=native -DPARCEL_MPI=0 -I/usr/include/openblas \
-c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
gfortran -O2 -fPIE -c examples/sample_f_mpi_stub.f90 -o sample_f_mpi_stub.o
gfortran -O2 -fPIE -c capi/parcel_sycl_mod.f90 -o parcel_sycl_mod.o
gfortran -O2 -fPIE -c examples/quick_start_f.f90 -o quick_start_f.o
acpp --acpp-targets=omp -o quick_start_f quick_start_f.o parcel_sycl_mod.o sample_f_mpi_stub.o \
wrapper_parcel_sycl.o -L$(dirname $(gfortran -print-file-name=libgfortran.so)) -lgfortran -lopenblas
OMP_NUM_THREADS=8 ./quick_start_f1.5.3 Building and Running (With MPI)
Compile the module and the main program with mpif90 and link with mpicxx, adding the MPI Fortran libraries and the Fortran runtime library (--showme:link is an Open MPI feature). With Fujitsu Fortran (frt, mpifrt), add -Kopenmp (Fortran in “Usage”). If the procedure without MPI was run in the same directory, first delete the mpi.mod made by the substitute module. Otherwise use mpi reads it instead of the real MPI module, and the link fails with __mpi_MOD_mpi_init and others not found.
export OMPI_CXX=acpp
rm -f mpi.mod # substitute module made by the procedure without MPI
mpicxx -O3 --acpp-targets=omp -march=native -I/usr/include/openblas \
-c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
mpif90 -O2 -fPIE -c capi/parcel_sycl_mod.f90 -o parcel_sycl_mod.o
mpif90 -O2 -fPIE -c examples/quick_start_f.f90 -o quick_start_f.o
mpicxx --acpp-targets=omp -o quick_start_f quick_start_f.o parcel_sycl_mod.o wrapper_parcel_sycl.o \
$(mpif90 --showme:link) -L$(dirname $(mpif90 -print-file-name=libgfortran.so)) -lgfortran -lopenblas
OMP_NUM_THREADS=2 mpirun -n 4 ./quick_start_fWith the OpenMP backend, do not set the environment variable OMP_PROC_BIND. In some environments it gathers all threads onto one core and slows down significantly (the “Installing a SYCL Compiler” chapter).
1.5.4 Output
The first line is the convergence status printed by the library; the other three lines are printed by the program.
converged_reason = 1 (CONVERGED_RTOL) , its = 500 , residual = 1.161602e-15 , true_residual = 1.183245e-13
rc = 0, reason = 1, iterations = 500, residual = 1.162E-15, true residual = 1.183E-13
x(1) = 500.000000 (exact 500.000000)
check: max relative error = 1.252E-15 (limit 1.0E-06) PASS
1.6 Python
1.6.1 Program
From Python, import the package parcel. The matrix is created from a SciPy CSR matrix with parcel.DistributedCSR and solved with parcel.solve(). The same script runs with both the version without MPI and the version with MPI (with MPI, from_global() extracts the rows owned by each rank). Only double precision is available. The source is examples/quick_start.py.
import numpy as np
import scipy.sparse as sp
import parcel
n_global = 1000
G = sp.diags([-1.0, 2.0, -1.0], [-1, 0, 1], shape=(n_global, n_global), format="csr")
A = parcel.DistributedCSR.from_global(G) # rows owned by this rank (all rows without MPI)
b = np.ones(A.n_local)
x, info = parcel.solve(A, b, method="pcg", tol=1e-9, maxiter=10000) # no preconditioner, double
# comparison with the exact solution. The matrix is the difference -x[i-1] + 2 x[i] - x[i+1] = 1 with zero values outside the boundary (Dirichlet), so
# the exact solution is x[i] = (i + 1) (n_global - i) / 2 (i is the global 0-based index). The maximum relative error over all components is computed over the owned rows,
# and the maximum over the ranks is reduced. For the limit 1e-6 see quick_start.cpp (condition number about 4.1e5, tolerance 1e-9)
err_limit = 1.0e-6
i = np.arange(A.row_start, A.row_start + A.n_local, dtype=float)
exact = 0.5 * (i + 1.0) * (n_global - i)
rel = np.abs(x - exact) / exact
rel[~np.isfinite(rel)] = np.inf # treat NaN as infinity (fail)
err_max = A.comm.allreduce(float(rel.max()), op="max")
ok = info.converged and err_max <= err_limit
if A.comm.rank == 0:
print(f"reason = {info.reason}, iterations = {info.iterations}, "
f"residual = {info.residual:.3e}, true residual = {info.true_residual:.3e}")
print(f"x[0] = {x[0]:.6f} (exact {0.5 * n_global:.6f})")
print(f"check: max relative error = {err_max:.3e} (limit {err_limit:.1e}) {'PASS' if ok else 'FAIL'}")
if not ok:
raise SystemExit(1)1.6.2 Installing and Running (Without MPI)
The package is installed by building its native layer from source for each environment. The default is the version without MPI. When installed with spack (previous section), it is available with spack load py-parcel-sycl~mpi.... To install with pip, run the following in python/ (details in “Python Package” in “Installation”).
python3 -m venv $HOME/venv_parcel && $HOME/venv_parcel/bin/python -m pip install -U pip
$HOME/venv_parcel/bin/python -m pip install scipy # used by the example script (parcel itself needs only NumPy)
cd python && $HOME/venv_parcel/bin/python -m pip install . && cd .. # default settings for CPU (omp)
OMP_NUM_THREADS=8 $HOME/venv_parcel/bin/python examples/quick_start.py1.6.3 Installing and Running (With MPI)
Install with the build setting PARCEL_MPI=1 and start with mpirun. All ranks run the same script.
export OMPI_CXX=acpp
python3 -m venv $HOME/venv_parcel_mpi && $HOME/venv_parcel_mpi/bin/python -m pip install -U pip scipy
cd python && $HOME/venv_parcel_mpi/bin/python -m pip install . -C cmake.define.PARCEL_MPI=1 && cd ..
OMP_NUM_THREADS=2 mpirun -n 4 $HOME/venv_parcel_mpi/bin/python examples/quick_start.pyWith the OpenMP backend, do not set the environment variable OMP_PROC_BIND. In some environments it gathers all threads onto one core and slows down significantly (the “Installing a SYCL Compiler” chapter).
1.6.4 Output
The Python package does not print the progress output of the library (rank 0 prints with print in the script).
reason = converged, iterations = 500, residual = 1.162e-15, true residual = 1.183e-13
x[0] = 500.000000 (exact 500.000000)
check: max relative error = 1.252e-15 (limit 1.0e-06) PASS
1.7 Where to Go Next
| Task | Section |
|---|---|
| Decide which solver and preconditioner to use | The “Choosing a Solver” chapter |
| Add a preconditioner | The “Preconditioners” chapter, set_precon_algo() |
| Lower the precision for speed (mixed precision) | “Specifying Precision” in the C++ section of “Usage” |
| Run on a GPU | “Running on GPUs” |
| Pass the matrix of your own problem | “Passing Your Own Matrix” |
| Run in jobs on compute nodes | “Running in Jobs” |
| Compute eigenvalues | The “Eigenvalue Solvers” chapter, examples/example_pcg_ks.cpp |
| Call from C / Fortran | C / Fortran in “Usage” |
| Call from Python | Python in “Usage” |
| Investigate non-convergence | “Common Member Functions” in the API Reference (get_converged_reason() and others), The “Numerical Accuracy and Reproducibility” chapter |