PARCEL SYCL v1.2 / User Guide User Guide API Reference Search Download 日本語

1 Quick Start

This chapter shows, with a minimal program that solves the linear system Ax=bAx = b for the finite-difference matrix of the one-dimensional Laplacian (a tridiagonal matrix) with the conjugate gradient method, the steps from installing a SYCL compiler to building and running. The same problem is solved in four languages (C++, C, Fortran and Python), each without MPI (one process) and with MPI (several processes). Names of machines and devices (s86, V100, etc.) and abbreviations are collected in the Glossary.

1.1 Overview of the Program

The problem is as follows. The programs in all four languages solve it and print the same results.

The right-hand side bb is symmetric under reversal (bi=bn−1−ib_i = b_{n-1-i}) and has components only along n/2n/2 eigenvectors of AA, so without rounding errors the conjugate gradient method reaches the exact solution xi=(i+1)(n−i)/2x_i = (i+1)(n-i)/2 (x0=n/2=500x_0 = n/2 = 500) in exactly n/2=500n/2 = 500 iterations. Every program prints the convergence status, the iteration count, the residuals and x0x_0, computes the maximum over all components of the relative error |xi−(i+1)(n−i)/2|/((i+1)(n−i)/2)|x_i - (i+1)(n-i)/2| / ((i+1)(n-i)/2) against the exact solution, and prints check: ... PASS if it is at most 1e-6 (with MPI, the maximum over the rows owned by each rank is reduced across the ranks). If the solver does not converge or the error exceeds the limit, it prints FAIL and exits with status 1. The limit is much smaller than the worst-case estimate 4e-4 from the condition number (about 4.1e5) and the tolerance 1e-9, but the actual error is at the rounding level (0 to 1.3e-15), and a wrong solution gives an error larger by orders of magnitude, so the two are distinguished. To build and run the examples in all four languages and get a pass/fail summary, use bash examples/run_quick_checks.sh (Installation, “Checking After the Build”).

The flow is the same in every language.

  1. Distribute the rows over the ranks. Each rank owns a contiguous block of rows. Without MPI, one process owns all rows.
  2. Build the owned rows of the coefficient matrix in CRS format (row pointers, column indices, values). Column indices are global.
  3. Set the solver (threshold, maximum iterations) and solve. Settings not given take the library defaults.
  4. Check the convergence status.

The programs hardly differ between without and with MPI. With MPI, each rank builds only its own rows, and all ranks call the same function with the same settings (a collective operation). Without MPI, one process owns everything. Which version is built is selected by the build setting PARCEL_MPI (“Default of MPI” in “Installation”).

Requirements:

1.2 Installing a SYCL Compiler

PARCEL is a header-only library compiled with a SYCL compiler. v1.2 uses AdaptiveCpp.

Choose the installation route by environment as follows.

Environment Route Notes
Test environments of this manual (s86, Odyssey, Aquarius, Miyabi-C, Miyabi-G, PS-A, PS-B) install_stack.sh (from scratch), or AdaptiveCpp into an existing spack The environment definitions spack/stack/envs/<env>/ are included. The procedure for an existing spack is in each section of the “Installing a SYCL Compiler” chapter
Other Linux systems (RHEL family, Ubuntu family, Windows WSL2) install_stack.sh with the generic template, or AdaptiveCpp into spack Confirmed on Ubuntu 24.04 and Rocky Linux 9 (both on WSL2). “Generic Linux”
macOS Homebrew “macOS (Homebrew)”
Python only pip (pip install . in python/), or py-parcel-sycl of spack Install AdaptiveCpp and LAPACKE first by one of the routes above. “Python Package”

The main routes are used as follows.

bash spack/stack/install_stack.sh s86 $HOME/parcel_stack 16     # <env> <prefix> [jobs]
. $HOME/parcel_stack/setup.sh                                   # when using it (every time)
spack load parcel-sycl~mpi~cuda~rocm py-parcel-sycl~mpi~cuda~rocm      # CPU version without MPI
brew install adaptivecpp open-mpi openblas gcc numpy scipy

The examples below assume a Linux environment (s86, etc.) with CPU execution (the OpenMP backend of AdaptiveCpp) and LAPACKE in /usr/include/openblas, and are run in the top directory of the package. Environment variables and modules differ by environment, so first apply the “Settings for use” of each environment in the “Installing a SYCL Compiler” chapter. The flags for macOS + Homebrew are in “macOS (Homebrew)” in “Installation”. CPU executables use the LLVM OpenMP runtime (libomp.so), but acpp does not embed its location in the executable. The “Settings for use” of the test environments resolve this by adding the LLVM lib/ to LD_LIBRARY_PATH. In your own environment, add the LLVM lib/ to LD_LIBRARY_PATH at run time or add -Wl,-rpath,<LLVM lib> at link time (without either, the OS libomp is used or the program fails to start). Also give the location of the Fortran runtime library (libgfortran) with -L (the examples find it with gfortran -print-file-name). For NVIDIA GPUs, change the target, e.g. --acpp-targets=cuda:sm_70; the program itself does not change. On V100 (sm_70), adding -Xcuda-ptxas -maxrregcount=255 is also recommended (Recommended Options).

1.2.1 Environments Whose Standard Compiler Is Not GCC

The examples below compile the C and Fortran main programs with gcc and gfortran (or mpicc and mpif90 of Open MPI). The same procedure works where the standard compilers are not GCC. The combinations used in the test environments are shown below; the detailed flags for each environment are in “Recommended Options” in “Installation”.

Standard compiler (test environment) C++ (PARCEL itself) C and Fortran main MPI wrappers Notes
GCC (most Linux systems such as s86) acpp (clang of LLVM 18) gcc, gfortran mpicxx, mpicc, mpif90 of Open MPI (OMPI_CXX=acpp) —
Fujitsu compilers (Odyssey) acpp --acpp-targets=omp.library-only; Fujitsu FCC (clang mode, given by ACPP_CPU_CXX) compiles everything including the kernels Fujitsu fcc, frt mpiFCC (OMPI_CXX=acpp), mpifcc, mpifrt Add -Kopenmp to fcc and frt (“Fortran” in “Usage”). The login and compute nodes have different architectures, so build on a compute node (“Odyssey”)
Intel oneAPI (PS-A) acpp (clang of LLVM 18; used after loading the site modules intel and intelmpi) gcc, gfortran (mpicc, mpif90 of Intel MPI) mpicxx of Intel MPI (I_MPI_CXX=acpp) Get the MPI Fortran libraries with mpif90 -show (“PS-A”)
Apple clang (macOS) acpp --acpp-targets=omp.library-only; Apple clang compiles everything clang (C), gfortran (Homebrew gcc) mpicxx, mpicc, mpif90 of Open MPI (Homebrew; OMPI_CXX=acpp) Add blocking of the reductions (-D_BLOCKING_LOOP_DOT_=1); -march=native is not needed (it makes no difference in speed); launch with mpirun --bind-to none (“macOS (Homebrew)” in “Installation”)

The thread counts in the examples below match the macOS + Homebrew machine on which the outputs were obtained (Apple M4, 10 cores): 8 threads without MPI, and 4 ranks × 2 threads with MPI. On your machine, choose them so that ranks × threads does not exceed the number of cores. The outputs were obtained with these two settings, and the printed lines were the same. The residuals change in the last digits with the environment and the number of processes (for example, the true residual is 2.49e-14 with 4 ranks on s86 CPU), but the iteration count 500 and x0=500x_0 = 500 do not. The per-iteration output and the time breakdown are omitted.

1.3 C++

1.3.1 Program

The source is examples/quick_start.cpp. Without and with MPI differ only in the four #if PARCEL_MPI blocks.

#include <algorithm>
#include <cmath>
#include <cstdio>
#include <vector>
#include "../Include/parcel_sycl.hpp"

int main(int argc, char *argv[]) {
#if PARCEL_MPI
  MPI_Init(&argc, &argv);
#else
  (void)argc; (void)argv;
#endif
  {
    // default device (determined by --acpp-targets); use an in-order queue
    //   (PARCEL submits kernels in order without events; solvers throw if the queue is not in-order)
    sycl::queue q{sycl::default_selector_v, sycl::property::queue::in_order{}};
#if PARCEL_MPI
    parcel::comm comm(MPI_COMM_WORLD, &q);
#else
    parcel::comm comm(&q);                          // without MPI: rank 0, one process
#endif
    const int myrank = comm.mpi_rank(), nproc = comm.mpi_size();

    // row distribution: each rank owns contiguous rows (the remainder goes to the last rank)
    const PARCEL_INT n_global = 1000;
    const PARCEL_INT nstart = (n_global / nproc) * myrank;
    const PARCEL_INT nend   = (myrank == nproc - 1) ? n_global - 1 : nstart + n_global / nproc - 1;
    const PARCEL_INT n      = nend - nstart + 1;

    // CRS of the owned rows (row pointers local 0-based, column indices global 0-based)
    std::vector<PARCEL_INT> rowptr(n + 1, 0), col;
    std::vector<double> val;
    for (PARCEL_INT i = nstart; i <= nend; i++) {
      if (i > 0)            { col.push_back(i - 1); val.push_back(-1.0); }
      col.push_back(i);       val.push_back( 2.0);
      if (i < n_global - 1) { col.push_back(i + 1); val.push_back(-1.0); }
      rowptr[i - nstart + 1] = (PARCEL_INT)col.size();
    }
    parcel::matrix<parcel_double> A(&q);
    A.set_crsMat<double>(&comm, nstart, nend, n, col.size(), rowptr.data(), col.data(), val.data());

    std::vector<double> x_h(n, 0.0), b_h(n, 1.0);
    parcel::vector<parcel_double> x(n, &q), b(n, &q);
    x.setValue(x_h.data());
    b.setValue(b_h.data());

    // conjugate gradient (all arithmetic in double)
    parcel::krylov::pcg<parcel_double, parcel_double, parcel_double, parcel_double> cg;
    cg.set_tolerance(1.0e-9);                       // relative residual ||r|| / ||b||
    cg.set_max_iteration(10000);
    q.wait();
    cg.solver<parcel_double, parcel_double, parcel_double,
              parcel_double, parcel_double, parcel_double>(A, x, b);
    x.getValue(x_h.data());
    q.wait();

    // comparison with the exact solution. The matrix is the difference -x[i-1] + 2 x[i] - x[i+1] = 1 with zero values outside the boundary (Dirichlet), so
    //   the exact solution is x[i] = (i + 1) (n_global - i) / 2 (i is the global 0-based index; x[0] = n_global / 2, maximum about n_global^2 / 8 at the center).
    //   The maximum of the relative error |x[i] - exact| / exact over all components is computed over the owned rows of each rank and reduced over the ranks.
    // limit 1e-6: the condition number is cot^2(pi / (2 (n_global + 1))) = about 4.1e5, and the worst-case estimate from tolerance 1e-9
    //   (relative error <= condition number x relative residual) is 4e-4. The actual error is far smaller: b = 1 is symmetric about the center, so the Krylov subspace
    //   is exhausted at dimension 500, pcg reaches an almost zero residual in 500 iterations, and the error is at the rounding level (0 to 1.3e-15 on s86 CPU and GPU, 1 and 2 ranks).
    //   A wrong solution (matrix, distribution or communication error) gives an error larger by orders of magnitude, so 1e-6 distinguishes them.
    const double err_limit = 1.0e-6;
    double err_local = 0.0;
    for (PARCEL_INT k = 0; k < n; k++) {
      const double i = (double)(nstart + k);
      const double exact = 0.5 * (i + 1.0) * ((double)n_global - i);
      double e = std::fabs(x_h[k] - exact) / exact;
      if (!(e == e)) e = HUGE_VAL;                  // treat NaN as infinity (fail)
      err_local = std::max(err_local, e);
    }
    double err_max = err_local;
#if PARCEL_MPI
    MPI_Allreduce(&err_local, &err_max, 1, MPI_DOUBLE, MPI_MAX, MPI_COMM_WORLD);
#endif
    const bool ok = cg.is_converged() && err_max <= err_limit;

    // convergence status and check (same on all ranks)
    if (myrank == 0) {
      std::printf("reason = %d (%s), iterations = %d, residual = %.3e, true residual = %.3e\n",
                  (int)cg.get_converged_reason(),
                  parcel::krylov::converged_reason_name(cg.get_converged_reason()),
                  (int)cg.get_iterations(), cg.get_residual(), cg.get_true_residual());
      std::printf("x[0] = %.6f (exact %.6f)\n", x_h[0], 0.5 * 1.0 * (double)n_global);
      std::printf("check: max relative error = %.3e (limit %.1e) %s\n", err_max, err_limit, ok ? "PASS" : "FAIL");
      std::fflush(stdout);                          // flush the output before MPI_Abort
    }
#if PARCEL_MPI
    if (!ok) { MPI_Abort(MPI_COMM_WORLD, 1); }
  }
  MPI_Finalize();
#else
    if (!ok) { return 1; }
  }
#endif
  return 0;
}

The SYCL queue must be an in-order queue (created with sycl::property::queue::in_order{}). PARCEL submits kernels to a single queue without sycl::event dependencies, and each kernel reads vectors written by the previous one, so it assumes execution in submission order. Passing a queue that is not in-order makes set_crsMat() and solver() of each solver throw std::invalid_argument.

The precision template arguments may be omitted; omitted precisions are double (parcel_double). The matrix, vectors and solver above can also be written as parcel::matrix<> A(&q);, parcel::vector<> x(n, &q);, parcel::krylov::pcg<> cg; and cg.solver(A, x, b); (“Precision Configurations”). When the threshold and the maximum number of iterations are not set, the defaults are 1.0e-8 and 10000.

1.3.2 Building and Running (Without MPI)

Compile directly with the SYCL compiler (acpp) with -DPARCEL_MPI=0 and run without mpirun.

acpp -O3 --acpp-targets=omp -march=native -DPARCEL_MPI=0 -I/usr/include/openblas \
     -o quick_start examples/quick_start.cpp -lopenblas
OMP_NUM_THREADS=8 ./quick_start

1.3.3 Building and Running (With MPI)

Set OMPI_CXX=acpp (Open MPI) so that the MPI wrapper (mpicxx) calls acpp, compile, and start with mpirun. PARCEL_MPI defaults to 1, so no option is needed.

export OMPI_CXX=acpp
mpicxx -O3 --acpp-targets=omp -march=native -I/usr/include/openblas \
       -o quick_start examples/quick_start.cpp -lopenblas
OMP_NUM_THREADS=2 mpirun -n 4 ./quick_start

With the OpenMP backend, do not set the environment variable OMP_PROC_BIND. In some environments it gathers all threads onto one core and slows down significantly (the “Installing a SYCL Compiler” chapter).

1.3.4 Output

The first line is the convergence status printed by the library; the other three lines are printed by the program.

 converged_reason = 1 (CONVERGED_RTOL) , its = 500 , residual = 1.161602e-15 , true_residual = 1.183245e-13
reason = 1 (CONVERGED_RTOL), iterations = 500, residual = 1.162e-15, true residual = 1.183e-13
x[0] = 500.000000 (exact 500.000000)
check: max relative error = 1.252e-15 (limit 1.0e-06) PASS

1.4 C

1.4.1 Program

From C, include capi/parcel_sycl_c.h and call parcel_solve_linear(). The settings go into the structure parcel_options (fill in the defaults with parcel_options_init() and change only what is needed), and the convergence status is returned in parcel_result. Integers such as row indices are the 64-bit parcel_int. Without MPI, the communicator argument (PARCEL_C_COMM) becomes int and its value is ignored. The source is examples/quick_start_c.c.

#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include "parcel_sycl_c.h"

int main(int argc, char *argv[]) {
#if PARCEL_MPI
  MPI_Init(&argc, &argv);
  int myrank, nproc;
  MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
  MPI_Comm_size(MPI_COMM_WORLD, &nproc);
  PARCEL_C_COMM comm = MPI_COMM_WORLD;
#else
  int myrank = 0, nproc = 1;
  PARCEL_C_COMM comm = 0;                 /* unused without MPI */
#endif
  /* row distribution: each rank owns contiguous rows (the remainder goes to the last rank) */
  const parcel_int n_global = 1000;
  const parcel_int nstart = (n_global / nproc) * myrank;
  const parcel_int nend   = (myrank == nproc - 1) ? n_global - 1 : nstart + n_global / nproc - 1;
  const parcel_int n      = nend - nstart + 1;

  /* CRS of the owned rows (row pointers local 0-based, column indices global 0-based) */
  parcel_int *rowptr = malloc((n + 1) * sizeof(parcel_int));
  parcel_int *col    = malloc(3 * n * sizeof(parcel_int));
  double *val = malloc(3 * n * sizeof(double));
  double *b   = malloc(n * sizeof(double));
  double *x   = calloc(n, sizeof(double));
  parcel_int nnz = 0;
  rowptr[0] = 0;
  for (parcel_int i = nstart; i <= nend; i++) {
    if (i > 0)            { col[nnz] = i - 1; val[nnz++] = -1.0; }
    col[nnz] = i;           val[nnz++] =  2.0;
    if (i < n_global - 1) { col[nnz] = i + 1; val[nnz++] = -1.0; }
    rowptr[i - nstart + 1] = nnz;
    b[i - nstart] = 1.0;
  }

  /* conjugate gradient (no preconditioner, double) */
  parcel_options opt;
  parcel_options_init(&opt);              /* fill in the defaults, then change what is needed */
  opt.tolerance     = 1.0e-9;             /* relative residual ||r|| / ||b|| */
  opt.max_iteration = 10000;
  parcel_result res;
  int rc = parcel_solve_linear(PARCEL_SOLVER_PCG, &opt, comm, n, nstart, n_global, nnz,
                               rowptr, col, val, b, x, &res);

  /* comparison with the exact solution. The matrix is the difference -x[i-1] + 2 x[i] - x[i+1] = 1 with zero values outside the boundary (Dirichlet), so
   *   the exact solution is x[i] = (i + 1) (n_global - i) / 2 (i is the global 0-based index). The maximum relative error over all components is computed over the owned rows,
   *   and the maximum over the ranks is reduced. For the limit 1e-6 see quick_start.cpp (condition number about 4.1e5, tolerance 1e-9). */
  const double err_limit = 1.0e-6;
  double err_local = 0.0, err_max;
  for (parcel_int k = 0; k < n; k++) {
    const double i = (double)(nstart + k);
    const double exact = 0.5 * (i + 1.0) * ((double)n_global - i);
    double e = fabs(x[k] - exact) / exact;
    if (!(e == e)) e = HUGE_VAL;          /* treat NaN as infinity (fail) */
    if (e > err_local) err_local = e;
  }
  err_max = err_local;
#if PARCEL_MPI
  MPI_Allreduce(&err_local, &err_max, 1, MPI_DOUBLE, MPI_MAX, MPI_COMM_WORLD);
#endif
  const int ok = (rc == PARCEL_OK && res.converged_reason > 0 && err_max <= err_limit);
  if (myrank == 0) {
    printf("rc = %d, reason = %d, iterations = %d, residual = %.3e, true residual = %.3e\n",
           rc, res.converged_reason, res.iterations, res.residual, res.true_residual);
    printf("x[0] = %.6f (exact %.6f)\n", x[0], 0.5 * (double)n_global);
    printf("check: max relative error = %.3e (limit %.1e) %s\n", err_max, err_limit, ok ? "PASS" : "FAIL");
  }
  free(rowptr); free(col); free(val); free(b); free(x);
#if PARCEL_MPI
  MPI_Finalize();
#endif
  return ok ? 0 : 1;
}

1.4.2 Building and Running (Without MPI)

Compile the implementation of the C API (the wrapper capi/wrapper_parcel_sycl.cpp) with the SYCL compiler, and link it with the C main program using the SYCL compiler. Give -DPARCEL_MPI=0 to both the wrapper and the main program (otherwise the types of the communicator argument do not match). acpp (clang) links as PIE, so add -fPIE to objects built with gcc.

acpp -O3 --acpp-targets=omp -march=native -DPARCEL_MPI=0 -I/usr/include/openblas \
     -c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
gcc  -O2 -fPIE -DPARCEL_MPI=0 -Icapi -c examples/quick_start_c.c -o quick_start_c.o
acpp --acpp-targets=omp -o quick_start_c quick_start_c.o wrapper_parcel_sycl.o -lopenblas
OMP_NUM_THREADS=8 ./quick_start_c

1.4.3 Building and Running (With MPI)

Compile the wrapper with mpicxx (which calls acpp) and the main program with mpicc, and link with mpicxx.

export OMPI_CXX=acpp
mpicxx -O3 --acpp-targets=omp -march=native -I/usr/include/openblas \
       -c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
mpicc  -O2 -fPIE -Icapi -c examples/quick_start_c.c -o quick_start_c.o
mpicxx --acpp-targets=omp -o quick_start_c quick_start_c.o wrapper_parcel_sycl.o -lopenblas
OMP_NUM_THREADS=2 mpirun -n 4 ./quick_start_c

With the OpenMP backend, do not set the environment variable OMP_PROC_BIND. In some environments it gathers all threads onto one core and slows down significantly (the “Installing a SYCL Compiler” chapter).

1.4.4 Output

The first line is the convergence status printed by the library; the other three lines are printed by the program.

 converged_reason = 1 (CONVERGED_RTOL) , its = 500 , residual = 1.161602e-15 , true_residual = 1.183245e-13
rc = 0, reason = 1, iterations = 500, residual = 1.162e-15, true residual = 1.183e-13
x[0] = 500.000000 (exact 500.000000)
check: max relative error = 1.252e-15 (limit 1.0e-06) PASS

1.5 Fortran

1.5.1 Program

From Fortran, use the module parcel_sycl_mod (capi/parcel_sycl_mod.f90) and call parcel_solve_linear(). The settings go into the derived type parcel_options (fill in the defaults with parcel_options_init and change only what is needed), and the convergence status is returned in parcel_result. This example sets index_base = 1 and passes the row pointers, column indices and the first owned row (nstart) 1-based, as usual in Fortran. Integers are integer(parcel_int_kind) (64 bits). Without MPI, use mpi is satisfied by a single-process substitute module (examples/sample_f_mpi_stub.f90), so the program does not change. The source is examples/quick_start_f.f90.

program quick_start_f
  use mpi
  use iso_c_binding
  use parcel_sycl_mod
  implicit none
  integer(parcel_int_kind), parameter :: n_global = 1000
  integer(parcel_int_kind) :: nstart, nend, n, nnz, i
  integer(parcel_int_kind), allocatable :: rowptr(:), col(:)
  real(c_double), allocatable :: val(:), b(:), x(:)
  type(parcel_options) :: opt
  type(parcel_result)  :: res
  integer :: myrank, nproc, ierr, rc
  ! limit of the comparison with the exact solution. See quick_start.cpp (condition number about 4.1e5, tolerance 1e-9)
  real(c_double), parameter :: err_limit = 1.0d-6
  real(c_double) :: xi, exact, e, err_local, err_max
  logical :: ok

  call MPI_Init(ierr)
  call MPI_Comm_rank(MPI_COMM_WORLD, myrank, ierr)
  call MPI_Comm_size(MPI_COMM_WORLD, nproc, ierr)

  ! row distribution: each rank owns contiguous rows (the remainder goes to the last rank); computed 0-based
  nstart = (n_global / nproc) * myrank
  if (myrank == nproc - 1) then
     nend = n_global - 1
  else
     nend = nstart + n_global / nproc - 1
  end if
  n = nend - nstart + 1

  ! CRS of the owned rows (index_base = 1: row pointers, column indices and nstart are passed 1-based)
  allocate(rowptr(n + 1), col(3 * n), val(3 * n), b(n), x(n))
  nnz = 0
  rowptr(1) = 1
  do i = nstart, nend
     if (i > 0) then
        nnz = nnz + 1; col(nnz) = i;     val(nnz) = -1.0d0     ! column i-1 (0-based) = i (1-based)
     end if
     nnz = nnz + 1;    col(nnz) = i + 1; val(nnz) =  2.0d0
     if (i < n_global - 1) then
        nnz = nnz + 1; col(nnz) = i + 2; val(nnz) = -1.0d0
     end if
     rowptr(i - nstart + 2) = nnz + 1
  end do
  b = 1.0d0
  x = 0.0d0

  ! conjugate gradient (no preconditioner, double)
  call parcel_options_init(opt)
  opt%index_base    = 1
  opt%tolerance     = 1.0d-9
  opt%max_iteration = 10000
  rc = parcel_solve_linear(PARCEL_SOLVER_PCG, opt, MPI_COMM_WORLD, n, nstart + 1, n_global, nnz, &
       rowptr, col, val, b, x, res)

  ! comparison with the exact solution. The matrix is the difference -x(i-1) + 2 x(i) - x(i+1) = 1 with zero values
  !   outside the boundary (Dirichlet), so the exact solution is x = (i + 1) (n_global - i) / 2 for the global 0-based
  !   index i. The maximum relative error over the owned rows is reduced over the ranks
  err_local = 0.0d0
  do i = nstart, nend
     xi = dble(i)
     exact = 0.5d0 * (xi + 1.0d0) * (dble(n_global) - xi)
     e = abs(x(i - nstart + 1) - exact) / exact
     if (e /= e) e = huge(e)                ! treat NaN as the largest value (fail)
     err_local = max(err_local, e)
  end do
  call MPI_Allreduce(err_local, err_max, 1, MPI_DOUBLE_PRECISION, MPI_MAX, MPI_COMM_WORLD, ierr)
  ok = rc == PARCEL_OK .and. res%converged_reason > 0 .and. err_max <= err_limit
  if (myrank == 0) then
     write(*, '(a,i0,a,i0,a,i0,a,es10.3,a,es10.3)') 'rc = ', rc, ', reason = ', res%converged_reason, &
          ', iterations = ', res%iterations, ', residual = ', res%residual, ', true residual = ', res%true_residual
     write(*, '(a,f12.6,a,f12.6,a)') 'x(1) = ', x(1), ' (exact ', 0.5d0 * dble(n_global), ')'
     if (ok) then
        write(*, '(a,es9.3,a,es7.1,a)') 'check: max relative error = ', err_max, ' (limit ', err_limit, ') PASS'
     else
        write(*, '(a,es9.3,a,es7.1,a)') 'check: max relative error = ', err_max, ' (limit ', err_limit, ') FAIL'
     end if
  end if
  call MPI_Finalize(ierr)
  if (.not. ok) stop 1
end program quick_start_f

1.5.2 Building and Running (Without MPI)

Compile the implementation of the Fortran API (the wrapper capi/wrapper_parcel_sycl.cpp, written in C++) with the SYCL compiler and -DPARCEL_MPI=0, compile the substitute mpi module, the PARCEL module and the main program with gfortran in this order, and link them using the SYCL compiler. acpp (clang) links as PIE, so add -fPIE to objects built with gfortran. Add the Fortran runtime library (-lgfortran) when linking.

acpp -O3 --acpp-targets=omp -march=native -DPARCEL_MPI=0 -I/usr/include/openblas \
     -c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
gfortran -O2 -fPIE -c examples/sample_f_mpi_stub.f90 -o sample_f_mpi_stub.o
gfortran -O2 -fPIE -c capi/parcel_sycl_mod.f90 -o parcel_sycl_mod.o
gfortran -O2 -fPIE -c examples/quick_start_f.f90 -o quick_start_f.o
acpp --acpp-targets=omp -o quick_start_f quick_start_f.o parcel_sycl_mod.o sample_f_mpi_stub.o \
     wrapper_parcel_sycl.o -L$(dirname $(gfortran -print-file-name=libgfortran.so)) -lgfortran -lopenblas
OMP_NUM_THREADS=8 ./quick_start_f

1.5.3 Building and Running (With MPI)

Compile the module and the main program with mpif90 and link with mpicxx, adding the MPI Fortran libraries and the Fortran runtime library (--showme:link is an Open MPI feature). With Fujitsu Fortran (frt, mpifrt), add -Kopenmp (Fortran in “Usage”). If the procedure without MPI was run in the same directory, first delete the mpi.mod made by the substitute module. Otherwise use mpi reads it instead of the real MPI module, and the link fails with __mpi_MOD_mpi_init and others not found.

export OMPI_CXX=acpp
rm -f mpi.mod                     # substitute module made by the procedure without MPI
mpicxx -O3 --acpp-targets=omp -march=native -I/usr/include/openblas \
       -c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
mpif90 -O2 -fPIE -c capi/parcel_sycl_mod.f90 -o parcel_sycl_mod.o
mpif90 -O2 -fPIE -c examples/quick_start_f.f90 -o quick_start_f.o
mpicxx --acpp-targets=omp -o quick_start_f quick_start_f.o parcel_sycl_mod.o wrapper_parcel_sycl.o \
       $(mpif90 --showme:link) -L$(dirname $(mpif90 -print-file-name=libgfortran.so)) -lgfortran -lopenblas
OMP_NUM_THREADS=2 mpirun -n 4 ./quick_start_f

With the OpenMP backend, do not set the environment variable OMP_PROC_BIND. In some environments it gathers all threads onto one core and slows down significantly (the “Installing a SYCL Compiler” chapter).

1.5.4 Output

The first line is the convergence status printed by the library; the other three lines are printed by the program.

 converged_reason = 1 (CONVERGED_RTOL) , its = 500 , residual = 1.161602e-15 , true_residual = 1.183245e-13
rc = 0, reason = 1, iterations = 500, residual =  1.162E-15, true residual =  1.183E-13
x(1) =   500.000000 (exact   500.000000)
check: max relative error = 1.252E-15 (limit 1.0E-06) PASS

1.6 Python

1.6.1 Program

From Python, import the package parcel. The matrix is created from a SciPy CSR matrix with parcel.DistributedCSR and solved with parcel.solve(). The same script runs with both the version without MPI and the version with MPI (with MPI, from_global() extracts the rows owned by each rank). Only double precision is available. The source is examples/quick_start.py.

import numpy as np
import scipy.sparse as sp
import parcel

n_global = 1000
G = sp.diags([-1.0, 2.0, -1.0], [-1, 0, 1], shape=(n_global, n_global), format="csr")
A = parcel.DistributedCSR.from_global(G)          # rows owned by this rank (all rows without MPI)
b = np.ones(A.n_local)
x, info = parcel.solve(A, b, method="pcg", tol=1e-9, maxiter=10000)   # no preconditioner, double

# comparison with the exact solution. The matrix is the difference -x[i-1] + 2 x[i] - x[i+1] = 1 with zero values outside the boundary (Dirichlet), so
#   the exact solution is x[i] = (i + 1) (n_global - i) / 2 (i is the global 0-based index). The maximum relative error over all components is computed over the owned rows,
#   and the maximum over the ranks is reduced. For the limit 1e-6 see quick_start.cpp (condition number about 4.1e5, tolerance 1e-9)
err_limit = 1.0e-6
i = np.arange(A.row_start, A.row_start + A.n_local, dtype=float)
exact = 0.5 * (i + 1.0) * (n_global - i)
rel = np.abs(x - exact) / exact
rel[~np.isfinite(rel)] = np.inf                   # treat NaN as infinity (fail)
err_max = A.comm.allreduce(float(rel.max()), op="max")
ok = info.converged and err_max <= err_limit
if A.comm.rank == 0:
    print(f"reason = {info.reason}, iterations = {info.iterations}, "
          f"residual = {info.residual:.3e}, true residual = {info.true_residual:.3e}")
    print(f"x[0] = {x[0]:.6f} (exact {0.5 * n_global:.6f})")
    print(f"check: max relative error = {err_max:.3e} (limit {err_limit:.1e}) {'PASS' if ok else 'FAIL'}")
if not ok:
    raise SystemExit(1)

1.6.2 Installing and Running (Without MPI)

The package is installed by building its native layer from source for each environment. The default is the version without MPI. When installed with spack (previous section), it is available with spack load py-parcel-sycl~mpi.... To install with pip, run the following in python/ (details in “Python Package” in “Installation”).

python3 -m venv $HOME/venv_parcel && $HOME/venv_parcel/bin/python -m pip install -U pip
$HOME/venv_parcel/bin/python -m pip install scipy                       # used by the example script (parcel itself needs only NumPy)
cd python && $HOME/venv_parcel/bin/python -m pip install . && cd ..     # default settings for CPU (omp)
OMP_NUM_THREADS=8 $HOME/venv_parcel/bin/python examples/quick_start.py

1.6.3 Installing and Running (With MPI)

Install with the build setting PARCEL_MPI=1 and start with mpirun. All ranks run the same script.

export OMPI_CXX=acpp
python3 -m venv $HOME/venv_parcel_mpi && $HOME/venv_parcel_mpi/bin/python -m pip install -U pip scipy
cd python && $HOME/venv_parcel_mpi/bin/python -m pip install . -C cmake.define.PARCEL_MPI=1 && cd ..
OMP_NUM_THREADS=2 mpirun -n 4 $HOME/venv_parcel_mpi/bin/python examples/quick_start.py

With the OpenMP backend, do not set the environment variable OMP_PROC_BIND. In some environments it gathers all threads onto one core and slows down significantly (the “Installing a SYCL Compiler” chapter).

1.6.4 Output

The Python package does not print the progress output of the library (rank 0 prints with print in the script).

reason = converged, iterations = 500, residual = 1.162e-15, true residual = 1.183e-13
x[0] = 500.000000 (exact 500.000000)
check: max relative error = 1.252e-15 (limit 1.0e-06) PASS

1.7 Where to Go Next

Task Section
Decide which solver and preconditioner to use The “Choosing a Solver” chapter
Add a preconditioner The “Preconditioners” chapter, set_precon_algo()
Lower the precision for speed (mixed precision) “Specifying Precision” in the C++ section of “Usage”
Run on a GPU “Running on GPUs”
Pass the matrix of your own problem “Passing Your Own Matrix”
Run in jobs on compute nodes “Running in Jobs”
Compute eigenvalues The “Eigenvalue Solvers” chapter, examples/example_pcg_ks.cpp
Call from C / Fortran C / Fortran in “Usage”
Call from Python Python in “Usage”
Investigate non-convergence “Common Member Functions” in the API Reference (get_converged_reason() and others), The “Numerical Accuracy and Reproducibility” chapter