4 Installation
This chapter describes how to install SYCL PARCEL.
PARCEL consists of header files only. It can be used simply by including ./Include/parcel_sycl.hpp and compiling with a SYCL-capable compiler. No prebuilding or installation as a library is required.
Note, however, the following three points.
- In C++, C and Fortran, MPI is used by default (
PARCEL_MPIdefaults to 1). The SYCL compiler (acpp) is invoked through an MPI wrapper (mpicxx, etc.). A single-process version without MPI can be built with-DPARCEL_MPI=0. Python and spack default to no MPI (“Default of MPI”, “Build without MPI”). - The eigenvalue solver
ks_arnoldi(Include/parcel_krylov_ks.hpp) includeslapacke.hand calls LAPACKE. Since all solvers are enabled by default, the LAPACKE header and library must be specified even when only linear systems are solved. Excludingks_arnoldiwith-DPARCEL_ENABLE_KS_ARNOLDI=0removes this requirement (“Selecting the Solvers to Build”). - The
#includedirectives in the source use relative paths. Compile the sample programs in the PARCEL top-level directory.
For installing a SYCL-capable compiler, see the previous chapter. Installing PARCEL together with its prerequisites with spack is described in “Installing with spack”, and the Python package in “Python Package”.
4.1 Package Layout
The layout of the distribution package (PARCEL_SYCL v1.2) is shown below. The library itself consists only of the headers in Include/; the rest are samples and tests.
| Location | Contents |
|---|---|
Include/ |
The library (headers only). Include parcel_sycl.hpp |
capi/ |
C / Fortran API: parcel_sycl_c.h (C declarations), parcel_sycl_mod.f90 (Fortran module), wrapper_parcel_sycl.cpp (implementation; build with a SYCL compiler), parcel_capi_default.hpp (compile-time defaults of the wrapper: internal precisions and the settings of the legacy API) |
examples/ |
quick_start.cpp, quick_start_c.c, quick_start_f.f90, quick_start.py (the C++, C, Fortran and Python examples of the Quick Start), example_pcg_ks.cpp (pcg and ks_arnoldi), sample_c.c / sample_f.f90 (call and verify all solvers through the C / Fortran API), sample_f_mpi_stub.f90 (single-process mpi module for building sample_f.f90 and quick_start_f.f90 without MPI), problem.f90 (matrix generation for the legacy API), build_examples.sh (builds all of the above; compilers, target and LAPACK are set with environment variables), run_quick_checks.sh (checking after the build; “Checking After the Build”) |
python/ |
The Python package parcel (“Python Package” in “Installation”, Python in “Usage”): parcel/ (Python layer), src/ (source of the native layer libparcel_py.so), envs/mac_brew.sh (settings for building on macOS + Homebrew), pyproject.toml / CMakeLists.txt (build settings for pip) |
spack/ |
spack recipes parcel-sycl / py-parcel-sycl (spack_repo/parcel/) and the scripts that install everything from fetching spack to PARCEL (stack/install_stack.sh, stack/check_stack.sh for checking, per-environment definitions stack/envs/<env>/) (“Installing with spack”) |
tests/verify/ |
The verification sample parcel_krylov_sample_verify.cpp, its defaults parcel_default.hpp, the memory report (diag_mem.hpp; PARCEL_MEM_REPORT), and the test matrix generators (Poisson, convection-diffusion, unstructured) |
tests/regression/ |
Generator of the regression-test job scripts (gen_regression.py), submission and summary scripts, the double-double check (ddcheck), the Python API test (sample_py.py), and a script that runs the regression tests on machines without a scheduler (run_local.sh) |
tests/perf/ |
Generators of the performance-test job scripts (gen_perf.py, and gen_perf_prec.py for mixed precision) and summary scripts |
tests/site_config.py |
Per-environment settings of the tests (queues, modules, compile flags, accounts). Edit it for your environment |
tests/check_version.py |
Version check (no build needed). Checks that the files that copy the version (the Python package, the spack recipes and the parcel-sycl@ specs of the environment definitions, CITATION.cff) agree with Include/parcel_version.hpp (“Checking the Version”) |
tools/make_tarball.py |
Script that creates the distribution tarball and its sha256 (“Verifying the Distribution”) |
README.md, CHANGELOG.md |
Overview and changes in v1.2 |
CITATION.cff, SUPPORT.md |
Citation information and how to get support (“Citation and Support”) |
doc/ |
HTML of the User Guide and the API Reference of this manual (Japanese and English; entry doc/index.html, English doc/index_en.html; open it directly in a web browser) |
COPYING, COPYING.LESSER |
License (LGPL-3.0) |
The descriptive md files (README.md, CHANGELOG.md, SUPPORT.md, spack/README.md, python/README.md, tests/*/README.md) are in English, with a Japanese version in the same place (README.ja.md etc., .ja before .md in the name).
How to run the tests is described in tests/regression/README.md and tests/perf/README.md. The regression tests consist of the verification sample (C++) and the tests of the C, Fortran and Python APIs (examples/sample_c.c, examples/sample_f.f90, tests/regression/sample_py.py); with or without MPI and the languages can be selected for generation, build, submission and summary. Besides the test environments of this manual, the bundled settings (tests/site_config.py) include examples for environments installed from scratch with spack (s86spack_cpu, s86spack_gpu, etc.) and an example for macOS + Homebrew without a scheduler (mac).
4.2 Checking After the Build
To check a build, use bash examples/run_quick_checks.sh. It builds the four Quick Start examples (C++, C, Fortran and Python) with examples/build_examples.sh, runs them as one process, and with an MPI build also runs the C++ example with two processes. Each example prints check: ... PASS if the maximum relative error against the exact solution is at most 1e-6 (“Program Overview”). At the end, the script prints PASS n / FAIL m (SKIP k) and the elapsed time on one line, and exits with status 0 if nothing failed. The Python example is SKIPPED (not failed) if parcel (with NumPy and SciPy) cannot be imported.
The build (build_examples.sh quick) runs the SYCL compilations (the C++ example and the wrapper of the C / Fortran API) at the same time (the number of concurrent compilations is BUILD_JOBS; the default is the smaller of the number of targets and the number of cores). After a successful build, the build settings (SYCL_TARGET, CXXFLAGS, PARCEL_MPI, the compilers and their locations, etc.) and the cksum of the sources (headers, API, examples) are recorded in examples/build/build_stamp.txt. From the second run on, if the executables exist and the record matches the current state, the build is skipped and only the runs are performed (less than a minute). If a setting or a source has changed, the differing items are shown and the examples are rebuilt. --rebuild always rebuilds.
export OMPI_CXX=acpp
bash examples/run_quick_checks.sh # with MPI. last line: === PASS 6 / FAIL 0 (SKIP 0) 所要 <sec> s
PARCEL_MPI=0 bash examples/run_quick_checks.sh # without MPI
bash examples/run_quick_checks.sh --rebuild # rebuild
NO_BUILD=1 bash examples/run_quick_checks.sh # run examples/build/ as is, without checking the recordThe environment variables for the compilers, target and LAPACK are the same as for build_examples.sh (for a GPU, e.g. SYCL_TARGET=--acpp-targets=cuda:sm_70). The MPI launcher is set with MPIEXEC (default mpirun), MPIEXEC_NP_FLAG (default -n), MPI_NP (default 2) and MPIEXEC_ARGS. For MPI libraries that cannot run MPI_Init without the launcher (e.g. Fujitsu MPI), MPIEXEC_ALL=1 starts the single-process runs through the launcher as well. Fujitsu mpiexec writes the standard output of the ranks to the job log, where the check lines cannot be read, so with Fujitsu mpiexec (MPIEXEC=mpiexec whose real file is under /opt/FJSV…) RC_ONLY=1 (pass if the exit status is 0) is the default (use RC_ONLY=0 to turn it off). When installed with spack, spack test run parcel-sycl py-parcel-sycl performs the same check (“Installing with spack”).
Number of threads and where it runs. If OMP_NUM_THREADS is not set, all available logical CPUs are used on a compute node (inside a job), and 4 on a login node (or the number available, if fewer than 4). For the MPI run this is divided by the number of processes so that the total stays the same (at least 1; 2 processes on a login node get 2 threads each). The number of available logical CPUs is the smallest of the CPU affinity (nproc), the CPU limit of the cgroup, and the job’s allocation on this node (PBS NCPUS, Slurm SLURM_CPUS_ON_NODE). Whether it is a compute node is decided by the presence of a job environment variable (PBS PBS_JOBID, Fujitsu PJM PJM_JOBID, Slurm SLURM_JOB_ID), and can be forced with QUICK_NODE=compute / QUICK_NODE=login. The decision and the values used are printed first. If OMP_NUM_THREADS is set, its value is used (in a job the scheduler may set it: PBS ompthreads (a queue default of 1 is possible), Fujitsu PJM --omp thread=). With Open MPI, --bind-to none is added unless MPIEXEC_ARGS specifies a placement (--bind-to, --map-by) (with 2 or fewer processes Open MPI binds each process to one core, which would pack several threads onto one core).
Where to run it. In CPU environments, a login node is fine (follow the rules of the site). In GPU environments and on A64FX (Odyssey), run it on a compute node (a short batch job or an interactive job): login nodes have no GPU, and A64FX executables do not run on the x86 login nodes. When the MPI run (2 processes) is done in a GPU environment, the script does not assign one GPU per process (as gpu_bind.sh in “Mapping One Rank to One Device” does), so both processes may use the same GPU.
4.3 Compile Options
This section describes the flags specified at compile time. The -D macros of the library itself (Include/) are described in the section Macros. The run-time options of the samples (environment variables PARCEL_SOLVER, PARCEL_NX, PARCEL_PRECON, etc.) are described in the “Usage” chapter.
Flags required in all environments
| Flag | Description |
|---|---|
--acpp-targets=<target> |
Kind of code to generate. omp (CPU, OpenMP), omp.library-only (A64FX + Fujitsu compiler), cuda:sm_70 (V100), cuda:sm_80 (A100), cuda:sm_90 (H100), hip:gfx942 (MI300A) |
-I<location of lapacke.h> |
Include path for lapacke.h. Example: -I/usr/include/openblas |
-l<LAPACK library> |
Library containing LAPACKE. Examples: -lopenblas, Fujitsu -lfjlapack, on PS -l:libopenblaso.so.0 |
-O3 |
Optimization. Specified in all environments |
Environment-dependent flags
| Flag | Environments | Description |
|---|---|---|
-march=native -mtune=native |
s86 CPU/GPU, Miyabi-C, Miyabi-G, PS-A | Optimization for the host CPU |
-Xcuda-ptxas -maxrregcount=255 |
s86 GPU (V100) | Raises the register limit of ptxas (the CUDA assembler). Recommended on V100. With the default allocation, the loads of the matrix and vectors in kernels such as the Jacobi preconditioner are split and serialized, which is slower. No spills occur and the results do not change. Not used on other GPUs (Register Limit on V100) |
-std=c++17 |
Odyssey | Specified for builds through the Fujitsu compiler |
-DNDEBUG -fno-math-errno -fno-trapping-math |
PS-B | Only floating-point flags that do not change values are used. -ffast-math is not used because it breaks the error terms of double-double and the detection of NaN (the build stops with #error); -fno-exceptions is not used because it is incompatible with the exceptions of the argument checks |
4.4 Macros
Since PARCEL is a header-only library, the configuration and behavior of the library (Include/) can be selected by adding -D<macro> or -D<macro>=<value> when compiling the user program. Every macro works with its default value, and normally nothing needs to be specified. There are the following four kinds by purpose.
- Selecting the solvers to build: exclude unused solvers to shorten compilation
- Integer type: select the widths of the global indices (64-bit by default) and the rank-local indices (32-bit by default)
- Optimization: select the loop structure of the kernels for each environment, and, since the defaults reduce memory, optimization options that use more memory for speed (Optimization)
- Checks and numerical operations: checks that prevent wrong results and the method of double-double operations (keep the defaults; Checks and Numerical Operations)
In addition, there are macros for diagnosis and isolation, which are not specified in normal builds.
4.4.1 Selecting the Solvers to Build (Include/parcel_solver_select.hpp)
All solvers are enabled by default, and normally nothing needs to be specified. Because all solvers go into a single translation unit, compilation takes a long time (6–14 minutes per configuration on A64FX); when the solvers to be used are known, for example during troubleshooting, disabling the others shortens the compilation.
Each macro builds the solver when set to 1 and excludes it when set to 0. A solver macro that is not specified takes the value of PARCEL_ENABLE_ALL_SOLVERS (default 1), so all solvers are built when nothing is specified. To build only some solvers, set PARCEL_ENABLE_ALL_SOLVERS=0 and set the solvers to be built to 1.
| Macro | Default | Description |
|---|---|---|
PARCEL_ENABLE_ALL_SOLVERS |
1 | Value used for each solver macro below when it is not specified. If set to 0, only the solvers set to 1 are built |
PARCEL_ENABLE_PCG |
1 (the value of PARCEL_ENABLE_ALL_SOLVERS) |
parcel::krylov::pcg |
PARCEL_ENABLE_PBICGSTAB |
1 (the value of PARCEL_ENABLE_ALL_SOLVERS) |
parcel::krylov::pbicgstab |
PARCEL_ENABLE_PGMRES |
1 (the value of PARCEL_ENABLE_ALL_SOLVERS) |
parcel::krylov::pgmres |
PARCEL_ENABLE_PCAGMRES |
1 (the value of PARCEL_ENABLE_ALL_SOLVERS) |
parcel::krylov::pcagmres (including the variant that does not form , CAGMRES_Q_IMPLICIT) |
PARCEL_ENABLE_POWER |
1 (the value of PARCEL_ENABLE_ALL_SOLVERS) |
parcel::krylov::power |
PARCEL_ENABLE_INVERSE_POWER |
1 (the value of PARCEL_ENABLE_ALL_SOLVERS) |
parcel::krylov::inverse_power |
PARCEL_ENABLE_KS_ARNOLDI |
1 (the value of PARCEL_ENABLE_ALL_SOLVERS) |
parcel::krylov::ks_arnoldi. If set to 0, LAPACKE is no longer required (except for the verification sample) |
Dependencies are resolved automatically: enabling inverse_power enables pcg and pgmres, and enabling ks_arnoldi enables pgmres. If no solver is enabled, a compile error occurs. There is no macro named PARCEL_ENABLE_CHOLQR_PCAGMRES; specifying it causes a compile error (#error).
# build only pgmres (exclude all others)
-DPARCEL_ENABLE_ALL_SOLVERS=0 -DPARCEL_ENABLE_PGMRES=1
# exclude only ks_arnoldi (build all others)
-DPARCEL_ENABLE_KS_ARNOLDI=04.4.2 Integer Type
Integers such as row indices, column indices and numbers of nonzeros are held in the type of the macro PARCEL_INT for global indices, and in the type of the macro PARCEL_LINT for indices within one rank (such as the CRS on the device). By default, PARCEL_INT is 64 bits (int64_t; MPI_INT64_T in MPI communication) and PARCEL_LINT is 32 bits (int32_t, MPI_INT32_T). The reasons for these defaults (integer overflow in large problems and the speed of the local kernels) are given in “Integer Type” in the Basic Structure chapter.
| Specification | PARCEL_INT |
PARCEL_LINT |
Notes |
|---|---|---|---|
| None (default) | int64_t |
int32_t |
|
-DPARCEL_LOCAL_INT64 |
int64_t |
int64_t |
-DPARCEL_LOCAL_INT64=0 is the same as the default |
-DPARCEL_INT32 |
int32_t |
int32_t |
-DPARCEL_INT32=0 is the same as the default |
-DPARCEL_INT |
int64_t |
int32_t |
Former way to make PARCEL_INT 64-bit. Accepted for compatibility (its value is not used) |
Both -DPARCEL_INT and -DPARCEL_INT32 |
— | — | Contradictory; stopped with #error |
Both -DPARCEL_INT32 and -DPARCEL_LOCAL_INT64 |
— | — | The rank-local type would be wider than the global type; stopped with #error |
The selected widths are available as the macros PARCEL_INT_BITS and PARCEL_LINT_BITS (64 or 32 each).
How to choose the type (the range of matrices, integer overflow in large problems, the effect on memory and on the speed of the local kernels, and the relation to the C and Fortran APIs) is described in “Integer Type” in the Basic Structure chapter.
4.4.3 Optimization
The macros of optimizations that increase speed without changing the computation (the algorithm and the number of iterations) are listed below: those that select the loop form of the kernels (the first eight) and those that make the solvers faster by using more memory (the last two; optimization options). How each optimization works, its effects and the defaults per environment are described in the “Optimization” chapter, and the compile commands per environment in Recommended Options. The defaults reduce memory.
| Macro | Default | Description |
|---|---|---|
_BLOCKING_LOOP_ |
0 | If 1, replaces all kernels except SpMV (dot products, axpy, preconditioners, etc.) with blocked versions (Blocking). For A64FX |
_BLOCKING_LOOP_DOT_ |
0 | If 1, replaces only the reductions (dot, dddot, max) with blocked versions. For CPUs other than A64FX. Do not specify on GPUs |
PARCEL_SPMV_LADDER |
0 | If 1, splits the inner loop of SpMV in a ladder (Ladder Splitting). For AMD GPUs and A64FX. Not used on NVIDIA GPUs |
PARCEL_PRECON_LADDER |
0 (the value of PARCEL_SPMV_LADDER) |
Ladder splitting of the iteration loops of the Jacobi preconditioner. Set explicitly to 0 on A64FX |
PARCEL_GEMV_AS_AXPY |
0 (the value of _BLOCKING_LOOP_) |
If 1, decomposes gemv into M axpy operations (Decomposing gemv). Changes the rounding order |
PARCEL_A64FX_AXPY_C |
1 | If 1, uses the versions of the axpy that takes coefficients by value and of the axpy decomposition of gemv rewritten for A64FX (C in Optimizations for A64FX). 0 gives the form before the rewrite. The results do not change. Effective only in _BLOCKING_LOOP_ builds |
PARCEL_CG_FUSED |
0 (the value of _BLOCKING_LOOP_) |
If 1, fuses the kernels of one iteration of pcg (the updates of and , the update of and ; D in Optimizations for A64FX). The results do not change. Can be 1 only in _BLOCKING_LOOP_ builds |
PARCEL_JACOBI_NODIV_UNIT |
1 | If 1, skips the division in the preparation of the right-hand side of the Jacobi preconditioner when the divisor is 1 (E in Optimizations for A64FX). The results do not change. Effective only in _BLOCKING_LOOP_ builds |
PARCEL_JACOBI_ROW2 |
0 (the value of _BLOCKING_LOOP_; 0 if PARCEL_PRECON_LADDER is 1) |
If 1, processes the Jacobi sweep two rows at a time alternately when the preconditioner matrix is single precision (F in Optimizations for A64FX). The results do not change. Not used for a double-precision preconditioner matrix or with on-the-fly computation |
PARCEL_JACOBI_ONTHEFLY |
1 (0 for A64FX) | The Jacobi preconditioner (PRECON_JACOBI) does not keep the values of and computes them from the values of and at every application. With 0 the values are stored (optimization option “without H”: uses the memory of the number of nonzeros of × size of the preconditioner matrix precision). With matrix scaling, a double-double coefficient matrix, preconditioner matrix or preconditioner computation precision, the values are stored even with 1. In builds for A64FX, where _BLOCKING_LOOP_ (or _BLOCKING_LOOP_SPMV_) is 1, the default is 0 (Choosing Between Memory and Speed) |
PARCEL_PCG_NO_XHIST |
1 | Default of the number of iterations (pstep) over which pcg batches the solution update. 1 gives 1 (one search direction, updated at every iteration); 0 gives 4 (optimization option “without G”: uses the memory of 3 vectors with halo). The run-time environment variable PARCEL_PSTEP takes precedence |
4.4.4 Default Orthogonalization of ks_arnoldi
The orthogonalization of the Arnoldi process of ks_arnoldi (Krylov-Schur method in the eigenvalue solver chapter) is selected in the program with set_qr_algo() (set_qr_algo() of ks_arnoldi in the API Reference). The following macro selects at build time the default used when it is not called. The environment variable PARCEL_KS_ORTHO is not read (if it is set, rank 0 prints a one-line warning to standard error at the first solver(); the behavior does not change).
| Macro | Default | Description |
|---|---|---|
PARCEL_KS_ORTHO_DEFAULT |
undefined (cgs2) |
cgs2 (classical Gram-Schmidt twice) or mgs (modified Gram-Schmidt twice). The build for PS-A (the spack environment psa, variant ks_ortho=mgs) uses mgs. When undefined, CGS2 (QR_CGS). A value given with set_qr_algo() takes precedence |
4.4.5 Checks and Numerical Operations
These macros select checks that prevent wrong results and the method of double-double operations. Keep the defaults; removing the checks can silently return wrong results, so specify them only when you understand the reasons.
| Macro | Default | Description |
|---|---|---|
PARCEL_REQUIRE_IN_ORDER |
1 | At the entry of set_crsMat() and of solver() of each solver, checks that the SYCL queue is in-order, and throws std::invalid_argument on violation (including when there is no queue). PARCEL submits kernels to one queue in order without sycl::event or depends_on, and each kernel reads the results of the previous one, so in-order execution is assumed. With the SYCL 2020 default (out-of-order) queue, the execution order of USM kernels that use no accessors is not guaranteed. 0 disables the check |
PARCEL_NO_CRS_CHECK |
0 | 1 disables the input checks of set_crsMat() (the normal version and the hi/lo version for double-double). The checks are: the local number of rows n equals nend-nstart+1 and is at least one (a rank with zero rows is not supported); rowptr[0] == 0, it is non-decreasing, and rowptr[n] == nnz; the row blocks of all ranks cover all rows without gaps or overlaps (in any rank order); and the column indices are within the global number of rows. The non-MPI version checks only rowptr and the column indices. On violation all ranks throw the same std::invalid_argument (the message names the offending rank and what is wrong). The cost is one pass over the nonzeros and two collective communications. If the checks are disabled and a broken input is given, the result is silently wrong because uninitialized receive buffers of the halo exchange are used, or memory outside the arrays is read |
PARCEL_ALLOW_FAST_MATH |
Undefined | If defined, allows builds with -ffast-math / -ffinite-math-only (__FAST_MATH__ / __FINITE_MATH_ONLY__). By default parcel_sycl.hpp stops with #error. Rewriting expressions by associativity eliminates the error terms of double-double, and dd silently falls to double precision. In addition, the NaN / Inf checks (std::isfinite) are removed and DIVERGED_NANORINF no longer works. If defined, the precision of dd and the detection of NaN are not guaranteed |
PARCEL_DD_NO_FMA |
Undefined | If defined, replaces the error-free transformation of double-double products (TwoProd) by fma (, , the default) with Dekker splitting. For CPUs without FMA instructions. Do not use it on GPUs (CUDA), where Dekker splitting breaks |
Do not use -fno-exceptions either. It is incompatible with the argument checks (std::invalid_argument, etc.).
4.5 Recommended Options
The recommended flags for each environment are shown below. Blocking of reductions is specified for CPUs other than A64FX, blocking of all kernels, ladder splitting of SpMV, and fusion of multiply-adds into FMA (-ffp-contract=fast) for A64FX, together with OMP_WAIT_POLICY=ACTIVE and KMP_BLOCKTIME=infinite at run time (Optimizations for A64FX), and ladder splitting for AMD GPUs. NVIDIA GPUs do not use ladder splitting, and V100 uses the register limit.
| Environment | --acpp-targets |
Kernel optimization | LAPACKE |
|---|---|---|---|
| s86 CPU | omp |
-D_BLOCKING_LOOP_DOT_=1 |
-I/usr/include/openblas -lopenblas |
| s86 GPU | cuda:sm_70 |
None (no ladder splitting) + -Xcuda-ptxas -maxrregcount=255 |
-I/usr/include/openblas -lopenblas |
| Odyssey | omp.library-only |
-D_BLOCKING_LOOP_=1 -DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=0 + -ffp-contract=fast |
-I/usr/include/openblas -lfjlapack |
| Aquarius | cuda:sm_80 |
None | -I/usr/include/openblas -lopenblas |
| Miyabi-C | omp |
-D_BLOCKING_LOOP_DOT_=1 |
-I/usr/include/openblas -lopenblas |
| Miyabi-G | cuda:sm_90 |
None | -I/usr/include/openblas -lopenblas |
| PS-A | omp |
-D_BLOCKING_LOOP_DOT_=1 |
lapacke.h from nvhpc + -l:libopenblaso.so.0 |
| PS-B | hip:gfx942 |
-DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=1 |
lapacke.h from nvhpc + -l:libopenblaso.so.0 |
The mechanisms of the optimizations and their measured effects per environment are described in the “Optimization” chapter.
The compile commands for each environment are shown below. The environment is set up with “Settings for use” of each environment in the chapter “Installing a SYCL Compiler”. Each is an example of building the verification sample tests/verify/parcel_krylov_sample_verify.cpp in double precision in the top-level directory of the package. The solver, problem size, and preconditioner are specified at run time (Run-Time Options of the Verification Sample). Without -DMAX_ITERATION, the maximum number of iterations is 10000 (the default in tests/verify/parcel_default.hpp, the same as the library default), and it can be changed at run time with PARCEL_MAX_ITER. Compilation uses neither MPI nor GPUs, so it is done on the login node except on A64FX. Replace <spack installation directory> and similar placeholders according to your environment.
4.5.1 s86 (CPU)
# Environment: apply "Settings for use" of "s86" in the chapter "Installing a SYCL Compiler"
mpicxx -O3 --acpp-targets=omp -march=native -mtune=native \
-D_BLOCKING_LOOP_DOT_=1 \
-I/usr/include/openblas \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas4.5.2 s86 (GPU)
# Environment: apply "Settings for use" of "s86" in the chapter "Installing a SYCL Compiler"
mpicxx -O3 --acpp-targets=cuda:sm_70 -march=native -mtune=native \
-Xcuda-ptxas -maxrregcount=255 \
-I/usr/include/openblas \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas4.5.3 Odyssey
On A64FX, the login nodes and compute nodes have different architectures, so the build is done on a compute node. An example build job script is shown below. The MPI wrapper mpiFCC is used, and OMPI_CXX=acpp makes it invoke acpp. With the omp.library-only target, acpp uses the Fujitsu compiler given in ACPP_CPU_CXX as the host compiler. As in the other environments, the location of the LAPACKE header is specified with -I/usr/include/openblas (with this, the build works on the compute node), and Fujitsu’s libfjlapack is used as the implementation.
#!/bin/bash
#PJM -L "rscgrp=short-o"
#PJM -L "node=1"
#PJM -L "elapse=2:00:00"
#PJM -g <group>
#PJM --omp thread=48
#PJM -j
# Environment: apply "Settings for use" of "Odyssey" in the chapter "Installing a SYCL Compiler"
cd ${PJM_O_WORKDIR}
LINKOPT="-L$fj_root/lib64 -Wl,-rpath,$fj_root/lib64 \
-L$sycl_root/lib -Wl,-rpath,$sycl_root/lib \
-L$llvm_root/lib -Wl,-rpath,$llvm_root/lib -lfjlapack"
mpiFCC -O3 -std=c++17 --acpp-targets=omp.library-only \
-D_BLOCKING_LOOP_=1 -DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=0 \
-ffp-contract=fast \
-I/usr/include/openblas \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp ${LINKOPT}A single build takes longer than on Xeon and needs several GiB of memory. The memory available to a single-node job is 28 GiB, so limit the number of concurrent builds when building several precision configurations at the same time.
-ffp-contract=fast makes the compiler fuse multiply-adds () into FMA instructions. The clang mode of the Fujitsu compiler does not fuse them by default. AdaptiveCpp passes the command-line -O3 to the Fujitsu compiler after ACPP_OMP_CXX_FLAGS, so the -Kfast in ACPP_OMP_CXX_FLAGS is canceled by the later -O3 and does not take effect. If fast-math is enabled, for example by passing -Kfast without a later -O3, double-double and the NaN detection break, so PARCEL stops the build with #error (“Do Not Use -ffast-math”). -ffp-contract=fast on the command line reaches the Fujitsu compiler. On one node of Odyssey (4 ranks × 12 threads, NX=256, pcg + Jacobi preconditioner), one iteration became 3 % faster (SpMV 10 %, the body of axpy 25 %). The number of iterations does not change, and the results change in the last digits (about 1e-11 relative in the residual). FMA adds without rounding the product, so the results are not bit-identical to a build without the flag. Use -ffp-contract=off if the results must agree. Inside the double-double functions, Include/ddlib.hpp still prohibits FMA (last paragraph of this section).
In builds for A64FX (-D_BLOCKING_LOOP_=1), the Jacobi preconditioner stores the values of by default (“Without H” in Choosing Between Memory and Speed; the default of PARCEL_JACOBI_ONTHEFLY is 0, so nothing needs to be specified). On A64FX the on-the-fly computation (a multiplication and a diagonal check per nonzero) shows up in the preconditioner time: on one node of Odyssey (NX=256, pcg + Jacobi preconditioner), storing the values makes the preconditioner section 0.77 times and the total time to convergence 0.81 times. In exchange, the memory grows by the size of the preconditioner matrix precision per nonzero of (8 bytes in double precision). To reduce memory, specify -DPARCEL_JACOBI_ONTHEFLY=1 explicitly.
At run time, set OMP_WAIT_POLICY=ACTIVE and KMP_BLOCKTIME=infinite, which keep the OpenMP threads spinning while they wait (recommended; this is not a library default, so set it in the job environment). On one node of Odyssey (4 ranks × 12 threads, one rank per CMG), the fixed cost per iteration that does not depend on the problem size drops from 5.2 ms to 3.0 ms. It helps when the number of rows per device is small. Waiting threads keep using their cores, so it is counterproductive when processes compete for cores (for example, other processes sharing the node). Together with the rewritten axpy and the fused pcg kernels that are enabled in this build, the effects and caveats are described in Optimizations for A64FX.
In code before 2026-10-10, the prohibition of FMA for the double-double arithmetic (#pragma clang fp contract(off)) did not end at the end of ddlib.hpp with the Fujitsu compiler, so no FMA instructions were used in any of the following kernels (the clang mode of the Fujitsu compiler does not know the push / pop of float_control). The current code limits the prohibition to the double-double functions, and the other kernels use FMA as the compiler options say (-ffp-contract=fast above). Results in double precision and other precisions differ from earlier builds in the last digits.
4.5.4 Aquarius
# Environment: apply "Settings for use" of "Aquarius" in the chapter "Installing a SYCL Compiler"
mpicxx -O3 --acpp-targets=cuda:sm_80 \
-I/usr/include/openblas \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas4.5.5 Miyabi-C
# Environment: apply "Settings for use" of "Miyabi-C" in the chapter "Installing a SYCL Compiler"
mpicxx -O3 --acpp-targets=omp -march=native -mtune=native \
-D_BLOCKING_LOOP_DOT_=1 \
-I/usr/include/openblas \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas4.5.6 Miyabi-G
# Environment: apply "Settings for use" of "Miyabi-G" in the chapter "Installing a SYCL Compiler"
mpicxx -O3 --acpp-targets=cuda:sm_90 -march=native -mtune=native \
-I/usr/include/openblas \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas4.5.7 PS-A
The PS compute nodes do not have /usr/include/openblas/lapacke.h (it exists only on the login nodes). In addition, MKL provides only mkl_lapacke.h. Therefore, the lapacke.h shipped with nvhpc is borrowed as the header, and the implementation is resolved to the OS OpenBLAS (the OpenMP version libopenblaso). Set OPENBLAS_NUM_THREADS=1 so that OpenBLAS threads are not nested inside PARCEL’s OpenMP threads. Since the MPI is Intel MPI, acpp is also specified with I_MPI_CXX.
# Environment: apply "Settings for use" of "PS-A" in the chapter "Installing a SYCL Compiler"
mpicxx -O3 --acpp-targets=omp -march=native -mtune=native \
-D_BLOCKING_LOOP_DOT_=1 \
-I/system/apps/rhel9/dev/nvhpc/25.3/Linux_x86_64/25.3/compilers/include/lp64 \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp -l:libopenblaso.so.04.5.8 PS-B
LAPACKE is handled in the same way as on PS-A.
# Environment: apply "Settings for use" of "PS-B" in the chapter "Installing a SYCL Compiler"
mpicxx -O3 --acpp-targets=hip:gfx942 \
-DNDEBUG -fno-math-errno -fno-trapping-math \
-DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=1 \
-I/system/apps/rhel9/dev/nvhpc/25.3/Linux_x86_64/25.3/compilers/include/lp64 \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp -l:libopenblaso.so.04.5.9 C and Fortran Programs
Programs that use the C or Fortran API compile the wrapper capi/wrapper_parcel_sycl.cpp with the same flags as the mpicxx command of each environment above (--acpp-targets, kernel optimization, LAPACKE), and link it with the C or Fortran main program using the SYCL compiler. Use the same --acpp-targets for compiling and linking the wrapper. acpp (clang) links as PIE, so add -fPIE to objects built with gcc or gfortran. An example for s86 CPU (Open MPI) follows; in other environments, replace the flags on the first line by those of the environment.
# Environment: apply "Settings for use" of "s86" in the chapter "Installing a SYCL Compiler"
FLAGS="-O3 --acpp-targets=omp -march=native -mtune=native -D_BLOCKING_LOOP_DOT_=1"
mpicxx ${FLAGS} -I/usr/include/openblas -c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
# C
mpicc -O2 -fPIE -Icapi -c prog.c -o prog.o
mpicxx --acpp-targets=omp -o prog prog.o wrapper_parcel_sycl.o -lopenblas -lm
# Fortran (compile the PARCEL module first)
mpif90 -O2 -fPIE -c capi/parcel_sycl_mod.f90 -o parcel_sycl_mod.o
mpif90 -O2 -fPIE -c prog.f90 -o prog.o
mpicxx --acpp-targets=omp -o prog prog.o parcel_sycl_mod.o wrapper_parcel_sycl.o \
$(mpif90 --showme:link) -L$(dirname $(mpif90 -print-file-name=libgfortran.so)) -lgfortran -lopenblas- The MPI Fortran libraries to add at link time are obtained from the MPI wrapper (
mpif90 --showme:linkfor Open MPI,mpif90 -showfor Intel MPI). - On PS-A and PS-B, specify LAPACKE (
-Iand-l:libopenblaso.so.0) as in the sections above. With Intel MPI on PS-A, usempif90(gfortran inside) as the Fortran wrapper. - With Fujitsu Fortran (frt,
mpifrt) and Fujitsu C (fcc,mpifcc), add-Kopenmp(Fortran in “Usage”). examples/build_examples.shbuilds the samples inexamples/with the same steps. The compilers, target and LAPACK are changed with environment variables (SYCL_TARGET,CXXFLAGS,LAPACK_INC,LAPACK_LIB,MPICXX/MPICC/MPIFC,FC_LINK).
4.5.10 macOS (Homebrew)
In an environment installed with Homebrew (“macOS (Homebrew)” in “Installing a SYCL Compiler”), use --acpp-targets=omp.library-only with blocking of the reductions only (-D_BLOCKING_LOOP_DOT_=1, as on other CPUs), and specify the Homebrew OpenBLAS for LAPACKE. With the Homebrew AdaptiveCpp, --acpp-targets=omp gives the same compilation. -march=native is not needed (it makes no difference in speed). Start with mpirun -n <processes> --bind-to none. The following was confirmed with the programs of the four languages in the Quick Start.
eval "$(/opt/homebrew/bin/brew shellenv)"
export OMPI_CXX=acpp
B=/opt/homebrew
T="--acpp-targets=omp.library-only"
LAPACK_INC="-I$B/opt/openblas/include"
LAPACK_LIB="-L$B/opt/openblas/lib -lopenblas -Wl,-rpath,$B/opt/openblas/lib"
FC_RT="-L$B/opt/gcc/lib/gcc/current -lgfortran -Wl,-rpath,$B/opt/gcc/lib/gcc/current"
# C++ (with MPI; without MPI, replace mpicxx by acpp and add -DPARCEL_MPI=0)
mpicxx -O3 $T -D_BLOCKING_LOOP_DOT_=1 ${LAPACK_INC} \
-o verify_double tests/verify/parcel_krylov_sample_verify.cpp ${LAPACK_LIB}
# C and Fortran (with MPI)
mpicxx -O3 $T -D_BLOCKING_LOOP_DOT_=1 ${LAPACK_INC} -c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
mpicc -O2 -Icapi -c prog.c -o prog_c.o
mpicxx $T -o prog_c prog_c.o wrapper_parcel_sycl.o ${LAPACK_LIB}
mpif90 -O2 -c capi/parcel_sycl_mod.f90 -o parcel_sycl_mod.o
mpif90 -O2 -c prog.f90 -o prog_f.o
mpicxx $T -o prog_f prog_f.o parcel_sycl_mod.o wrapper_parcel_sycl.o $(mpif90 --showme:link) ${FC_RT} ${LAPACK_LIB}
mpirun -n 4 --bind-to none ./prog_cThe gfortran runtime library on macOS is a .dylib, so its location ($B/opt/gcc/lib/gcc/current) is given with -L and -Wl,-rpath. Sourcing python/envs/mac_brew.sh puts the same values into environment variables (SYCL_TARGET, CXXFLAGS, LAPACK_INC, LAPACK_LIB, FC_LINK), so examples/build_examples.sh can build the samples. The Python package is described in “Python Package”.
4.6 Default of MPI
Whether PARCEL is built with MPI or as the single-process version without MPI is decided by the macro PARCEL_MPI (1: with, 0: without). The default differs by usage as follows.
| Usage | Default of PARCEL_MPI |
Where it is decided | How to build without / with MPI |
|---|---|---|---|
C++ (including Include/parcel_sycl.hpp) |
1 (with MPI) | Headers in Include/ |
Add -DPARCEL_MPI=0 |
C (capi/parcel_sycl_c.h) |
1 (with MPI) | parcel_sycl_c.h and the wrapper |
Add -DPARCEL_MPI=0 to both the wrapper and the main program |
Fortran (capi/parcel_sycl_mod.f90) |
1 (with MPI) | The wrapper (the module does not depend on MPI) | Add -DPARCEL_MPI=0 to the wrapper and satisfy use mpi with the substitute module |
examples/build_examples.sh |
1 (with MPI) | Environment variable PARCEL_MPI |
PARCEL_MPI=0 bash examples/build_examples.sh |
Python (python/, pip) |
0 (without MPI) | python/CMakeLists.txt |
-C cmake.define.PARCEL_MPI=1 for MPI |
spack (parcel-sycl, py-parcel-sycl) |
0 (without MPI; variant ~mpi) |
The recipe variant mpi |
+mpi for MPI |
C++, C and Fortran default to MPI so that programs written as for v1.0 (compiled with the MPI wrappers and started with mpirun) run as they are. Python and spack default to no MPI so that the package can be installed and tried immediately on machines without MPI (local computers, macOS, etc.). To use the version with MPI, specify it as above. In C and Fortran, if the wrapper and the main program are built with different settings, the types of the communicator argument do not match and the program does not work correctly (this is not detected at link time), so build both with the same setting. C++ is a header-only library, so the setting of the user program is used as it is. The computation is the same in both versions, and the results of the MPI version run with one process and of the version without MPI agree bit for bit.
4.7 Build without MPI
With -DPARCEL_MPI=0, a single-process version that does not use MPI can be built. It does not include mpi.h or link the MPI library, so compile it directly with the SYCL compiler (acpp, etc.) instead of the MPI compiler wrapper (mpicxx), and run it without mpirun. Without the option (the default), MPI is used. The computation is the same as running the MPI build with one process, and the results agree bit for bit.
acpp -O3 --acpp-targets=omp -march=native -DPARCEL_MPI=0 \
-I/usr/include/openblas -o quick_start examples/quick_start.cpp -lopenblas
./quick_start
# the four programs in examples/ (C / Fortran are compiled with gcc / gfortran)
PARCEL_MPI=0 bash examples/build_examples.shTo build C and Fortran programs by hand, add -DPARCEL_MPI=0 to both the wrapper and the C main program, and use acpp, gcc and gfortran instead of the MPI wrappers (mpicxx, mpicc, mpif90). use mpi in Fortran is satisfied by the single-process substitute module examples/sample_f_mpi_stub.f90 (which implements MPI_Init, MPI_Comm_rank, MPI_Comm_size, MPI_Finalize and others for one process). Example commands are in the “without MPI” parts of “C” and “Fortran” of the Quick Start.
User programs are written in the same way as with MPI. The differences are:
- C++: create the communicator with
parcel::comm comm(&q);(corresponding toparcel::comm comm(MPI_COMM_WORLD, &q);with MPI).mpi_rank()returns 0 andmpi_size()returns 1.set_crsMat(&comm, nstart, nend, n, nnz, ...)can be used as is, but since one process owns everything, usenstart = 0andnend = n - 1(otherwisestd::invalid_argument). Functions that use MPI types (comm::mpi_comm(), etc.) do not exist. Guard the MPI calls in the user program (MPI_Init, etc.) with#if PARCEL_MPI(seeexamples/quick_start.cpp). - C: compile both the wrapper (
capi/wrapper_parcel_sycl.cpp) and the program that includesparcel_sycl_c.hwith-DPARCEL_MPI=0(otherwise the type of the communicator argument does not match). The function names and argument order are unchanged; the communicator argument (PARCEL_C_COMM) becomesintand its value is ignored (pass 0). Usenstart = index_baseandn_local = n_global(otherwisePARCEL_ERR_DISTRIBUTION). Exceptions inside the library do not callMPI_Abort; a message is printed andPARCEL_ERR_RUNTIMEis returned.MPI_Initis not needed. - Python: the version without MPI is the default (Python Package in “Installation”). Run with
python script.pywithoutmpirun. - Fortran: use
capi/parcel_sycl_mod.f90as is (the communicator integer is ignored). Compileexamples/sample_f.f90unchanged together withexamples/sample_f_mpi_stub.f90, a single-process substitute for thempimodule, using gfortran.
4.8 Python Package
The Python package parcel (python/ in the distribution package) is installed by building, from source, its native layer (libparcel_py.so) that calls the C++ library of PARCEL directly. You need an environment in which PARCEL can be built (“Settings for use” in the chapter Installing a SYCL Compiler), Python 3.8 or later, and NumPy. SciPy is convenient for building matrices, and the Python test of the regression tests (tests/regression/sample_py.py) uses it. The native layer is built with AdaptiveCpp. The default is the single-process version without MPI; the build setting PARCEL_MPI=1 gives the version with MPI. There are three ways to install it. Usage is described in Python in “Usage”.
- Install it with the spack recipe
py-parcel-sycl(“Installing with spack”). The script that installs everything from scratch,spack/stack/install_stack.sh, installs this package together with the C++ headers and the C / Fortran API. - Run
pip install .inpython/(“Without MPI (Default)” below). - Build the native layer directly without pip and give its location with environment variables (same section).
4.8.1 Settings per Environment
The native layer is built for each environment (SYCL target and with or without MPI). The build settings are given with pip’s -C cmake.define.<name>=<value> (“Build Settings”). The values used in the test environments of this manual are shown below (the same as in Recommended Options). Blank cells can be left at their defaults.
| System | PARCEL_TARGET |
PARCEL_KERNEL_FLAGS |
PARCEL_EXTRA_FLAGS |
|---|---|---|---|
| s86 (CPU), Miyabi-C, PS-A | (omp) |
(-D_BLOCKING_LOOP_DOT_=1) |
|
| s86 (V100) | cuda:sm_70 |
(-DPARCEL_SPMV_LADDER=0 -DPARCEL_PRECON_LADDER=0; no ladder splitting) |
-Xcuda-ptxas -maxrregcount=255 |
| Aquarius (A100) | cuda:sm_80 |
(same as above) | |
| Miyabi-G (GH200) | cuda:sm_90 |
(same as above) | |
| PS-B (MI300A) | hip:gfx942 |
(-DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=1) |
|
| macOS (Apple silicon) + Homebrew | omp.library-only |
(-D_BLOCKING_LOOP_DOT_=1) |
Values in parentheses are the defaults (for PARCEL_KERNEL_FLAGS, the value chosen from the target). Give the location of LAPACKE (LAPACKE_INCLUDE_DIR, LAPACK_LIBS) for your environment (for example -l:libopenblaso.so.0 on PS-A and PS-B; see the section of each environment in Recommended Options). Build, install and run in the same environment (“Settings for use” in the chapter Installing a SYCL Compiler; the AdaptiveCpp, LAPACK and MPI libraries must be found).
On macOS, in an environment installed with Homebrew (“macOS (Homebrew)” in “Installing a SYCL Compiler”), sourcing python/envs/mac_brew.sh sets the values in the table above and the location of the Homebrew OpenBLAS in environment variables; then use pip or the direct build below.
4.8.2 Without MPI (Default)
Run pip in python/. pip uses scikit-build-core and CMake 3.18 or later for the build (and fetches them if missing), so the package index must be reachable. Upgrade pip in the venv first (an old pip may not accept -C, which passes the build settings). It may be run on a login node and takes 1 to 2 minutes.
python3 -m venv $HOME/venv_parcel
$HOME/venv_parcel/bin/python -m pip install -U pip
$HOME/venv_parcel/bin/python -m pip install scipy # optional; convenient for building matrices (parcel depends only on NumPy)
cd python
$HOME/venv_parcel/bin/python -m pip install . -C cmake.define.PARCEL_TARGET=cuda:sm_70 \
-C "cmake.define.PARCEL_EXTRA_FLAGS=-Xcuda-ptxas -maxrregcount=255" # example for s86 (V100)The native layer is built directly with acpp and does not link the MPI library.
Without pip, build the native layer directly, and set the environment variable PARCEL_PY_LIB to its location and add python/ to PYTHONPATH (run in the top directory of the package; the Python jobs of the regression tests work this way, and tests/regression/sub_build_lang_<environment>.sh builds bin/<environment>/libparcel_py.so).
acpp -O3 --acpp-targets=omp -DPARCEL_MPI=0 -march=native -mtune=native -D_BLOCKING_LOOP_DOT_=1 \
-fPIC -shared -IInclude -I/usr/include/openblas -o libparcel_py.so python/src/parcel_py.cpp -lopenblas
PYTHONPATH=python PARCEL_PY_LIB=$PWD/libparcel_py.so python script.pyCheck the installation as follows. parcel.has_mpi() returns True for the version with MPI, parcel.build_info() returns the build settings, and parcel.device_name() returns the name of the device used.
$HOME/venv_parcel/bin/python -c "import parcel; print(parcel.has_mpi(), parcel.build_info(), parcel.device_name())"The Python API test tests/regression/sample_py.py performs, from Python, the same problem, items and checks as the C sample examples/sample_c.c (the 7 solvers, pcagmres without building in two ways, ks_arnoldi with shift-invert, etc.). It can be run as the language python of the regression tests (Package Layout) or on its own. The argument is NX.
PARCEL_TEST_FROM_SOURCE=1 PARCEL_PY_LIB=$PWD/libparcel_py.so python tests/regression/sample_py.py 16 # reads python/ of the package
python tests/regression/sample_py.py 16 # reads the installed parcel4.8.3 With MPI
With the build setting PARCEL_MPI=1, the native layer is built with the MPI C++ wrapper (mpicxx by default). Set OMPI_CXX=acpp (Open MPI) or the equivalent beforehand so that the wrapper calls acpp. mpi4py is optional; to use it, install one built with the same MPI into the venv.
export OMPI_CXX=acpp
python3 -m venv $HOME/venv_parcel_mpi # a separate venv from the version without MPI
$HOME/venv_parcel_mpi/bin/python -m pip install -U pip
cd python
$HOME/venv_parcel_mpi/bin/python -m pip install . -C cmake.define.PARCEL_MPI=1Without pip, replace acpp by mpicxx in the build line above and remove -DPARCEL_MPI=0. Running is described in “Running with MPI”.
Test status: both the package installed with pip and the one built from source passed all items in the version with MPI on s86 (CPU, V100), Miyabi-C, Miyabi-G, PS-A, PS-B and Aquarius. The version without MPI was confirmed on s86, Aquarius and macOS (Homebrew; also the version with MPI). The checks of py-parcel-sycl installed with spack are given in “Installing with spack”.
4.8.4 Build Settings
The names given with pip’s -C cmake.define.<name>=<value> are listed below (the same list is at the top of python/CMakeLists.txt). Names not given take the defaults in the right column.
| Name | Meaning | Default |
|---|---|---|
PARCEL_MPI |
0: single-process version without MPI, 1: with MPI | 0 |
PARCEL_TARGET |
The value of --acpp-targets |
omp |
PARCEL_KERNEL_FLAGS |
Kernel optimization macros | Chosen from PARCEL_TARGET (-D_BLOCKING_LOOP_DOT_=1 for omp, the two ladder macros for hip, no ladder splitting for cuda) |
PARCEL_HOST_FLAGS |
Host optimization | -march=native -mtune=native |
PARCEL_EXTRA_FLAGS |
Other flags | None |
PARCEL_INCLUDE_DIR |
The Include/ of PARCEL |
Include/ next to python/ |
LAPACKE_INCLUDE_DIR |
Location of lapacke.h |
/usr/include/openblas |
LAPACK_LIBS |
LAPACK libraries | -lopenblas |
PARCEL_MPICXX |
C++ compiler | acpp without MPI, mpicxx with MPI |
4.9 Installing with spack
spack/ in the distribution package contains recipes for installing PARCEL with spack, and a script that installs everything from scratch, from fetching spack through LLVM, AdaptiveCpp, OpenBLAS and Python to PARCEL. The original description is spack/README.md. The target is spack 1.x (package repository API v2.2).
4.9.1 Recipes
| Package | Contents |
|---|---|
parcel-sycl |
Headers (include/), C API (lib/libparcel_sycl_c.so and include/parcel_sycl_c.h), Fortran module (include/parcel_sycl_mod.mod and lib/libparcel_sycl_f.a) |
py-parcel-sycl |
The Python package parcel (builds python/ with scikit-build-core; “Python Package”) |
The variants are common to both.
| Variant | Default | Meaning |
|---|---|---|
mpi |
off | With MPI (the default is the single-process version without MPI) |
cuda + cuda_arch=<n> |
off | --acpp-targets=cuda:sm_<n>. With cuda_arch=70 (V100), -Xcuda-ptxas -maxrregcount=255 is added automatically |
rocm + amdgpu_target=<gfx> |
off | --acpp-targets=hip:<gfx> |
(neither cuda nor rocm) |
--acpp-targets=omp (CPU) |
|
native |
on | -march=native -mtune=native |
kernel |
auto |
Kernel optimization (“Optimization”): auto (blocking_dot on CPUs, ladder on AMD GPUs (+rocm), none on NVIDIA GPUs (+cuda), a64fx for +fj), blocking_dot, a64fx, ladder, none |
fj |
off | For A64FX: compiles the host side with the Fujitsu compiler (FCC) and builds the Fortran module with frt (confirmed with the Odyssey definition wis_ody). Adds -std=c++17 -ffp-contract=fast (Recommended Options) |
capi (parcel-sycl only) |
on | Builds the C / Fortran API |
scipy (py-parcel-sycl only) |
on | Also installs SciPy (used by DistributedCSR.from_scipy() / from_global() and examples/quick_start.py; corresponds to parcel[scipy] with pip) |
ks_ortho |
cgs2 |
Default orthogonalization of ks_arnoldi (“Default Orthogonalization of ks_arnoldi”). mgs adds -DPARCEL_KS_ORTHO_DEFAULT=mgs (also in PARCEL_CXXFLAGS). A value given by the program with set_qr_algo() takes precedence. py-parcel-sycl is built with parcel-sycl of the same value |
To add them to an existing spack environment, use the local distribution package with spack develop.
spack -e <env> repo add <package>/spack/spack_repo/parcel
spack -e <env> config add "concretizer:unify:when_possible" # when installing CPU and GPU versions side by side
spack -e <env> develop --no-clone --path <package> parcel-sycl@1.2.0
spack -e <env> develop --no-clone --path <package> py-parcel-sycl@1.2.0
spack -e <env> add "parcel-sycl@1.2.0 ~mpi ~cuda ~rocm" "py-parcel-sycl@1.2.0 ~mpi ~cuda ~rocm" # CPU
spack -e <env> add "parcel-sycl@1.2.0 ~mpi +cuda cuda_arch=70" "py-parcel-sycl@1.2.0 ~mpi +cuda cuda_arch=70" # GPU
spack -e <env> concretize -f && spack -e <env> installTo use them, spack load with the configuration specified. spack load parcel-sycl sets PARCEL_CXXFLAGS (target, optimization, with or without MPI, header location) and PARCEL_LDFLAGS (RPATHs of the libraries needed at run time (LLVM libomp, AdaptiveCpp, LAPACK, MPI) and LAPACK) so that user programs are compiled with the same settings as PARCEL. The version with MPI also sets OMPI_CXX=acpp etc., so compile with mpicxx in the same way. The RPATH of libomp is not added through the MPI wrapper, so always add PARCEL_LDFLAGS. spack load does not set the locations of the headers (parcel_sycl_c.h, parcel_sycl_mod.mod) and libraries of the C and Fortran APIs; find them with spack location -i and give them with -I, -L and -Wl,-rpath.
spack env activate <env>
spack load parcel-sycl+cuda py-parcel-sycl+cuda # load with the configuration specified
acpp $PARCEL_CXXFLAGS -O3 -o prog prog.cpp $PARCEL_LDFLAGS # C++
P=$(spack location -i parcel-sycl+cuda) # C (version without MPI): give the location yourself
gcc -I$P/include -o prog prog.c -L$P/lib -lparcel_sycl_c -Wl,-rpath,$P/lib
python script.py # PythonNotes:
- Specify the root specs down to
~cuda ~rocm. With onlyparcel-sycl ~mpi, unification may merge it with the GPU version. - CPU and GPU versions in the same environment collide in the view, so exclude
parcel-syclandpy-parcel-syclfrom the view of the environment (view: default: {root: .spack-env/view, exclude: [parcel-sycl, py-parcel-sycl]}). - Adding specs to an already concretized environment may cause an internal error (unsatisfiable) in
concretize. Re-concretize withconcretize -f. - Use a single MPI and a single
hipsycl(AdaptiveCpp) in an environment (packages:mpi:require:including the version and configuration,packages:hipsycl:require:). With several external MPIs, another configuration (such as a CUDA build) may be selected. - The CUDA used by AdaptiveCpp is recorded from
ACPP_CUDA_PATH(or the spackcuda) at build time, andspack loadsets the same value. To avoid a CUDA newer than the driver, specify it as the externalcudapackage or with this variable. - The LAPACKE headers are copied to
include/lapacke/(for systems whose compute nodes lacklapacke.h). - Running tests inside the
spack developsource (which createsbin/,__pycache__, etc.) makes spack consider it modified, and PARCEL is rebuilt at the nextinstall. Run tests in a copy of the source.
4.9.2 Installing from the Distribution Tarball
The recipes hold the url and sha256 of the distribution tarball. The url points to the tarball on the GitHub Release (https://github.com/parcel-jaea/PARCEL-release/releases/download/v1.2.0/parcel_sycl-1.2.0.tar.gz), and tools/make_tarball.py --pin writes the sha256 into the recipes (“Verifying the Distribution”). Without spack develop, spack fetches the tarball from the url (parcel-sycl and py-parcel-sycl use the same tarball). When the computer cannot connect to the outside, place the tarball in the layout of a spack mirror (<mirror>/_source-cache/archive/<first two characters of the sha256>/<sha256>.tar.gz) and register it with spack mirror add. spack checks the fetched tarball against the sha256 in the recipe and stops if it differs. The sha256 is part of the spack hash, so rewriting it rebuilds PARCEL (also when it is used with spack develop). The procedure is given in the section “Installing from the Release Tarball” of spack/README.md.
4.9.3 Choosing the Configurations
The configurations to install are chosen by the combination of packages (parcel-sycl, py-parcel-sycl) and variants (the spec). Examples of specs by purpose are shown below (the version @1.2.0 is omitted).
| Purpose | Spec |
|---|---|
| CPU, without MPI (C++, C, Fortran) | parcel-sycl ~mpi ~cuda ~rocm |
| CPU, with MPI | parcel-sycl +mpi ~cuda ~rocm |
| NVIDIA GPU (example: A100) | parcel-sycl ~mpi +cuda cuda_arch=80 |
| NVIDIA GPU, with MPI | parcel-sycl +mpi +cuda cuda_arch=80 |
| AMD GPU (MI300A) | parcel-sycl ~mpi +rocm amdgpu_target=gfx942 |
| C++ (headers) only, without the C and Fortran APIs | parcel-sycl ~capi ~mpi ~cuda ~rocm |
| Python as well | Add py-parcel-sycl with the same configuration (e.g., py-parcel-sycl ~mpi ~cuda ~rocm) |
- Choosing languages: C++ (headers) is always installed with
parcel-sycl. The C and Fortran APIs are installed or omitted together by the variantcapi(on by default); one of them alone cannot be chosen. Python is chosen by whether the separate packagepy-parcel-syclis added. - With
+capi(the default), a Fortran compiler is required (to build the Fortran module). On systems without a Fortran compiler, use~capi. py-parcel-syclusesparcel-syclfor its build, soparcel-syclis installed even when only Python is wanted. In that caseparcel-syclhas the default+capi. On systems without a Fortran compiler, add the variant ofparcel-sycl, as inpy-parcel-sycl ~mpi ~cuda ~rocm ^parcel-sycl ~capi(confirmed on s86 that it concretizes and thatparcel-syclthen no longer uses a Fortran compiler). OpenBLAS, used for LAPACK, also uses a Fortran compiler by default, so on systems with no Fortran compiler at all also specifyopenblas ~fortran(as in the Odyssey definitionwis_ody).+cudaand+rocmcannot be specified together (the recipe raises an error). Without either, the CPU (OpenMP) version is built. For GPUs, specify onecuda_arch(CUDA compute capability: V100 70, A100 80, GH200 90) oramdgpu_target.+mpirequires MPI. Register the system MPI as an external package in the environment definition (use a single MPI and a singlehipsyclin an environment; notes above).mpi,cuda,rocm, etc. ofpy-parcel-syclmust match those of theparcel-syclit is combined with (the recipe requiresparcel-syclwith the same configuration).
With installation from scratch, the configurations installed are those listed in specs: of spack/stack/envs/<env>/spack.yaml. The bundled definitions list all configurations available on each system (for example, 8 lines on s86: CPU / GPU × without / with MPI × parcel-sycl / py-parcel-sycl). To narrow or change the configurations, edit specs: before running install_stack.sh. For example, to install only C++, C, Fortran and Python for the CPU with MPI, write as follows. If the list is edited after installation, rerunning install_stack.sh with the same command re-concretizes and installs only what is missing.
spack:
specs:
- parcel-sycl@1.2.0 +mpi ~cuda ~rocm # CPU with MPI (C++, C, Fortran)
- py-parcel-sycl@1.2.0 +mpi ~cuda ~rocm # Python with the same configuration4.9.4 Installing from Scratch
Without an existing spack, spack/stack/install_stack.sh installs everything from fetching spack to PARCEL in one run. The arguments are the environment name (a directory under spack/stack/envs/), the installation directory and the number of parallel jobs.
bash spack/stack/install_stack.sh s86 $HOME/parcel_stack 16 # <env> <prefix> [jobs]
bash spack/stack/check_stack.sh $HOME/parcel_stack $HOME/parcel_check # check on the login node
. $HOME/parcel_stack/setup.sh # when using it
spack load parcel-sycl+mpi+cuda py-parcel-sycl+mpi+cuda- Built by spack: LLVM 18.1.8, AdaptiveCpp (
hipsycl24.10.0), OpenBLAS, binutils, Python 3.11 with NumPy, SciPy and pip, PARCEL itself, the C / Fortran API and the Python package. - Taken from each system: compilers, CUDA / ROCm and MPI. They are tied to the network and drivers of the machine, and are written as external packages in
spack/stack/envs/<env>/spack.yaml. - spack itself (1.1.0.dev0) and the recipe collection (spack-packages) are pinned to the confirmed versions (at the top of
install_stack.sh). - spack, its configuration and caches are all placed in the installation directory, and
~/.spackand/etc/spackare not read (SPACK_DISABLE_LOCAL_CONFIG=1). Existing spack environments neither affect it nor are affected. - The PARCEL source is the directory two levels above
install_stack.sh(the top of the distribution package, which must containInclude/,capi/andpython/). To use another location, givePARCEL_SRC=<directory>. envs/<env>/site_env.shis read during the installation and whensetup.shis read (site modules for MPI, etc.).after_activate.shis read after the environment is activated.compilers.yamllists compilers installed before the main environment (to build a newer GCC on systems whose OS GCC is old).- If interrupted, rerun the same command to resume (installed packages are reused). With
CONCRETIZE_ONLY=1, only concretization is performed. check_stack.shrunsspack loadfor each installed configuration and checks thatparcelcan be imported from Python and thatsample_c.c,sample_f.f90andquick_start.cppinexamples/build and find all libraries needed at run time (they are not run). The Fortran sample without MPI is compiled together withexamples/sample_f_mpi_stub.f90.- The post-installation test is
spack test run parcel-sycl py-parcel-sycl(results withspack test results). It builds and runs the Quick Start examples with the installed headers and libraries (parcel-sycl: C++, and C and Fortran with+capi;py-parcel-sycl:quick_start.py) and checks that the maximum relative error against the exact solution is at most 1e-6 (the same as “Checking After the Build”). Versions with MPI are started withMPIEXEC(default:bin/mpiexecof the MPI),MPIEXEC_NP_FLAG,MPI_NP(default 2) andMPIEXEC_ARGS. On a node without a GPU (a login node), the GPU versions only test the build and the runs are SKIPPED, so run the test on a GPU compute node as well.PARCEL_TEST_RUN=0disables the runs. The number of threads and the decision on where it runs are the same as in “Checking After the Build” (ifOMP_NUM_THREADSis not set, all available logical CPUs on a compute node and 4 on a login node; forced withQUICK_NODE), and the decision and the values used appear in the output ofspack test results -l. For+fj +mpi(Fujitsumpiexec),RC_ONLY=1is the default (useRC_ONLY=0to turn it off). With Intel MPI, the C and Fortran tests are built with the generic wrappers in the same place (mpicc,mpif90), and the underlying compiler is set to spack’s compiler (gfortran, etc.) withI_MPI_CCandI_MPI_F90(andI_MPI_FC) (thempiifxreturned by spack calls ifx, which cannot readparcel_sycl_mod.modbuilt by gfortran). Run the GPU versions and+fjon compute nodes (a short batch job or an interactive job). The CPU versions may be run on a login node (follow the rules of the site).
4.9.5 Provided Environments
envs/ |
System | Configurations | Taken from the system |
|---|---|---|---|
s86 |
s86 (Xeon + V100) | CPU / GPU (cuda_arch=70) × without / with MPI |
GCC 9.5.0, CUDA 12.6, Open MPI 4.1.6 |
miyc |
Miyabi-C | CPU × without / with MPI | GCC 11.4.1, Open MPI 4.1.6 |
miyg |
Miyabi-G | GPU (cuda_arch=90) × without / with MPI |
GCC 11.4.1, CUDA 12.6, Open MPI 4.1.6 (CUDA build) |
psa |
PS-A | CPU × without / with MPI (ks_ortho=mgs: MGS2 is the default orthogonalization of ks_arnoldi) |
GCC 11.5.0, Intel MPI 2021.18 |
psb |
PS-B | GPU (amdgpu_target=gfx942) × without / with MPI |
GCC 11.5.0, ROCm 6.3.3, Open MPI 5.0.7 (ROCm build) |
wis_aqu |
Aquarius | GPU (cuda_arch=80) × without / with MPI |
GCC 8.3.1 (used only to build GCC 12), CUDA 12.2, Open MPI 4.1.1 |
wis_ody |
Odyssey | CPU (+fj) × without / with MPI |
GCC 8.3.1, Fujitsu compilers, MPI and LAPACK (tcsds-1.2.43) |
generic |
Other Linux systems (template) | CPU × without MPI (with MPI and NVIDIA GPUs added by uncommenting) | Edit to use (GCC, MPI, CUDA) |
- In all seven environments, the installation with
install_stack.shand the check withcheck_stack.shpassed, and the regression tests or the samples (with and without MPI) passed with the installed PARCEL. With the recipes added to existing spack environments, the samples also passed on compute nodes of the six environments other than Odyssey. - The locations of external packages (
prefix:) and module names are those of each system. Edit them for other machines. - Odyssey (
wis_ody) has x86 login nodes, so it is installed by a job on an A64FX compute node (envs/wis_ody/install_job.shis an example; the notes on parallelism, memory limit and time are the same as in the Odyssey section of “Installing a SYCL Compiler”). LLVM 18 is built with the OS GCC 8.3.1, and its clang builds AdaptiveCpp and the C / C++ parts of NumPy and SciPy. Building a newer GCC such as GCC 12 would make the Fujitsu compiler (FCC) unable to read its libstdc++ and unable to link C++ programs. PARCEL is built+fj: the host side with FCC, MPI withmpiFCCand LAPACK with the Fujitsu-lfjlapack. - On Linux systems other than the seven environments above, edit and use the
generictemplate (below). - On macOS, spack is not used; install with Homebrew (“macOS (Homebrew)” in “Installing a SYCL Compiler”).
- Example settings for running the regression and performance tests in the installed environment are in
tests/site_config.py(s86spack_cpu,s86spack_gpu, etc.; the section on spack environments intests/regression/README.md).
4.9.6 Other Systems
On Linux systems other than the seven environments above, copy and edit the template spack/stack/envs/generic/, and install with install_stack.sh. The template follows the definitions for s86 and Miyabi-C. On Ubuntu 24.04 (WSL2; OS GCC 13.3.0, Open MPI 4.1.6, CUDA 12.6 from the NVIDIA repository, RTX A6000), it was edited as in the steps below and the eight configurations CPU and GPU (cuda_arch=86) × without and with MPI were installed; check_stack.sh, the Quick Start (C++, C, Python) and the verification sample (pcg, pgmres, ks_arnoldi) passed. On Rocky Linux 9.8 (compatible with RHEL 9; WSL2; OS GCC 11.5.0, Open MPI 4.1.1, CUDA 12.6 from the NVIDIA repository), the same eight configurations were installed and the same tests passed. Register the OS Open MPI of the RHEL family after putting it, including the headers, in one directory (“Registering MPI and CUDA”).
cp -r spack/stack/envs/generic spack/stack/envs/mysite # copy and edit
CONCRETIZE_ONLY=1 bash spack/stack/install_stack.sh mysite $HOME/parcel_stack # try concretization only
bash spack/stack/install_stack.sh mysite $HOME/parcel_stack 16 # installThe places to edit are listed as steps in the comments at the top of spack.yaml.
- The version and location of the compiler (GCC) (check with
gcc --versionandwhich gcc g++ gfortran). - To use MPI, uncomment the two
+mpilines ofspecs:and thempiandopenmpilines ofpackages:, and edit the version and location (see “Generic Linux” for the format). - To use NVIDIA GPUs, uncomment the
+cudalines and thecudalines, editcuda_arch(found as in “Running on GPUs”) and the version and location of CUDA, and change thehipsyclandllvmlines for the GPU. For AMD GPUs, start from the PS-B definitionpsb(external ROCm). - If the CPU is not x86_64, edit
targetsofllvm(aarch64for Arm). - On systems that load MPI etc. with modules, write
module loadinsite_env.sh(module load mpi/openmpi-x86_64for the OS Open MPI of the RHEL family; thesite_env.shof the template has an example). On systems with little memory, lower the number of concurrent compilations withPARCEL_STACK_JOBS.