PARCEL SYCL v1.2 / User Guide User Guide API Reference Search Download 日本語

4 Installation

This chapter describes how to install SYCL PARCEL.
PARCEL consists of header files only. It can be used simply by including ./Include/parcel_sycl.hpp and compiling with a SYCL-capable compiler. No prebuilding or installation as a library is required.

Note, however, the following three points.

For installing a SYCL-capable compiler, see the previous chapter. Installing PARCEL together with its prerequisites with spack is described in “Installing with spack”, and the Python package in “Python Package”.

4.1 Package Layout

The layout of the distribution package (PARCEL_SYCL v1.2) is shown below. The library itself consists only of the headers in Include/; the rest are samples and tests.

Location Contents
Include/ The library (headers only). Include parcel_sycl.hpp
capi/ C / Fortran API: parcel_sycl_c.h (C declarations), parcel_sycl_mod.f90 (Fortran module), wrapper_parcel_sycl.cpp (implementation; build with a SYCL compiler), parcel_capi_default.hpp (compile-time defaults of the wrapper: internal precisions and the settings of the legacy API)
examples/ quick_start.cpp, quick_start_c.c, quick_start_f.f90, quick_start.py (the C++, C, Fortran and Python examples of the Quick Start), example_pcg_ks.cpp (pcg and ks_arnoldi), sample_c.c / sample_f.f90 (call and verify all solvers through the C / Fortran API), sample_f_mpi_stub.f90 (single-process mpi module for building sample_f.f90 and quick_start_f.f90 without MPI), problem.f90 (matrix generation for the legacy API), build_examples.sh (builds all of the above; compilers, target and LAPACK are set with environment variables), run_quick_checks.sh (checking after the build; “Checking After the Build”)
python/ The Python package parcel (“Python Package” in “Installation”, Python in “Usage”): parcel/ (Python layer), src/ (source of the native layer libparcel_py.so), envs/mac_brew.sh (settings for building on macOS + Homebrew), pyproject.toml / CMakeLists.txt (build settings for pip)
spack/ spack recipes parcel-sycl / py-parcel-sycl (spack_repo/parcel/) and the scripts that install everything from fetching spack to PARCEL (stack/install_stack.sh, stack/check_stack.sh for checking, per-environment definitions stack/envs/<env>/) (“Installing with spack”)
tests/verify/ The verification sample parcel_krylov_sample_verify.cpp, its defaults parcel_default.hpp, the memory report (diag_mem.hpp; PARCEL_MEM_REPORT), and the test matrix generators (Poisson, convection-diffusion, unstructured)
tests/regression/ Generator of the regression-test job scripts (gen_regression.py), submission and summary scripts, the double-double check (ddcheck), the Python API test (sample_py.py), and a script that runs the regression tests on machines without a scheduler (run_local.sh)
tests/perf/ Generators of the performance-test job scripts (gen_perf.py, and gen_perf_prec.py for mixed precision) and summary scripts
tests/site_config.py Per-environment settings of the tests (queues, modules, compile flags, accounts). Edit it for your environment
tests/check_version.py Version check (no build needed). Checks that the files that copy the version (the Python package, the spack recipes and the parcel-sycl@ specs of the environment definitions, CITATION.cff) agree with Include/parcel_version.hpp (“Checking the Version”)
tools/make_tarball.py Script that creates the distribution tarball and its sha256 (“Verifying the Distribution”)
README.md, CHANGELOG.md Overview and changes in v1.2
CITATION.cff, SUPPORT.md Citation information and how to get support (“Citation and Support”)
doc/ HTML of the User Guide and the API Reference of this manual (Japanese and English; entry doc/index.html, English doc/index_en.html; open it directly in a web browser)
COPYING, COPYING.LESSER License (LGPL-3.0)

The descriptive md files (README.md, CHANGELOG.md, SUPPORT.md, spack/README.md, python/README.md, tests/*/README.md) are in English, with a Japanese version in the same place (README.ja.md etc., .ja before .md in the name).

How to run the tests is described in tests/regression/README.md and tests/perf/README.md. The regression tests consist of the verification sample (C++) and the tests of the C, Fortran and Python APIs (examples/sample_c.c, examples/sample_f.f90, tests/regression/sample_py.py); with or without MPI and the languages can be selected for generation, build, submission and summary. Besides the test environments of this manual, the bundled settings (tests/site_config.py) include examples for environments installed from scratch with spack (s86spack_cpu, s86spack_gpu, etc.) and an example for macOS + Homebrew without a scheduler (mac).

4.2 Checking After the Build

To check a build, use bash examples/run_quick_checks.sh. It builds the four Quick Start examples (C++, C, Fortran and Python) with examples/build_examples.sh, runs them as one process, and with an MPI build also runs the C++ example with two processes. Each example prints check: ... PASS if the maximum relative error against the exact solution is at most 1e-6 (“Program Overview”). At the end, the script prints PASS n / FAIL m (SKIP k) and the elapsed time on one line, and exits with status 0 if nothing failed. The Python example is SKIPPED (not failed) if parcel (with NumPy and SciPy) cannot be imported.

The build (build_examples.sh quick) runs the SYCL compilations (the C++ example and the wrapper of the C / Fortran API) at the same time (the number of concurrent compilations is BUILD_JOBS; the default is the smaller of the number of targets and the number of cores). After a successful build, the build settings (SYCL_TARGET, CXXFLAGS, PARCEL_MPI, the compilers and their locations, etc.) and the cksum of the sources (headers, API, examples) are recorded in examples/build/build_stamp.txt. From the second run on, if the executables exist and the record matches the current state, the build is skipped and only the runs are performed (less than a minute). If a setting or a source has changed, the differing items are shown and the examples are rebuilt. --rebuild always rebuilds.

export OMPI_CXX=acpp
bash examples/run_quick_checks.sh                 # with MPI. last line: === PASS 6 / FAIL 0 (SKIP 0)  所要 <sec> s
PARCEL_MPI=0 bash examples/run_quick_checks.sh    # without MPI
bash examples/run_quick_checks.sh --rebuild       # rebuild
NO_BUILD=1 bash examples/run_quick_checks.sh      # run examples/build/ as is, without checking the record

The environment variables for the compilers, target and LAPACK are the same as for build_examples.sh (for a GPU, e.g. SYCL_TARGET=--acpp-targets=cuda:sm_70). The MPI launcher is set with MPIEXEC (default mpirun), MPIEXEC_NP_FLAG (default -n), MPI_NP (default 2) and MPIEXEC_ARGS. For MPI libraries that cannot run MPI_Init without the launcher (e.g. Fujitsu MPI), MPIEXEC_ALL=1 starts the single-process runs through the launcher as well. Fujitsu mpiexec writes the standard output of the ranks to the job log, where the check lines cannot be read, so with Fujitsu mpiexec (MPIEXEC=mpiexec whose real file is under /opt/FJSV…) RC_ONLY=1 (pass if the exit status is 0) is the default (use RC_ONLY=0 to turn it off). When installed with spack, spack test run parcel-sycl py-parcel-sycl performs the same check (“Installing with spack”).

Number of threads and where it runs. If OMP_NUM_THREADS is not set, all available logical CPUs are used on a compute node (inside a job), and 4 on a login node (or the number available, if fewer than 4). For the MPI run this is divided by the number of processes so that the total stays the same (at least 1; 2 processes on a login node get 2 threads each). The number of available logical CPUs is the smallest of the CPU affinity (nproc), the CPU limit of the cgroup, and the job’s allocation on this node (PBS NCPUS, Slurm SLURM_CPUS_ON_NODE). Whether it is a compute node is decided by the presence of a job environment variable (PBS PBS_JOBID, Fujitsu PJM PJM_JOBID, Slurm SLURM_JOB_ID), and can be forced with QUICK_NODE=compute / QUICK_NODE=login. The decision and the values used are printed first. If OMP_NUM_THREADS is set, its value is used (in a job the scheduler may set it: PBS ompthreads (a queue default of 1 is possible), Fujitsu PJM --omp thread=). With Open MPI, --bind-to none is added unless MPIEXEC_ARGS specifies a placement (--bind-to, --map-by) (with 2 or fewer processes Open MPI binds each process to one core, which would pack several threads onto one core).

Where to run it. In CPU environments, a login node is fine (follow the rules of the site). In GPU environments and on A64FX (Odyssey), run it on a compute node (a short batch job or an interactive job): login nodes have no GPU, and A64FX executables do not run on the x86 login nodes. When the MPI run (2 processes) is done in a GPU environment, the script does not assign one GPU per process (as gpu_bind.sh in “Mapping One Rank to One Device” does), so both processes may use the same GPU.

4.3 Compile Options

This section describes the flags specified at compile time. The -D macros of the library itself (Include/) are described in the section Macros. The run-time options of the samples (environment variables PARCEL_SOLVER, PARCEL_NX, PARCEL_PRECON, etc.) are described in the “Usage” chapter.

Flags required in all environments

Flag Description
--acpp-targets=<target> Kind of code to generate. omp (CPU, OpenMP), omp.library-only (A64FX + Fujitsu compiler), cuda:sm_70 (V100), cuda:sm_80 (A100), cuda:sm_90 (H100), hip:gfx942 (MI300A)
-I<location of lapacke.h> Include path for lapacke.h. Example: -I/usr/include/openblas
-l<LAPACK library> Library containing LAPACKE. Examples: -lopenblas, Fujitsu -lfjlapack, on PS -l:libopenblaso.so.0
-O3 Optimization. Specified in all environments

Environment-dependent flags

Flag Environments Description
-march=native -mtune=native s86 CPU/GPU, Miyabi-C, Miyabi-G, PS-A Optimization for the host CPU
-Xcuda-ptxas -maxrregcount=255 s86 GPU (V100) Raises the register limit of ptxas (the CUDA assembler). Recommended on V100. With the default allocation, the loads of the matrix and vectors in kernels such as the Jacobi preconditioner are split and serialized, which is slower. No spills occur and the results do not change. Not used on other GPUs (Register Limit on V100)
-std=c++17 Odyssey Specified for builds through the Fujitsu compiler
-DNDEBUG -fno-math-errno -fno-trapping-math PS-B Only floating-point flags that do not change values are used. -ffast-math is not used because it breaks the error terms of double-double and the detection of NaN (the build stops with #error); -fno-exceptions is not used because it is incompatible with the exceptions of the argument checks

4.4 Macros

Since PARCEL is a header-only library, the configuration and behavior of the library (Include/) can be selected by adding -D<macro> or -D<macro>=<value> when compiling the user program. Every macro works with its default value, and normally nothing needs to be specified. There are the following four kinds by purpose.

In addition, there are macros for diagnosis and isolation, which are not specified in normal builds.

4.4.1 Selecting the Solvers to Build (Include/parcel_solver_select.hpp)

All solvers are enabled by default, and normally nothing needs to be specified. Because all solvers go into a single translation unit, compilation takes a long time (6–14 minutes per configuration on A64FX); when the solvers to be used are known, for example during troubleshooting, disabling the others shortens the compilation.

Each macro builds the solver when set to 1 and excludes it when set to 0. A solver macro that is not specified takes the value of PARCEL_ENABLE_ALL_SOLVERS (default 1), so all solvers are built when nothing is specified. To build only some solvers, set PARCEL_ENABLE_ALL_SOLVERS=0 and set the solvers to be built to 1.

Macro Default Description
PARCEL_ENABLE_ALL_SOLVERS 1 Value used for each solver macro below when it is not specified. If set to 0, only the solvers set to 1 are built
PARCEL_ENABLE_PCG 1 (the value of PARCEL_ENABLE_ALL_SOLVERS) parcel::krylov::pcg
PARCEL_ENABLE_PBICGSTAB 1 (the value of PARCEL_ENABLE_ALL_SOLVERS) parcel::krylov::pbicgstab
PARCEL_ENABLE_PGMRES 1 (the value of PARCEL_ENABLE_ALL_SOLVERS) parcel::krylov::pgmres
PARCEL_ENABLE_PCAGMRES 1 (the value of PARCEL_ENABLE_ALL_SOLVERS) parcel::krylov::pcagmres (including the variant that does not form QQ, CAGMRES_Q_IMPLICIT)
PARCEL_ENABLE_POWER 1 (the value of PARCEL_ENABLE_ALL_SOLVERS) parcel::krylov::power
PARCEL_ENABLE_INVERSE_POWER 1 (the value of PARCEL_ENABLE_ALL_SOLVERS) parcel::krylov::inverse_power
PARCEL_ENABLE_KS_ARNOLDI 1 (the value of PARCEL_ENABLE_ALL_SOLVERS) parcel::krylov::ks_arnoldi. If set to 0, LAPACKE is no longer required (except for the verification sample)

Dependencies are resolved automatically: enabling inverse_power enables pcg and pgmres, and enabling ks_arnoldi enables pgmres. If no solver is enabled, a compile error occurs. There is no macro named PARCEL_ENABLE_CHOLQR_PCAGMRES; specifying it causes a compile error (#error).

# build only pgmres (exclude all others)
-DPARCEL_ENABLE_ALL_SOLVERS=0 -DPARCEL_ENABLE_PGMRES=1
# exclude only ks_arnoldi (build all others)
-DPARCEL_ENABLE_KS_ARNOLDI=0

4.4.2 Integer Type

Integers such as row indices, column indices and numbers of nonzeros are held in the type of the macro PARCEL_INT for global indices, and in the type of the macro PARCEL_LINT for indices within one rank (such as the CRS on the device). By default, PARCEL_INT is 64 bits (int64_t; MPI_INT64_T in MPI communication) and PARCEL_LINT is 32 bits (int32_t, MPI_INT32_T). The reasons for these defaults (integer overflow in large problems and the speed of the local kernels) are given in “Integer Type” in the Basic Structure chapter.

Specification PARCEL_INT PARCEL_LINT Notes
None (default) int64_t int32_t
-DPARCEL_LOCAL_INT64 int64_t int64_t -DPARCEL_LOCAL_INT64=0 is the same as the default
-DPARCEL_INT32 int32_t int32_t -DPARCEL_INT32=0 is the same as the default
-DPARCEL_INT int64_t int32_t Former way to make PARCEL_INT 64-bit. Accepted for compatibility (its value is not used)
Both -DPARCEL_INT and -DPARCEL_INT32 — — Contradictory; stopped with #error
Both -DPARCEL_INT32 and -DPARCEL_LOCAL_INT64 — — The rank-local type would be wider than the global type; stopped with #error

The selected widths are available as the macros PARCEL_INT_BITS and PARCEL_LINT_BITS (64 or 32 each).

How to choose the type (the range of matrices, integer overflow in large problems, the effect on memory and on the speed of the local kernels, and the relation to the C and Fortran APIs) is described in “Integer Type” in the Basic Structure chapter.

4.4.3 Optimization

The macros of optimizations that increase speed without changing the computation (the algorithm and the number of iterations) are listed below: those that select the loop form of the kernels (the first eight) and those that make the solvers faster by using more memory (the last two; optimization options). How each optimization works, its effects and the defaults per environment are described in the “Optimization” chapter, and the compile commands per environment in Recommended Options. The defaults reduce memory.

Macro Default Description
_BLOCKING_LOOP_ 0 If 1, replaces all kernels except SpMV (dot products, axpy, preconditioners, etc.) with blocked versions (Blocking). For A64FX
_BLOCKING_LOOP_DOT_ 0 If 1, replaces only the reductions (dot, dddot, max) with blocked versions. For CPUs other than A64FX. Do not specify on GPUs
PARCEL_SPMV_LADDER 0 If 1, splits the inner loop of SpMV in a ladder (Ladder Splitting). For AMD GPUs and A64FX. Not used on NVIDIA GPUs
PARCEL_PRECON_LADDER 0 (the value of PARCEL_SPMV_LADDER) Ladder splitting of the iteration loops of the Jacobi preconditioner. Set explicitly to 0 on A64FX
PARCEL_GEMV_AS_AXPY 0 (the value of _BLOCKING_LOOP_) If 1, decomposes gemv into M axpy operations (Decomposing gemv). Changes the rounding order
PARCEL_A64FX_AXPY_C 1 If 1, uses the versions of the axpy that takes coefficients by value and of the axpy decomposition of gemv rewritten for A64FX (C in Optimizations for A64FX). 0 gives the form before the rewrite. The results do not change. Effective only in _BLOCKING_LOOP_ builds
PARCEL_CG_FUSED 0 (the value of _BLOCKING_LOOP_) If 1, fuses the kernels of one iteration of pcg (the updates of xx and pp, the update of rr and ⟨r,r⟩\langle r, r \rangle; D in Optimizations for A64FX). The results do not change. Can be 1 only in _BLOCKING_LOOP_ builds
PARCEL_JACOBI_NODIV_UNIT 1 If 1, skips the division in the preparation of the right-hand side of the Jacobi preconditioner when the divisor is 1 (E in Optimizations for A64FX). The results do not change. Effective only in _BLOCKING_LOOP_ builds
PARCEL_JACOBI_ROW2 0 (the value of _BLOCKING_LOOP_; 0 if PARCEL_PRECON_LADDER is 1) If 1, processes the Jacobi sweep two rows at a time alternately when the preconditioner matrix is single precision (F in Optimizations for A64FX). The results do not change. Not used for a double-precision preconditioner matrix or with on-the-fly computation
PARCEL_JACOBI_ONTHEFLY 1 (0 for A64FX) The Jacobi preconditioner (PRECON_JACOBI) does not keep the values of B=DA−1RB = D_A^{-1}R and computes them from the values of AA and DA−1D_A^{-1} at every application. With 0 the values are stored (optimization option “without H”: uses the memory of the number of nonzeros of AA × size of the preconditioner matrix precision). With matrix scaling, a double-double coefficient matrix, preconditioner matrix or preconditioner computation precision, the values are stored even with 1. In builds for A64FX, where _BLOCKING_LOOP_ (or _BLOCKING_LOOP_SPMV_) is 1, the default is 0 (Choosing Between Memory and Speed)
PARCEL_PCG_NO_XHIST 1 Default of the number of iterations (pstep) over which pcg batches the solution update. 1 gives 1 (one search direction, updated at every iteration); 0 gives 4 (optimization option “without G”: uses the memory of 3 vectors with halo). The run-time environment variable PARCEL_PSTEP takes precedence

4.4.4 Default Orthogonalization of ks_arnoldi

The orthogonalization of the Arnoldi process of ks_arnoldi (Krylov-Schur method in the eigenvalue solver chapter) is selected in the program with set_qr_algo() (set_qr_algo() of ks_arnoldi in the API Reference). The following macro selects at build time the default used when it is not called. The environment variable PARCEL_KS_ORTHO is not read (if it is set, rank 0 prints a one-line warning to standard error at the first solver(); the behavior does not change).

Macro Default Description
PARCEL_KS_ORTHO_DEFAULT undefined (cgs2) cgs2 (classical Gram-Schmidt twice) or mgs (modified Gram-Schmidt twice). The build for PS-A (the spack environment psa, variant ks_ortho=mgs) uses mgs. When undefined, CGS2 (QR_CGS). A value given with set_qr_algo() takes precedence

4.4.5 Checks and Numerical Operations

These macros select checks that prevent wrong results and the method of double-double operations. Keep the defaults; removing the checks can silently return wrong results, so specify them only when you understand the reasons.

Macro Default Description
PARCEL_REQUIRE_IN_ORDER 1 At the entry of set_crsMat() and of solver() of each solver, checks that the SYCL queue is in-order, and throws std::invalid_argument on violation (including when there is no queue). PARCEL submits kernels to one queue in order without sycl::event or depends_on, and each kernel reads the results of the previous one, so in-order execution is assumed. With the SYCL 2020 default (out-of-order) queue, the execution order of USM kernels that use no accessors is not guaranteed. 0 disables the check
PARCEL_NO_CRS_CHECK 0 1 disables the input checks of set_crsMat() (the normal version and the hi/lo version for double-double). The checks are: the local number of rows n equals nend-nstart+1 and is at least one (a rank with zero rows is not supported); rowptr[0] == 0, it is non-decreasing, and rowptr[n] == nnz; the row blocks of all ranks cover all rows without gaps or overlaps (in any rank order); and the column indices are within the global number of rows. The non-MPI version checks only rowptr and the column indices. On violation all ranks throw the same std::invalid_argument (the message names the offending rank and what is wrong). The cost is one pass over the nonzeros and two collective communications. If the checks are disabled and a broken input is given, the result is silently wrong because uninitialized receive buffers of the halo exchange are used, or memory outside the arrays is read
PARCEL_ALLOW_FAST_MATH Undefined If defined, allows builds with -ffast-math / -ffinite-math-only (__FAST_MATH__ / __FINITE_MATH_ONLY__). By default parcel_sycl.hpp stops with #error. Rewriting expressions by associativity eliminates the error terms of double-double, and dd silently falls to double precision. In addition, the NaN / Inf checks (std::isfinite) are removed and DIVERGED_NANORINF no longer works. If defined, the precision of dd and the detection of NaN are not guaranteed
PARCEL_DD_NO_FMA Undefined If defined, replaces the error-free transformation of double-double products (TwoProd) by fma (p=fma(a,b,0)p = \mathrm{fma}(a,b,0), e=fma(a,b,−p)e = \mathrm{fma}(a,b,-p), the default) with Dekker splitting. For CPUs without FMA instructions. Do not use it on GPUs (CUDA), where Dekker splitting breaks

Do not use -fno-exceptions either. It is incompatible with the argument checks (std::invalid_argument, etc.).

The recommended flags for each environment are shown below. Blocking of reductions is specified for CPUs other than A64FX, blocking of all kernels, ladder splitting of SpMV, and fusion of multiply-adds into FMA (-ffp-contract=fast) for A64FX, together with OMP_WAIT_POLICY=ACTIVE and KMP_BLOCKTIME=infinite at run time (Optimizations for A64FX), and ladder splitting for AMD GPUs. NVIDIA GPUs do not use ladder splitting, and V100 uses the register limit.

Environment --acpp-targets Kernel optimization LAPACKE
s86 CPU omp -D_BLOCKING_LOOP_DOT_=1 -I/usr/include/openblas -lopenblas
s86 GPU cuda:sm_70 None (no ladder splitting) + -Xcuda-ptxas -maxrregcount=255 -I/usr/include/openblas -lopenblas
Odyssey omp.library-only -D_BLOCKING_LOOP_=1 -DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=0 + -ffp-contract=fast -I/usr/include/openblas -lfjlapack
Aquarius cuda:sm_80 None -I/usr/include/openblas -lopenblas
Miyabi-C omp -D_BLOCKING_LOOP_DOT_=1 -I/usr/include/openblas -lopenblas
Miyabi-G cuda:sm_90 None -I/usr/include/openblas -lopenblas
PS-A omp -D_BLOCKING_LOOP_DOT_=1 lapacke.h from nvhpc + -l:libopenblaso.so.0
PS-B hip:gfx942 -DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=1 lapacke.h from nvhpc + -l:libopenblaso.so.0

The mechanisms of the optimizations and their measured effects per environment are described in the “Optimization” chapter.

The compile commands for each environment are shown below. The environment is set up with “Settings for use” of each environment in the chapter “Installing a SYCL Compiler”. Each is an example of building the verification sample tests/verify/parcel_krylov_sample_verify.cpp in double precision in the top-level directory of the package. The solver, problem size, and preconditioner are specified at run time (Run-Time Options of the Verification Sample). Without -DMAX_ITERATION, the maximum number of iterations is 10000 (the default in tests/verify/parcel_default.hpp, the same as the library default), and it can be changed at run time with PARCEL_MAX_ITER. Compilation uses neither MPI nor GPUs, so it is done on the login node except on A64FX. Replace <spack installation directory> and similar placeholders according to your environment.

4.5.1 s86 (CPU)

# Environment: apply "Settings for use" of "s86" in the chapter "Installing a SYCL Compiler"

mpicxx -O3 --acpp-targets=omp -march=native -mtune=native \
  -D_BLOCKING_LOOP_DOT_=1 \
  -I/usr/include/openblas \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas

4.5.2 s86 (GPU)

# Environment: apply "Settings for use" of "s86" in the chapter "Installing a SYCL Compiler"

mpicxx -O3 --acpp-targets=cuda:sm_70 -march=native -mtune=native \
  -Xcuda-ptxas -maxrregcount=255 \
  -I/usr/include/openblas \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas

4.5.3 Odyssey

On A64FX, the login nodes and compute nodes have different architectures, so the build is done on a compute node. An example build job script is shown below. The MPI wrapper mpiFCC is used, and OMPI_CXX=acpp makes it invoke acpp. With the omp.library-only target, acpp uses the Fujitsu compiler given in ACPP_CPU_CXX as the host compiler. As in the other environments, the location of the LAPACKE header is specified with -I/usr/include/openblas (with this, the build works on the compute node), and Fujitsu’s libfjlapack is used as the implementation.

#!/bin/bash
#PJM -L "rscgrp=short-o"
#PJM -L "node=1"
#PJM -L "elapse=2:00:00"
#PJM -g <group>
#PJM --omp thread=48
#PJM -j

# Environment: apply "Settings for use" of "Odyssey" in the chapter "Installing a SYCL Compiler"

cd ${PJM_O_WORKDIR}
LINKOPT="-L$fj_root/lib64 -Wl,-rpath,$fj_root/lib64 \
  -L$sycl_root/lib -Wl,-rpath,$sycl_root/lib \
  -L$llvm_root/lib -Wl,-rpath,$llvm_root/lib -lfjlapack"

mpiFCC -O3 -std=c++17 --acpp-targets=omp.library-only \
  -D_BLOCKING_LOOP_=1 -DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=0 \
  -ffp-contract=fast \
  -I/usr/include/openblas \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp ${LINKOPT}

A single build takes longer than on Xeon and needs several GiB of memory. The memory available to a single-node job is 28 GiB, so limit the number of concurrent builds when building several precision configurations at the same time.

-ffp-contract=fast makes the compiler fuse multiply-adds (a×b+ca \times b + c) into FMA instructions. The clang mode of the Fujitsu compiler does not fuse them by default. AdaptiveCpp passes the command-line -O3 to the Fujitsu compiler after ACPP_OMP_CXX_FLAGS, so the -Kfast in ACPP_OMP_CXX_FLAGS is canceled by the later -O3 and does not take effect. If fast-math is enabled, for example by passing -Kfast without a later -O3, double-double and the NaN detection break, so PARCEL stops the build with #error (“Do Not Use -ffast-math”). -ffp-contract=fast on the command line reaches the Fujitsu compiler. On one node of Odyssey (4 ranks × 12 threads, NX=256, pcg + Jacobi preconditioner), one iteration became 3 % faster (SpMV 10 %, the body of axpy 25 %). The number of iterations does not change, and the results change in the last digits (about 1e-11 relative in the residual). FMA adds without rounding the product, so the results are not bit-identical to a build without the flag. Use -ffp-contract=off if the results must agree. Inside the double-double functions, Include/ddlib.hpp still prohibits FMA (last paragraph of this section).

In builds for A64FX (-D_BLOCKING_LOOP_=1), the Jacobi preconditioner stores the values of BB by default (“Without H” in Choosing Between Memory and Speed; the default of PARCEL_JACOBI_ONTHEFLY is 0, so nothing needs to be specified). On A64FX the on-the-fly computation (a multiplication and a diagonal check per nonzero) shows up in the preconditioner time: on one node of Odyssey (NX=256, pcg + Jacobi preconditioner), storing the values makes the preconditioner section 0.77 times and the total time to convergence 0.81 times. In exchange, the memory grows by the size of the preconditioner matrix precision per nonzero of AA (8 bytes in double precision). To reduce memory, specify -DPARCEL_JACOBI_ONTHEFLY=1 explicitly.

At run time, set OMP_WAIT_POLICY=ACTIVE and KMP_BLOCKTIME=infinite, which keep the OpenMP threads spinning while they wait (recommended; this is not a library default, so set it in the job environment). On one node of Odyssey (4 ranks × 12 threads, one rank per CMG), the fixed cost per iteration that does not depend on the problem size drops from 5.2 ms to 3.0 ms. It helps when the number of rows per device is small. Waiting threads keep using their cores, so it is counterproductive when processes compete for cores (for example, other processes sharing the node). Together with the rewritten axpy and the fused pcg kernels that are enabled in this build, the effects and caveats are described in Optimizations for A64FX.

In code before 2026-10-10, the prohibition of FMA for the double-double arithmetic (#pragma clang fp contract(off)) did not end at the end of ddlib.hpp with the Fujitsu compiler, so no FMA instructions were used in any of the following kernels (the clang mode of the Fujitsu compiler does not know the push / pop of float_control). The current code limits the prohibition to the double-double functions, and the other kernels use FMA as the compiler options say (-ffp-contract=fast above). Results in double precision and other precisions differ from earlier builds in the last digits.

4.5.4 Aquarius

# Environment: apply "Settings for use" of "Aquarius" in the chapter "Installing a SYCL Compiler"

mpicxx -O3 --acpp-targets=cuda:sm_80 \
  -I/usr/include/openblas \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas

4.5.5 Miyabi-C

# Environment: apply "Settings for use" of "Miyabi-C" in the chapter "Installing a SYCL Compiler"

mpicxx -O3 --acpp-targets=omp -march=native -mtune=native \
  -D_BLOCKING_LOOP_DOT_=1 \
  -I/usr/include/openblas \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas

4.5.6 Miyabi-G

# Environment: apply "Settings for use" of "Miyabi-G" in the chapter "Installing a SYCL Compiler"

mpicxx -O3 --acpp-targets=cuda:sm_90 -march=native -mtune=native \
  -I/usr/include/openblas \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp -lopenblas

4.5.7 PS-A

The PS compute nodes do not have /usr/include/openblas/lapacke.h (it exists only on the login nodes). In addition, MKL provides only mkl_lapacke.h. Therefore, the lapacke.h shipped with nvhpc is borrowed as the header, and the implementation is resolved to the OS OpenBLAS (the OpenMP version libopenblaso). Set OPENBLAS_NUM_THREADS=1 so that OpenBLAS threads are not nested inside PARCEL’s OpenMP threads. Since the MPI is Intel MPI, acpp is also specified with I_MPI_CXX.

# Environment: apply "Settings for use" of "PS-A" in the chapter "Installing a SYCL Compiler"

mpicxx -O3 --acpp-targets=omp -march=native -mtune=native \
  -D_BLOCKING_LOOP_DOT_=1 \
  -I/system/apps/rhel9/dev/nvhpc/25.3/Linux_x86_64/25.3/compilers/include/lp64 \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp -l:libopenblaso.so.0

4.5.8 PS-B

LAPACKE is handled in the same way as on PS-A.

# Environment: apply "Settings for use" of "PS-B" in the chapter "Installing a SYCL Compiler"

mpicxx -O3 --acpp-targets=hip:gfx942 \
  -DNDEBUG -fno-math-errno -fno-trapping-math \
  -DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=1 \
  -I/system/apps/rhel9/dev/nvhpc/25.3/Linux_x86_64/25.3/compilers/include/lp64 \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp -l:libopenblaso.so.0

4.5.9 C and Fortran Programs

Programs that use the C or Fortran API compile the wrapper capi/wrapper_parcel_sycl.cpp with the same flags as the mpicxx command of each environment above (--acpp-targets, kernel optimization, LAPACKE), and link it with the C or Fortran main program using the SYCL compiler. Use the same --acpp-targets for compiling and linking the wrapper. acpp (clang) links as PIE, so add -fPIE to objects built with gcc or gfortran. An example for s86 CPU (Open MPI) follows; in other environments, replace the flags on the first line by those of the environment.

# Environment: apply "Settings for use" of "s86" in the chapter "Installing a SYCL Compiler"
FLAGS="-O3 --acpp-targets=omp -march=native -mtune=native -D_BLOCKING_LOOP_DOT_=1"
mpicxx ${FLAGS} -I/usr/include/openblas -c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o

# C
mpicc  -O2 -fPIE -Icapi -c prog.c -o prog.o
mpicxx --acpp-targets=omp -o prog prog.o wrapper_parcel_sycl.o -lopenblas -lm

# Fortran (compile the PARCEL module first)
mpif90 -O2 -fPIE -c capi/parcel_sycl_mod.f90 -o parcel_sycl_mod.o
mpif90 -O2 -fPIE -c prog.f90 -o prog.o
mpicxx --acpp-targets=omp -o prog prog.o parcel_sycl_mod.o wrapper_parcel_sycl.o \
       $(mpif90 --showme:link) -L$(dirname $(mpif90 -print-file-name=libgfortran.so)) -lgfortran -lopenblas

4.5.10 macOS (Homebrew)

In an environment installed with Homebrew (“macOS (Homebrew)” in “Installing a SYCL Compiler”), use --acpp-targets=omp.library-only with blocking of the reductions only (-D_BLOCKING_LOOP_DOT_=1, as on other CPUs), and specify the Homebrew OpenBLAS for LAPACKE. With the Homebrew AdaptiveCpp, --acpp-targets=omp gives the same compilation. -march=native is not needed (it makes no difference in speed). Start with mpirun -n <processes> --bind-to none. The following was confirmed with the programs of the four languages in the Quick Start.

eval "$(/opt/homebrew/bin/brew shellenv)"
export OMPI_CXX=acpp
B=/opt/homebrew
T="--acpp-targets=omp.library-only"
LAPACK_INC="-I$B/opt/openblas/include"
LAPACK_LIB="-L$B/opt/openblas/lib -lopenblas -Wl,-rpath,$B/opt/openblas/lib"
FC_RT="-L$B/opt/gcc/lib/gcc/current -lgfortran -Wl,-rpath,$B/opt/gcc/lib/gcc/current"

# C++ (with MPI; without MPI, replace mpicxx by acpp and add -DPARCEL_MPI=0)
mpicxx -O3 $T -D_BLOCKING_LOOP_DOT_=1 ${LAPACK_INC} \
  -o verify_double tests/verify/parcel_krylov_sample_verify.cpp ${LAPACK_LIB}

# C and Fortran (with MPI)
mpicxx -O3 $T -D_BLOCKING_LOOP_DOT_=1 ${LAPACK_INC} -c capi/wrapper_parcel_sycl.cpp -o wrapper_parcel_sycl.o
mpicc  -O2 -Icapi -c prog.c -o prog_c.o
mpicxx $T -o prog_c prog_c.o wrapper_parcel_sycl.o ${LAPACK_LIB}
mpif90 -O2 -c capi/parcel_sycl_mod.f90 -o parcel_sycl_mod.o
mpif90 -O2 -c prog.f90 -o prog_f.o
mpicxx $T -o prog_f prog_f.o parcel_sycl_mod.o wrapper_parcel_sycl.o $(mpif90 --showme:link) ${FC_RT} ${LAPACK_LIB}
mpirun -n 4 --bind-to none ./prog_c

The gfortran runtime library on macOS is a .dylib, so its location ($B/opt/gcc/lib/gcc/current) is given with -L and -Wl,-rpath. Sourcing python/envs/mac_brew.sh puts the same values into environment variables (SYCL_TARGET, CXXFLAGS, LAPACK_INC, LAPACK_LIB, FC_LINK), so examples/build_examples.sh can build the samples. The Python package is described in “Python Package”.

4.6 Default of MPI

Whether PARCEL is built with MPI or as the single-process version without MPI is decided by the macro PARCEL_MPI (1: with, 0: without). The default differs by usage as follows.

Usage Default of PARCEL_MPI Where it is decided How to build without / with MPI
C++ (including Include/parcel_sycl.hpp) 1 (with MPI) Headers in Include/ Add -DPARCEL_MPI=0
C (capi/parcel_sycl_c.h) 1 (with MPI) parcel_sycl_c.h and the wrapper Add -DPARCEL_MPI=0 to both the wrapper and the main program
Fortran (capi/parcel_sycl_mod.f90) 1 (with MPI) The wrapper (the module does not depend on MPI) Add -DPARCEL_MPI=0 to the wrapper and satisfy use mpi with the substitute module
examples/build_examples.sh 1 (with MPI) Environment variable PARCEL_MPI PARCEL_MPI=0 bash examples/build_examples.sh
Python (python/, pip) 0 (without MPI) python/CMakeLists.txt -C cmake.define.PARCEL_MPI=1 for MPI
spack (parcel-sycl, py-parcel-sycl) 0 (without MPI; variant ~mpi) The recipe variant mpi +mpi for MPI

C++, C and Fortran default to MPI so that programs written as for v1.0 (compiled with the MPI wrappers and started with mpirun) run as they are. Python and spack default to no MPI so that the package can be installed and tried immediately on machines without MPI (local computers, macOS, etc.). To use the version with MPI, specify it as above. In C and Fortran, if the wrapper and the main program are built with different settings, the types of the communicator argument do not match and the program does not work correctly (this is not detected at link time), so build both with the same setting. C++ is a header-only library, so the setting of the user program is used as it is. The computation is the same in both versions, and the results of the MPI version run with one process and of the version without MPI agree bit for bit.

4.7 Build without MPI

With -DPARCEL_MPI=0, a single-process version that does not use MPI can be built. It does not include mpi.h or link the MPI library, so compile it directly with the SYCL compiler (acpp, etc.) instead of the MPI compiler wrapper (mpicxx), and run it without mpirun. Without the option (the default), MPI is used. The computation is the same as running the MPI build with one process, and the results agree bit for bit.

acpp -O3 --acpp-targets=omp -march=native -DPARCEL_MPI=0 \
  -I/usr/include/openblas -o quick_start examples/quick_start.cpp -lopenblas
./quick_start

# the four programs in examples/ (C / Fortran are compiled with gcc / gfortran)
PARCEL_MPI=0 bash examples/build_examples.sh

To build C and Fortran programs by hand, add -DPARCEL_MPI=0 to both the wrapper and the C main program, and use acpp, gcc and gfortran instead of the MPI wrappers (mpicxx, mpicc, mpif90). use mpi in Fortran is satisfied by the single-process substitute module examples/sample_f_mpi_stub.f90 (which implements MPI_Init, MPI_Comm_rank, MPI_Comm_size, MPI_Finalize and others for one process). Example commands are in the “without MPI” parts of “C” and “Fortran” of the Quick Start.

User programs are written in the same way as with MPI. The differences are:


4.8 Python Package

The Python package parcel (python/ in the distribution package) is installed by building, from source, its native layer (libparcel_py.so) that calls the C++ library of PARCEL directly. You need an environment in which PARCEL can be built (“Settings for use” in the chapter Installing a SYCL Compiler), Python 3.8 or later, and NumPy. SciPy is convenient for building matrices, and the Python test of the regression tests (tests/regression/sample_py.py) uses it. The native layer is built with AdaptiveCpp. The default is the single-process version without MPI; the build setting PARCEL_MPI=1 gives the version with MPI. There are three ways to install it. Usage is described in Python in “Usage”.

4.8.1 Settings per Environment

The native layer is built for each environment (SYCL target and with or without MPI). The build settings are given with pip’s -C cmake.define.<name>=<value> (“Build Settings”). The values used in the test environments of this manual are shown below (the same as in Recommended Options). Blank cells can be left at their defaults.

System PARCEL_TARGET PARCEL_KERNEL_FLAGS PARCEL_EXTRA_FLAGS
s86 (CPU), Miyabi-C, PS-A (omp) (-D_BLOCKING_LOOP_DOT_=1)
s86 (V100) cuda:sm_70 (-DPARCEL_SPMV_LADDER=0 -DPARCEL_PRECON_LADDER=0; no ladder splitting) -Xcuda-ptxas -maxrregcount=255
Aquarius (A100) cuda:sm_80 (same as above)
Miyabi-G (GH200) cuda:sm_90 (same as above)
PS-B (MI300A) hip:gfx942 (-DPARCEL_SPMV_LADDER=1 -DPARCEL_PRECON_LADDER=1)
macOS (Apple silicon) + Homebrew omp.library-only (-D_BLOCKING_LOOP_DOT_=1)

Values in parentheses are the defaults (for PARCEL_KERNEL_FLAGS, the value chosen from the target). Give the location of LAPACKE (LAPACKE_INCLUDE_DIR, LAPACK_LIBS) for your environment (for example -l:libopenblaso.so.0 on PS-A and PS-B; see the section of each environment in Recommended Options). Build, install and run in the same environment (“Settings for use” in the chapter Installing a SYCL Compiler; the AdaptiveCpp, LAPACK and MPI libraries must be found).

On macOS, in an environment installed with Homebrew (“macOS (Homebrew)” in “Installing a SYCL Compiler”), sourcing python/envs/mac_brew.sh sets the values in the table above and the location of the Homebrew OpenBLAS in environment variables; then use pip or the direct build below.

4.8.2 Without MPI (Default)

Run pip in python/. pip uses scikit-build-core and CMake 3.18 or later for the build (and fetches them if missing), so the package index must be reachable. Upgrade pip in the venv first (an old pip may not accept -C, which passes the build settings). It may be run on a login node and takes 1 to 2 minutes.

python3 -m venv $HOME/venv_parcel
$HOME/venv_parcel/bin/python -m pip install -U pip
$HOME/venv_parcel/bin/python -m pip install scipy           # optional; convenient for building matrices (parcel depends only on NumPy)
cd python
$HOME/venv_parcel/bin/python -m pip install . -C cmake.define.PARCEL_TARGET=cuda:sm_70 \
    -C "cmake.define.PARCEL_EXTRA_FLAGS=-Xcuda-ptxas -maxrregcount=255"     # example for s86 (V100)

The native layer is built directly with acpp and does not link the MPI library.

Without pip, build the native layer directly, and set the environment variable PARCEL_PY_LIB to its location and add python/ to PYTHONPATH (run in the top directory of the package; the Python jobs of the regression tests work this way, and tests/regression/sub_build_lang_<environment>.sh builds bin/<environment>/libparcel_py.so).

acpp -O3 --acpp-targets=omp -DPARCEL_MPI=0 -march=native -mtune=native -D_BLOCKING_LOOP_DOT_=1 \
     -fPIC -shared -IInclude -I/usr/include/openblas -o libparcel_py.so python/src/parcel_py.cpp -lopenblas
PYTHONPATH=python PARCEL_PY_LIB=$PWD/libparcel_py.so python script.py

Check the installation as follows. parcel.has_mpi() returns True for the version with MPI, parcel.build_info() returns the build settings, and parcel.device_name() returns the name of the device used.

$HOME/venv_parcel/bin/python -c "import parcel; print(parcel.has_mpi(), parcel.build_info(), parcel.device_name())"

The Python API test tests/regression/sample_py.py performs, from Python, the same problem, items and checks as the C sample examples/sample_c.c (the 7 solvers, pcagmres without building QQ in two ways, ks_arnoldi with shift-invert, etc.). It can be run as the language python of the regression tests (Package Layout) or on its own. The argument is NX.

PARCEL_TEST_FROM_SOURCE=1 PARCEL_PY_LIB=$PWD/libparcel_py.so python tests/regression/sample_py.py 16   # reads python/ of the package
python tests/regression/sample_py.py 16                                                                # reads the installed parcel

4.8.3 With MPI

With the build setting PARCEL_MPI=1, the native layer is built with the MPI C++ wrapper (mpicxx by default). Set OMPI_CXX=acpp (Open MPI) or the equivalent beforehand so that the wrapper calls acpp. mpi4py is optional; to use it, install one built with the same MPI into the venv.

export OMPI_CXX=acpp
python3 -m venv $HOME/venv_parcel_mpi                       # a separate venv from the version without MPI
$HOME/venv_parcel_mpi/bin/python -m pip install -U pip
cd python
$HOME/venv_parcel_mpi/bin/python -m pip install . -C cmake.define.PARCEL_MPI=1

Without pip, replace acpp by mpicxx in the build line above and remove -DPARCEL_MPI=0. Running is described in “Running with MPI”.

Test status: both the package installed with pip and the one built from source passed all items in the version with MPI on s86 (CPU, V100), Miyabi-C, Miyabi-G, PS-A, PS-B and Aquarius. The version without MPI was confirmed on s86, Aquarius and macOS (Homebrew; also the version with MPI). The checks of py-parcel-sycl installed with spack are given in “Installing with spack”.

4.8.4 Build Settings

The names given with pip’s -C cmake.define.<name>=<value> are listed below (the same list is at the top of python/CMakeLists.txt). Names not given take the defaults in the right column.

Name Meaning Default
PARCEL_MPI 0: single-process version without MPI, 1: with MPI 0
PARCEL_TARGET The value of --acpp-targets omp
PARCEL_KERNEL_FLAGS Kernel optimization macros Chosen from PARCEL_TARGET (-D_BLOCKING_LOOP_DOT_=1 for omp, the two ladder macros for hip, no ladder splitting for cuda)
PARCEL_HOST_FLAGS Host optimization -march=native -mtune=native
PARCEL_EXTRA_FLAGS Other flags None
PARCEL_INCLUDE_DIR The Include/ of PARCEL Include/ next to python/
LAPACKE_INCLUDE_DIR Location of lapacke.h /usr/include/openblas
LAPACK_LIBS LAPACK libraries -lopenblas
PARCEL_MPICXX C++ compiler acpp without MPI, mpicxx with MPI


4.9 Installing with spack

spack/ in the distribution package contains recipes for installing PARCEL with spack, and a script that installs everything from scratch, from fetching spack through LLVM, AdaptiveCpp, OpenBLAS and Python to PARCEL. The original description is spack/README.md. The target is spack 1.x (package repository API v2.2).

4.9.1 Recipes

Package Contents
parcel-sycl Headers (include/), C API (lib/libparcel_sycl_c.so and include/parcel_sycl_c.h), Fortran module (include/parcel_sycl_mod.mod and lib/libparcel_sycl_f.a)
py-parcel-sycl The Python package parcel (builds python/ with scikit-build-core; “Python Package”)

The variants are common to both.

Variant Default Meaning
mpi off With MPI (the default is the single-process version without MPI)
cuda + cuda_arch=<n> off --acpp-targets=cuda:sm_<n>. With cuda_arch=70 (V100), -Xcuda-ptxas -maxrregcount=255 is added automatically
rocm + amdgpu_target=<gfx> off --acpp-targets=hip:<gfx>
(neither cuda nor rocm) --acpp-targets=omp (CPU)
native on -march=native -mtune=native
kernel auto Kernel optimization (“Optimization”): auto (blocking_dot on CPUs, ladder on AMD GPUs (+rocm), none on NVIDIA GPUs (+cuda), a64fx for +fj), blocking_dot, a64fx, ladder, none
fj off For A64FX: compiles the host side with the Fujitsu compiler (FCC) and builds the Fortran module with frt (confirmed with the Odyssey definition wis_ody). Adds -std=c++17 -ffp-contract=fast (Recommended Options)
capi (parcel-sycl only) on Builds the C / Fortran API
scipy (py-parcel-sycl only) on Also installs SciPy (used by DistributedCSR.from_scipy() / from_global() and examples/quick_start.py; corresponds to parcel[scipy] with pip)
ks_ortho cgs2 Default orthogonalization of ks_arnoldi (“Default Orthogonalization of ks_arnoldi”). mgs adds -DPARCEL_KS_ORTHO_DEFAULT=mgs (also in PARCEL_CXXFLAGS). A value given by the program with set_qr_algo() takes precedence. py-parcel-sycl is built with parcel-sycl of the same value

To add them to an existing spack environment, use the local distribution package with spack develop.

spack -e <env> repo add <package>/spack/spack_repo/parcel
spack -e <env> config add "concretizer:unify:when_possible"          # when installing CPU and GPU versions side by side
spack -e <env> develop --no-clone --path <package> parcel-sycl@1.2.0
spack -e <env> develop --no-clone --path <package> py-parcel-sycl@1.2.0
spack -e <env> add "parcel-sycl@1.2.0 ~mpi ~cuda ~rocm" "py-parcel-sycl@1.2.0 ~mpi ~cuda ~rocm"                    # CPU
spack -e <env> add "parcel-sycl@1.2.0 ~mpi +cuda cuda_arch=70" "py-parcel-sycl@1.2.0 ~mpi +cuda cuda_arch=70"     # GPU
spack -e <env> concretize -f && spack -e <env> install

To use them, spack load with the configuration specified. spack load parcel-sycl sets PARCEL_CXXFLAGS (target, optimization, with or without MPI, header location) and PARCEL_LDFLAGS (RPATHs of the libraries needed at run time (LLVM libomp, AdaptiveCpp, LAPACK, MPI) and LAPACK) so that user programs are compiled with the same settings as PARCEL. The version with MPI also sets OMPI_CXX=acpp etc., so compile with mpicxx in the same way. The RPATH of libomp is not added through the MPI wrapper, so always add PARCEL_LDFLAGS. spack load does not set the locations of the headers (parcel_sycl_c.h, parcel_sycl_mod.mod) and libraries of the C and Fortran APIs; find them with spack location -i and give them with -I, -L and -Wl,-rpath.

spack env activate <env>
spack load parcel-sycl+cuda py-parcel-sycl+cuda                   # load with the configuration specified
acpp $PARCEL_CXXFLAGS -O3 -o prog prog.cpp $PARCEL_LDFLAGS        # C++
P=$(spack location -i parcel-sycl+cuda)                           # C (version without MPI): give the location yourself
gcc -I$P/include -o prog prog.c -L$P/lib -lparcel_sycl_c -Wl,-rpath,$P/lib
python script.py                                                  # Python

Notes:

4.9.2 Installing from the Distribution Tarball

The recipes hold the url and sha256 of the distribution tarball. The url points to the tarball on the GitHub Release (https://github.com/parcel-jaea/PARCEL-release/releases/download/v1.2.0/parcel_sycl-1.2.0.tar.gz), and tools/make_tarball.py --pin writes the sha256 into the recipes (“Verifying the Distribution”). Without spack develop, spack fetches the tarball from the url (parcel-sycl and py-parcel-sycl use the same tarball). When the computer cannot connect to the outside, place the tarball in the layout of a spack mirror (<mirror>/_source-cache/archive/<first two characters of the sha256>/<sha256>.tar.gz) and register it with spack mirror add. spack checks the fetched tarball against the sha256 in the recipe and stops if it differs. The sha256 is part of the spack hash, so rewriting it rebuilds PARCEL (also when it is used with spack develop). The procedure is given in the section “Installing from the Release Tarball” of spack/README.md.

4.9.3 Choosing the Configurations

The configurations to install are chosen by the combination of packages (parcel-sycl, py-parcel-sycl) and variants (the spec). Examples of specs by purpose are shown below (the version @1.2.0 is omitted).

Purpose Spec
CPU, without MPI (C++, C, Fortran) parcel-sycl ~mpi ~cuda ~rocm
CPU, with MPI parcel-sycl +mpi ~cuda ~rocm
NVIDIA GPU (example: A100) parcel-sycl ~mpi +cuda cuda_arch=80
NVIDIA GPU, with MPI parcel-sycl +mpi +cuda cuda_arch=80
AMD GPU (MI300A) parcel-sycl ~mpi +rocm amdgpu_target=gfx942
C++ (headers) only, without the C and Fortran APIs parcel-sycl ~capi ~mpi ~cuda ~rocm
Python as well Add py-parcel-sycl with the same configuration (e.g., py-parcel-sycl ~mpi ~cuda ~rocm)

With installation from scratch, the configurations installed are those listed in specs: of spack/stack/envs/<env>/spack.yaml. The bundled definitions list all configurations available on each system (for example, 8 lines on s86: CPU / GPU × without / with MPI × parcel-sycl / py-parcel-sycl). To narrow or change the configurations, edit specs: before running install_stack.sh. For example, to install only C++, C, Fortran and Python for the CPU with MPI, write as follows. If the list is edited after installation, rerunning install_stack.sh with the same command re-concretizes and installs only what is missing.

spack:
  specs:
  - parcel-sycl@1.2.0 +mpi ~cuda ~rocm         # CPU with MPI (C++, C, Fortran)
  - py-parcel-sycl@1.2.0 +mpi ~cuda ~rocm      # Python with the same configuration

4.9.4 Installing from Scratch

Without an existing spack, spack/stack/install_stack.sh installs everything from fetching spack to PARCEL in one run. The arguments are the environment name (a directory under spack/stack/envs/), the installation directory and the number of parallel jobs.

bash spack/stack/install_stack.sh s86 $HOME/parcel_stack 16           # <env> <prefix> [jobs]
bash spack/stack/check_stack.sh $HOME/parcel_stack $HOME/parcel_check  # check on the login node
. $HOME/parcel_stack/setup.sh                                          # when using it
spack load parcel-sycl+mpi+cuda py-parcel-sycl+mpi+cuda

4.9.5 Provided Environments

envs/ System Configurations Taken from the system
s86 s86 (Xeon + V100) CPU / GPU (cuda_arch=70) × without / with MPI GCC 9.5.0, CUDA 12.6, Open MPI 4.1.6
miyc Miyabi-C CPU × without / with MPI GCC 11.4.1, Open MPI 4.1.6
miyg Miyabi-G GPU (cuda_arch=90) × without / with MPI GCC 11.4.1, CUDA 12.6, Open MPI 4.1.6 (CUDA build)
psa PS-A CPU × without / with MPI (ks_ortho=mgs: MGS2 is the default orthogonalization of ks_arnoldi) GCC 11.5.0, Intel MPI 2021.18
psb PS-B GPU (amdgpu_target=gfx942) × without / with MPI GCC 11.5.0, ROCm 6.3.3, Open MPI 5.0.7 (ROCm build)
wis_aqu Aquarius GPU (cuda_arch=80) × without / with MPI GCC 8.3.1 (used only to build GCC 12), CUDA 12.2, Open MPI 4.1.1
wis_ody Odyssey CPU (+fj) × without / with MPI GCC 8.3.1, Fujitsu compilers, MPI and LAPACK (tcsds-1.2.43)
generic Other Linux systems (template) CPU × without MPI (with MPI and NVIDIA GPUs added by uncommenting) Edit to use (GCC, MPI, CUDA)

4.9.6 Other Systems

On Linux systems other than the seven environments above, copy and edit the template spack/stack/envs/generic/, and install with install_stack.sh. The template follows the definitions for s86 and Miyabi-C. On Ubuntu 24.04 (WSL2; OS GCC 13.3.0, Open MPI 4.1.6, CUDA 12.6 from the NVIDIA repository, RTX A6000), it was edited as in the steps below and the eight configurations CPU and GPU (cuda_arch=86) × without and with MPI were installed; check_stack.sh, the Quick Start (C++, C, Python) and the verification sample (pcg, pgmres, ks_arnoldi) passed. On Rocky Linux 9.8 (compatible with RHEL 9; WSL2; OS GCC 11.5.0, Open MPI 4.1.1, CUDA 12.6 from the NVIDIA repository), the same eight configurations were installed and the same tests passed. Register the OS Open MPI of the RHEL family after putting it, including the headers, in one directory (“Registering MPI and CUDA”).

cp -r spack/stack/envs/generic spack/stack/envs/mysite           # copy and edit
CONCRETIZE_ONLY=1 bash spack/stack/install_stack.sh mysite $HOME/parcel_stack   # try concretization only
bash spack/stack/install_stack.sh mysite $HOME/parcel_stack 16                  # install

The places to edit are listed as steps in the comments at the top of spack.yaml.