PARCEL SYCL v1.2 / User Guide User Guide API Reference Search Download 日本語

3 Installing a SYCL Compiler

SYCL PARCEL consists of header files only, and a SYCL-capable compiler is required to use it. In the v1.2 tests, AdaptiveCpp (formerly hipSYCL / Open SYCL) was used in all environments. AdaptiveCpp uses LLVM (clang) as its front end and can generate code for CPUs (OpenMP), NVIDIA GPUs (CUDA), and AMD GPUs (HIP) from a single source.

AdaptiveCpp can also be built from source, but in all test environments for this version it was installed with spack. The spack package name is hipsycl. Whether to use a spack environment (spack env), its name, and the spack installation directory are up to the user. The examples below write them as <spack environment name> and <spack installation directory> (the names used in the tests are shown in the table of each environment). When no spack environment is used, omit the spack env activate line and remove --add from spack install. The compiler driver is acpp, and the kind of code to generate is specified with --acpp-targets=.

The procedure in this chapter installs AdaptiveCpp into an existing spack. Alternatively, spack/stack/install_stack.sh in the distribution package installs everything from fetching spack through LLVM, AdaptiveCpp, OpenBLAS and Python to PARCEL itself and its C, Fortran and Python APIs into a new spack environment separated from any existing spack (MPI, CUDA / ROCm and compilers are taken from each system; “Installing with spack” in the Installation chapter). On macOS, spack is not used; AdaptiveCpp is installed with Homebrew (“macOS (Homebrew)”). The procedure for Linux systems other than the test environments is in “Generic Linux”, and how to install LAPACKE in “Installing LAPACKE”.

Example configurations of the environments used for testing are shown in the table at the beginning of each environment’s section below (see the Glossary for the names of environments and devices; the hardware column is the configuration of the compute nodes). Module versions and the like may change with site updates. AdaptiveCpp in all eight environments is the same commit 7677cf6e (24.10.0+git.7677cf6e.20241220).

When the login nodes and compute nodes have different CPU architectures (e.g., Odyssey), build both AdaptiveCpp and PARCEL on the compute nodes. In environments where the architectures match, building on the login node is fine. However, when -march=native is specified, make sure that the login node and compute nodes have the same CPU generation.

When the OpenMP backend is used with --acpp-targets=omp, the run-time thread-binding behavior depends on the version of the linked libomp. With the libomp of LLVM 18.1.8 (s86 CPU, PS-A), setting OMP_PROC_BIND causes all worker threads to be packed onto a single core, which results in a large slowdown. To avoid this, leave OMP_PROC_BIND unset or set it to false at run time (other values such as spread show the same problem). If the cores of each rank are assigned by the binding of mpirun (such as --bind-to core), the threads stay within that range.

3.1 s86 (SGI8600)

s86 CPU

Item Value
Hardware (CPU / GPU) Xeon Gold 6242R ×2
OS RHEL 8.10 (x86_64)
Host compiler GCC 9.5.0 (gnu/9.5.0)
MPI Open MPI 4.1.6 + UCX 1.16.0 (openmpi/cur)
CUDA / ROCm —
AdaptiveCpp 24.10.0 (+cuda; shared with GPU)
LLVM (clang) 18.1.8 (spack, built with GCC 9.5.0)
Installation method (spack environment) spack, sycl (~/spack)
BLAS / LAPACK OpenBLAS 0.3.15 (OS)
--acpp-targets omp

s86 GPU

Item Value
Hardware (CPU / GPU) Xeon Gold 6248R ×2 + V100-SXM2-32GB ×4
OS RHEL 8.10 (x86_64)
Host compiler GCC 9.5.0
MPI Open MPI 4.1.6 + UCX 1.16.0 (openmpi/cur)
CUDA / ROCm CUDA 12.6 (/home/app/cuda/12.6)
AdaptiveCpp 24.10.0 (+cuda; shared with CPU)
LLVM (clang) 18.1.8 (spack)
Installation method (spack environment) spack, sycl (shared with CPU)
BLAS / LAPACK OpenBLAS 0.3.15 (OS)
--acpp-targets cuda:sm_70

The s86 CPU nodes have Intel Xeon Gold 6242R (20 cores ×2), and the GPU nodes have Intel Xeon Gold 6248R (24 cores ×2) + NVIDIA V100-SXM2-32GB ×4. LLVM 18 and AdaptiveCpp are installed with spack. There is no need to install separate CPU and GPU versions: a single AdaptiveCpp with the CUDA backend (+cuda) is placed in the spack environment (sycl in the tests) and used for both --acpp-targets=omp and --acpp-targets=cuda:sm_70.

Installation (once; shared by CPU and GPU)

module purge
module load gnu/9.5.0
module load openmpi/cur
. <spack installation directory>/share/spack/setup-env.sh
spack env create <spack environment name>      # first time only
spack env activate <spack environment name>
spack compiler find
spack external find
spack install --add llvm@18
spack install --add hipsycl +cuda % llvm@18

The default CUDA path of this hipsycl points to spack’s cuda@12.9.0, but for V100 builds it is overridden by specifying the site’s CUDA 12.6 with ACPP_CUDA_PATH.

Settings for use (CPU)

module purge
module load gnu/9.5.0
module load openmpi/cur
. <spack installation directory>/share/spack/setup-env.sh
spack env activate <spack environment name>
spack load hipsycl
export sycl_root=`spack location -i hipsycl`
export llvm_root=`spack location -i llvm`
export LD_LIBRARY_PATH=$sycl_root/lib/:$llvm_root/lib/:${LD_LIBRARY_PATH}
export KMP_STACKSIZE=1G
export MPICXX_CXX=acpp
export OMPI_CXX=acpp

Settings for use (GPU)

module purge
module load gnu/9.5.0
module load openmpi/cur
module load cuda/12.6
. <spack installation directory>/share/spack/setup-env.sh
spack env activate <spack environment name>
spack load hipsycl
export sycl_root=`spack location -i hipsycl`
export llvm_root=`spack location -i llvm`
export cuda_root=/home/app/cuda/12.6
export ACPP_CUDA_PATH=$cuda_root
export LD_LIBRARY_PATH=$sycl_root/lib/:$llvm_root/lib/:$cuda_root/lib64:${LD_LIBRARY_PATH}
export KMP_STACKSIZE=1G
export MPICXX_CXX=acpp
export OMPI_CXX=acpp

3.2 Odyssey (Wisteria-Odyssey, A64FX)

Item Value
Hardware (CPU / GPU) A64FX (48 cores)
OS RHEL 8 (compute nodes)
Host compiler Fujitsu FCC tcsds-1.2.43 (fj/1.2.43, clang mode)
MPI Fujitsu MPI 1.2.43 (fjmpi/1.2.43)
CUDA / ROCm —
AdaptiveCpp 24.10.0 (~cuda)
LLVM (clang) 18.1.8 (spack, built with GCC 8.3.1)
Installation method (spack environment) spack, odyssey (built on compute nodes)
BLAS / LAPACK Fujitsu LAPACK (-lfjlapack)
--acpp-targets omp.library-only

The Odyssey compute nodes are A64FX (48 cores, 4 CMGs), whose architecture differs from that of the login nodes. Therefore, both the installation of AdaptiveCpp and the build of PARCEL are performed on the compute nodes (spack is run in a batch job or an interactive job).

On Odyssey, --acpp-targets=omp.library-only is used. In this mode, AdaptiveCpp acts only as the SYCL runtime library, and the entire code, including kernels, is compiled by the host C++ compiler. The Fujitsu compiler (FCC, clang mode) is specified as the host compiler with ACPP_CPU_CXX. This enables the optimization of the Fujitsu compiler for A64FX and the Fujitsu OpenMP runtime. However, AdaptiveCpp appends -O3 after ACPP_OMP_CXX_FLAGS, so -Kfast does not take effect. Fusion of multiply-adds into FMA is specified with -ffp-contract=fast on the command line, as in the Recommended Options. LLVM 18.1.8 was installed with spack and placed in the spack environment (odyssey in the tests). Fujitsu MPI (mpiFCC) is used for MPI, and Fujitsu LAPACK (-lfjlapack) for LAPACK.

Installation (once; on a compute node)

LLVM and AdaptiveCpp for A64FX are built on a compute node. The spack environment used in the tests is defined by the two specs llvm@18 and hipsycl %clang@18: LLVM 18.1.8 is built with the OS-bundled GCC 8.3.1, and AdaptiveCpp 24.10.0 (without CUDA; ~cuda is the default) is built with that clang. The following batch job creates the same installation (spack 1.1.0.dev0 was used).

When building on a compute node, take care of the parallelism, the memory limit and the processing time.

#!/bin/bash
#PJM -L "rscgrp=<resource group>"
#PJM -L "node=1"
#PJM -L "elapse=24:00:00"           # about 3 hours for LLVM with -j12; use a long resource group
#PJM -g <group>
#PJM -j

. /etc/profile.d/modules.sh
module purge
module load odyssey
JOBS=12                               # concurrent compilations; 32 or more runs out of memory (about 28 GiB) for LLVM
export SPACK_ROOT=<spack installation directory>
# (optional) place the spack user configuration and cache outside the home directory
export SPACK_USER_CONFIG_PATH=<spack user configuration directory>
export SPACK_USER_CACHE_PATH=<spack cache directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env create <spack environment name>      # first time only
spack env activate <spack environment name>
spack compiler find                  # registers the OS GCC 8.3.1 and others
spack install -j ${JOBS} --add llvm@18
spack install -j ${JOBS} --add hipsycl %clang@18

Settings for use

On Odyssey, the following settings are also made on a compute node (inside the job), as is the build.

. /etc/profile.d/modules.sh
module purge
module load odyssey
export SPACK_ROOT=<spack installation directory>
# (optional) place the spack user configuration and cache outside the home directory
export SPACK_USER_CONFIG_PATH=<spack user configuration directory>
export SPACK_USER_CACHE_PATH=<spack cache directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env activate <spack environment name>
export sycl_root=`spack location -i hipsycl`
export llvm_root=`spack location -i llvm@18`
export fj_root=/opt/FJSVxtclanga/tcsds-1.2.43
export LD_LIBRARY_PATH=$sycl_root/lib/:$llvm_root/lib/:$fj_root/lib64/:${LD_LIBRARY_PATH}
export MPICXX_CXX=acpp
export OMPI_CXX=acpp
export ACPP_CPU_CXX="$fj_root/bin/FCC"
export ACPP_OMP_CXX_FLAGS="-O3 -Kopenmp -Kfast -std=c++17 -Nclang"

3.3 Aquarius (Wisteria-Aquarius, A100)

Item Value
Hardware (CPU / GPU) Xeon Platinum 8360Y ×2 + A100-SXM4-40GB ×8
OS RHEL 8.10 (login node)
Host compiler GCC 8.3.1 (gcc/8.3.1)
MPI Open MPI 4.1.1 (ompi/4.1.1, non-CUDA-aware build)
CUDA / ROCm CUDA 12.2 (cuda/12.2)
AdaptiveCpp 24.10.0 (+cuda)
LLVM (clang) 18.1.8 (spack); 13.0.1 also installed
Installation method (spack environment) spack, aquarius
BLAS / LAPACK OpenBLAS 0.3.15 (OS)
--acpp-targets cuda:sm_80

Since llvm@13.0.1 and llvm@18.1.8 coexist in the spack environment, specify the version explicitly, as in spack location -i llvm@18. A CUDA-aware Open MPI is separately available as ompi-cuda/4.1.5-12.2.

The Aquarius compute nodes have Intel Xeon Platinum 8360Y (36 cores ×2) + NVIDIA A100-SXM4-40GB ×8. AdaptiveCpp (with the CUDA backend) and LLVM 18.1.8 were installed in the spack environment (aquarius in the tests), and CUDA 12.2 was used. The host compiler is the OS-bundled GCC 8.3.1. For OpenMPI, the site’s ompi module is used (the default at the time of the tests was 4.1.1, a non-CUDA-aware build), and spack’s hwloc is added to LD_LIBRARY_PATH.

Installation (once)

The spack environment used in the tests is defined by the two specs llvm@18 %gcc and hipsycl+cuda %clang@18: LLVM 18.1.8 is built with the OS-bundled GCC 8.3.1, and AdaptiveCpp 24.10.0 with the CUDA backend is built with that clang. The site CUDA 12.2 of the cuda module is used as a spack external package, and OS tools such as cmake, perl and python were also registered as externals. The following steps create the same installation (spack 1.1.0.dev0 was used).

. /etc/profile.d/modules.sh
module purge
module load aquarius
module load cuda
export SPACK_ROOT=<spack installation directory>
# (optional) place the spack user configuration and cache outside the home directory
export SPACK_USER_CONFIG_PATH=<spack user configuration directory>
export SPACK_USER_CACHE_PATH=<spack cache directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env create <spack environment name>      # first time only
spack env activate <spack environment name>
spack compiler find                  # registers the OS GCC 8.3.1 and others
spack external find                  # registers OS tools and the CUDA 12.2 of the cuda module as externals
spack install --add llvm@18 %gcc
spack install --add hipsycl+cuda %clang@18

Settings for use

. /etc/profile.d/modules.sh
module purge
module load aquarius
module load cuda
module load ompi
export SPACK_ROOT=<spack installation directory>
# (optional) place the spack user configuration and cache outside the home directory
export SPACK_USER_CONFIG_PATH=<spack user configuration directory>
export SPACK_USER_CACHE_PATH=<spack cache directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env activate <spack environment name>
export cuda_root=/work/opt/local/x86_64/cores/cuda/12.2
export sycl_root=`spack location -i hipsycl`
export llvm_root=`spack location -i llvm@18`
export hwloc_root=`spack location -i hwloc`
export ACPP_CUDA_PATH=$cuda_root
export LD_LIBRARY_PATH=$hwloc_root/lib:$sycl_root/lib/:$llvm_root/lib/:$cuda_root/lib64:${LD_LIBRARY_PATH}
export KMP_STACKSIZE=1G
export MPICXX_CXX=acpp
export OMPI_CXX=acpp

The spack hwloc ($hwloc_root/lib) is placed at the head of LD_LIBRARY_PATH to avoid a version mismatch of libraries. The site Open MPI 4.1.1 uses an old hwloc; if it is loaded first, the LLVM OpenMP runtime (libomp.so), built against a newer hwloc, fails to load and the CPU backend of AdaptiveCpp becomes unusable (runs with MPI stop even with builds for the GPU). Searching the spack hwloc first makes both use the newer hwloc. The spack recipe parcel-sycl makes the same setting at spack load (“Installing with spack” in “Installation”).

With OpenMPI 4.1 on Aquarius, the mpirun processes of jobs that start at the same time may compete for the session directory in /tmp and fail to start. In run jobs, give each job its own session directory before mpirun.

PARCEL_SESS=/tmp/parcel_${PJM_JOBID:-$$}
mkdir -p ${PARCEL_SESS}
export OMPI_MCA_orte_tmpdir_base=${PARCEL_SESS}
export OMPI_MCA_btl_vader_backing_directory=${PARCEL_SESS}
trap 'rm -rf ${PARCEL_SESS}' EXIT

3.4 Miyabi-C (Xeon Max)

Item Value
Hardware (CPU / GPU) Xeon CPU Max 9480 ×2
OS RHEL 9.8 (x86_64)
Host compiler GCC 11.4.1 (gcc/11.4.1)
MPI Open MPI 4.1.6 (ompi/4.1.6)
CUDA / ROCm —
AdaptiveCpp 24.10.0 (~cuda; built with oneAPI 2023.2.0)
LLVM (clang) 17.0.6 (bundled with the OS, /usr/bin/clang++)
Installation method (spack environment) spack, miyc
BLAS / LAPACK OpenBLAS 0.3.21 (OS)
--acpp-targets omp

The Miyabi-C compute nodes have Intel Xeon CPU Max 9480 (56 cores ×2, HBM2e). AdaptiveCpp was installed on top of the site’s gcc / ompi modules as the spack environment (miyc in the tests). AdaptiveCpp itself was built with oneAPI 2023.2.0, and kernels are compiled with the clang 17.0.6 bundled with the OS (RHEL9) (/usr/bin/clang++; registered in spack as an external). The spack LLVM is not used, and the OpenMP runtime is also the OS-bundled libomp 17.0.6, so there is no need to add LLVM to LD_LIBRARY_PATH.

Installation (once)

The spack environment used in the tests is defined by the single spec hipsycl %oneapi: AdaptiveCpp 24.10.0 (without CUDA) was built with the site Intel oneAPI 2023.2.0. The oneAPI compilers, the OS clang 17.0.6 (/usr/bin/clang, /usr/bin/clang++) and Open MPI are written as external packages in the definition of the spack environment (spack.yaml) as follows. The following procedure creates the same installation (spack 1.1.0.dev0 was used). It may be run on a login node.

  packages:
    intel-oneapi-compilers:
      externals:
      - spec: intel-oneapi-compilers@2023.2.0
        prefix: /work/opt/local/x86_64/cores/intel/2023.2.0
    llvm:
      externals:
      - spec: llvm@17.0.6+clang~flang+lld~lldb
        prefix: /usr
    openmpi:
      externals:
      - spec: openmpi@4.1.6~cuda fabrics=ucx schedulers=slurm
        prefix: /work/opt/local/x86_64/apps/gcc/11.4.1/ompi/4.1.6
      buildable: false
    hipsycl:
      require: '%oneapi'
. /etc/profile.d/modules.sh
module purge
module load gcc
module load ompi
export SPACK_ROOT=<spack installation directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env create <spack environment name>      # first time only; add the packages above to spack.yaml
spack env activate <spack environment name>
spack compiler find                  # registers the OS GCC 11.4.1 and others
spack install --add hipsycl %oneapi

Settings for use

. /etc/profile.d/modules.sh
module purge
module load gcc
module load ompi
export SPACK_ROOT=<spack installation directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env activate <spack environment name>
spack load hipsycl
export sycl_root=`spack location -i hipsycl`
export LD_LIBRARY_PATH=$sycl_root/lib/:${LD_LIBRARY_PATH}
export KMP_STACKSIZE=1G
export MPICXX_CXX=acpp
export OMPI_CXX=acpp

3.5 Miyabi-G (GH200)

Item Value
Hardware (CPU / GPU) GH200 (Grace 72 cores + H100 96GB)
OS RHEL 9.8 (aarch64)
Host compiler GCC 11.4.1 (gcc/11.4.1)
MPI Open MPI 4.1.6 CUDA-aware build (ompi-cuda/4.1.6-12.6)
CUDA / ROCm CUDA 12.6 (cuda/12.6)
AdaptiveCpp 24.10.0 (+cuda)
LLVM (clang) 18.1.8 (spack, built with GCC 11.4.1)
Installation method (spack environment) spack, miyg
BLAS / LAPACK OpenBLAS 0.3.21 (OS)
--acpp-targets cuda:sm_90

Each Miyabi-G compute node has one NVIDIA GH200 (Grace 72 cores + H100 96GB). The CPU is Arm (Neoverse V2). AdaptiveCpp (with the CUDA backend) and LLVM 18.1.8 were installed on top of the site’s gcc / cuda / ompi-cuda modules (the defaults at the time of the tests were cuda/12.6 and ompi-cuda/4.1.6-12.6) as the spack environment (miyg in the tests). The CUDA used by AdaptiveCpp is explicitly set to 12.6 with ACPP_CUDA_PATH.

Installation (once)

The spack environment used in the tests is defined by the two specs llvm@18 and hipsycl+cuda %clang@18: LLVM 18.1.8 is built with the site GCC 11.4.1, and AdaptiveCpp 24.10.0 with the CUDA backend is built with that clang. CUDA and the CUDA-aware Open MPI of the site are used as external packages (written in spack.yaml as follows). The following procedure creates the same installation (spack 1.1.0.dev0 was used). It may be run on a login node (Grace, aarch64); building LLVM takes several hours.

  packages:
    cuda:
      externals:
      - spec: cuda@12.6.85
        prefix: /work/opt/local/aarch64/cores/cuda/12.6
    openmpi:
      externals:
      - spec: openmpi@4.1.6+cuda fabrics=ucx schedulers=slurm
        prefix: /work/opt/local/aarch64/apps/gcc/11.4.1/openmpi-cuda/4.1.6-12.6
      buildable: false
. /etc/profile.d/modules.sh
module purge
module load gcc
module load cuda
module load ompi-cuda
export SPACK_ROOT=<spack installation directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env create <spack environment name>      # first time only; add the packages above to spack.yaml
spack env activate <spack environment name>
spack compiler find                  # registers the site GCC 11.4.1 and others
spack external find                  # registers the OS tools as external packages
spack install --add llvm@18
spack install --add hipsycl+cuda %clang@18

Settings for use

. /etc/profile.d/modules.sh
module purge
module load gcc
module load cuda
module load ompi-cuda
export SPACK_ROOT=<spack installation directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env activate <spack environment name>
spack load hipsycl
export sycl_root=`spack location -i hipsycl`
export llvm_root=`spack location -i llvm`
export cuda_root=/work/opt/local/aarch64/cores/cuda/12.6
export ACPP_CUDA_PATH=$cuda_root
export LD_LIBRARY_PATH=$sycl_root/lib/:$llvm_root/lib/:$cuda_root/lib64:${LD_LIBRARY_PATH}
export KMP_STACKSIZE=1G
export MPICXX_CXX=acpp
export OMPI_CXX=acpp

3.6 PS-A (Plasma Simulator Subsystem A)

Item Value
Hardware (CPU / GPU) Xeon 6980P ×2
OS RHEL 9.6 (x86_64)
Host compiler Intel oneAPI DPC++/C++ 2026.1.0 (intel/2026.1); OS GCC 11.5.0
MPI Intel MPI 2021.18.1 (intel / intelmpi)
CUDA / ROCm —
AdaptiveCpp 24.10.0 (~cuda~rocm)
LLVM (clang) 18.1.8 (spack, built with GCC 11.5.0)
Installation method (spack environment) spack, sysa (~/spack)
BLAS / LAPACK OpenBLAS 0.3.26 (OS OpenMP build libopenblaso) + lapacke.h of nvhpc 25.3
--acpp-targets omp

The PS-A compute nodes have Intel Xeon 6980P (128 cores ×2). Intel MPI is used for MPI. AdaptiveCpp and LLVM 18 series were installed on top of the site’s intel / intelmpi modules as the spack environment (sysa in the tests).

Installation (once)

The spack environment used in the tests contains LLVM 18.1.8 and AdaptiveCpp 24.10.0 (without CUDA or ROCm), both built with the OS-bundled GCC 11.5.0. The following steps create the same installation (spack 1.1.0.dev0 was used).

. /etc/profile.d/modules.sh
module purge
module load intel
module load intelmpi
. <spack installation directory>/share/spack/setup-env.sh
spack env create <spack environment name>      # first time only
spack env activate <spack environment name>
spack compiler find                  # registers the OS GCC 11.5.0, the oneAPI compilers and others
spack external find                  # registers OS tools as externals
spack install --add llvm@18 %gcc
spack install --add hipsycl %gcc

Settings for use

. /etc/profile.d/modules.sh
module purge
module load intel
module load intelmpi
. <spack installation directory>/share/spack/setup-env.sh
spack env activate <spack environment name>
spack load hipsycl
export sycl_root=`spack location -i hipsycl`
export llvm_root=`spack location -i llvm`
export LD_LIBRARY_PATH=$sycl_root/lib/:$llvm_root/lib/:${LD_LIBRARY_PATH}
export KMP_STACKSIZE=1G
export MPICXX_CXX=acpp
export OMPI_CXX=acpp
export I_MPI_CXX=acpp
export OPENBLAS_NUM_THREADS=1

3.7 PS-B (Plasma Simulator Subsystem B)

Item Value
Hardware (CPU / GPU) MI300A ×4
OS RHEL 9.6 (x86_64)
Host compiler GCC 11.5.0 (OS)
MPI Open MPI 5.0.7 (openmpi/5.0.7/rocm6.3.3)
CUDA / ROCm ROCm 6.3.3 (rocm/6.3.3, /opt/rocm-6.3.3)
AdaptiveCpp 24.10.0 (+rocm, amdgpu_target=gfx942)
LLVM (clang) 18.1.8 (spack, built with GCC 11.5.0; shared with PS-A)
Installation method (spack environment) spack, sysb (~/spack)
BLAS / LAPACK OpenBLAS 0.3.26 (OS OpenMP build libopenblaso) + lapacke.h of nvhpc 25.3
--acpp-targets hip:gfx942

Each PS-B compute node has four AMD Instinct MI300A (APU, 24 cores + GPU). AdaptiveCpp is used with the HIP backend, and the target is hip:gfx942. AdaptiveCpp was installed on top of the ROCm 6.3.3-enabled OpenMPI module as the spack environment (sysb in the tests).

Installation (once)

The spack environment used in the tests contains LLVM 18.1.8, shared with PS-A, and AdaptiveCpp 24.10.0 with the HIP backend (+rocm amdgpu_target=gfx942). Both are built with the OS-bundled GCC 11.5.0, and hip, hsa-rocr-dev, comgr and llvm-amdgpu of ROCm 6.3.3 (/opt/rocm-6.3.3) are used as externals. Since several versions of ROCm are installed on the login node, the version is given explicitly. The following steps create the same installation (spack 1.1.0.dev0 was used). LLVM is the same as that of PS-A.

. /etc/profile.d/modules.sh
module purge
module load openmpi/5.0.7/rocm6.3.3    # also loads rocm/6.3.3
. <spack installation directory>/share/spack/setup-env.sh
spack env create <spack environment name>      # first time only
spack env activate <spack environment name>
spack compiler find                  # registers the OS GCC 11.5.0 and others
spack external find                  # registers OS tools as externals
spack external find hip hsa-rocr-dev comgr llvm-amdgpu   # registers ROCm 6.3.3 as externals
spack install --add llvm@18.1.8+clang~cuda+lld+lldb~python
spack install --add hipsycl@24.10.0~cuda+rocm amdgpu_target=gfx942 ^hip@6.3.3 ^llvm@18.1.8 ^llvm-amdgpu@6.3.3

Settings for use

. /etc/profile.d/modules.sh
module purge
module load openmpi/5.0.7/rocm6.3.3
. <spack installation directory>/share/spack/setup-env.sh
spack env activate <spack environment name>
spack load hipsycl
export sycl_root=`spack location -i hipsycl`
export llvm_root=`spack location -i llvm`
export LD_LIBRARY_PATH=$sycl_root/lib/:$llvm_root/lib/:${LD_LIBRARY_PATH}
export KMP_STACKSIZE=1G
export MPICXX_CXX=acpp
export OMPI_CXX=acpp
export OPENBLAS_NUM_THREADS=1

3.8 macOS (Homebrew)

Item Value
Device (CPU / GPU) Apple M4 (10 cores: 4 performance, 6 efficiency), 32 GB memory
OS macOS 15 (arm64)
Host compiler Apple clang 17 (Xcode Command Line Tools)
MPI Open MPI 5.0 (Homebrew open-mpi)
CUDA / ROCm —
AdaptiveCpp 25.10 (Homebrew adaptivecpp; built without an embedded LLVM)
LLVM (clang) — (the host Apple clang is used)
Installation Homebrew (prebuilt binaries)
BLAS / LAPACK OpenBLAS (Homebrew openblas)
Fortran gfortran (Homebrew gcc)
Python Homebrew Python 3.14 with NumPy and SciPy
--acpp-targets omp.library-only

On macOS, spack is not used; everything is installed as prebuilt binaries with Homebrew. The Homebrew AdaptiveCpp is built without an embedded LLVM, and the host C++ compiler (Apple clang) compiles the whole code including the kernels. Therefore --acpp-targets=omp.library-only is used as on Odyssey (omp gives the same compilation), and the reductions are blocked as on other CPUs (-D_BLOCKING_LOOP_DOT_=1; “Blocking” in the “Optimization” chapter). With the default form, in which one work item handles one row, dot products and others become extremely slow. The GPU (Apple GPU) cannot be used.

Installation (once)

Install the Xcode Command Line Tools (xcode-select --install) and Homebrew. With the default Homebrew location (/opt/homebrew), the following packages are installed as prebuilt binaries in a few minutes. With Homebrew in another location, AdaptiveCpp, GCC, Open MPI and Python are built from source and take a long time.

brew install adaptivecpp open-mpi openblas gcc numpy scipy

Settings for use

Put the Homebrew bin on PATH and let the MPI wrappers call acpp. Sourcing python/envs/mac_brew.sh in the distribution package sets, in addition, the location of OpenBLAS, the location of the Fortran runtime library, and the build settings used by the Python package and examples/build_examples.sh (omp.library-only, -D_BLOCKING_LOOP_DOT_=1) in environment variables. The default is without MPI; set PARCEL_MPI=1 before sourcing it for the settings with MPI.

eval "$(/opt/homebrew/bin/brew shellenv)"
export OMPI_CXX=acpp
# to also set the build settings for Python and the examples, source the following instead
#   (give PARCEL_BREW if Homebrew is not in /opt/homebrew; PARCEL_MPI=1 for MPI)
# export PARCEL_MPI=0
# . python/envs/mac_brew.sh

Open MPI 5 on macOS cannot bind processes to cores (--bind-to core stops with “binding not supported”), so start with, e.g., mpirun -n 4 --bind-to none. The GNU coreutils timeout command is not available. It was confirmed in the one environment of the table above (CPU) with the C++, C, Fortran and Python samples (Quick Start) and the regression tests (with and without MPI). An example setting for running the regression tests in this environment is mac in tests/site_config.py of the distribution package.

3.9 Generic Linux (RHEL Family, Ubuntu Family)

This section shows how to install AdaptiveCpp on Linux systems other than the test environments of this manual. There are two ways.

The procedure below generalizes that of the test environments (“Installation (once)” of each environment). Both this procedure and the method with the generic template were tried on Ubuntu 24.04 and Rocky Linux 9.8 (compatible with RHEL 9), both on WSL2 (x86_64, NVIDIA RTX A6000), for the CPU and the GPU, without and with MPI. The package names were confirmed on these two and may differ on other OS versions.

3.9.1 Prerequisites

Item RHEL family example Ubuntu family example Notes
C, C++ and Fortran compilers dnf install gcc gcc-c++ gcc-gfortran apt install build-essential gfortran Used to build LLVM 18. GCC 8.3.1 to 13.3.0 were used in the tests
Tools used by spack dnf install git python3 make patch bzip2 xz unzip libcurl-devel zlib-devel perl apt install git python3 make patch bzip2 xz-utils unzip libcurl4-openssl-dev spack external find registers the OS curl, zlib, perl and so on, and spack uses them to build cmake and others. The development files of curl and zlib and perl with its modules (a minimal RHEL family system may have the interpreter only) are therefore needed
MPI (when used with MPI) The MPI of the system (Open MPI, etc.). From OS packages: install with dnf install openmpi-devel and load with module load mpi/openmpi-x86_64 The MPI of the system. From OS packages: apt install openmpi-bin libopenmpi-dev Tied to the network of the system, so use the installed one
CUDA (for NVIDIA GPUs) The CUDA of the system. To install it in the OS, cuda-toolkit-12-6 etc. from the dnf repository of NVIDIA (rhel9 for the RHEL 9 family; the same repository on WSL2) The CUDA of the system. To install it in the OS, cuda-toolkit-12-6 etc. from the apt repository of NVIDIA (the repository for WSL on WSL2) A version supported by the driver or older (“Running on GPUs”)
LAPACKE “Installing LAPACKE” Same The OpenBLAS of spack can also be used

On the RHEL 9 family, the development package of OpenBLAS (openblas-devel) is in the CRB repository, which is disabled by default. Enable it as follows before installing (CRB is the name on Rocky Linux and AlmaLinux; on RHEL itself it corresponds to the CodeReady Linux Builder repository, which is enabled with subscription-manager instead of dnf config-manager; the repository name follows the example in Red Hat’s documentation). EPEL is not needed. dnf config-manager is also used to add the CUDA repository of NVIDIA.

dnf install dnf-plugins-core                      # makes dnf config-manager available
dnf config-manager --set-enabled crb              # enables CRB (Rocky Linux, AlmaLinux)
# subscription-manager repos --enable codeready-builder-for-rhel-9-$(arch)-rpms   # on RHEL itself (instead of the line above)
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo   # when installing CUDA in the OS

Behind a proxy of your organization, set http_proxy and https_proxy (and no_proxy); spack, git, curl and apt use them (for dnf, write proxy=<URL> in /etc/dnf/dnf.conf). Behind a proxy that inspects TLS, register the certificate of the proxy (the CA of the organization) in the OS. On the Ubuntu family, put the CA certificate (PEM) at /usr/local/share/ca-certificates/<name>.crt and run update-ca-certificates; on the RHEL family, put it in /etc/pki/ca-trust/source/anchors/ and run update-ca-trust.

3.9.2 Fetching and Configuring spack

spack 1.x is targeted (the version used by install_stack.sh is pinned at its top; the procedure of this section was confirmed with the development version 1.3.0.dev0 at the time of fetching). Fetch spack, create an environment, and register the compilers and the OS tools.

git clone -c feature.manyFiles=true https://github.com/spack/spack.git <spack installation directory>
export SPACK_ROOT=<spack installation directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env create <spack environment name>      # first time only
spack env activate <spack environment name>
spack compiler find                  # register GCC and other compilers found in PATH
spack external find                  # register OS tools such as cmake, perl and python as external packages

OS tools registered with spack external find are used without being built by spack, which shortens the installation.

3.9.3 Registering MPI and CUDA

MPI and CUDA are taken from the system and are not built by spack. spack external find openmpi cuda may find them, but on systems that provide them as modules they are often not found; write them directly into the definition of the spack environment (spack.yaml, opened with spack config edit). Adjust the versions and locations to your system.

spack:
  packages:
    mpi:
      require: [openmpi]
    openmpi:
      buildable: false
      externals:
      - spec: openmpi@4.1.6~cuda
        prefix: /path/to/openmpi/4.1.6     # one level above bin/mpicc
    cuda:
      buildable: false
      externals:
      - spec: cuda@12.6.20
        prefix: /path/to/cuda/12.6
O=$HOME/openmpi-rhel                              # any location; the OS files are not changed
mkdir -p $O
ln -s /usr/lib64/openmpi/bin $O/bin
ln -s /usr/lib64/openmpi/lib $O/lib
ln -s /usr/lib64/openmpi/share $O/share
ln -s /usr/include/openmpi-x86_64 $O/include

3.9.4 Installing LLVM and AdaptiveCpp

Build LLVM 18 with the OS GCC, and build AdaptiveCpp (spack package hipsycl) with its clang. To use GPUs, select the backend with a variant.

spack install --add llvm@18 %gcc
spack install --add hipsycl %clang@18                              # CPU only
# spack install --add hipsycl+cuda %clang@18                       # also NVIDIA GPUs (instead of the above)
# spack install --add hipsycl+rocm amdgpu_target=gfx942 %clang@18  # also AMD GPUs (MI300A example)

Building LLVM takes a long time and much memory. The number of concurrent compilations is set with spack install -j <n>. If the build stops for lack of memory, lower the number and run the same command again (installed packages are reused; see the “Odyssey” section for an example on Odyssey).

3.9.5 Settings for Use

export SPACK_ROOT=<spack installation directory>
. $SPACK_ROOT/share/spack/setup-env.sh
spack env activate <spack environment name>
export sycl_root=`spack location -i hipsycl`
export llvm_root=`spack location -i llvm@18`
export LD_LIBRARY_PATH=$sycl_root/lib:$llvm_root/lib:${LD_LIBRARY_PATH}
export ACPP_CUDA_PATH=<CUDA location>       # for NVIDIA GPUs
export OMPI_CXX=acpp MPICXX_CXX=acpp          # make the MPI wrapper (Open MPI) call acpp

Check the installation by building and running the C++ program of the Quick Start. On GPUs, also check the name of the device used (“Running on GPUs”).

3.9.6 Windows WSL2

Linux on Windows WSL2 (Ubuntu, Rocky Linux) can also use the CPU and NVIDIA GPUs with the procedure of this section and with install_stack.sh (confirmed with Ubuntu 24.04 and Rocky Linux 9, using two RTX A6000). The differences from Linux systems follow.

3.10 Installing LAPACKE

The eigenvalue solver ks_arnoldi of PARCEL uses LAPACKE (the C interface of LAPACK). All solvers are built by default, so the header lapacke.h and the library are needed even when only linear solvers are used (they are unnecessary if ks_arnoldi is excluded with -DPARCEL_ENABLE_KS_ARNOLDI=0; “Selecting the Solvers to Build”). Examples of installing it follow. Package names may differ between OS versions and additional repositories (such as CRB for the RHEL family). The RHEL family row was confirmed on Rocky Linux 9, and the Ubuntu family row on Ubuntu 24.04.

OS etc. Example
RHEL family dnf install openblas-devel (after enabling CRB; “Prerequisites”)
Ubuntu family apt install libopenblas-dev liblapacke-dev
macOS brew install openblas (“macOS (Homebrew)”)
spack spack install openblas (OpenBLAS includes LAPACKE)
Fujitsu environment (Odyssey) Nothing to install. The implementation is Fujitsu LAPACK (-lfjlapack, included in the Fujitsu language environment), and the header lapacke.h of the OS OpenBLAS is borrowed (see below)

Find the installed header and library as follows.

find / -name lapacke.h 2>/dev/null               # find the header
rpm -ql openblas-devel | grep -E 'lapacke.h|\.so$'   # RHEL family: files installed by the package
dpkg -L liblapacke-dev | grep -E 'lapacke.h|\.so$'   # Ubuntu family: files installed by the package
ldconfig -p | grep -E 'liblapacke|libopenblas'        # Ubuntu family: the .so of OpenBLAS is in another package, so look in the loader's list
ls $(spack location -i openblas)/include $(spack location -i openblas)/lib   # spack