Skip to content

LAVA Installation for External Users

If you do not have NAS system access, install LAVA using the pre-compiled tarballs from the NASA Software Catalog.

If you are a NAS user, go to NAS Users (Module-Based Access).

Acquiring LAVA

Download the appropriate tarball for your system from the NASA Software Catalog. Tarballs are named using the following convention:

LAVA_<version>_<compiler>_<mpi>_<arch>_<DateStamp>.tar.gz

Example: LAVA_v1.0.0_NVIDIA_OpenMPI_x86_64_2026.03.02.tar.gz

Version numbers follow the format [MAJOR].[MINOR].[PATCH].

Available Build Configurations

LAVA provides four pre-built configurations optimized for different hardware platforms:

Tarball Name Compiler MPI Architecture Target Systems
LAVA_*_NVIDIA_OpenMPI_x86_64_*.tar.gz NVHPC 25.3 OpenMPI 4.1.6 x86_64 (AVX2) NVIDIA A100 GPUs, RTX 5090, RTX 4090
LAVA_*_NVIDIA_HPCX_aarch64_*.tar.gz NVHPC 25.3 HPC-X aarch64 (ARM) NVIDIA GH200 (Grace-Hopper)
LAVA_*_Intel_OpenMPI_x86_64_*.tar.gz Intel 2023.2.1 OpenMPI 4.1.5 x86_64 (AVX2 & AVX512) Intel/AMD CPUs (Haswell, Skylake, Rome, Milan, Genoa)
LAVA_*_Intel_Cray-MPICH_x86_64_*.tar.gz Intel 2023.2.1 Cray-MPICH x86_64 (AVX512) AMD Turin (Zen 5)

Note

Companies in tarball names above were not involved in generating the LAVA binaries. They only convey compiler information.

Tip

While the Intel and Nvidia binaries were compiled with compiler versions that are older than the most recent available compilers, at runtime the user can choose to use the latest Intel or Nvidia runtime and the binaries should be compatible.

Which Tarball to Download

GPU Systems

Your Hardware Minimum Compute Capability Supported Recommended Tarball
NVIDIA A100, RTX 5090, RTX 4090 (x86_64 systems) SM 8.0+ LAVA_*_NVIDIA_OpenMPI_x86_64_*.tar.gz
NVIDIA GH200 (Grace-Hopper ARM systems) SM 9.0+ LAVA_*_NVHPC_HPCX_aarch64_*.tar.gz

CPU-Only Systems (Intel/AMD)

Your CPU Instruction Set Support Recommended Tarball
Intel Haswell, Broadwell, Skylake (non-X)
AMD Zen, Zen 2, Zen 3, Zen 4 (Rome, Milan, Genoa)		 AVX2 LAVA_*_Intel_OpenMPI_x86_64_*.tar.gz
(use AVX2 executables)
Intel Skylake-X, Cascade Lake, Ice Lake, Sapphire Rapids
AMD Zen 5 (Turin)		 AVX512 LAVA_*_Intel_OpenMPI_x86_64_*.tar.gz
(use AVX512 executables)
or LAVA_*_Intel_Cray-MPICH_x86_64_*.tar.gz

Note

The Intel_OpenMPI_x86_64 tarball includes both AVX2 and AVX512 executables with double and single precision variants. Use AVX2 versions for broader compatibility if unsure of your CPU capabilities.

Note

LAVA and its tools require CPUs with AVX2 instruction set at a minimum.

Installation Steps

Extract the tarball

tar -xzf LAVA_<version>_<compiler>_<mpi>_<arch>_<DateStamp>.tar.gz
cd LAVA_<version>_<compiler>_<mpi>_<arch>

The extracted directory contains:

  • bin/ - All LAVA executables and utility scripts
  • lib/ - Curvilinear libraries and Python packages (Intel builds only)
  • species/ - Species database files

Set up your environment

Load the required modules and set environment variables like LD_LIBRARY_PATH based on your configuration.

Environment Setup for NVIDIA + OpenMPI (x86_64 GPU Systems)

Tarball: LAVA_*_NVIDIA_OpenMPI_x86_64_*.tar.gz

Required Software

  • OpenMPI 4.1.6 or later
  • NVIDIA GPU driver version 570.26 or later (required for CUDA 12.8 support)
  • See NVIDIA CUDA-Driver Compatibility for driver requirements
  • NVIDIA runtime libraries (any version compatible with CUDA 12.8+)

Note

This tarball was compiled with NVHPC 25.3, but you can use newer NVHPC versions (e.g., 25.11 or later) at runtime. The NVHPC runtime libraries are backward compatible.

Environment Setup: 

# If using modules (recommended)
module load nvhpc/25.11
module load openmpi/4.1.6

# Set LAVA necessary env variables
export LAVA_INSTALL_DIR=/path/to/LAVA/install
export LAVA_OUTPUT_INTERVAL=1000000

# Add LAVA binaries to PATH
export PATH=/path/to/LAVA/bin:${PATH}

# If NOT using modules, manually set LD_LIBRARY_PATH
# The following paths are required (adjust versions/paths for your installation):
export LD_LIBRARY_PATH=\
/path/to/nvhpc/compilers/lib:\
/path/to/nvhpc/cuda/lib64:\
/path/to/openmpi/lib:\
${LD_LIBRARY_PATH}

# Example for a typical NVHPC 25.3 installation:
# export LD_LIBRARY_PATH=\
# /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/compilers/lib:\
# /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/cuda/lib64:\
# /opt/openmpi/4.1.6/lib:\
# ${LD_LIBRARY_PATH}

Required Runtime Libraries

The LAVA executable depends on the following dynamic libraries:

  • NVHPC Compiler Libraries: libcudafor.so, libcudafor_120.so, libnvf.so, libaccdevice.so, libacchost.so, libaccdevaux.so, libnvomp.so, libnvhpcatm.so, libnvc.so, libnvcpumath-avx2.so, libcudadevice.so, libcudanvhpc.so
  • CUDA Runtime: libcudart.so.12
  • OpenMPI Libraries: libmpi.so.40, libmpi_mpifh.so.40, libmpi_usempif08.so.40, libmpi_usempi_ignore_tkr.so.40
  • System Libraries: libz.so.1, librt.so.1, libdl.so.2, libstdc++.so.6, libpthread.so.0, libc.so.6, ld-linux-x86-64.so.2, libgcc_s.so.1, libm.so.6

Verifying Your Setup: 

# Check if all dependencies are resolved
ldd /path/to/LAVA/bin/lavacart.3D.NS1

# All libraries should show resolved paths (no "not found")
# If you see "not found", ensure the corresponding library path is in LD_LIBRARY_PATH

Environment Setup for NVIDIA + HPC-X (aarch64 GPU Systems)

Tarball: LAVA_*_NVIDIA_HPCX_aarch64_*.tar.gz

Required Software

  • HPC-X MPI (bundled with NVHPC or standalone)
  • NVIDIA GPU driver version 570.26 or later (required for CUDA 12.8 support)
  • See NVIDIA CUDA-Driver Compatibility for driver requirements
  • NVHPC runtime libraries (any version compatible with CUDA 12.8+)

Note

This tarball was compiled with NVHPC 25.3, but you can use newer NVHPC versions (e.g., 25.11 or later) at runtime. The NVHPC runtime libraries are backward compatible.

Environment Setup: 

# If using modules (recommended)
module load nvhpc-hpcx-cuda12/25.7

# Set LAVA necessary env variables
export LAVA_INSTALL_DIR=/path/to/LAVA/install
export LAVA_OUTPUT_INTERVAL=1000000

# Add LAVA binaries to PATH
export PATH=/path/to/LAVA/bin:${PATH}

# If NOT using modules, manually set LD_LIBRARY_PATH
# The following paths are required (adjust versions/paths for your installation):
export LD_LIBRARY_PATH=\
/path/to/nvhpc/compilers/lib:\
/path/to/nvhpc/cuda/lib64:\
/path/to/nvhpc/comm_libs/12.8/hpcx/hpcx-2.22.1/ompi/lib:\
${LD_LIBRARY_PATH}

# Example for a typical NVHPC 25.3 ARM installation:
# export LD_LIBRARY_PATH=\
# /opt/nvidia/hpc_sdk/Linux_aarch64/25.3/compilers/lib:\
# /opt/nvidia/hpc_sdk/Linux_aarch64/25.3/cuda/lib64:\
# /opt/nvidia/hpc_sdk/Linux_aarch64/25.3/comm_libs/12.8/hpcx/hpcx-2.22.1/ompi/lib:\
# ${LD_LIBRARY_PATH}

Required Runtime Libraries

The LAVA executable depends on the following dynamic libraries:

  • NVHPC Compiler Libraries: libcudafor.so, libcudafor_120.so, libnvf.so, libaccdevice.so, libacchost.so, libaccdevaux.so, libnvomp.so, libnvhpcatm.so, libnvc.so, libnvcpumath-armv8.so, libnvmalloc.so, libcudadevice.so, libcudanvhpc.so
  • CUDA Runtime: libcudart.so.12
  • HPC-X MPI Libraries: libmpi.so.40, libmpi_mpifh.so.40, libmpi_usempif08.so.40, libmpi_usempi_ignore_tkr.so.40
  • System Libraries: Standard Linux libraries (typically in /usr/lib64 or /lib64)

Verifying Your Setup: 

# Check if all dependencies are resolved
ldd /path/to/LAVA/bin/lavacart.3D.NS1

# All libraries should show resolved paths (no "not found")
# If you see "not found", ensure the corresponding library path is in LD_LIBRARY_PATH

Environment Setup for Intel + OpenMPI (CPU Systems)

Tarball: LAVA_*_Intel_OpenMPI_x86_64_*.tar.gz

Required Software

  • OpenMPI 4.1.5 or later

Environment Setup: 

# If using modules (recommended)
module load intel/2022.1.0
module load openmpi/4.1.5

# Set LAVA necessary env variables
export LAVA_INSTALL_DIR=/path/to/LAVA/install
export LAVA_OUTPUT_INTERVAL=1000000

# Add LAVA binaries to PATH
export PATH=/path/to/LAVA/bin:${PATH}

# If NOT using modules, manually set LD_LIBRARY_PATH
# The following paths are required (adjust versions/paths for your installation):
export LD_LIBRARY_PATH=\
/path/to/openmpi/lib:\
${LD_LIBRARY_PATH}

# Example for a typical OpenMPI 4.1.5 installation:
# export LD_LIBRARY_PATH=\
# /opt/openmpi/4.1.5/lib:\
# ${LD_LIBRARY_PATH}

Required Runtime Libraries

The LAVA executable depends on the following dynamic libraries:

  • OpenMPI Libraries: libmpi.so.40, libmpi_usempif08.so.40, libmpi_usempi_ignore_tkr.so.40, libmpi_mpifh.so.40
  • System Libraries: libz.so.1, libdl.so.2, libstdc++.so.6, libm.so.6, libgcc_s.so.1, libpthread.so.0, libc.so.6, ld-linux-x86-64.so.2

Verifying Your Setup: 

# Check if all dependencies are resolved (use AVX2 or AVX512 as appropriate)
ldd /path/to/LAVA/bin/lavacart.3D.NS1.AVX2
# or
ldd /path/to/LAVA/bin/lavacart.3D.NS1.AVX512

# All libraries should show resolved paths (no "not found")
# If you see "not found", ensure the corresponding library path is in LD_LIBRARY_PATH

Choosing AVX2 vs AVX512 Executables:

  • This tarball includes both AVX2 and AVX512 versions of each executable in double and single precision
  • AVX512 executables: lavaunst.3D.NS1.AVX512, lavaunst.3D.NS1.AVX512.FLOAT
  • AVX2 executables: lavacart.3D.NS1.AVX2, lavacart.3D.NS1.AVX2.FLOAT
  • Use AVX512 versions on supported CPUs for better performance (10-30% faster)
  • Use AVX2 versions for broader compatibility

Environment Setup for Intel + Cray-MPICH (Cray/HPE Systems)

Tarball: LAVA_*_Intel_Cray-MPICH_x86_64_*.tar.gz

Required Software

  • Cray MPICH (typically on Cray/HPE systems)

Environment Setup: 

# If using modules (recommended)
module load intel-classic/2023.2.1
module load PrgEnv-intel

# Set LAVA necessary env variables
export LAVA_INSTALL_DIR=/path/to/LAVA/install
export LAVA_OUTPUT_INTERVAL=1000000

# Add LAVA binaries to PATH
export PATH=/path/to/LAVA/bin:${PATH}

# If NOT using modules, manually set LD_LIBRARY_PATH
# The following paths are required (adjust versions/paths for your installation):
export LD_LIBRARY_PATH=\
/opt/cray/pe/lib64:\
${LD_LIBRARY_PATH}

# Example for a typical Cray MPICH installation:
# export LD_LIBRARY_PATH=\
# /opt/cray/pe/lib64:\
# ${LD_LIBRARY_PATH}

Required Runtime Libraries

The LAVA executable depends on the following dynamic libraries:

  • Cray MPI Libraries: libmpi_intel.so.12, libmpifort_intel.so.12
  • System Libraries: libz.so.1, libstdc++.so.6, libm.so.6, libgcc_s.so.1, libc.so.6, ld-linux-x86-64.so.2

Verifying Your Setup: 

# Check if all dependencies are resolved
ldd /path/to/LAVA/bin/lavacart.3D.NS1.AVX512

# All libraries should show resolved paths (no "not found")
# If you see "not found", ensure the corresponding library path is in LD_LIBRARY_PATH

Note: This configuration is optimized for AMD Turin (Zen 5) processors with AVX512 support.

Troubleshooting

error while loading shared libraries: libmpi.so.XX: cannot open shared object file

Solution: Ensure your MPI module is loaded and matches the tarball's MPI version.

Illegal instruction error

Please verify that both the operating system and the processor support Intel(R) X87, CMOV, MMX, FXSAVE, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT and AVX instructions.
Solution: Your CPU may not support the required instruction set. Try the AVX2 executable instead of AVX512, or check CPU capabilities with lscpu | grep -i flags. If AVX2 is not available, the CPU is not supported; LAVA requires AVX2 or newer.

CUDA errors on GPU systems

Solution:

  • Verify CUDA version compatibility (nvidia-smi)
  • Check that CUDA_HOME and LD_LIBRARY_PATH are set correctly
  • Ensure GPU drivers are up to date

Lot of files like lout.*

Solution: Ensure the environment variable LAVA_OUTPUT_INTERVAL is set to a large value. This controls the rank stride for the lout files.

Using non scheduler aware MPI runtime

Solution: When running with MPI runtime that is not SLURM or PBS aware, you might need to specify additional options like hostfile and pass environment variables through. Please refer to the mpiexec line in the pbs script for GH200 nodes on NAS for reference.

For more troubleshooting tips please refer here.