Running LAVA on NASA Advanced Supercomputing (NAS) Resources

This guide provides NAS-specific information for running LAVA simulations, including PBS job scripts, queue limits, node configurations, and job sizing guidelines.

Note: For LAVA installation instructions, see the LAVA Access Guide. For LAVA Voronoi installation, see the LAVA Voronoi Installation Guide.

Example Usage on NAS Systems

On A100 GPU Systems (Cabeus x86_64)

#!/bin/bash
#PBS -S /bin/bash
#PBS -N lava_a100_job
#PBS -q gpu_normal
#PBS -l select=4:ncpus=128:model=mil_a100:ngpus=4:mem=500gb
#PBS -W group_list=<your project group ID>
#PBS -l place=scatter:excl
#PBS -l walltime=02:00:00
#PBS -j oe
#PBS -k oed
#PBS -r n

# Load LAVA module
module use /swbuild/LAVA/LAVA_modules
module load LAVA

# Set environment
export OMP_NUM_THREADS=32

# Change to working directory
cd $PBS_O_WORKDIR

# Run LAVA
mpiexec -N 4 --bind-to none --report-bindings mbind.x -t $OMP_NUM_THREADS \
    lavacart.3D.NS1 cart.inputs |& tee run.log

Submit with: qsub submit_a100.pbs

On GH200 GPU Systems (Cabeus Grace-Hopper ARM)

#!/bin/bash
#PBS -S /bin/bash
#PBS -N lava_gh200_job
#PBS -q gpu_normal
#PBS -l select=16:ncpus=72:mpiprocs=1:model=gh200:ngpus=1:mem=460gb
#PBS -W group_list=<your project group ID>
#PBS -l walltime=04:00:00
#PBS -j oe
#PBS -k oed
#PBS -r n

# Load LAVA module
module use /swbuild/LAVA/LAVA_modules
module load LAVA

# Set environment
export OMP_NUM_THREADS=72

# Change to working directory
cd $PBS_O_WORKDIR

# Run LAVA with NUMA binding (1 MPI rank per node, 1 GPU per node)
mpiexec -N 1 --report-bindings --hostfile $PBS_NODEFILE \
    -x PATH -x LD_LIBRARY_PATH -x CUDA_VISIBLE_DEVICES \
    numactl --cpunodebind 0 --membind 0 \
    mbind.x -t $OMP_NUM_THREADS lavaunst.3D.NS1 unst.inputs |& tee run.log

Submit with: qsub submit_gh200.pbs

On CPU Systems - Milan (Aitken)

#!/bin/bash
#PBS -S /bin/bash
#PBS -N lava_milan_job
#PBS -q normal
#PBS -l select=16:ncpus=128:model=mil_ait
#PBS -W group_list=<your project group ID>
#PBS -l walltime=02:00:00
#PBS -j oe
#PBS -k oed
#PBS -r n

# Load LAVA module
module use /swbuild/LAVA/LAVA_modules
module load LAVA

# Set environment
export OMP_NUM_THREADS=32

# Change to working directory
cd $PBS_O_WORKDIR

# Run LAVA (4 MPI ranks per node, 32 threads each)
mpirun -npernode 4 mbind.x -t $OMP_NUM_THREADS \
    lavaunst.3D.NS1.AVX2 unst.inputs |& tee run.log

Submit with: qsub submit_milan.pbs

On CPU Systems - Skylake (Electra)

#!/bin/bash
#PBS -S /bin/bash
#PBS -N lava_skylake_job
#PBS -q normal
#PBS -l select=32:ncpus=40:model=sky_ele
#PBS -W group_list=<your project group ID>
#PBS -l walltime=04:00:00
#PBS -j oe
#PBS -k oed
#PBS -r n

# Load LAVA module
module use /swbuild/LAVA/LAVA_modules
module load LAVA

# Set environment
export OMP_NUM_THREADS=20

# Change to working directory
cd $PBS_O_WORKDIR

# Run LAVA (2 MPI ranks per node, 20 threads each)
mpirun -npernode 2 mbind.x -t $OMP_NUM_THREADS \
    lavacart.3D.NS1.AVX512 cart.inputs |& tee run.log

Submit with: qsub submit_skylake.pbs

On AMD Turin Systems (Athena)

#!/bin/bash
#PBS -S /bin/bash
#PBS -N lava_turin_job
#PBS -q normal
#PBS -l select=16:ncpus=512:mpiprocs=4:model=tur_ath
#PBS -W group_list=<your project group ID>
#PBS -l walltime=04:00:00
#PBS -j oe
#PBS -k oed
#PBS -r n

# Load LAVA module
module use /swbuild/LAVA/LAVA_modules
module load LAVA

# Set environment
export OMP_NUM_THREADS=128

# Change to working directory
cd $PBS_O_WORKDIR

# Run LAVA (4 MPI ranks per node, 128 threads each)
mpiexec -ppn 4 --cpu-bind none mbind.x -cs -t $OMP_NUM_THREADS \
    lavacart.3D.NS1.AVX512 cart.inputs |& tee run.log

Submit with: qsub submit_turin.pbs

Tip

When running on Athena make sure that you have cd $PBS_O_WORKDIR because on athena compute nodes you land on home directory by default.

Available NAS Resources

The following NAS resources are available for running LAVA simulations:

Node Identifier	CPU	GPU	Number of Nodes	SBUs*	Front End
Electra
bro_ele	2x Intel Broadwell	-	1152	1.0	pfe
sky_ele	2x Intel Skylake	-	2304	1.59	pfe
Aitken
cas_ait	2x Intel Cascade Lake	-	1152	1.64	pfe
rom_ait	2x AMD Rome	-	2048	4.06	pfe
mil_ait	2x AMD Milan	-	512	4.38	afe
Athena
tur_ath	2x AMD Turin	-	1024	12.19	athfe
Cabeus				[GPU]
mil_a100	1x AMD Milan	4× NVIDIA A100	128	37.86	cfe0[1-2]
gh200	1x NVIDIA Grace	1× NVIDIA H100	350	18.34	cghfe0[1-2]

*CPU and GPU SBU allocations are separate and thus not directly comparable

For detailed information about hardware specifications, policies, and usage guidelines, refer to the NAS HECC Knowledge Base.

---

Queue Limits

For the latest queue limits and policies, please refer to the following links:

• Aitken and Electra: PBS Job Queue Structure
• Athena: PBS Job Queue Structure
• Cabeus A100: Preparing to Run on Cabeus A100 GPU Nodes
• Cabeus GH200: Preparing to Run on Cabeus Grace Hopper Nodes

---

Running Configuration Table

The table below describes the recommended number of MPI ranks and OpenMP threads to use for each model type. The table also lists the executable suffix to use based on the node architecture.

Model	PBS Select Line	MPI Ranks per Node	OpenMP Threads per Rank	Executable Suffix
bro_ele	ncpus=28:model=bro_ele	2	14	AVX2
sky_ele	ncpus=40:model=sky_ele	2	20	AVX512
cas_ait	ncpus=40:model=cas_ait	2	20	AVX512
rom_ait	ncpus=128:model=rom_ait	4	32	AVX2
mil_ait	ncpus=128:model=mil_ait	4	32	AVX2
tur_ath	ncpus=512:model=tur_ath	4	128	AVX512
mil_a100	ncpus=128:model=mil_a100:ngpus=4:mem=500gb	4	32
gh200	ncpus=72:model=gh200:ngpus=1:mem=460gb	1	72

Tip

Optimum configuration may vary depending on your specific simulation. Consider performance testing to optimize resource allocation.

---