Preparing the FUN3D CFD Solver for the Exascale Era

Overview

NASA’s CFD Vision 2030 study emphasizes the key role that leadership-class computing must play in achieving the simulation goals of the aerospace community. In a joint collaboration with NVIDIA Corporation, Old Dominion University, ParaTools, Inc., the Department of Defense High Performance Computing Modernization Program (HPCMP) user Productivity Enhancement, Technology Transfer, and Training (PETTT) program, and the Oak Ridge Leadership Computing Facility (OLCF), researchers at NASA’s Langley Research Center recently demonstrated impressive next-generation computational performance for the NASA unstructured-grid Navier-Stokes flow solver FUN3D. In this multi-year effort, the team explored a broad range of programming models appropriate for the many-core architectures driving the current landscape of leadership-class high-performance computing (HPC) facilities.

Project Details

We used mini-applications based on critical kernels drawn from the full FUN3D application to investigate several programming models, including OpenACC, CUDA Fortran, CUDA C/C++, PTX, and Kokkos. The team ultimately used a CUDA-based approach to port and optimize the complete suite of FUN3D kernels required to discretize and solve the Navier-Stokes equations. In this manner, the entire solution is computed on the GPUs with minimal data transfer between CPUs and GPUs. Communication between GPUs is performed using NVIDIA’s GPUDirect RDMA to minimize latency and maximize throughput.

Results and Impact

The team showed that a single NVIDIA Tesla V100 GPU delivers a computational speedup of approximately 4.3 times that of a heavily optimized code running on a dual-socket Intel Xeon Gold 6148 Skylake processor containing 40 cores. To demonstrate the efficiency of the implementation at scale, the team leveraged early access to Summit, an IBM AC922 leadership-class system with NVIDIA Tesla V100 GPUs, installed at OLCF and ranked as the world's most powerful supercomputer in June 2018. Experiments were performed using as many as 1,024 hardware nodes, comprising a total of 6,144 Tesla V100 GPUs.

We compared the performance of the GPU approach with that of a conventional hybrid MPI/OpenMP formulation, running exclusively on the dual-socket IBM POWER9 CPUs with 44 cores each. Tests showed a nominal 30 times speedup advantage for the GPU approach—or a total throughput equivalent to that of about a million CPU cores. The success of the current approach offers a valuable opportunity to compute a broad range of applications in significantly less wall time.

Why HPC Matters

NASA’s Pleiades and Electra supercomputers were used extensively for node-level, Xeon-based optimization and profiling exercises. Critical reviews from hardware experts at the NASA Advanced Supercomputing facility have also been invaluable to the effort.

What's Next

The team is working to demonstrate the efficiency of their overall approach by applying the methodology to a broad class of applications relevant to multiple NASA missions. Additional kernels and workflow components typically required by common science and engineering workflows will continue to be developed as needed.

This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.

More Information

• FUN3D website
• Oak Ridge Leadership Computing Facility
• See the poster for this demonstration (PDF, 1.8MB). As shown in the NASA booth at SC18.