Climate Dynamics on the Intel Xeon Phi

Overview

The use of high-performance computing (HPC) is essential to climate simulations. As computational resources increase, more accurate simulations are possible, and program execution times decrease. For example, if you want to predict the weather for tomorrow, your simulation needs to take less than a day! The latest trend in HPC is the introduction of hardware accelerators such as graphics processing units (GPUs) and coprocessors such as Intel's Xeon Phi. In order to continue to utilize modern supercomputers with heterogeneous environments, it is necessary to develop optimized codes suitable for these architectures. To this end, our work focused on modifying NASA climate dynamics codes to run optimally on the Intel Xeon Phi.

Project Details

The Intel Xeon Phi is the company's latest coprocessor based on the Many Integrated Core (MIC) architecture. It houses 60 compute cores and, with hyperthreading, is capable of running 240 threads concurrently. Hardware saturation requires creating parallel regions capable of running 240 OpenMP (OMP) threads with an iteration space several times the number of running OMP threads. Prior to this work, the climate dynamics codes offered parallel OMP regions capable of executing fewer than 100 threads. To increase the OMP concurrency, we developed a high-level blocking scheme that offers the capability to decompose the 2D domain available to each Message Passing Interface (MPI) rank. With this modification, it is now possible to launch hundreds to thousands of OMP threads and utilize multi-core architectures such as the Phi.

Results and Impact

In this work, we have created a hybrid OMP/MPI finite volume dynamical core. This approach allows the possibility of creating hundreds to thousands of OMP threads, which is necessary to run efficiently on the Phi. In our experience with the Phi, we found that it is easy to get a working program, but difficult to get good performance. Currently, the three dominant subroutines have been optimized and run at the level of an Intel Xeon Sandy Bridge node. End-to-end to performance of the dynamical core on the Phi at this time is still substantially slower than the Sandy Bridge node (~6x) due to remaining “scaffolding” and unoptimized subroutines. We hope to finish these optimizations in the near future.

Role of High-End Computing Resources

At NASA Goddard Space Flight Center, our challenges include limited space for computational resources and the need to cool the compute nodes when they are running. These challenges are faced by all computational facilities, and it is an active area of research in the HPC community. The introduction of hardware accelerators has helped overcome these constraints, as they take up relatively little room and consume less power. As more computational resources begin housing accelerators and similar architectures, it will be important to have codes suitable for these architectures.