Improving the Scalability of the GEOS-5 Earth System Model

Overview

The Goddard Earth Observing System (GEOS-5) is a global climate model and assimilation system maintained by NASA's Global Modeling and Assimilation Office (GMAO). GEOS-5 is used as both a near real-time forecast and a comprehensive reanalysis platform. A current GMAO effort is aimed at examining the performance of GEOS-5 at finer resolutions—sub-10-kilometer (km) cell width—that will become the future operational standard. To better utilize high-performance computing (HPC) resources, we are exploring ways to improve the scaling of the model as HPC evolves toward using many-core architectures.

Project Details

At present, GEOS-5 is parallelized purely using the Message Passing Interface (MPI), the standard mechanism used in HPC to parallelize applications on multi-node clusters. However, the advent of nodes with many-core (20-60+ cores) chips and large amounts of memory leads us to investigate using a hybrid programming paradigm by incorporating OpenMP shared-memory programming to better manage intra-node computation in combination with MPI for inter-node communication.

Our scaling studies focus on the use and impact of OpenMP with MPI in GEOS-5, starting with the cubed-sphere dynamics and a solar radiation physics routine as representative elements. Our results examining these two components will allow us to determine how MPI+OpenMP can be applied to GEOS-5 as a whole.

Results and Impact

In the past several months, we have run benchmarks on a variety of HPC hardware at the NASA Center for Climate Simulation (NCCS) and NASA Advanced Supercomputing (NAS) facilities. We have found that:

• In general, better scaling of the cubed-sphere dynamics is achieved using MPI+OpenMP at high core count, but MPI alone is better for low core count.
• For the dynamics, the core count where MPI+OpenMP becomes beneficial, and the best-performing number of threads per MPI process, is dependent on the model resolution.
• The solar radiation routine appears to scale well using both pure MPI and hybrid MPI+OpenMP. This gives hope that other column-physics routines will be amenable for conversion.

Why HPC Matters

GMAO is determined to continue moving GEOS-5 to higher resolutions to better model and forecast the Earth's atmosphere and climate. However, each forecast must be completed within a certain timeframe, so higher resolutions using more HPC resources must do so as efficiently as possible. Studying GEOS-5's scaling characteristics will allow us to re-architect the model in order to fully utilize all the HPC resources available to us now and in the future.

What's Next

As GEOS-5 continues to increase its operational resolution, the need to better utilize HPC resources will become a more pressing concern. We plan to take the lessons learned from these preliminary studies and begin applying the hybrid MPI+OpenMP paradigm to more of GEOS-5.