GPU Acceleration of the Goddard Earth Observing System Atmospheric Model

Overview

The Goddard Earth Observing System 5 (GEOS-5) is the atmospheric model used by the Global Modeling and Assimilation Office (GMAO) for a variety of applications—from long-term climate prediction at relatively coarse resolution, to data assimilation and numerical weather prediction, to very high-resolution cloud-permitting simulations.

GEOS-5 is being ported to a graphics processing unit (GPU) cluster at the NASA Center for Climate Simulation (NCCS). By utilizing GPU co-processor technology, we expect to increase the throughput of GEOS-5 by at least an order of magnitude, and potentially much more as implementations improve and become easier to use.

Project Details

Using 12,000 Intel Westmere processor cores on the NCCS Discover supercomputer, GEOS-5 completes a 10-kilometer (km) five-day forecast within the three-hour operational window required to complete a weather forecast data assimilation cycle. As we move below 10-km resolution, the number of cores required to maintain this operational throughput increases rapidly, exceeding 6 million cores as we approach the ultimate goal of a 1-km, global cloud-resolving model.

The primary physics components of GEOS-5 have been ported to the GPU cluster using the Portland Group, Inc. (PGI) Fortran compiler. This has been accomplished with minimal code changes and is designed to co-exist with the conventional CPU code. Initial results suggest speedups on the order of 15–40 times faster than the CPU code for a 50-km test case.

A larger challenge is GEOS-5's finite-volume dynamical core. A PGI Fortran implementation of the hydrostatic dynamics is nearing completion, and initial results suggest speedups of 30–40 times faster than the CPU code. This gain is particularly substantial when we consider that the dynamics portions of GEOS-5 simulations consume more than 90% of the computational cycles at cloud-permitting resolutions below 7 km.

Results and Impact

Having a full implementation of GEOS-5 on GPUs provides a useful benchmark for the scalability of global atmospheric models on GPUs, and facilitates evaluation of future system architecture configurations. Furthermore, the GPU implementation of GEOS-5 will accelerate the process of scientific exploration across all global modeling scales, including:

• The large-scale, high-end application of non-hydrostatic, cloud-permitting modeling at 10- to 1-km global resolutions
• Intermediate-resolution seasonal climate and weather prediction at 50- to 25-km on small clusters of GPUs
• Long-range, coarse-resolution climate modeling, enabled on a small box of GPUs for individual researchers

Role of High-End Computing Resources

NASA supercomputing resources allow the development of high-end, global, cloud-permitting implementations of GEOS-5 at 3.5-km resolutions—among the highest resolutions of any global model to date. The availability of an "evaluation" GPU cluster at NCCS also enables us to explore new technologies and computational methods. Using more than 3 million processor-hours on the NCCS Discover cluster, we have been able to complete a two-year, 10-km "Nature Run" (the highest resolution of its kind) for use in observing system simulation experiments.