Optimizing User Applications for NASA's HECC Environment

Overview

It is amazing to consider how many things must go "just right" in order to make the most effective use of high-performance computing systems. For example, the way variables are laid out in memory can have a large impact on performance. Compute resources can be added to speed things up, but this does not solve code performance issues. Are we minimizing time-to-solution, or the use of resources, or both? Once that question is answered, others emerge, including the question of how to approach optimization. The High End Computing Capability (HECC) Project's Application Performance and Productivity (APP) team works closely with NASA users to understand performance issues and optimize their code.

Project Details

The APP team provides a range of custom consulting services for HECC users, such as helping them port codes to NASA supercomputing resources, and diagnosing and finding solutions for job failures. When a user asks for help with code optimization, the APP team requests the code, a representative dataset, and expected results. The team then begins an iterative process to identify, diagnose, and resolve performance issues with the code running on HECC supercomputers at the NASA Advanced Supercomputing (NAS) facility at Ames Research Center.

Performance analysis tools are essential to this process. These software programs can help identify bottlenecks by reporting where the code is spending wall-clock time; determine whether a "hot spot" has been compiled using scalar or vector instructions; and find imbalances between different threads of execution. Each analysis program has strengths and weaknesses, so the APP team uses a variety of different tools during the optimization process.

Results and Impact

Many HECC users have requested code optimization help from our APP team consultants over the past year. The team has responded with a number of notable performance successes, including:

• 10x speedup for the Goddard Cumulus Ensemble (GCE), an atmospheric weather model that is used to research the role of water and energy cycles in the tropical climate system, aerosol impact on cloud and precipitation in different environments, and other precipitation-related phenomena.
• 15x speed-up for Screen, a load-balancing code used to calculate the transient response of International Space Station interfaces during applied dynamic loading events, such as docking and orbit boost.
• 34% performance improvement for StellarBox, a 3D magnetohydrodynamics code used to simulate the convection zone and lower atmosphere of the Sun and other stars.

In many cases, performance improvements achieved by the APP team enable scientific or engineering analyses that were previously intractable. In the case of GCE, for example, improvements allowed scientists to investigate the connection between large-scale weather conditions and internal cloud dynamics, which previously required more memory than was available. A simulation that would have taken 10¬–12 months was completed in just in one month using the optimized code—enabling the GCE project to meet its goals, as the original code would have taken more wall-clock time than was available for the duration of the project.

What's Next

The APP team plans to initiate very lightweight performance instrumentation in all Message Passing Interface (MPI) jobs running on the NASA's Pleiades supercomputer. The resulting information will help us proactively identify cases with severe resource utilization problems resulting from poor load balancing, incorrect process pinning, or other causes.