Improving HPC Throughput with Memory Profiling

Overview

Analyzing per-thread memory usage of a high-performance computing (HPC) application is an important component in balancing memory load and improving performance. At the same time, controlling swap usage in HPC systems using memory swap is also necessary to help avoid network file system hangs. The Linux kernel 'cgroups' facility is a commonly used method of controlling per-user swap usage, but this approach does not provide any memory profile information. We describe an efficient technique for controlling swap space usage on a per-user basis, while providing profiling information to the user. The user has the option of generating plots with both memory usage per node and memory usage per thread.

Project Details

Technical staff at the NASA Center for Climate Simulation (NCCS) have developed a memory-monitoring tool that tracks a specific job associated with a user's process in the Slurm resource manager and monitors both the memory high-water mark per thread and the per-node memory usage. The memory usage samples for each process ID are recorded in an XML file that can be used to produce graphs. This approach gives the user the opportunity to visually review the memory usage for a specific application. Using these graphs, the user can make the necessary code changes to improve memory distribution in computational processing threads.

The memory-monitoring tool runs in the background with low overhead, providing the opportunity for the user to control the swap ratio per node requested by the resource manager. In addition, we developed it in a low-level programming language using the native Slurm API functions, making the tool shell independent and able to run on various architectures.

Results and Impact

Our memory-monitoring tool provides the HPC user with the following:

• The opportunity to control the memory swap of their run, thereby protecting the HPC network file system from hangs.
• The ability to monitor each thread's memory high-water mark.
• The ability to produce thread memory utilization graphs to help balance thread memory usage.
• The option to use the tool only when a job is initiated and not run it continuously.

Use of this tool at the NCCS has reduced the number of network file system hangs from a daily occurrence to an occasional event.

Why HPC Matters

HPC provides scientists with the means for solving complex research and engineering problems. The responsible utilization of memory resources and the development of well-balanced HPC applications from the memory perspective improve the throughput of users' jobs and decrease expensive downtimes associated with out-of-memory network file system hangs.