Porting Earth Science Applications to Next-Generation Intel Xeon Phi Coprocessors

Overview

The NASA Center for Climate Simulation (NCCS) delivers modern parallel systems and services to support NASA's Earth exploration missions. Heterogeneous computing with multiple levels of parallelism is key for designing next-generation exascale systems that contain both multi-core chips and accelerators, such as graphics processing units (GPUs) or coprocessors like Intel's Xeon Phi. However, adapting scientific applications to take advantage of the performance provided by heterogeneous computing resources is no easy task.

Project Details

In late 2012, NCCS augmented its Discover supercomputer with 480 Intel Xeon Phi coprocessors integrated with 480 Intel Xeon Sandy Bridge nodes. To prepare the user community for effectively and efficiently utilizing Phi, we focused on porting existing scientific applications and characterizing the porting and optimization efforts required for a wide range of applications, including both Fortran and C codes, applications with high Intel Math Kernel Library (MKL) usage, Open Multi-Processing (OpenMP) codes, and pure Message Passing Interface (MPI) codes without OpenMP threading.

Results and Impact

Porting a number of scientific applications to the Intel Xeon Phi is relatively straightforward, but it takes significant optimization efforts to reach Phi's potential performance capacity. We find that regardless of the programming model we are using (offload, native, or symmetric), there are four steps we must apply to attain good performance on Phi:

• Characterization of the applications. We use profiling tools like Vtune and Tuning and Analysis Utilities (TAU) to detect hot spots and evaluate the events and metrics of selected hot spots using the hardware counters.
• Thread-level optimization. For pure MPI codes, we add OpenMP directives into hot loops. OpenMP collapse-clause is used for some loops to increase thread parallelism in order to to facilitate the large number of threads on Phi.
• Data-parallel implementation using Single Instruction, Multiple Data (SIMD). Removing dependencies between loops, using SIMD or IVDEP directives when necessary, and avoiding non-unit stride access or random access is critical to achieving fine-level SIMD parallelism. We also use Vtune hardware counters to provide vectorization intensity metrics for guiding our optimization.
• Optimization for memory access. Optimization techniques we have used for smart memory access include data alignment to a 64-byte boundary, a blocking/tiling data structure to increase data reuse in cache, the use of huge pages, and software prefetching.

By applying these techniques and lessons learned, we achieve performance improvement for several user codes. In addition, optimizing for the Intel Xeon Phi also enables users to run their applications more efficiently on standard Intel Xeon processors.