Testing Code at Exascale with Only a Workstation

Overview

Unfortunately, some critical software defects are only exhibited when applications run at very high resolutions on large numbers of cores. However, debugging and unit testing at exascale continues to be extremely expensive in terms of hardware resources, licenses for debuggers, developer time, etc. We are exploring two technologies that may reduce these costs in the future. The first approach is to develop a "mock" of the message-passing layer and thereby eliminate the dependency on the actual exascale environment for a given process. A single process would still use grid resolutions and local data structures identical to those that would exist in the true exascale environment. The second approach is to develop an advanced parallel Fortran logging utility that flexibly supports activation/deactivation of low-level application-specific diagnostics.

Project Details

• MockMPI: We are extending our parallel Fortran Unit testing framework (pFUnit) by creating a configurable "mock" implementation of the Message Passing Interface (MPI). Users specify a sequence of MPI interactions that a single process will experience during its execution. The application is linked with MockMPI instead of MPI and then exercised in a purely serial context. Unit testing and debugging can then proceed on smaller platforms, e.g., workstations.
• pFlogger: We have developed a parallel Fortran logging utility that is largely based on the design of the python logger. pFlogger has crucial extensions for parallel execution including rerouting and/or annotating messages from different processes and aggregating messages across multiple processes. pFlogger uses a simple configuration file that allows low-level diagnostic messages to be selectively activated at run-time on a component-by-component basis.

Results and Impact

Using the technologies we are developing should significantly reduce the costs of certain types of software debugging and testing. Locally run unit-tests can guard against defects that would otherwise be undetectable except on large systems. Debugging can be done on local workstations with serial debuggers without the need to wait for large, interactive batch sessions on a mainframe.

Why HPC Matters

Ideally all use of high-end computing (HEC) would be to produce new scientific/technical results. Unfortunately, a significant portion of resources is actually used for debugging and testing software in preparation for production runs. Approaches such as those described here can increase the proportion of HEC resources that are used for actual results.

What's Next

• Mock MPI is in an early prototype stage. Expanding the functionality to a larger subset of MPI is straightforward, but more work is required for the user interface to simplify specification of complex sequences of MPI interactions. Ease-of-use is essential for user adoption.
• pFlogger is currently in the process of being released as NASA open-source software. Widespread deployment within our own scientific applications is waiting on certain compiler fixes that impact build times for large codes.