Continuous Enhancements to the Pleiades Supercomputer for Maximum Uptime

Overview

When the Pleiades supercomputer was initially installed at the NASA Advanced Supercomputing (NAS) facility in 2008, the system comprised a homogeneous cluster with 51,000 cores and Double Data Rate InfiniBand (IB) links. Since then, the system size has been increased by nearly 2.5 times–to 126,000 cores with four generations of processors and three generations (Double, Quad, and Fourteen Data Rate) of IB links in a single cluster. This heterogeneous architecture has challenged NAS systems experts to develop increasingly effective tools and processes to maintain a productive, operational supercomputer with maximum uptime.

Project Details

Operating an InfiniBand cluster of nearly 12,000 nodes requires the NAS systems team to ensure that the Pleiades IB interconnect is robust enough to handle fault conditions and is able to recover gracefully. One of the enhancements we developed is a modification to the IB Subnet Manager that enables the "live" integration of new nodes or removal of nodes from the IB fabric. This capability allows the system to continue providing computing cycles to users while resources are being added to the system or removed for maintenance activities.

A recent system enhancement is the "suspend and resume" feature in the batch scheduler software. By suspending jobs on the system, some maintenance activities can be completed without needing to drain the system of running jobs–an expensive process in terms of lost computing cycles. By suspending jobs on the system, the overhead maintenance time can be greatly reduced, with minor impact on users.

Another improvement related to the batch scheduler is the ability to have flexible wall-time for batch jobs. For situations when it is not feasible to suspend jobs and we must drain jobs on the system, the ability to run jobs as close as possible to the maintenance window is desirable. By specifying both a minimum and a maximum wall-time, the batch scheduler will allow jobs to start if the minimum wall-time can be completed before the system must be drained. This technique also enables more efficient use of the system.

Results and Impact

By developing continuous enhancements to the batch scheduler and InfiniBand software, the NAS systems team is able to maximize the computing cycles available for scientists and engineers working on important NASA missions in aeronautics, Earth science, human space exploration, and study of the universe.