Great Scott! 1.21 Megawatts and 88 Gigabytes per Second: Time Travel with a Time Series Database

Overview

Time Series Databases (TSDBs) are databases optimized for storing colossal volumes of data indexed by time. Modern TSDBs can ingest hundreds of thousands of data points per second using commodity hardware. This is an ideal fit for monitoring a supercomputer, which may have thousands—if not tens of thousands—of components, each with many distinct metrics.

Project Details

The NASA Center for Climate Simulation (NCCS) Operations Team is using a TSDB to help monitor the Discover supercomputer with its 3,600+ nodes and additional infrastructure components. Every 10 seconds, the TSDB stores:

• 85,000+ measurements relating to its General Parallel File System (GPFS). The Discover supercomputer's disk systems have an aggregate read bandwidth of approximately 88 gigabytes per second.
• 11,000+ power-related measurements from rack power distribution units (PDUs). The peak power utilization of Discover's four newest scalable units is approximately 1.2 megawatts.

Results and Impact

High-resolution historical data has proven invaluable for both real-time monitoring and post-mortem capabilities, effectively allowing a systems administrator to "time travel" to the moment of a specific event. From a tripped PDU breaker to a user-reported filesystem slowdown, an administrator can examine data from the event in high resolution as if it were occurring in the present moment. Having access to historical data at this resolution has enabled problem diagnosis and root cause analysis that was not possible before now.

The collected power usage data has enabled the NCCS to create an aggregated view of power consumption across compute racks, individual Scalable Compute Units (SCUs), room PDUs, and facility uninterruptible power supplies (UPSs). This information has yielded a better understanding of overall power load trends across the facility.

Why HPC Matters

High-performance computing (HPC) resources provide scientists with the means for solving complex research and engineering problems and increasing the resolution of existing models. Detailed monitoring of these computing systems allows administrators to quickly understand, diagnose, and resolve system problems, minimizing disruptions to the NCCS scientific community as they pursue a better understanding of the world around us.

What's Next

Administrators are exploring using TSDBs to store additional information such as node-level performance data and detailed job resource usage summaries. This data could then be used to provide additional information to NCCS support staff and the NCCS user community via web-based portals.