Millions of Messages per Minute! Surviving the NCCS’s Log Avalanche

Overview

With rapid increases in the scale and complexity of high-performance computing (HPC) systems, log data processing has become an exciting and intriguing challenge for system administrators. During the last year, the NASA Center for Climate Simulation (NCCS) has increased the number of services that it offers, intensifying the workload of log messages to process. Parsing, ingesting, and processing these logs enable extracting a range of system details for system administrators to solve potential issues. To meet the NCCS’s HPC demands while using commodity-based hardware, the NCCS security team has enhanced the existing log analysis infrastructure with new hardware, tuning, and a modified architecture.

Project Details

The cluster has four main components: forwarding, parsing, processing, and storage. For the forwarding component, we have two Dell R640s, each with NVMe and SSD disks running rsyslog and working as failover to each other. We tested and tuned these systems against UDP and TCP connections using both imudp and imtcp inputs.

Parsing and processing are performed by Logstash and Graylog respectively. These are two R420 systems equipped with 10-Gbps Intel 8259x network interface controllers (NICs). Storage resides in a Hot/Warm Elasticsearch cluster incorporating three Dell C6100 management nodes, one Dell R740 hot indexing node, and one Dell R710 warm data node.

In summary, logs are sent to the rsyslog servers, forwarded to Logstash for parsing, received by Graylog for an additional processing step, and stored by Elasticsearch. This infrastructure provides a stable, reliable environment that can expand to accommodate the NCCS’s rapidly growing needs for HPC processing and storage.

Results and Impact

After tuning the existing rsyslog forwarding servers, we were able to receive approximately 23 million messages/minute UDP and 27 million messages/minute TCP. These represent a substantial improvement over the observed rate of 220,000 messages/minute of our previous systems. Inclusion of management, hot, and warm nodes in the Elasticsearch cluster allows us to manage slow and fast storage capabilities. Fine-tuned Logstash and Graylog systems are now working in combination with hot/warm indexes.

The existing log analysis infrastructure exceeds NCCS assets workload requirements of approximately 200 million messages/day, leaving enough resources to process and store more messages from new services that could be integrated in the near future.

Why HPC Matters

Consolidating logs from multiple environments has proven to be the most effective approach at the NCCS, and for HPC as a whole. By having an infrastructure capable of processing instruments and services information into human-readable representations, we have been able to identify and prevent issues that could not be observed manually. Our enhanced infrastructure puts us ahead of the game as demand increases for more compute and data resources. In the end, the ability to process this huge number of logs opens greater possibilities of deploying more services at the NCCS without compromising monitoring capabilities.

What's Next

Future work includes integration of new hardware into the cluster to continue expanding our capacity according to NCCS growth. We will test additional tuning features to achieve even greater performance. In the near future, we are seeking to process firewall and forensic-related logs to expand our monitoring capabilities. Additional analytics tools will be closely monitored and applied as appropriate.

More Information

• See the poster for this demonstration (PDF, 1.9MB). As shown in the NASA booth at SC18.