Neural Network Training at Scale With Deep Learning Frameworks
Overview
Improvements in instrument technology have resulted in enormous amounts of scientific data being generated on a regular basis. Currently, terabytes of data are produced from just a day’s worth of experiments or observations. This situation has created new opportunities to learn more about the universe and to think differently about data so that we can do better science, and NASA’s scientific community is becoming increasingly aware of modern industrial practices to leverage large amounts of data to answer key questions. With this fresh interest in data science, the High-End Computing Capability (HECC) Project’s Data Science team created a framework to facilitate the translation and formatting of data pipelines to work with the deep learning frameworks currently being developed in industry.
In deep learning, a subset of data science, nested neural networks are built to create models based on large datasets. Deep learning is an enabling technology for artificial intelligence and is credited for recent advances in areas such as image recognition, pattern recognition, and anomaly detection—tools that can assist NASA in the development of new technologies.
Project Details
The Data Science team aims to provide NASA researchers with deep learning tools to support advancements in scientific discovery by maintaining software stacks for use in the HECC environment at the NASA Advanced Supercomputing (NAS) facility. This environment supports software languages such as Python and R, and the software stacks are built using Anaconda, an open-source and extensible framework for managing various software packages and environments. We currently support eight different conda environments as part as the deep learning framework we maintain for our users. The framework caters to users’ varying requirements, providing them with ready access to environments that contain the latest versions of TensorFlow, PyTorch, and other popular utilities used in deep learning. Recently, we added the distributed training utility Horovod as a standalone environment that allows users to distribute their training across multiple nodes up to three times faster than using the TensorFlow training distribution method. This software stack is used on 49 NVIDIA V100 GPU nodes at the NAS facility, with a total of 200 GPUs.
Deep learning is an enabling technology for artificial intelligence and is credited for recent advances in areas such as image recognition, pattern recognition, and anomaly detection—building tools that will assist NASA in unprecedented ways in future missions.
Results and Impact
More than 120 users have accessed our framework, creating over 140 unique environments by cloning the conda environments we maintain. These modified environments allow the researchers to train deep learning models on the NAS facility supercomputers. One example of the science supported by these environments is the development of carbon nanotube chemical (CNT) sensors. NASA recently developed inexpensive, low-power CNT sensors for use on the International Space Station (ISS) to monitor cabin air quality. The CNT surfaces can be engineered to be highly adsorptive to ammonia gas or carbon dioxide. When these gases adsorb onto the surface, the electrical properties of the CNT changes; however, it is challenging to convert the CNT signals to human-readable indicators of air quality and safety. Using our conda environments, researchers trained a neural network to learn the CNT’s response to the presence of ammonia or carbon dioxide, enabling it to predict the gas concentration the CNT was exposed to. With initial results showing predictive accuracy better than 94%, the researchers determined that neural networks can be a very effective way to interpret CNT data—a major step toward real-world testing on the ISS.
Our new Horovod environment enables neural network training at scale by allowing for multi-node GPU training of machine learning models developed in either TensorFlow or PyTorch. Horovod allows for easy distribution of training across multiple nodes and is faster than TensorFlow’s built in multi-node distributions. After applying minor code changes, Horovod can easily scale distribution to an optimal number of nodes—in one test, Hovorod processed 19 times more image files from the dataset by distributing training from one node to 15 nodes.
More Information
Researchers
- Shubha Ranjan, NASA Ames Research Center Please enable Javascript to see email address.
- Thaddeus Norman, NASA Ames Research Center Please enable Javascript to see email address.
- Jonathan Gee, NASA Ames Research Center Please enable Javascript to see email address.
