Ziggy: A Powerful New Tool for Data Analysis Pipeline Management
NASA projects and missions that generate vast amounts of data, such as the agency’s ground- and space-borne observatories, require efficient science data pipelines to process and analyze the information they gather. Researchers have historically written their pipeline management software from scratch—a challenging undertaking for scientists who may not have extensive software development experience. To help relieve future missions of this obligation, a team of researchers at NASA’s Ames Research Center have developed a purpose-built application, called Ziggy, for management of data analysis pipelines in any subject matter area. Ziggy provides easy-to-use tools to define, configure, and manage data analysis pipelines, track execution outcomes, preserve logs and other state information, and organize raw mission data, results, and instrument models. Descended from the pipeline management software used so successfully by the Kepler Space Telescope and Transiting Exoplanet Survey Satellite (TESS) missions, Ziggy is lightweight enough to run on a laptop, and provides built-in support for submitting and managing processing algorithms on high-performance computing systems at the NASA Advanced Supercomputing (NAS) facility.
Quick Facts
The Kepler and TESS missions would not have been possible without pipeline management tools that supported processing on NAS supercomputers. Ziggy brings those same capabilities to future missions of all kinds, lowering the entry barriers for data analysis projects that want to use NASA’s enormous compute resources to expand their science reach.
- Ziggy integrates full support for execution of compute-intensive algorithms on the supercomputers at the NAS facility.
- Ziggy’s tools simplify the process of configuring NAS jobs: users enter their parameters such as required memory per process and select an optimization such as cost or wait times; then, Ziggy converts the user parameters to NAS-side parameters, such as architecture, number of nodes, wall times, and so on.
- The software makes it simple and easy to develop a data analysis pipeline, deploy it to NAS systems, and run it for the duration of a mission.
- Future developments will include support for Amazon Web Services (AWS) and for hybrid NAS/AWS execution, in which a single instance of Ziggy will execute tasks on AWS or NAS depending on which is more appropriate.
Researchers
- Peter Tenenbaum, NASA Ames Research Center Please enable Javascript to see email address.
- Jon Jenkins, NASA Ames Research Center Please enable Javascript to see email address.
