Supercomputing

Accelerating Satellite Data Discovery for NASA Science Using Parallel Processing

Image from map video representing satellite images as they are captured over a three-year period
Every day, the NASA Center for Climate Simulation (NCCS) receives between 5 and 25 terabytes of satellite imagery into our library of over 25 million satellite images. The map in this video represents the location of satellite imagery captured over a three-year period. As the colors in particular regions change from green to cyan and purple, this indicates new imagery acquired for those areas during a given month. Erin Goodnough, NASA/Goddard

At the NASA Center for Climate Simulation (NCCS), they manage a vast library of satellite imagery of Earth for scientific research. With up to 25 terabytes of data coming into the NCCS daily from various sources, the library holds approximately 25 million images and their associated metadata records, taking up 20% of the 72-petabyte disk storage space. Reducing excess and incomplete data is critical for maintaining a healthy library for efficient data discovery and scientific use.

To optimize this massive library, the NCCS adapted a methodology from the University of Minnesota’s Polar Geospatial Center (PGC) that compared metadata for each of the 25 million images to find candidate files for deletion. NASA’s adapted methodology incorporated NCCS’s architecture and data policies and processes and implemented parallel processing in a cloud computing environment leveraging multiple Windows nodes. They identified 4.7 million files for deletion, freeing up 1.3 petabytes of associated disk space, and completed the analysis and deletions in approximately one month. This work streamlined the library, so scientists can find and process data more efficiently.

Quick Facts

Without utilizing multiple nodes and parallel processing on the Explore/ADAPT science cloud, this work would have taken a year of continuous processing, while new satellite data requiring evaluation for completeness and quality is still coming into the system every day.

Erin Goodnough,
NASA Goddard Space Flight Center
  • This project used the NCCS Explore/ADAPT science cloud computing environment, which includes Windows computing nodes, Python libraries, and access to the 72-petabyte Centralized Storage System (CSS) file system and the open-source PostgreSQL geospatial database containing the metadata for those files.
  • Successfully optimizing the library required understanding where NASA partners’ and end users’ best practices and data architecture came from, how this context informed their existing processes, and how NCCS’s data architecture and workflows differ.
  • The team restored 1.3 petabytes of space after deleting 4.7 million satellite images from the file system and will regularly use this new process to keep the file system streamlined, thus shortening the time for data discovery and enabling scientists to learn more about our world even faster.
  • The NCCS continues refining data management processes from ingest through archival, including validating satellite imagery for appropriate geospatial attributes and file permissions, and ensuring data is available for retrieval after it is archived.

Researcher

  • Erin Goodnough, NASA Goddard Space Flight Center

More Information