NAS Data Portal: Distributing Large, High‑Volume Computational Datasets

Overview

The NAS Data Portal is a web-based resource for sharing high-volume computational datasets in order to promote collaboration, data dissemination, and the sharing of results obtained from projects run on the high-performance computing systems at the NASA Advanced Supercomputing (NAS) facility. The shared dataset files are staged to a public-facing interface outside of the NAS facility’s restricted compute enclave in a safe and secure manner, enabling researchers to share datasets with the public at large or with specific collaborators. Since the original dataset files are staged rather than duplicated, there is no restriction on the amount of data that can be shared. Currently, over four petabytes (PB) of shared data is publicly available, in disciplines such as heliophysics, planetary research, Earth science, and oceanography.

"The NAS Data Portal has significantly helped scientists and engineers supporting NASA missions to disseminate and share computational results, without having to be concerned with the technical issues associated with sharing large datasets."

Project Details

Development of the NAS Data Portal was motivated by a desire to provide a turn-key system for computational scientists to easily and securely share their large datasets without having to be concerned with technical issues typically associated with migration and sharing of very large datasets. The data portal is comprised of an upload server that provides an interface for submitting, approving, and updating datasets, and two download servers for navigating and downloading the files, which provide load balancing during periods of high traffic. Dataset files are staged from the restricted servers in the NAS secure enclave out to the public-facing interface using a “re-exporter” host and in‑house software, which creates mount points on the restricted filesystems that are visible on the external web servers. Dataset metadata is stored on a dedicated database that serves file sizes and creation times without having to reference the original filesystem, and securely tracks which files are currently available for download. The data portal was primarily built using the dynamic Django web framework, which creates on-the-fly webpages from database contents.

Results and Impact

Since its inception, the NAS Data Portal has significantly helped scientists and engineers supporting NASA missions to disseminate and share computational results, often as an adjunct reference associated with publication of the results. With this resource, the researchers do not need to spend time dealing with the technical issues associated with sharing massive data volumes. Currently, there are 15 datasets publicly available on the data portal in various disciplines—primarily in Earth and geosciences, planetary geophysics, and heliophysics. The datasets range from 400 kilobytes (KB) to 4 PB in size and are comprised of anywhere from one single file to 185 million files. In 2020, over 140 terabytes (TB) of data has been downloaded. A typical week may see approximately 150,000 to over 800,000 file downloads.

Why HPC Matters

The high-performance computing (HPC) environments at the NAS facility enable massive computational studies that produce enormous volumes of output data. This HPC infrastructure provides the necessary resources to build a capable, efficient and secure environment for sharing complete study results, including the re-exporter, fileservers, web servers, and database—as well as the staff expertise needed to build, maintain and expand the data portal—representing a significant new capability for the NAS facility.

What's Next

Near-future enhancements include the addition of more hardware servers for greater stability, throughput, and load-balancing during peak download times, and expansions to the re-exporter for better stability and efficiency in filesystem staging. New user-friendly features are also being developed to assist with high-volume dataset expansion (thousands of files per day), and to enable automatic approval and publishing of new dataset files.

We are also planning dynamic directories for near-real-time publication of data, via data pipelines operating within the secure compute enclave, in order to share large numbers of new files every day. The long-term vision for the NAS Data Portal is to serve as a more general data discovery platform, incorporating dataset querying, spatiotemporal subsetting, and on-the-fly data visualization services.

Related Links

• NAS Data Portal