NASA Center for Climate Simulation Debuts Centralized Storage With Curated Earth Science Datasets

By Jarrett Cohen, NASA Goddard Space Flight Center

Now available to all NASA Center for Climate Simulation (NCCS) computing platforms is a new resource hosting notable earth science datasets.

Centralized Storage System extends across
several hardware racks and currently holds 10PB
of curated Earth science datasets.
(Photo by Laura Carriere, NCCS)

NCCS users can access the Centralized Storage System (CSS) from the Discover supercomputer, the ADAPT Science Cloud, the DataPortal, and a new GPU cluster focused on Artificial Intelligence (AI) and Machine Learning (ML) applications that will be available on August 10, 2020. As of July 2020, CSS holds 10PB of curated datasets from NASA, other agencies, and national and international research programs.

“NCCS previously provided long-term storage on tape media, which resulted in data that was difficult for users to discover and integrate into compute workflows,” said Bennett Samowich, lead, NCCS CSS. “By holding curated data products that are archived at other locations, CSS allows fast access to datasets in support of scientific research.“

“In particular, AI and ML workflows can take advantage of the availability of these datasets,” Samowich noted. “CSS is the best location for final data products that can become input to other projects.”

CSS-hosted datasets span range of types and sources, including:

• Satellite observations, e.g., Landsat, MODIS, and NGA.

• Airborne observations, e.g., IceBridge.

• NASA and ensemble climate reanalyses, e.g., MERRA-2 and CREATE-IP.

• Climate projections, e.g., CMIP5.

The path ‘/css’ provides access to all of the CSS data, which is stored using IBM’s Spectrum Scale General Parallel File System (GPFS). fast internal IB and 40GbE networks connect CSS to the NCCS compute and data services environments

While CSS is available as read-only, its 15PB capacity (growing to 30PB later this summer) will allow NCCS users to make their own data available to other users. Besides Earth science data, NCCS is open to hosting users’ astrophysics datasets in the future. NCCS encourages users to share their data by requesting CSS access and moving their final data products to CSS. Data must fall under an active Data Management Plan; contact the NCCS User Services Group to get help setting up a plan.

In addition to CSS, “NCCS will continue to provide local storage on both Discover and ADAPT for intermediate datasets,” Samowich said. “This local storage will continue to support workflows that require fast I/O during computation as well as analysis of research results.”