National Center for Atmospheric Research Announcing Powerful HPC, Cheyenne, for Scientific Discovery

The National Center for Atmospheric Research (NCAR) has selected its next supercomputer for advancing atmospheric and Earth science, following a competitive open procurement process.

The machine will help scientists lay the groundwork for improved predictions of a range of phenomena, from hour-by-hour risks associated with thunderstorm outbreaks to the timing of the 11-year solar cycle and its potential impacts on GPS and other sensitive technologies.

The system, named Cheyenne, will be installed this year at the NCAR-Wyoming Super-computing Center (NWSC) and become operational at the beginning of 2017.

Cheyenne will be built by Silicon Graphics International Corp. in conjunction with centralized file system and storage components provided by DataDirect Networks, Inc.. The SGI HPC will be a 5.34-petaflop system, meaning it can carry out 5.34 quadrillion calculations per second. It will be capable of more than 2.5 times the amount of scientific computing performed by Yellowstone, the current NCAR supercomputer.

Funded by the National Science Foundation (NSF) and the state of Wyoming through an appropriation to the University of Wyoming, Cheyenne will be a critical tool for researchers across the country studying climate change, severe weather, geomagnetic storms, seismic activity, air quality, wildfires, and other important geoscience topics. Since the super-computing facility in Wyoming opened its doors in 2012, more than 2,200 scientists from more than 300 universities and federal labs have used its resources.

“We’re excited to bring more super-computing power to the scientific community,” said Anke Kamrath, director of operations and services, NCAR’s computational and information systems laboratory. “Whether it’s the threat of solar storms or a heightened risk in certain severe weather events, this new system will help lead to improved predictions and strengthen society’s resilience to potential disasters.“

“Researchers at the University of Wyoming will make great use of the new system as they continue their work into better understanding such areas as the surface and subsurface flows of water and other liquids, cloud processes, and the design of wind energy plants,” said William Gern, VP of research and economic development, University of Wyoming. “UW’s relationship with NCAR through the NWSC has greatly strengthened our scientific computing and data-centric research. It’s helping us introduce the next generation of scientists and engineers to these endeavors.“

The NWSC is located in Cheyenne, and the name of the system was chosen to honor the support that it has received from the people of that city. It also commemorates the upcoming 150th anniversary of the city, which was founded in 1867 and named for the American Indian Cheyenne nation.

Increased power, greater efficiency
The storage system for Cheyenne will be integrated with NCAR’s existing GLADE file system. The DDN storage will provide an initial capacity of 20PB, expandable to 40PB with the addition of extra drives. This, combined with the current 16PB of GLADE, will total 36PB of high-speed storage. The new DDN system also will transfer data at the rate of 200GB/s, which is more than twice as fast as the current file system’s rate of 90GB/s.

The system will include Intel Xeon processors, whose performance will be augmented through optimization work that has been done by NCAR and the University of Colorado Boulder. NCAR and the university performed this work through their participation in the Intel Parallel Computing Centers program.

Even with its increased power, Cheyenne will be three times more energy efficient (in floating point operations per second, or flops, per watt) than Yellowstone, its predecessor, which is itself highly efficient.

“The new system will have a peak computation rate of over 3 billion calculations per second for every watt of power consumed,” said NCAR’s Irfan Elahi, project manager, Cheyenne and section manager for high-end super-computing services.

More detailed predictions
HPC such as Cheyenne allow researchers to run increasingly detailed models that simulate complex processes and how they might unfold in the future. These predictions give resource managers and policy experts valuable information for planning ahead and mitigating risk.

Some of the areas in which Cheyenne is expected to accelerate research:

Stream-flow: Year-ahead predictions of stream-flows and associated reservoir levels at a greater level of detail will provide water managers, farmers, and other decision makers with vital information about likely water availability and the potential for drought or flood impacts.
Severe weather: By conducting multiple simultaneous runs (or ensembles) of high-resolution forecast models, scientists will lay the groundwork for more specific predictions of severe weather events, such as the probability that a cluster of intense thunderstorms with the risk of hail or flooding will strike a county at a particular hour.
Solar energy: Specialized models of solar irradiance and cloud cover will be run more frequently and at higher resolution, producing research that will help utilities predict how much energy will be generated by major solar arrays hours to days in advance.
Regional climate change: Scientists will conduct multiple simulations with detailed climate models, predicting how particular regions around the world will experience changing patterns of precipitation and temperature, along with potential impacts from sea level rise, stream-flow, and runoff.
Decadal prediction: Ensembles of detailed climate models will also help scientists predict the likelihood of certain climate patterns over a 10-year period, such as the risk of drought for a certain region or changes in Arctic sea ice extent.
Air quality: Scientists will be able to simulate the movement and evolution of air pollutants in far more detail, thereby better understanding the potential health effects of particular types of emissions and working toward improved forecasts of air quality.
Subsurface flows: More accurate and detailed models will enable researchers to better simulate the subsurface flows of water, oil, and gas, leading to a greater understanding of these resources.
Solar storms: Innovative, three-dimensional models of the sun will lay the groundwork for predictions of the timing and strength of the sun’s 11-year cycle as well as for days-ahead forecasts of solar disturbances that can generate geomagnetic storms in Earth’s upper atmosphere.

“Super-computing is vital to NCAR’s scientific research and applications, giving us a virtual laboratory in which we run experiments that would otherwise be impractical or impossible to do,” said NCAR director James Hurrell. “Cheyenne will be a key component of the research infrastructure of the United States through its provision of super-computing specifically tailored for the atmospheric, geospace, and related sciences. The capabilities of this new system will be central to the continued improvement of our ability to understand and predict changes in weather, climate, air quality, and space weather, as well as their impacts on people, ecosystems, and society.“

Key features of Cheyenne supercomputer system:

34-petaflop SGI ICE XA Cluster featuring a future Intel ‘Broadwell’ processor
More than 4K compute nodes
20% of the compute nodes have 128GB memory and the remaining ~80% have 64GB memory
313TB of total memory
Mellanox EDR IB high-speed interconnect
Partial 9D Enhanced Hypercube interconnect topology
SUSE Linux Enterprise Server OS
Altair PBS Professional Workload Manager
Intel Parallel Studio XE compiler suite
SGI Management Center & SGI Development Suite
Mellanox Unified Fabric Manager