Texas Advanced Computing Center to Deploy HPC “Stampede” With Dell and Intel

The Texas Advanced Computing Center (TACC) at The University of Texas at Austin will deploy and support a world-class supercomputer with computing and visualization capabilities for the open science community, as part of the National Science Foundation’s (NSF) ‘eXtreme Digital’ (XD) program.

The new system, called Stampede, will be built by TACC in partnership with Dell, Inc. and Intel Corp. to support for four years the nation’s scientists in addressing the most challenging scientific and engineering problems. NSF is providing $27.5 million immediately and Stampede is expected to be up and running in January 2013. The estimated investment will be more than $50 million over four years; the Stampede project may be renewed in 2017, which would enable four additional years of open science research on a successor system.

"Stampede will be one of the most powerful systems in the world and will be uniquely comprehensive in its technological capabilities," said TACC Director Jay Boisseau. "Many researchers will leverage Stampede not only for massive computational calculations, but for all of their scientific computing, including visualization, data analysis, and data-intensive computing. We expect the Stampede system to be a model for supporting petascale simulation-based science and data-driven science."

When Stampede is deployed in 2013, it will be the most powerful system in the NSF XD environment, currently the most advanced, comprehensive, and robust collection of integrated digital resources and services enabling open science research in the world. As a critical part of XD, the Extreme Science and Engineering Discovery Environment (XSEDE) consortium comprising more than a dozen universities and two research laboratories, has now replaced the TeraGrid as the integrating fabric for the bulk of the NSF’s high-end digital resources. Researchers from any U.S. open science institution can apply to use Stampede for a variety of novel scientific and educational activities through the XSEDE project.

"We at NSF are gratified to fund such a powerful combination of system and services for open science research," said NSF Program Manager Irene Qualters. "The technological capacity is important; but even more important is that the scientific community-those on the front lines of cutting edge research on a number of multi-disciplinary areas critical to addressing society’s greatest challenges-have open access in order to push the frontiers of science and engineering."

Computational science has become the third pillar of scientific discovery, complementing theory and physical experimentation, allowing scientists to explore phenomena that are too big, small, fast, or dangerous to investigate in the laboratory. Advanced computing enables complex computational science and engineering research, and is critical in areas such as weather forecasting, climate modeling, energy exploration and production, drug discovery, new materials design and manufacturing, and more efficient and safer automobiles and airplanes.

"Stampede will usher forth a new era of computational simulation in which observational and experimental data are assimilated into large-scale models to create better predictions, accompanied by quantified uncertainties," said Omar Ghattas, professor of Geological Sciences and Mechanical Engineering, and director of the Center for Computational Geosciences in the Institute for Computational Engineering and Sciences at The University of Texas at Austin. "My group, in particular, is excited about the opportunities Stampede offers to greatly accelerate our work in quantifying uncertainties in computer models of dynamics of polar ice sheets, global seismic wave propagation, and whole-earth plate tectonics."

When completed, Stampede will comprise several thousand Dell Zeus servers with each server having dual 8-core processors from the forthcoming Intel Xeon Processor E5 Family (formerly codenamed Sandy Bridge-EP) and each server with 32 gigabytes of memory. This production system will offer almost 2 petaflops of peak performance, which is double the current top system in XD, and the real performance of scientific applications will see an even greater performance boost due to the newer processor and interconnect technologies. The cluster will also include a new innovative capability: Intel Many Integrated Core (MIC) co-processors codenamed Knights Corner, providing an additional 8 petaflops of performance. Intel MIC co-processors are designed to process highly parallel workloads and provide the benefits of using the most popular x86 instruction set. This will simplify the task of porting and optimizing applications on Stampede to utilize the performance of both the Intel Xeon processors and Intel MIC co-processors.

Additionally, Stampede will offer 128 next-generation NVIDIA graphics processing units (GPUs) for remote visualization, 16 Dell servers with 1 terabyte of shared memory and 2 GPUs each for large data analysis, and a Lustre file system for data-intensive computing. All components will be integrated with an InfiniBand FDR 56Gb/s network for scalability.

Altogether, Stampede will have a peak performance of 10 petaflops, 272 terabytes (272,000 gigabytes) of total memory, and 14 petabytes (14 million gigabytes) of disk storage.

As part of the Stampede project, future generations of Intel MIC processors will be added when they become available, increasing Stampede’s aggregate peak performance to at least 15 petaflops. Stampede also has potential for additional upgrades.

"Stampede is the most powerful x86-based Linux HPC cluster announced for deployment in the U.S. at this time," Boisseau said. "The system builds on TACC’s history of continuously deploying larger and more powerful x86 Linux clusters that enable new scientific breakthroughs. It will also be the first large-scale implementation of Intel’s MIC architecture-based products."

"We’re thrilled about partnering with The University of Texas and TACC to build Stampede, an unprecedented HPC system that will be more powerful than any of the current top HPC systems in the world," said John Mullen, Dell vice president and general manager of Major Public Accounts, Education, State and Local Government.

"Intel is proud to be a core part of enabling the next-generation of scientific discovery for XSEDE’s users," said Anthony Neal-Graves, vice president and general manager of Workstations and MIC Computing at Intel. "Our goal is to provide consistency with the next-generation of Intel processors, co-processors and software so that our nation’s best scientists can focus on scientific discovery and not computer science."

In order to sustain tens of petaflops of performance and achieve exascale computing, Boisseau said, industry and the science community will have to move to highly data parallel processors. "You get so much potential performance out of one of these highly data parallel processors like a GPU or an Intel MIC architecture-based product," he said. "We think this architecture is the quickest path for many applications to sustain petascale performance and to eventually build exascale systems."

Recognizing the importance of advanced computing in enabling breakthrough discoveries, The University of Texas at Austin has pledged additional support for the project, including a new data center to house Stampede set to break ground in November 2011 at the J.J. Pickle Research Campus, home to TACC’s main offices as well as the Ranger and Lonestar supercomputers.

"This grant solidifies The University of Texas at Austin’s role as one of the world’s supercomputing hubs," said William Powers Jr., president of The University of Texas at Austin. "Scientists from around the country will be able to use our supercomputer to tackle some of society’s greatest challenges. It will promote collaboration and problem solving at UT and beyond. Congratulations to Jay Boisseau and his entire team at TACC."

Stampede will support more than a thousand projects in computational and data-driven science and engineering from across the U.S. It will also allow researchers to develop advanced methods for petascale computing, including Intel MIC architecture optimization, and will foster new expertise in data-intensive computing. Finally, Stampede will be used to help train the next generation of researchers in advanced computational science and technology, expanding the use of advanced computing across disciplines and into new communities and domains.

Stampede and the other computational resources of XSEDE are made freely available to researchers across the country through a peer review system. Unlike telescopes or particle accelerators, systems like Stampede help researchers across all disciplines, including the humanities, and are critical to the expansion of knowledge and innovation.

Stampede will be operated and supported for four years by TACC, Dell, and a team of cyberinfrastructure experts at The University of Texas at Austin, Clemson University, University of Colorado at Boulder, Cornell University, Indiana University, Ohio State University, and The University of Texas at El Paso.