Fujitsu Computer System at Institute for Cosmic Ray Research of University of Tokyo

Fujitsu Ltd announced that it has completed construction of a new computer system for the Institute for Cosmic Ray Research (ICRR) of the University of Tokyo.

This x86 cluster system, forming the backbone for ICRR’s research on cosmic rays (1), began operation on January 1, 2014.

The new system will store and analyze observational data on cosmic rays, which include protons, gamma rays, and neutrinos, and will be used for ICRR’s various domestic and international joint research projects in astrophysics and particle physics.

The computation server is an x86 cluster comprised of 136 Fujitsu Server Primergy CX250 S2 servers, acting in tandem with a high-speed distributed file system of 33 Fujitsu Storage Eternus DX80 S2 units running the FEFS (2) file system for speed, reliability, and scalability.

Integer-math performance of the computation server is rated at 96,426 using the SPECint_rate2006 benchmark (3), about ten times the performance of the previous system. It also boasts a file server capacity of 4.4PB, which is roughly 3.2 times more capacity than the previous system, and has a data-transfer speed of 18GB/s, an approximately 30-fold improvement.

This system will support the KAGRA Project (4) and contribute to advances in other research in astrophysics and particle physics.

Background
ICRR installed its first x86 cluster system in January 1991 as a collaborative research environment for multiple projects in the field of particle physics and cosmic-ray research, conducted primarily by the University of Tokyo. At locations both in and outside of Japan, ICRR has conducted research on high-energy cosmic rays, high-energy gamma rays, and neutrinos; for example, the Super-Kamiokande (5) Project has discovered neutrino oscillations and the Tibet AS-gamma Experiment used cosmic rays to observe the sun’s cosmic-ray shadow (6).

Together with expanding its research in the fields of astrophysics and particle physics, ICRR plans to move forward on new experimental projects, such as the KAGRA project, which aims to detect the gravitational wave for the first time in history, and the ground-based gamma ray observatory, CTA (7), which is designed to help understand galactic and extragalactic very-high energy gamma-ray sources.

To meet the needs of these new projects, ICRR required a new computer system that could keep pace with growth expected in the volume of observational data and of analyses. It would be essential for the new system to boost analytical performance, have a greater storage capacity and to enhance data input and output performance.

To satisfy these requirements, the new system started with new hardware that has approximately ten times the integer-math performance of its predecessor, enough storage capacity to accommodate roughly six years’ worth of observational data in cosmic-ray research, and the FEFS scalable file system to sidestep the bottlenecks created by centralized data I/O loads, all of which result in an efficient environment that enables more accurate research.

System Overview
The new system offers two main functions: as a computation server that handles data analysis, and as a file server that stores observational data.

Computation server
An x86 cluster of 136 PRIMERGY CX250 S2 servers with up-to-date CPUs results in integer-math performance rated at 96,426 using the SPECint_rate2006 benchmark, an approximately tenfold improvement over its predecessor.

File server
With 10 Fujitsu Server Primergy RX300 S7 servers and 33 Eternus DX80 S2 storage units, the system has 4.4PB of storage capacity and an 18GB/s data-transfer speed, giving it roughly 3.2 times the storage capacity of its predecessor and about 30 times the data transfer performance. The FEFS scalable file system includes functions that allow storage capacity and bandwidth to be allocated on a per-user and per-group basis, creating a research environment optimized for multiple research projects. The FEFS file system demonstrated its value in Super-Kamiokande‘s system in 2011.

Additional components of the system include 12 Fujitsu Server Primergy RX200 S8 servers used as a login environment for 450 ICRR researchers, a Fujitsu Storage Eternus NR1000 F3220 storage system with 22TB for networked storage, 15 x86 servers that provide e-mail, web, and other network services, a storage unit, and additional network switches.

Future Prospects
The new system will be used in ICRR’s projects both in and outside of Japan, and will help illuminate the mechanisms underlying ultra-high energy phenomena such as where and how high-energy cosmic rays originate and accelerate, through deeper, broader studies of the cosmos. Furthermore, inquiries into the properties of neutrinos and into dark matter particles will lead to even further advances in the study of cosmic rays and particle physics. The world’s first gravitational wave detections promise to reveal new aspects of the universe, bringing us closer to testing the theory of general relativity, and to probe the beginning of the universe.

Composition of New System
ICRR will be the primary user of the new system for multiple research projects.

Composition of the New System

Kimihiro Okumura, associate professor, ICRR, University of Tokyo, explains: “At ICRR, we expect to begin many new research projects. Accordingly, it is anticipated that there will be a significant increase in observational data. To handle the storage and high-speed processing of this data, we needed to upgrade the performance of our computer system. In particular, our goal was to dramatically increase the speed of data transmission, and for a high-performance, highly reliable storage system that meets these requirements, we employed the Fujitsu FEFS file system. We expect the research groups using the computer system to announce many new observational results in the future that generate new insights into hitherto unknown phenomena relating to the universe and cosmic ray particles.“

(1) Cosmic rays: Nuclei and elementary particles originating both inside and outside the Milky Way galaxy. Where they were made and how they are accelerated to such high energies is still not understood.
(2) FEFS: A high-speed distributed file system that can be shared over as many as 100,000 nodes.
(3) 96,426 using the SPECint_rate2006 benchmark: The estimated value using SPECint_rate2006 as a benchmark of CPU performance.
(4) KAGRA Project: The Large-Scale Cryogenic Gravitational Wave Telescope. Selected in 2012 for the Research Infrastructure Program by the Ministry of Education, Sports, Science, Culture, and Technology, facilities for this ICRR project are currently under construction, and due to go into full operation in 2017-2018.
(5) Super-Kamiokande: A facility consisting of a subterranean pool used as a Cherenkov-ray collector located in Kamioka, Gifu Prefecture, Japan.
(6) Sun’s cosmic-ray shadow: A phenomenon in which cosmic rays (high-energy charged particles that move through space) are deflected by the sun’s magnetic field.
(7) Ground-based gamma ray observatory, CTA: A future ground-based gamma-ray observatory planned to come online in 2020 with 10 times the sensitivity of existing gamma-ray telescopes. The sites will be decided in March 2013, and the construction will start in 2015.