Akquinet, MT-C and Fenix Consortium Team Up to Enhance Multifunctional Data Mover NODEUM

Akquinet AG, MT-C SA and the Fenix consortium have teamed-up to enhance the multifunctional Data Mover NODEUM for realising a data mover service within the Fenix infrastructure.

The consortium have teamed-up to enhance the multi-functional Data Mover NODEUM for realising a data mover service within the Fenix infrastructure. This service will allow users to manage data stored in performance parallel file systems as well as on-premise cloud data repositories.

The Jülich Supercomputing Centre (JSC), as representative of the Fenix consortium, has decided to commission Akquinet and MT-C with development tasks for the implementation of part of the Fenix infrastructure. The aim is to simplify the data transfer between cloud object stores with a Swift interface and fast parallel file systems with a POSIX interface. For this purpose, the multifunctional data mover NODEUM is to be used with the corresponding expansion of specific functionalities.

The goal of the project is to realise the Data Mover service at different Fenix locations, including the European supercomputing centres JSC (Jülich Supercomputing Centre, Germany), BSC (Barcelona Supercomputing Center, Spain), CINECA – Inter-University Consortium (Italy), CEA (France) and CSCS (Swiss National Supercomputing Centre, Switzerland). Federated cloud object stores are used at these locations, which enable researchers to exchange their data. With the Data Mover, researchers can copy their data locally to the existing parallel file systems in order to process them on fast HPC systems. In addition, researchers can copy data, which have been generated on these HPCs to the cloud object stores in order to make them accessible to other researchers over the world.

Today, the main storage service provided by HPC centres is realised by internal but fast parallel file systems based on, e.g., GPFS or Lustre. They thus work as individual silos in internal infrastructures. To make data externally accessible and sharable, cloud object stores technologies are better suitable. The scope of the project is to connect the 2 worlds with the NODEUM Data Mover and to transfer data from the world of the parallel file systems to the new world of Posix/S3 – considering all AAI requirements.