Alluxio Enhanced Hybrid Cloud Solution Based on Optane Persistent Memory

Alluxio, Inc. announced a go-to-market solution in collaboration with Intel Corp. to offer an in-memory acceleration layer with 2nd Gen Xeon Scalable processors and Intel Optane persistent memory (PMem).

The solution eliminates performance degradation of analytics clusters that are increasingly built on disaggregated compute and storage architecture.

“Today’s disaggregated cloud storage lacks efficient file system semantics support like ‘rename’. Additionally, disaggregated cloud storage typically can’t leverage compute side storage media such as DRAM and SSD for use as buffers and page caches,” said Haoyuan Li, founder and CEO. “Adding Alluxio Data Orchestration System and Intel’s Optane persistent memory solves both issues, enabling maximum benefit for cloud storage and achieving competitive and even better performance than traditional on-premises configurations. This is particularly helpful for hybrid cloud environments when data is remote.“

Benchmarking results show 2.13x faster completion compared to local HDFS and a 1.92x speedup over DRAM cache for 4TB decision support queries when adding the company and Intel persistent memory ⁽¹⁾. Using Storage over App Direct, a feature of PMem App Direct mode, allows the company to access high-performance block storage without the latency of moving data across the I/O bus to provide the data acceleration and reduction in query runtime. With the firm and Xeon Scalable processors, an I/O intensive benchmark delivers a 3.4x speedup over disaggregated S3 object storage and a 1.3x speedup over a co-located compute and storage architecture ⁽²⁾.

In addition, the company has joined into a collaboration with Intel aimed at improving their joint customers’ experience with managing and processing their data, such as optimizations for Intel Deep Learning Boost, the AI acceleration technology built into Xeon Scalable processors. Together, the partners will bring solutions to market to help fuel next-gen data, analytics and AI applications and use cases.

“With this new collaboration we bring to market a solution that enables enterprise-grade shared storage and faster time to insights to solve for the challenges we see around bounded storage and compute resources on Hadoop,” said Rowan Scranage, chief business officer. “Together with Intel, we plan to disrupt the advanced analytics and AI status quo with an in-memory data accelerator layer to accelerate intermediate data access and ease data bottlenecks that many of our customers are highlighting as key challenges with their increasing big data requirements.“

“We are thrilled to work with Alluxio and help bring this compelling solution to market,” said Arijit Bandyopadhyay, CTO, enterprise analytics and AI, data platform group, Intel. “As we see more companies building advanced analytics and AI applications that are faced with performance challenges, the Alluxio-based in-memory data accelerator coupled with Intel’s Optane persistent memory will bring data locality, data accessibility, and data elasticity back into the environment. Based on the performance gains we are seeing with this joint solution, we are looking forward to bringing it to the market at scale.“

Resources:
Solution brief: Get insights faster with Alluxio and Intel    
Blog: Speeding Big Data Analytics on the Cloud with In-Memory Data Accelerator   
Tech Talk: On-demand tech talk: Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage (registration required)
Webinar: Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & Alluxio

⁽¹⁾ Footnote: Tested by Intel as on 12/06/2019 with the following configuration. Detailed white paper will follow.
Optane PMem cache: 2 socket Intel Xeon Gold 6240 Processor, 18 cores HT On Turbo ON Total Memory 192GB (12 slots/ 16GB/2666 MHz), DCPMM 1TB (8 slots/ 128GB/ 2666MHz), DCPMM firmware version: 01.02.00.5410, BIOS: SE5C620.86B.0X.02.0094.102720191711 (ucode:0x500002c), BKC version: ww08.2019, Fedora 29 (Server Edition), 4.20.6-200.fc29.x86_64, Storage for application: 11x1TB HDD (ST1000NX0313) for Ceph OSD, Hadoop version: Hadoop 3.1.2, Alluxio version: Alluxio 2.0.0, Spark version: Spark 2.3.0, Hive version: Hive 3.1.1, Ceph version: Ceph 12.2.12
DRAM cache: 2 socket Intel Xeon Gold 6240 Processor, 18 cores HT On Turbo ON Total Memory 768GB (24 slots/ 32GB/ 2666MHz), BIOS: SE5C620.86B.0X.02.0094.102720191711 (ucode:0x500002c), Fedora 29 (Server Edition), 4.20.6-200.fc29.x86_64, Storage for application: 11x1TB HDD (ST1000NX0313) for Ceph OSD, Hadoop version: Hadoop 3.1.2, Alluxio version: Alluxio 2.0.0, Spark version: Spark 2.3.0, Hive version: Hive 3.1.1, Ceph version: Ceph 12.2.12
Without cache: 2 socket Intel Xeon Gold 6240 Processor, 18 cores HT On Turbo ON Total Memory 192GB (12 slots/16GB/2666 MHz), BIOS: SE5C620.86B.0X.02.0094.102720191711 (ucode:0x500002c), Fedora 29 (Server Edition), 4.20.6-200.fc29.x86_64, Storage for application: 11x1TB HDD (ST1000NX0313) for Ceph OSD, Hadoop version: Hadoop 3.1.2, Alluxio version: Alluxio 2.0.0, Spark version: Spark 2.3.0, Hive version: Hive 3.1.1, Ceph version: Ceph 12.2.12
⁽²⁾ Details and configs

Read also:
Extended Series B Funding of $15.5 Million for Alluxio
Now totaling $23 million
April 17, 2020 | Press Release
Availability of Alluxio Structured Data Service Featuring Data Catalog Service and Transformation Service
Provides just-in-time data transform of data to be compute-optimized for applications like Presto, independent of storage solution or format.
March 20, 2020 | Press Release