Micron Open-Source Heterogeneous-Memory Storage Engine Designed for SSDs and Storage-Class Memory

Micron Technology, Inc. announced an open-source Heterogeneous-memory Storage Engine (HSE), designed for SSDs and storage-class memory (SCM).

Click to enlarge

Legacy storage engines born in the era of HDDs failed to architecturally provide for the increased performance and reduced latency of next-gen non-volatile media. HSE, originally developed by the company and available to the open-source community, is for developers using all-flash infrastructure who require the benefits of open-source software, including the ability to customize or enhance code for their use cases.

“As the only company developing storage class memory, flash and DRAM technologies, Micron is uniquely positioned to build a software stack that accelerates applications running in today’s flash-based storage environments as well as storage class memory-based infrastructure of the future,” said Derek Dicker, corporate VP and GM, storage business unit. “We have delivered a first-of-its-kind innovation for open-source storage developers, unlocking the potential of high-performance storage applications.“

In addition to delivering performance and endurance improvements, HSE reduces latency, especially for large-scale data sets, through intelligent data placement. It improves throughput of particular storage applications by up to 6x, reduces latency 11x ⁽¹⁾ and improves SSD endurance by 7x. It can also take advantage of multiple classes of media concurrently, such as flash and 3D XPoint technology. When a firm’s X100 NVMe SSD, a fast SSD, is added to a set of four 5210 QLC SSDs, throughput more than doubles and read latency improves nearly 4x. ⁽²⁾

“We see enormous potential in the technologies being introduced by Micron, especially as it takes an innovative approach in lowering the latency between compute, memory and storage resources,” said Stefanie Chiras, VP and GM, Red Hat Enterprise Linux, Red Hat, Inc. “We look forward to working with Micron in the open-source community to further build out these innovations, ultimately bringing new choices to the storage world derived from open standards and concepts.“

“As the demand for object-based storage continues to increase and it’s deployed for more and more workloads, it’s no surprise that our customers are increasingly interested in fast object storage,” said Brad King, field CTO and co-founder, Scality S.A. “While our storage software can support ‘cheap and deep’ on the lowest-cost commodity hardware for the simplest workloads, it can also exploit the performance benefits of technologies like flash, storage class memory and SSDs for very demanding workloads. Micron’s HSE technology enhances our ability to continue optimizing flash performance, latency and SSD endurance without trade-offs.“

Features and benefits of Heterogenous-Memory Storage Engine:

Integration with MongoDB, an NoSQL database, delivers performance improvements, reduces latency and takes advantage of modern memory and storage technologies. It can also be integrated with other storage applications like NoSQL databases and object stores.
HSE is ideal when performance at scale matters, including very large data size, large key counts (billions), high operation concurrency (thousands), or deployment of multiple classes of media.
The platform is designed to be extensible to new interfaces and new storage devices, enabling use with a range of applications and solutions, including databases, IoT, 5G, AI, HPC, and object storage.
HSE is capable of delivering additional performance for SDS, such as Red Hat Ceph Storage and Scality RING, that enables cloud-native applications through containerized platforms like Red Hat OpenShift as well as tiered performance for file, block and object storage protocols for multiple use cases.
HSE is delivered as an embeddable key-value database; the company will maintain the code repository on GitHub.

Get started developing at www.github.com/hse-project or optimize MongoDB for flash and SCM with HSE.

Resources :
Blog: Micron Contributes Performance-Improving Software to Open Source Community. Why?
Video: Heterogeneous-Memory Storage Engine
Technical brief: Opening up Storage Performance in an Open Source World (PDF)

⁽¹⁾ Tests performed using HSE v1.7.0 vs RocksDB v6.6.4 using YCSB v0.17.0 and 4 Micron 9300 SSDs. Data represents the maximum performance from various YCSB workload tests.
⁽²⁾ Tests performed using HSE v1.7.0 vs RocksDB v6.6.4 using YCSB v0.17.0 and 4 Micron 5210 SSDs with and without an X100 SSD. Data represents the performance from YCSB Workload A tests.