Non-Volatile DIMM as Storage, In-Memory Database

In a test of in-memory database system (IMDS) speed and recoverability using the emerging Non-Volatile DIMM (NVDIMM) technology, McObject LLC and AgigA Tech, Inc., a subsidiary of Cypress Semiconductor Corp., announced McObject has benchmarked its eXtremeDB Financial Edition IMDS using AgigA Tech’s AGIGARAM NVDIMM as main memory storage.

The tests included ‘pulling the plug’ mid-execution, which confirmed the AGIGARAM product’s ability to save data persistently in the event of system failure, and to facilitate recovery.

In addition, eXtremeDB Financial Edition’s speed managing data in AgigA Tech’s NVDIMM proved equal to using conventional memory (DRAM). In contrast, alternative IMDS mechanisms that safeguard data from system failure, such as transaction logging, introduce latency. By delivering peak performance along with full data durability, the combination of the eXtremeDB Financial Edition in-memory database system and the AGIGARAM NVDIMM opens the door to a new and powerful approach to managing data in demanding capital markets applications such as algorithmic trading, order matching, in-memory analytics and more.

McObject has published the results of its benchmark tests using eXtremeDB Financial Edition and AgigA Tech’s NVDIMM in a report available for download.

In-memory database systems store records in main memory, leveraging DRAM’s superior R/W speed while eliminating the file I/O, cache management, data transfer and other overhead inherent in traditional database management systems (DBMSs) that incorporate persistent storage. As a result, IMDSs perform an order of magnitude faster than their disk-based counterparts.

But some applications require a higher level of data durability than DRAM can provide. As a solution, IMDS vendors offer transaction logging, which keeps a record of changes to the database that can be used to recover after a crash. But this logging requires writes to persistent storage (such as SSD or HDD), which adds some latency. IMDSs with transaction logging still outperform traditional on-disk DBMSs (by up to 23x, depending on storage devices used, according to McObject’s earlier published findings) but they lag ‘pure’ in-memory database systems deployed in DRAM.

One existing type of non-volatile RAM uses a battery to maintain data in memory during a power failure. However, disadvantages of this battery-backed RAM include restrictive temperature requirements, leakage risk, limited storage time, long re-charge cycles, finite battery shelf life, and overall high cost-of-ownership.

AgigA Tech was formed to develop technology that would overcome battery-backed RAM’s limitations. Its battery-free AGIGARAM NVDIMM combines standard DRAM with NAND flash and an ultracapacitor power source. When a system operates normally, AGIGARAM provides all the capabilities of high speed DRAM. But in the event of power loss, the ultracapacitor provides a burst of electricity used to write main memory contents to the NAND flash chip, where it can be held virtually indefinitely, with the AGIGARAM NVDIMM restoring data from NAND flash to DRAM upon system recovery.

McObject tested eXtremeDB Financial Edition using AgigA Tech’s 4GB AGIGARAM DDR3 NVDIMM for main memory storage, on an Intel Oak Creek Canyon reference motherboard equipped with a Pentium Dual Core CPU 1407 @ 2.8GHz processor, as well as 8GB of conventional memory. The benchmark application measured speed in separate tests of database inserts, updates, deletes, index searches and table traversals. In each test, the AGIGARAM NVDIMM delivered speed comparable to the results obtained using conventional memory.

To test response to failure, the system was re-booted during execution. Upon re-start, the test application launched automatically, accessed the database in its pre-failure state (upon recovery, the AGIGARAM NVDIMM had loaded it from flash into its DRAM), checked for database consistency and resumed operation.

Test application code enabling database recovery leveraged an eXtremeDB feature that supports the IMDS’s use with battery-backed RAM. It enables a process to re-connect to an NVRAM-hosted eXtremeDB Financial Edition database, initiate any needed cleanup, and resume normal operation after a system reboots. An application’s recovery algorithm assumes that the memory block of the database memory device assigned as MCO_MEMORY_ASSIGN_DATABASE can be re-used after a crash or power failure by re-opening it as an existing database.

“The idea that there must be a tradeoff between performance and persistence/durability has become so ingrained in the database field that it is rarely questioned. McObject’s test of the eXtremeDB Financial Edition in-memory database system with AgigA Tech’s AGIGARAM NVDIMM shows that mission critical applications needn’t accept latency as the price for recoverability. Developers working in a variety of capital markets application categories will view this as a breakthrough, and we look forward to our customers enjoying the benefits of the joint eXtremeDB Financial Edition/AGIGARAM NVDIMM solution,” Steve Graves, McObject CEO, said.

“An in-memory database system is perhaps the ideal application to take advantage of the features of NVDIMMs,” said Ron Sartore, CEO, AgigA Tech. “Our AGIGARAM NVDIMMs provide the speed and endurance of DRAM with the persistence of flash. McObject’s impressive benchmark reveals the value of this combination.”