First NVMe SSDs Announced – What’s NVMe?

This article was written
by Dennis Martin, president, Demartek, LLC.

Demartek Comments on IDF2013 and NVMe

I recently attended the Intel Developer Forum (IDF) 2013 in San
Francisco, CA, in order to follow-up on the progress of NVM Express
(NVMe). While there, I attended an NVMe technical session and spent a
large amount of time in the NVMe Pavilion discussing NVMe with most of
the vendors there. NVMe has made good progress since last year (last year’s comments), with the first NVMe SSD announced and a robust ecosystem developing to support this technology.

NVMe – What Is It?
NVMe is a scalable host controller interface designed for enterprise and
client systems that use PCIe SSDs. NVMe is being developed by an open
industry consortium of 90+ members and is directed by a 13 company
Promoter Group including Cisco, Dell, EMC, Intel, LSI, Micron, NetApp,
Oracle, PMC-Sierra, Samsung, SanDisk, Seagate and sTec (now a part of
Western Digital). The original NVMe 1.0 specification was published in
March 2011. The 1.0c specification was published in February 2012.
Version 1.1 of the NVMe specification was published in October of 2012.
The NVMe workgroup is continuing to develop the specification and is
expecting the next revision to be completed approximately mid-year 2014.

   (Source: NVMe)

PCIe
As we have been explaining for some time, acceptance of SSDs is growing
in enterprise datacenters. Although there are several good interfaces
available for SSD technology, PCIe in particular provides very good
bandwidth and very low latency characteristics, with the ability to
support the next generation of non-volatile memory.

The current generation of PCIe (3.0) provides 1GB/s of bandwidth per
lane, and many PCIe devices in this category use four or eight lanes of
PCIe, which means that 4GB/s or 8GB/s is achievable with these storage
devices.

The current generation of servers using PCIe 3.0 can support up to 40
lanes of PCIe per processor, meaning that many of today’s two-socket
servers support 80 lanes of PCIe. Many of these servers have several
PCIe slots that can support more than one of these PCIe SSDs, along with
other types of adapters.

Hardware and Software Latency
The designers of NVMe are anticipating some interesting changes with
respect to performance of the entire hardware and software
infrastructure. With today’s technology, an I/O request spends the
majority of its time working through the hardware infrastructure,
including the NAND flash media, the flash controller, the adapter and
related hardware. A small portion of the time is spent working through
the host software I/O stack.

Future non-volatile memory technologies, some of which are beginning to
emerge, have the potential of performing much faster than today’s NAND
flash technology. When this occurs, using current software stacks and
protocols, an I/O request will spend the majority of its time working
through the software stack, rather than through the hardware
infrastructure.

NVMe is designed to streamline the I/O stack in order to reduce the
software stack latency. Intel compared the Linux NVMe software stack to
the Linux SCSI/SAS software stack. They found that NVMe reduced the
software overhead by more than 50% in their tests, from 6.0µs to 2.8µs.
The NVMe stack also reduced the number of instruction cycles from 19,500
to 9,100.

Because NVMe is designed for non-volatile memory technologies, it is
designed with support for many more queues than other protocols that
were designed for slower storage technologies. NVMe supports 64K
commands per queue and 64K queues so that it can fully support very fast
hardware.

NVMe is designed to work with non-volatile memory technology that
includes today’s NAND flash technology and future non-volatile memory
technologies. NVMe also currently assumes that the storage is connected
to the PCIe bus. However, I have seen at least two other technologies,
NVDIMM and TeraDIMM (also known as ULLtraDIMM), that put flash memory on
the memory bus and can present itself as storage. These could
potentially take advantage of the NVMe stack in the future.

Enterprise vs. Client Devices
Just as with HDD drive technology, there are two basic classes of SSD
technology: enterprise and client, each having a different purpose and
having different operating characteristics. We explain the technical
differences between enterprise and client devices in our SSD Deployment
Guide. NVMe provides optional features that are designed to support both
enterprise SSDs and client SSDs.

NVMe 1.1 Specification Features
The NVMe 1.1 specification includes features that support end-to-end
data protection, multi-path support including reservations, power
optimizations and enhancements to the NVMe command set. The multi-path
support, used primarily in enterprise devices, supports dual-port
devices and supports devices behind a PCIe switch. The reservations
feature supports multi-host environments and is compatible with
implementations that use SCSI reservations.

The NVMe 1.1 specification includes power optimization features that are
directed towards client SSDs. The Autonomous Power State Transition can
direct the NVMe controller to automatically transition to a lower power
state after a certain idle period.

Future revisions to the NVMe specification are expected to support
namespace management, live firmware updates and other features.

NVMe Drivers
Driver development has progressed nicely over the last year, and there
continues to be a great deal of activity in this area. NVMe device
drivers include:

• Windows: ‘in-box’ for Windows 8.1 and Windows Server 2012 R2 (StorNVMe)
• Linux: native O.S. driver since kernel 3.3 (January 2012)
• Unix: FreeBSD driver upstream and ready for release
• Solaris: planned for version S12
• VMware: vmklinux driver expected by December 2013
• UEFI: open source driver available on SourceForge

NVMe Plugfests
The first NVMe plugfest was held in May 2013 with 11 companies
participating, and another plugfest is being planned for 2014. Companies
with products passing the Compliance Tests and Interoperability
procedures can choose to list their products on the NVMe Integrators List.

SNIA NVM Programming TWG
The Storage Networking Industry Association (SNIA) has a Non Volatile
Memory (NVM) Programming Technical Work Group (TWG) that is working on a
common set of software interfaces that provide access to non-volatile
memory. The TWG is not limiting its work to NVMe, but exposes some of the features of NVMe to applications through attributes. This TWG has a working draft of the NVM Programming Model
available on the SNIA website.

Samsung
Samsung announced their first NVMe SSD earlier this year, claiming
faster sequential read speed and random read speed than their other
enterprise SSDs.

Microsoft
Microsoft plainly stated that they are committed to NVMe, and presented
some impressive performance gains using NVMe compared to SAS and SATA
interface devices. They also discussed their roadmap for features
planned for both enterprise and client NVMe devices.

Microsoft showed data that produced a more than 4x improvement in IO/s
using an NVMe device with their driver compared to SAS and SATA devices.
They also showed reduction in end-to-end latency by more than 8x in the
same tests.

EMC
I spent some time talking with EMC in their booth about some interesting
things that they are doing with NVMe. In addition to the obvious use
case of flash-based SSDs, it turns out that NVMe can support other
applications such as non-volatile RAM. EMC took existing NVMe components
including the FreeBSD driver, utility and a standard reference
controller to create a faster NVRAM software and controller package that
can become a component of several of their storage systems.

NVMe Protocol and Electrical Tools
I also spent time with several other companies at their booths
discussing NVMe. Two companies, Agilent Technologies and Teledyne LeCroy
were demonstrating NVMe protocol analyzers, while one company,
Tektronix, was demonstrating NVMe electrical validation tools, all of
which are available today.

NVMe Technology Demonstrations
I also saw technology demonstrations, or prototypes, of NVMe SSDs and
NVMe infrastructure from Dell, Intel, LSI, SanDisk, sTec and Western
Digital. These included NVMe controllers, PCIe card form factor SSDs and
drive form factor SSDs. All of the prototype SSDs I saw were enterprise NVMe SSDs, but LSI was showing their NVMe controller that supports both enterprise and client SSDs. I
was told that some of these products are getting close to shipping,
some possibly by the end of this calendar year or in the first half of
2014. Discussions with the other companies who were present all lead me
to believe that we will see many NVMe products announced throughout
2014. From what I was able to observe and learn, the first batch of NVMe
products will be primarily enterprise products, with client products
following a few months later.

The UEFI 2.4 specification contains updates for NVMe, and full BIOS support of NVMe devices is expected to appear in products beginning in 2014. In addition, a native VMware vSphere driver is expected in 2014.

The companies with booths in the IDF 2013 NVMe Pavilion included Agilent
Technologies, Dell, EMC, Intel, LSI, Micron, PMC-Sierra, SanDisk, SNIA,
sTec, Tektronix, Teledyne LeCroy, Viking Technology and Western
Digital.

Demartek NVMe-Related Publications
We plan to include an NVMe section in the 2013 edition of our SSD
Deployment Guide, and we expect to provide quite a bit more information
on NVMe products in 2014. There is additional NVMe technical information
on the Demartek Storage Interface Comparison page.

NVMExpress.org
The NVMe website has news, specifications, drivers, whitepapers and more