Lightbits: NVMe/TCP Storage for Kubernetes, Match Made in Heaven

By Carol Platz, VP marketing, Lightbits Labs Ltd.

As Kubernetes (K8) becomes more popular as the preferred container orchestration system for cloud-native applications, it’s important to implement a persistent storage platform that can deliver performance and scalability without compromising portability.

Managing persistent storage for K8 is challenging, with some solutions leading to tradeoffs. SDS, that is architected for NVMe over TCP (NVMe/TCP) protocol, is proving itself to be a solution that delivers performance, low latency, and scalability without sacrificing application portability.

Quick rundown on K8s
K8s is an open-source system for orchestrating software containers. Containers are analogous to VMs, but they are simpler and use fewer system resources. They are particularly suited to microservices architectures and cloud-native applications.

With K8s, one can deploy and scale software more easily than is possible with traditional VMs. Admins can separate K8s-based apps into parts and scale those parts independently. This enables greater efficiency and resource utilization. Further, K8s provides orchestrated portability for containers. It is relatively easy to move a K8s-based app between cloud platforms and operating systems. For these reasons, K8s is growing increasingly popular, with the technology accounting for 56% of container platforms [source: Statista, Kubernetes Statistics and Facts, February 7, 2022] running on the public cloud.

What are challenges in architecting storage for K8?
Not implementing the right storage platform can obstruct the benefits of K8s. For instance, using HDDs as stateful storage with K8s is not optimal. The performance is not good enough for apps like MySQL, Redis, Apache Spark, and MongoDB. Managing traditional HDDs in support of K8s also adds complexity to storage management. One is forever re-allocating storage as K8s-based apps scale up in pieces and migrate across platforms.

A further difficulty arises from the preference for local DAS in K8s environments. To hit desired application performance levels for MySQL, MongoDB, and others, it is necessary to provision storage with high IO/s and low latency levels. This capability, in turn, must be combined with data resiliency, especially when the storage must support hundreds of K8s clusters.

To meet these criteria, the vendors who make the applications recommend direct-attached flash storage. Here’s the problem with this approach: while direct-attached flash storage for K8s on bare-metal servers delivers the performance apps like MongoDB need, the use of persistent storage on the application server basically destroys K8s application portability. This nullifies one of K8s most attractive features. It also results in sub-optimal utilization of flash storage capacity.

What are characteristics of best storage for K8s?
There are alternatives to DAS for K8 storage. What should the solution look like? Low latency performance is a foundational requirement. The performance should be close to that of local DAS flash drives. While delivering this performance, the solution must also support K8s portability.

Operationally, the solution should reduce storage management challenges for K8s. It has to be relatively easy to allocate storage to ever-shifting K8s clusters. And, this must all happen with cost efficiency and good resource utilization.

The best approach is to deploy a cloud-native storage architecture. This should make some intuitive sense. Many, if not most applications running on K8s are cloud-native. A cloud-native storage solution offers the kind of flexibility that matches the portability and agility of K8s. The issue is performance. Storage performance in the cloud does not always match that of DAS in a bare metal deployment. This problem can be solved, however. More on that further down in the blog.

What are various options and protocols for K8s storage?
Alternatives to DAS for K8s are available from a variety of technologies. Some K8s persistent storage solutions use the Container Storage Interface (CSI), a standard that exposes arbitrary block and file storage systems to containerized workloads on K8s and other container orchestration systems. CSI makes the K8s volume layer extensible. It enables 3rd-party storage providers to write plugins for K8s without having to deal with the core K8s code. This simplifies the process of creating storage solutions for K8s.

Convenient as it may be for solution developers, CSI can cause problems for K8s system owners. For example, the iSCSI transport protocol works with the CSI driver. However, this approach tends to increase complexity and cut down on performance – while potentially locking the user into a specific vendor.

Another option is Container-Native Storage (CNS), a Kubernetes-native, SDS solution that runs in a container on each node of a Kubernetes cluster. They are similar to VMs but are more lightweight. CNS is persistent and can support mission-critical workloads. I won’t go into CNS in this blog, but you should know it exists.

Storage solutions for K8s typically connect storage to K8s using different protocols.

These include:

• Network File System (NFS) – a networking protocol designed for distributed file sharing. NFS for K8s defines how K8s-based apps will store and retrieve files from storage devices across networks. With NFS, admins can share some or all of a file system with the K8s apps, which are treated as remote users. K8s client apps that have authorization can access the system and mount NFS shares.

• ISCSI – a transport layer protocol that adapts the earlier SCSI to the Internet and determines how packets from a K8s app will be transported over a TCP/IP network. With iSCSI, it is possible to build a shared storage network that supports more than one K8s app, each of which can access centralized storage resources, but do so as if the resource was a locally connected device.

• Samba – a free open-source version of the SMB networking protocol. It runs on many different operating systems, but it is best known for Unix and Linux.

• NVMe/TCP – a technology that connects NVMe storage devices to K8s apps using TCP over TCP/IP. This approach is advantageous because it uses existing TCP/IP networks, which are ubiquitous. It also requires no special network cards or other hardware, which is the case with iSCSI. NVMe/TCP is standards-based, so there is no vendor lock-in. Indeed, all the protocols except NVMe/TCP require investments in infrastructure with proprietary hardware and drivers that tend to lock in vendors. Proprietary storage drivers are also tightly coupled with specific versions of the K8s kernel. Or, they need RDMA networking, a technology that does not always align well with K8s agility and portability.

Why is Lightbits NVMe/TCP storage good option for K8s?

It’s simple to support data-intensive, cloud-native applications without compromise. Lightbits Labs Ltd. offers an NVMe/TCP storage solution that makes it possible to enjoy the advantage of K8s portability without compromising on storage performance or scalability. The firm’s solution provides low-latency performance that is as close as possible to local flash or DAS. The NVMe/TCP SDS solution can support hundreds of K8s clusters from a single Lightbits storage cluster without inhibiting K8s container portability model.

Portability comes from the solution’s disaggregated design. The company’s solution uses CSI to allow K8s pods to move between servers on the network. If a K8s pod moves to a new physical server, the same Lightbits persistent volume will be attached to that new machine and remain available to the pod. Additionally, in contrast to other K8s storage solutions, a single Lightbits cluster can support multiple container orchestration platforms, including Red Hat OpenShift Container Platform (OCP), VMware Tanzu, OpenStack, and VMware vSphere, as well as bare-metal applications.

The solution comprises a integration between Lightbits, optimized for performance hardware platforms, and Intel hardware. These include Xeon Scalable processors with Intel DL Boost and 100GbE Intel Ethernet 800 Series network adapters with Application Device Queues (ADQ) technology.

By utilizing low-cost QLC NAND SSDs, The company’s solution runs cost-effectively. It further keeps management simple and economical by utilizing standard NVMe/TCP drivers that are part of all major Linux distributions. The solution’s ‘Intelligent Flash Management’ capabilities improve flash endurance, which extends the lifespan of the investment. Data-reduction features improve efficiency. The disaggregated architecture leads to improved resource utilization.

In architectural terms, the firm’s storage solution for K8s features a control plane that is designed to function in an asynchronous mode – handling thousands of requests at any given moment. This approach is suited to large-scale K8s clusters that must handle Persistent Volume Claims (PVCs) that are created, modified, or deleted by the thousands every day.

The metrics reveal the solution’s capabilities. The company’s storage solution for K8s delivers up to 4x higher throughput for 4K reads, along with up to 17x more throughput for 4K writes and up to 14x more throughput for 8K read/write workloads. For 32K read/write workloads the solution provides up to 5x more throughput. The solution also scales efficiently, which is critical for success with K8s.

Conclusion
Storage has the potential to be a drag on K8s potential value. Local flash or DAS may offer the best performance for K8s, but it invariably leads to poor storage resource utilization and high costs. DAS also threatens to impede K8s portability, which is one of the technology’s most compelling benefits. Alternatives to DAS are available, however. The most effective is NVMe/TCP in the company’s SDS solution, that offers the levels of performance required by popular apps running on K8s, while preserving K8s portability and agility.

Resources:
Boost Performance for Stateful Apps Deployed in Kubernetes   
Supporting and Enabling Modern Applications with Lightbits    
Migrating Traditional Virtualized Apps To Kubernetes? You Can Consolidate Your Storage Too