Quobyte Distributed File System With TensorFlow Plug-In

Quobyte Inc. announced that tits Data Center File System is a distributed file system to offer a TensorFlow plug-in, providing increased throughput performance and linear scalability for ML-powered applications to enable faster training across larger data sets while achieving higher-accuracy results.

TensorFlow is an open source library for numerical computation and large-scale machine learning used across industries such as autonomous vehicles, robotics, financial services, healthcare, government, aerospace, defense. Using the company’s storage with TensorFlow helps to simplify and streamline the operation of machine learning.

The TensorFlow File System plug-in allows TensorFlow applications to talk directly to the company’s platform, bypassing the OS kernel to reduce kernel mode context switches and lower CPU usage. While Quobyte storage can be used with all stages of ML, the resulting increased GPU utilization from the TensorFlow plug-in speeds up model training of ML workflows.

The company provides users the flexibility to train anywhere and move models into production to better support ML workloads from the data center to the cloud to the edge. The TensorFlow plug-in can be used to train models locally on sample data sets and use the Google Cloud Platform for training at scale because the firm’s platform runs on-prem and in the cloud. Additionally, because it bypasses the kernel entirely, the plug-in works with both current and older versions of Linux, providing a range of flexible deployment options for use in ML. Using the TensorFlow-plug-in since there are no application modifications required.

“As more and more businesses look to leverage ML to increase innovation, achieve a faster time to market and provide a more positive customer experience, there is an increasing need for storage infrastructures that offer higher performance and increased flexibility that these workloads need,” said Frederic Van Haren, lead analyst, HPC and AI systems, Evaluator Group. “Vendors, like Quobyte, that offer high performance, broad platform support and flexibility of deployment options are well positioned to help companies handle bigger data sets, achieve more accurate results and run ML workloads in any environment.“

With the company, there is no need for specialized storage systems to get the most out of ML. Its platform is a single storage system that addresses many different performance profiles, including the high-throughput, low-latency requirement of ML’s model training stage, as well as large block sequential, small block random or mixed general workloads. It supports a broad set of access protocols and clients, such as S3, Linux, Hadoop, Windows and NFS for greater platform flexibility and more complete data ingest and preparation. Data is readily available at any stage all within a single global namespace and all managed through a management console.

Additional benefits of TensorFlow File System plug-in include:

• Ability to leverage HDD and SSD to get the best price-performance ratio without cumbersome tiering

• Prefetching of training data can deliver substantial performance improvement. Much machine-generated data uses a sequential naming convention that makes it for prefetching.

• Infinite scalability that allows users to grow storage in terms of throughput and capacity when they need it. As ML project requirements change – oftentimes more quickly than anticipated – the Quobyte installation will adapt. Disks or servers can be quickly and added when needed to provide more capacity or performance without interruption to applications or services.

• Multi-tenancy that provides additional security by allowing users to define isolated namespaces and physical separation of data/workloads inside the same cluster. Administrators can further isolate tenants by controlling to which physical hardware they have access in order to ensure performance and that data is not accessible to any unauthorized users on the network.

“By providing the first distributed file system with a TensorFlow plug-in, we are ensuring as much as a 30% faster throughput performance improvement for ML training workflows, helping companies better meet their business objectives through improved operational efficiency,” said Bjorn Kolbeck, CEO, Quobyte. “With the higher accuracy of results, scalability to handle bigger data sets and flexibility to run on-prem to the cloud, and edge, we believe we are providing an optimal experience that allows customers to fully leverage the value of their Machine Learning infrastructure investments.“

Read also:
Quobyte Data Center File System Available on Google Cloud Platform Marketplace     
Enabling Google Cloud users to configure hyperscale distributed storage platform in few clicks
December 13, 2018 | Press Release