IDC MarketScape: WW Object-Based Storage 2018

This report was written by Amita Potnis, research manager, IDC Corp.

IDC MarketScape Worldwide Object-Based Storage Vendor Assessment

(Source: IDC, 2018)

IDC opinion
In this report, we stated that the problems of storing and managing unstructured data are not going away. This remains true in 2018.

According to File- and Object-Based Storage Survey Findings, 2017: Adoption and Workloads Trends, Part 1 (IDC #US43630018, March 2018), more than 50% of the 450 respondents adopted object-based storage (OBS) because existing NAS was too complex to manage. More than 40% of the respondents also indicated that scalability was a major concern for their organizations given the ever-growing data sets. Organizations large and small are now aware of the advantages of OBS and the flexibility of offerings of this technology.

As adoption of OBS continues to increase, much of the feature functions are now standard across products. Examples of such features are S3 compatibility, erasure coding, replication, and custom metadata. IDC believes that the OBS market segment has reached a maturity point, and going forward the strategic vision in terms of partnerships with public cloud providers, file system capabilities, a portfolio of diverse solutions that support a variety of use cases across verticals, and so forth will play a strategic role in market penetration. Based on several customer interviews, IDC expects OBS will take a more prominent role in production workloads and this is where the OBS suppliers will see growth.

This IDC MarketScape assesses the present commercial OBS supplier (suppliers that deliver software-defined OBS solutions as software or appliances much like other storage platforms) landscape. Cloud-based storage services based on OBS are not included in this IDC MarketScape. Since the publication of the last OBS IDC MarketScape, the landscape has changed dramatically – a fragmented OBS market has now somewhat consolidated. As demand for traditional external storage systems continues to decline, several mainstream storage suppliers enhanced their portfolio to include OBS solutions by developing them in-house or via acquisitions. This IDC MarketScape assesses 13 OBS suppliers that are owners of IP.

IDC believes that going forward, the longevity and success of any OBS supplier lie in its strategy and roadmap. Today, there is a definitive shift to compete together as opposed to commercial OBS suppliers competing against public cloud. In recent months, several OBS suppliers (listed in this IDC MarketScape) have partnered with public cloud storage service providers to support end users’ multicloud initiatives as well as a wider range of workloads. Therefore, this IDC MarketScape gives additional weight to strategy than capability. In earlier IDC MarketScapes, strategy and capability criteria carried equal weightage.

When considering an OBS solution, IDC recommends that end users look for the following capabilities and business strategies of OBS suppliers that are representative of the market:

Security: What data would move to the OBS platform, and what kind of security services does the potential product support (encryption, user authentication, protection against ransomware, unforeseen events, etc.)?
Multicloud strategy initiatives: Does the commercial OBS solution support your organization’s multicloud initiative by supporting data tiering to public cloud? Is data moved in native format supported by the public cloud, or does it need to be accessed through the OBS solution’s proprietary interface?
Performance: Will the OBS solution support higher performance needs for specific production workloads? More specifically, does it support file access protocols (NFS and SMB) natively via a file system?
Information life-cycle management (ILM): Does the OBS solution support a robust set of ILM policies not just for on-premises deployments but also in the public cloud/tape or optical media?
Cost: Does the OBS solution lend itself to lower capex or opex, and will it sustain cost savings in a 5- to 10-year period?

IDC MarketScape vendor inclusion criteria
This study assesses the capabilities and business strategies of key suppliers in the (scale-out) OBS market segment – which is part of the overall file- and object-based storage market. This evaluation is based on a comprehensive framework and a set of parameters that gauge the success of a supplier in delivering an OBS solution in the market. This study includes analysis of 13 notable players in the commercial OBS market, with broader portfolios and global scale.

The suppliers enlisted in this study are (in alphabetical order) Caringo, Cloudian Inc., Cohesity, Dell EMC, Hitachi Data Systems (HDS), IBM, NetApp, Red Hat, Scality, StorageCraft, SUSE, SwiftStack, and Western Digital (WD).

To make this list, the suppliers need to have an OBS platform that:

Conforms to IDC’s taxonomy on OBS platforms. According to IDC’s taxonomy, software-based OBS platforms can run on any commodity x86 platforms and do not have any specific hardware customizations (like custom ASICs or SoCs) mated to the software stack, and they leverage an OBS data organization scheme.
Has been developed in-house or owned by way of an acquisition. In other words, the supplier needs to be the IP owner of that platform.
Is delivered as software, hardware (appliance or gateway), and/or as (private or public) cloud based. Additional points were granted to solutions supporting data tiering to and from public clouds through a simple intuitive UI as well as those that supported search and data analytics reporting capabilities.
Is sold as licensed software directly to buyers or indirectly via OEM/channel partners and not just as a service. Additional points were granted if the supplier had partnerships with as-a- service providers to deliver it as a cloud offering.
Was generally available as a current offering at the time IDC undertook this study in early 2018 with revenue of $20 million+.

This study is designed to evaluate each supplier for its OBS offering as opposed to the breadth of products and services of the firm. In other words, it should be observed that this study evaluates each participating supplier as an entity within the OBS market.

In addition, some suppliers did not make the list because they did not meet one or more of the selection criteria. The supplier that gets an honorable mention in this study is DataDirect Networks. Other suppliers in adjunct markets such as data protection are also mentioned.

Advice for technology buyers
As end users move toward a hybrid cloud/multicloud storage strategy to support the needs of their ever-increasing data sets while supporting traditional and next-generation workloads, IDC urges you to think about the following when choosing OBS solutions:

Automated information life-cycle management (ILM): Policy-driven ILM capabilities that enable management, repair, and deletion of data will help increase efficiency when managing petabyte- or exabyte-scale data sets.

Solutions portfolio: A strong technology (hardware and software) partnership portfolio lends itself to a given OBS offering being able to support many use cases across verticals. An OBS supplier with several ISV and server-hardware partnerships provides end users the ease of procuring and deploying solutions in a quick and efficient manner.

Data access: A flexible OBS solution will allow native data access capability for files and objects where data can be stored or accessed as file or objects or native public cloud format.

Data tiering: Data can be tiered to multiple public cloud storage services while being managed through a single pane of glass giving customers choice and ease of management to use the right platforms for their workloads.

Ability to support production workloads: OBS solutions are slowly but surely being deployed for production use cases. As customers look at OBS solutions, it is imperative that they consider solutions that can support the performance, bandwidth, scale, and other requirements of production workloads.

Scality
The company is positioned as a Leader in this IDC MarketScape for object-based storage. It is a privately-held, long standing, provider of software-only OBS called the RING. In addition to financial backing from venture capital firms (most recently in April 2018), Scality has a strong engineering and partner agreement with HPE that allows RING to be listed on the HPE bill of materials. This has given Scality access to a wide net of HPE’s channel thus seeing an increase in sales, as well as a resell arrangement with Cisco for the RING on UCS Servers.

The RING is designed to scale out from three servers to hundreds of servers with hundreds of petabyte-scale environments managed by a single namespace, with the choice of capacity or performance-optimized nodes, varied data durability levels, and support for large as well as small objects. It is optimized such that multiple applications can be deployed while using different storage policies. It also offers native scale-out file system capability. The company claims that half the business comes from scale-out file system capability that customers adopt to either augment or replace existing NAS appliances and the other half from object (S3 or REST)-based applications. IDC’s research supports this claim that customers are looking for an OBS solution with native file system capabilities when replacing their NAS systems that allows them to run their traditional file-based applications on OBS without having to rewrite them.

Scality is expanding its strategy to include newer products, and in July 2017, the company brought to market a multicloud data management controller called Zenko. This new product allows users to mobilize data from origin storage locations to public cloud storage targets of choice. Customers claim that the RING and Zenko provide a multicloud solution for advanced workflow integration to all AWS, Microsoft Azure, and Google Cloud public clouds (and others including the Wasabi Cloud and Digital Ocean), by supporting one-to-many replication (via the S3 cross region replication API) from RING (or a local cache) to all three clouds simultaneously. Eventually, Zenko will support movement of data to third-party S3-enabled private cloud and NAS offerings as well as tape.

The S3 Connector is another product in Scality’s product portfolio that provides scalable, highly available, and reliable storage services (S3 API compatibility, enterprise security, high-performance buckets, etc.) for data generated by cloud applications. The firm caters to a variety of customers across verticals such as financial services, government, healthcare, media and entertainment, and life science. Overall, Scality’s uniqueness comes from bringing products to market that cater to the market requirements and developing a portfolio that addresses the needs of today and the future.

Strengths
Scality’s strength is in its brand recognition in the market, go-to-market strategy, and several successful petabyte scale deployments. Strength also comes from the ability to recognize and execute on new found opportunities. For example, Scality brought to market scale-out file system, Zenko, and S3 Connector to address specific needs of the market that can potentially boost the company’s business.

Challenges
Scality’s challenge comes from an already saturated OBS competitive landscape that also includes traditional storage vendors and newer ISVs. The public cloud continues to be a challenge along with rapid innovation within the competitive landscape. Scality will have to differentiate itself via cloud-centric innovation while leveraging technical and channel partnerships to deepen market share.

Market Definition

According to Worldwide File- and Object-Based Storage Forecast, 2016-2020 (IDC #US41685816, September 2016), object-based storage capacity is expected to grow at a CAGR of 30.7% for 2016-2020, reaching 293.7EB in 2020.

IDC classifies OBS platforms as part of the scale-out file- and OBS (FOBS) market segment. It uses the classification scheme to classify newer software-based file- and object-based storage platforms.

Scale-out FOBS refers to FOBS solutions that use a distributed data placement mechanism to span multiple independent server hosts or controllers while presenting a single data access namespace. Such architectures are also called shared nothing (or shared data) architectures. Such architectures allow for flexible scalability in performance and capacity independent of each other using commodity components. Data sharing and distribution mechanisms (such as local and geographic replication and local and distributed erasure coding) account for one or more concurrent component failures. Scale-out FOBS solutions are made up of two variants: scale-out FBS solutions and scale-out OBS solutions. There are two principal differences between the two types: how data is organized, and how data is accessed.

Scale-out FBS solutions use distributed file systems with hierarchical structures to organize and store data. These structures are akin to mechanisms used by monolithic file systems, which in most cases follow a root directory (folder) and inverted tree structure. In contrast, scale-out OBS solutions use flat structures to organize data. Such structures are higher-level structures in which data is often organized using an ‘account, container, object’ approach wherein ‘objects”‘are analogous to ‘files’ in FBS solutions. Accounts, containers, and objects are referenced by a metadata repository that stores and manages attributes of data stored in that structure. The level at which OBS solutions operate varies from platform to platform. Many OBS solutions operate on a per-object level (i.e., allow each object to be treated independently as far as policy management is concerned), whereas others operate at a container or account level (i.e., only allow policies to be applied at a container or account level). Several OBS solutions also leverage NoSQL databases as metadata repositories and persistent data stores (instead of storing chunks in the file systems).

Because of the need to manage objects with a comprehensive set of attributes, most OBS solutions use a different set of data interfaces than their FBS counterparts that mostly leverage NFS, SMB (CIFS), or FTP protocols. It is common for many OBS solutions to support HTTP/REST, CDMI, Amazon S3, and other object-specific interfaces.

IDC’s File- and Object-Based Storage User Survey Findings
In 2017, IDC conducted a survey of 450 concurrent users of file and object-based storage in North America. The study, called IDC’s File- and Object-Based Storage Survey Findings, reported:

52.9% of 450 North America-based respondents started using OBS in the past two years.
41% of the respondents also indicated that they are currently using either an OBS appliance or a commercial software-only offering, while 42% are using public cloud or managed private cloud services.
Over 26% of 254 respondents plan to integrate flash into their OBS environment, while 56.4% have already done so. The primary reasons to integrate flash are higher storage density, bandwidth (to move large data sets faster), and performance (throughput). Flash is integrated into OBS to support data analytics of unstructured data, content applications, business applications, and others.
48% of 281 respondents indicate that they use OBS in the public cloud to support a new initiative, while 42.7% indicate (lower) cost as a secondary reason to adopt OBS in the public cloud. Better data security and data protection are other top reasons to adopt public cloud.
422 respondents deployed OBS either as traditional on-/off-premises or as private cloud storage. 37.7% of users deploying traditional on-/off-premises OBS and 34.6% of users deploying private cloud indicate the reason for this choice of platform is because of sensitivity of data (cannot be pushed into public cloud).

The research also indicates that the actual benefits of deploying OBS in terms of avoiding vendor lock- in and flexibility of platform (private/public cloud, appliance, software only, open source software, etc.) are greater than perceived benefits.

Backup, DR, archive, and active archive are still dominant use cases for OBS. Newer use case such as data analytics for unstructured data, media streaming, and web serving are deployed on off-premises OBS and are expected to be revenue-generating workloads in the future.