Object Storage Dead, S3 Eats Everything

Is it a provocation? Think about that and reread carefully the title.

If you follow me for several years, you have probably read the article I published in The Register in 2016 about the genesis, roots and story of object storage but also this one on my own blog in 2015. If you didn’t have read them already, please do so now and come back here right after you do it.

First, above all, it’s fundamental to restate what users need: a scalable storage system, super reliable, offering multiple standard access methods at an attractive cost. All other aspects are non relevant at all. So the important words here are scalable, reliable, multiple standard access methods and cost.

We find easily 20+ products/vendors offering an object storage solution and this number is too big for the market they address meaning that several of them continues to be non profitable after 10+ years of business and several rounds of VC money.

We see companies with more than 7 investment rounds. If a market is hot, we should see more M&As and more rapid profitability.

I hope to not see a new VC round from these vendors as it will indicate again and again their survival mode before a bankrupt or a ‘negative’ sale…

This category illustrates a bad bet for investors, some of them told me the nightmare they live having realized the perspective of a bad exit.

Only Cleversafe did the perfect exit as Chris Gladwin understood before all others that his exit opportunity arrives once a life.

Also some vendors died even with famous leaders and investors and potentially good product ideas. In other words, especially from the back of the room, I mean ‘old’ object storage vendors, who is successful, give me numbers? Y/Y growth for the last 5 years? Tough and not very famous even with lots of promises several years ago.

Commercial vs. open source is a famous debate that shakes all established positions in all IT segments. You have name in mind  and for object storage it makes sense to select open source software when you need to consider high capacity system and commodity hardware versus a very controlled closed environment that hold last many years. Data survive to systems and companies for sure, so we understand why users finally pick standard stuff.

A key question related buying decisions? A big company buys from a big vendor, this is safe for both. A small company buys from anyone and is pretty proud to be ‘considered’ by a big vendor even with channel in the middle. But a big company buys only from a small vendor when it didn’t find any solution from big ones, when it finds an innovation the big one is unable to develop.

With several M&As, this attractiveness of the small player is gone. For object storage, differentiators are gone as all vendors sell same stuff from a user perspective. Think again about what they expect mentioned above.

In the past, we found vendors with different design, data consistency, data protection with replication and others with erasure coding, some offers with various interfaces, data services… but these few things disappeared with the arrival of SDS, commodity hardware and open source that finally burned margin these vendors can expect and hope.

Erasure coding for instance is a must have and became a standard. You find very good open source libraries or you use the Intel ISA-L product.

Imagine you don’t have it, you’re not considered as a serious player but if you have it, nothing changes, no additional consideration.

New NAS and file servers were back for a few years and some advantages claimed by object storage vendors suddenly disappeared as well. File interface is the default application mechanism when this application needs to write/read/store data, there is no need for integration as it works immediately and transparently in most cases.

SQL was back a few years ago. People thought it will disappear with a new paradigm coming and even Hadoop offered a SQL capability, students learned SQL and it is present now again everywhere in the new application stack with current IT trends.

For storage it was the same, people though NFS or NAS globally will be replaced by object interfaces and it was and is not the case at all.

Some vendors tried to offer dedicated access methods and some others chose very early S3 as the coming winner, this is the case for Cloudian and more recently Minio.

And again users don’t care about internal stuff, they ask about a standard interface – S3 in that case – that can be consumed by their applications. We see also a vendor who offers S3 on top of various clouds as each of them have their own interface, and finally in that example S3 becomes the ingest method for multi-clouds. OK, different S3s exist. We find better than other implementations but at the end of the day, all are S3 and the need is S3 nothing else.

Internal beauty is gone and most of the time introduces some clear constraints and restrictions completely orthogonal with the flexibility required and SDS promises. So the object storage battle is over, for quite some time to be honest, as S3 is the clear winner.

Some vendors forgot their past choices and tell the planet that they’re the preferred, de-facto… blablabla best S3 compatible solution. And as you can see everything is S3 with again new common data storage service more and more considered as an utility meaning erosion of prices and margins for vendors. Could you list who will be present in 2 or 5 years?

It reminds several vendors who claimed to do primary storage workloads with their object storage. Guess what, who continue to promote primary storage as use cases have shifted to secondary storage without any doubt in mind.

Just put NVMe drives in an object storage system and see what you obtain. A lot of disappointment for many products, just a few, very few, are doing well with flash and can continue to market some vertical use cases requiring flash, capacity, resiliency and S3.

The recent S3 SELECT is a perfect illustration of the combination of two winning standards – here S3 and SQL – as both of them are widely adopted and present on the market, perfect for some use cases with data processing like Spark, cloud data warehouse with Snowflake, ML engines with TensorFlow or H2O and even security.

Who is the current leader in object storage? Do we limit to pure or extend to unified storage vendors? Does it mean something as some vendors from HCI (Nutanix, etc.) to other storage categories (Datera, iXsystems, Qumulo, etc.) to Hadoop (MapR, etc.) to developers platform (Pivotal, etc.) have started to embed and offer a S3 service? What are the metrics to elect category leaders? Capacity, number of deployments, number of customers, number of downloads, number of integrations, number of partners, revenue, company size by employees, etc.? Two dimensions are more important than others: downloads and deployments. Another key aspect is workloads and use cases.

There is no object storage market, it’s a technology category driven by S3, capacity and ubiquitous access needs. Some analysts have made a mistake to merge and list file servers and object storage vendors in same studies.

The market is always right and tells the truth.