Backblaze HDD Failure Rates for 3Q16

This study was published on the blog of Backblaze, Inc. on November 15, 2016, and written by Andy Klein.

HDD Stats for Q3 2016: Less is More

In our last report for Q2 2016, we counted 68,813 spinning HDDs in operation. For Q3 2016 we have 67,642 drives, or 1,171 fewer HDDs.

Stop, put down that Twitter account, Backblaze is not shrinking. In fact, we’re growing very nicely and are approaching 300PB of data under our management. We have fewer drives because over the last quarter we swapped out more than 3,500 2TB HGST and WDC HDDs for 2,400 8TB Seagate drives. So we have fewer drives, but more data. Lots more data.

We’ll get into the specifics a little later on, but first, let’s take a look at our Q3 2016 drive stats.

Backblaze HDD reliability stats for Q3 2016
Below is the HDD failure data for Q3 2016. This chart is just for the period of Q3 2016. The HDD models listed below are data drives, not boot drives. We only list drive models that have 45 or more of that model deployed.

Backblaze HDD Failure Rate for 3Q16
(reporting period July 1, 2016 – September 30, 2016)

A couple of comments on the chart:
    •    The models that have an annualized failure rate of 0.00% had zero HDD failures in Q3 2016.
    •    The ‘annualized failure rate’ is computed as follows: ((Failures)/(Drive Days/365))*100. Therefore, consider the number of ‘Failures’ and ‘Drive Days’ before reaching any conclusions about the failure rate.

Less is more: the move to 8TB drives
In our Q2 2016 drive stats post we covered the beginning of our process to migrate the data on our aging 2TB HDDs to new 8TB HDDs. At the end of Q2, the migration was still in process. All of the 2TB drives were still in operation, along with 2,720 of the new 8TB drives – the migration target. In early Q3, that stage of the migration project was completed and the ’empty’ 2TB HDDs were removed from service.

We then kicked off a second wave of migrations. This wave was smaller but continued the process of moving data from the remaining 2TB HDDs to the 8TB-based systems. As each migration finished we decommissioned the 2TB drives and they stopped reporting daily drive stats. By the end of Q3, we had only 180 of the 2TB drives left – four Storage Pods with 45 drives each.

The following table summarizes the shift over the 2nd and 3rd quarters.

Migration from 2TB to 8TB HDDs

As you can see, during Q3 we ‘lost’ over 1,100 HDDs from Q2, but we gained about 12PB of storage. Over the entire migration project (Q2 and Q3) we added about 900 total drives while gaining 32PB of storage.

Drive migration and HDD failure rates
A four-fold storage density increase takes care of much of the math in justifying the migration project. Even after factoring drive cost, migration costs, drive recycling, electricity, and all the other incidentals, the migration still made economic sense. The only wildcard was the failure rates of the HDDs in question. Why? The 2TB HGST drives had performed very well. Drive failure is to be expected, but our costs go up if the new drives fail at twice or three times the rate of the 2TB drives. With that in mind let’s take a look at the failure rates of the drives involved in the migration project.

Failure Rates of Drives in Migration Project

The Seagate 8TB drives are doing very well. Their annualized failure rate compares favorably to the HGST 2TB HDDs. With the average age of the HGST drives being 66 months, their failure rate was likely to rise, simply because of normal wear and tear. The average age of the Seagate 8TB HDDs is just 3 months, but their 1.6% failure rate during the first few months bodes well for a continued low failure rate going forward.

What about the 60 drive Storage Pods?
In Q3 we deployed 2,400 8TB drives into two Backblaze Vaults. We used 60 drive Storage Pods in each vault. In other words, each Backblaze Vault had 1,200 HDDs and each HDD was 8TB. That’s 9.6PB of storage in one Backblaze Vault.

As a reminder, each Backblaze Vault consists of 20 Storage Pods logically grouped together to act as one storage system. Storage Pods are spread out across a data center in different cabinets, on different circuits and on different network switches to maximize data durability. Backblaze Vaults are the backbone that powers both our cloud backup and B2 cloud storage services.

Our Q3 switch to 60-drive Storage Pods signals the end of the line for our 45-drive systems. They’ve had a good long run. We put together the history of our Storage Pods for anyone who is interested. Over the next couple of years, all of our 45 drive Storage Pods will be replaced by 60 drive systems. Most likely this will be done as we migrate from 3TB and 4TB drives to larger HDDs. I hear 10TB HAMR drives are just around the corner, although we might have to wait for the price to drop a bit.

Cumulative HDD failure rates by model
Regardless of the drive size or the Storage Pod used, we’ll continue to track and publish our data on our HDD test data web page. If you’re not into wading through several million rows of HDD data, the table below shows the annualized drive failure rate over the lifetime of each of the data drive models we currently have under management. This is based on data from April 2013 through September 2016 for all data drive models with active drives as of September 30, 2016.

Annualized Hard Drive Failure Rates
(Cumulative from April 10, 2013 thru September 30, 2016 – ordered by drive size)

Recap
Less is more. The migrations are finished for the moment, although we are evaluating the migration from 3TB drives to 10TB drives. First though, we’d like to give our data center team a chance to catch their breath. The early returns on the Seagate 8TB drives look good. The 1.6% failure rate at the 3-month point is the best we’ve seen from any Seagate drive we’ve used at the same average age. We’ll continue to track this going forward.
Next time we’ll cover our Q4 drive stats, along with a recap of the lifetime performance of every data drive we’ve used past and present. That should be fun.

Looking for the tables, charts, and images from this post? You can download them from Backblaze B2 as a ZIP file (2.3MB).