Backblaze Stats on 150,757 HDDs

This report, published on October 20, 2020, was written by
Andy Klein, director of compliance, on the blog of Backblaze, Inc.

Backblaze Hard Drive Stats 3Q20

As of September 30, 2020, Backblaze had 153,727 spinning HDDs in our cloud storage ecosystem spread across four data centers. Of that number, there were 2,780 boot drives and 150,947 data drives. This review looks at the 3Q20 and lifetime HDD failure rates of the data drive models currently in operation in our data centers and provides a handful of insights and observations along the way.

Quarterly HDD Failure Stats for 3Q20
At the end of 3Q20, Backblaze was using 150,974 HDDs to store customer data. For our evaluation we remove from consideration those drive models for which we did not have at least 60 drives (more on that later). This leaves us with 150,757 HDDs in our review. The table below covers what happened in 3Q20.

Observations on the 3Q20 Stats
There are several models with zero drive failures in the quarter. That’s great, but when we dig in a little we get different stories for each of the drives.
• The 18TB Seagate model (ST18000NM000J) has 300 drive days and they’ve been in service for about 12 days. There were no out of the box failures which is a good start, but that’s all you can say.
• The 16TB Seagate model (ST16000NM001G) has 5,428 drive days which is low, but they’ve been around for nearly 10 months on average. Still, I wouldn’t try to draw any conclusions yet, but a quarter or two more like this and we might have something to say.
• The 4TB Toshiba model (MD04ABA400V) has only 9,108 drive days, but they have been putting up zeros for 7 quarters straight. That has to count for something.
• The 14TB Seagate model (ST14000NM001G) has 21,120 drive days with 2,400 drives, but they have only been operational for less than 1 month. Next quarter will give us a better picture.
• The 4TB HGST (model: HMS5C4040ALE640) has 274,923 drive days with no failures this quarter. Everything else is awesome, but hold on before you run out and buy one. Why? You’re probably not going to get a new one and if you do, it will really be at least 3 years old, as HGST/WDC hasn’t made these drives in at least that long. If someone from HGST/WDC can confirm or deny that for us in the comments that would be great. There are stories dating back to 2016 where folks tried to order this drive and got a refurbished drive instead. If you want to give a refurbished drive a try, that’s fine, but that’s not what our numbers are based on.

The 3Q20 annualized failure rate (AFR) of 0.89% is slightly higher than last quarter at 0.81%, but lower than the 2.07% from a year ago. Even with the lower drive failure rates, our data center techs are not bored. In this quarter they added nearly 11,000 new drives totaling over 150PB of storage, all while operating under strict Covid-19 protocols. We’ll cover how they did that in a future post, but let’s just say they were busy.

The Island of Misfit Drives
There were 190 drives (150,947 minus 150,757) that were not included in the 3Q20 Quarterly Chart above because we did not have at least 60 drives of a given model. Here’s a breakdown:

Nearly all of these drives were used as replacement drives. This happens when a given drive model is no longer available for purchase, but we have many in operation and we need a replacement. For example, we still have 3 WDC 6TB drives in use; they are installed in 3 different Storage Pods, along with 6TB drives from Seagate and HGST. Most of these drives were new when they were installed, but sometimes we reuse a drive that was removed from service, typically via a migration. Such drives are, of course, reformatted, wiped, and then must pass our qualification process to be reinstalled.

There are two “new” drives on our list. These are drives that are qualified for use in our data centers, but we haven’t deployed in quantity yet. In the case of the 10TB HGST drive, the availability and qualification of multiple 12TB models has reduced the likelihood that we would use more of this drive model. The 16TB Toshiba drive model is more likely to be deployed going forward as we get ready to deploy the next wave of big drives.

The Big Drives Are Here
When we first started collecting HDD data back in 2013, a big drive was 4TB, with 5TB and 6TB drives just coming to market. Today, we’ll define big drives as 14TB, 16TB, and 18TB drives. The table below summarizes our current utilization of these drives.

The total of 19,878 represents 13.2% of our operational data drives. While most of these are the 14TB Toshiba drives, all of the above have been qualified for use in our data centers.

For all of the drive models besides the Toshiba 14TB drive, the number of drive days is still too small to conclude anything, although the Seagate 14TB model, the Toshiba 16TB model, and the Seagate 18TB model have experienced no failures to date.

We will continue to add these large drives over the coming quarters and track them along the way. As of 3Q20, the lifetime AFR for this group of drives is 1.04%, which as we’ll see, is below the lifetime AFR for all of the drive models in operation.

Lifetime HDD Failure Rates
The table below shows the lifetime AFR for the HDD models we had in service as of September 30, 2020. All of the drive models listed were in operation during this time frame.

The lifetime AFR as of 3Q20 was 1.58%, the lowest since we started keeping track in 2013. That is down from 1.73% one year ago, and down from 1.64% last quarter.

We added back the average age column as “Avg Age.” This is in months and is the average age of the drives used to compute the data in the table and is based on the amount of time they have been in operation. One thing to remember is that our environment is very dynamic with drives being added: being migrated, and leaving on a regular basis and this could impact the average age. For example, we could retire a Storage Pod with mostly older drives and that could lower the average age of the remaining drives of that model while those remaining drives got older.

Looking at the average age, the 6TB Seagate drives are the oldest cohort, averaging nearly 5 and a half years of service each. They have actually gotten better over the last couple years and are aging well with a current lifetime AFR of 1.0%.

The complete data set used to create the information used in this review is available on our HDD Test Data webpagehttps://www.backblaze.com/b2/hard-drive-test-data.html.

If you just want the summarized data used to create the tables and charts in this blog post, you can download the Zip file containing the MS Excel spreadsheet.

Read also:
Backblaze Tested 139,867 HDDs in 2Q20
Lowest historical annualized failure rate of 0.81% vs. 1.07% in 1Q20, no failure on Toshiba 4TB drives since 4Q18
August 20, 2020 | Press Release