HDD Failure Rates Based on 124,956 HDDs at Backblaze

This report was published on February 11, 2020 and authored by Andy Klein, director of compliance, on a blog of Backblaze, Inc.

Backblaze HDD Stats for 2019

As of December 31, 2019, Backblaze had 124,956 spinning HDDs. Of that number, there were 2,229 boot drives and 122,658 data drives.

This review looks at the HDD annualized failure rates (AFR) for the data drive models in operation in our data centers. In addition, we’ll take a look at how our 12 and 14TB drives are doing and get a look at the new 16TB drives we started using in 4Q19. Along the way we’ll share observations and insights on the data presented.

2019 HDD AFR
At the end of 2019, Backblaze was monitoring 122,658 HDDs used to store data. For our evaluation, we remove from consideration those drives that were used for testing purposes and those drive models for which we did not have at least 5,000 drive days during 4Q19 (see notes and observations for why). This leaves us with 122,507 HDDs.

The table below covers what happened in 2019.

Notes and Observations
There were 151 drives (122,658 minus 122,507) that were not included in the list above. They were either used for testing or did not have at least 5,000 drive days during 4Q19. The 5,000 drive-day limit removes those drive models where we only have a limited number of drives working a limited number of days during the period of observation.

The data for all drives, data drives, boot drives, etc., is available for download on the HDD Test Data webpage.

The only drive model not to have a failure during 2019 was the 4TB Toshiba model MD04ABA400V.

That’s very good, but the data sample is still somewhat small. For example, if there had been just 1 drive failure during the year, the AFR for that Toshiba model would be 0.92% – still excellent, not 0%.

The Toshiba 14TB drive, model MG07ACA14TA, is performing very well at a 0.65% AFR, similar to the rates put up by the HGST drives. For their part, the Seagate 6TB and 10TB drive continue to be solid performers with AFR of 0.96% and 1.00% respectively.

The AFR for 2019 for all drive models was 1.89% which is much higher than 2018. We’ll discuss that later in this review.

Beyond the 2019 Chart – “Hidden” Drive Models
There are a handful of HDD models that didn’t make it to the 2019 chart because they hadn’t recorded enough drive-days in operation. We wanted to take a few minutes to shed some light on these drive models and where they are going in our environment.

Seagate 16TB HDDs
In 4Q19 we started qualifying Seagate 16TB drives, model ST16000NM001G. As of the end of 4Q19 we had 40 drives in operation, with a total of 1,440 drive days-well below our 5,000 drive day threshold for 4Q19, so they didn’t make the 2019 chart. There have been 0 failure through 4Q19, making the AFR 0%, a good start for any drive. Assuming they continue to pass our drive qualification process, they will be used in the 12TB migration project and to add capacity as needed in 2020.

Toshiba 8TB HDDs
In 4Q19 there were 20 Toshiba 8TB drives model HDWF180. These drives have been installed for nearly 2 years. In 4Q19, they only had 1,840 drive days, below the reporting threshold, but lifetime they do have 13,994 drive days with only 1 drive failure, giving us an AFR of 2.6%. We like these drives, but by the time they were available to us in quantity, we could buy 12TB drives at the same cost per terabyte. More density, same price. Given we are moving to 16TB drives and beyond, we most likely will not be buying any of these drives in the future.

HGST 10TB HDDs
There are 20 HGST 10TB drives model HUH721010ALE600 in the operation. These drives have been in service a little over one year. They reside in the same Backblaze Vault as the Seagate 10TB drives. The HGST units recorded only 1,840 drive days in 4Q19 and a total of 8,042 since being installed. There have been 0 failure. As with the Toshiba 8TB, purchasing more of these 10TB drives is unlikely.

Toshiba 16TB HDDs
You won’t find these in the 4Q19 stats, but in 1Q20 we added 20 Toshiba 16TB drives model MG08ACA16TA. They have logged a total of 100 drive days, so it is way too early to say anything other than more to come in the 1Q20 report.

Comparing HDD Stats for 2017, 2018, and 2019
The chart below compares the AFR for each of the last 3 years. The data for each year is inclusive of that year only and for the drive models present at the end of each year.

Rising AFR in 2019
The total AFR rose significantly in 2019. About 75% of the different drive models experienced a rise in AFR from 2018 to 2019.

There are 2 primary drivers behind this rise. First, the 8TB drives as a group seem to be having a mid-life crisis as they get older, with each model exhibiting their highest failure rates recorded. While none of the rates is cause for worry, they contribute roughly one fourth of the drive days to the total, so any rise in their failure rate will affect the total.

The second factor is the Seagate 12TB drives, this issue is being aggressively addressed by the 12TB migration project reported on previously.

Migration Slows, but Growth Doesn’t
In 2019 we added 17,729 net new drives. In 2018, a majority of the 14,255 drives added were due to migration. In 2019, less than half of the new drives were for migration with the rest being used for new systems. In 2019 we decommissioned 8,800 drives totaling 37PB of storage and replaced them with 8,800 drives, all 12TB, totaling about 105PB, then we added an additional 181PB in 2019 using 12TB and 14TB drives.

Drive Diversity
Manufacturer diversity across drive brands increased slightly in 2019. In 2018, Seagate drives were 78.15% of the drives in operation, by the end of 2019 that percentage had decreased to 73.28%. HGST went from 20.77% in 2018, to 23.69% in 2019, and Toshiba increased form 1.34% in 2018 to 3.03% in 2019. There were no Western Digital branded drives in the data center in 2019, but as WDC rebrands the newer large-capacity HGST drives, we’ll adjust our numbers accordingly.

Lifetime HDD Stats
While comparing the AFR of HDDs over multiple years is a great way to spot trends, we also look at the lifetime AFR of our HDDs. The chart below shows the AFR of all of the drives models in production as of December 31, 2019.