We just lost 3TB of data on a SanDisk Extreme SSD(theverge.com)
theverge.com
We just lost 3TB of data on a SanDisk Extreme SSD
https://www.theverge.com/22291828/sandisk-extreme-pro-portable-my-passport-failure-continued
51 comments
Uh... is one broken drive such a big issue? Hardware will fail, usually when it is very new or very old (Bathtub curve). Or are there more such events the article didn't mention?
Yes, here are the linked articles indicating it's a WD issue and they're continuing to push this product with sales:
https://www.theverge.com/2023/5/22/23733267/sandisk-extreme-...
https://arstechnica.com/gadgets/2023/05/sandisk-extreme-ssds...
https://www.theverge.com/2023/5/22/23733267/sandisk-extreme-...
https://arstechnica.com/gadgets/2023/05/sandisk-extreme-ssds...
If you read the article, the first paragraph explains that this was the replacement drive that WD sent him when his previous one failed.
So it’s at least 2 drives in a row in a short period of time. Statistically that will happen, but it’s like gambling - if you win too much you get shot.
So it’s at least 2 drives in a row in a short period of time. Statistically that will happen, but it’s like gambling - if you win too much you get shot.
Which points to a firmware bug in the SSD that the use-case managed to trigger repeatedly. Firmware bugs were around in abundance when performant SSDs first came to market more than a decade ago, and while manufacturers have gotten better at testing SSD firmware, it is still inevitable that bugs will occasionally make it into shipping hardware. Moral of the story: don't trust your valued data to any one storage device. I've tried to use drives from different vendors in RAID 1 mirrors to minimize exposure to firmware bugs, and I've still seen SSDs die at unexpected times in the last few years.
yes there are, and they mention an Ars Technica story, as well as reddit users complaining.
I lost 1TB on such a SSD I hardly used.
new drives should never crash or lose data, also WD and SanDisk lie through their teeth, there is no integrity.
> new drives should never crash or lose data,
In your small sample size maybe.
In your small sample size maybe.
It'll be true for sufficiently short periods of time, or for ages if you never put any data on it.
About 1/4 of hard drives I buy die in the first 24 hours since I unsportingly run badblocks on them to see whether they're on the start of the failure curve.
About 1/4 of hard drives I buy die in the first 24 hours since I unsportingly run badblocks on them to see whether they're on the start of the failure curve.
What do you mean you run badblocks? Can you talk more about your drive testing process?
Not the person you asked, but I do something similar. This is the guide I like to follow for new drives: https://www.truenas.com/community/resources/hard-drive-burn-...
It's on the TrueNAS forum, but it'll work on basically any Linux system. I'm currently using Unraid, which is based on Slackware.
(I must be luckier than the parent, because I've only had a single drive in a dozen or more die during testing - and that was a used one off ebay!)
It's on the TrueNAS forum, but it'll work on basically any Linux system. I'm currently using Unraid, which is based on Slackware.
(I must be luckier than the parent, because I've only had a single drive in a dozen or more die during testing - and that was a used one off ebay!)
badblocks has a destructive write mode. It writes numbers to disk then reads them back a while later and tells you if they differ. That seems to work hard drives fairly hard - they warm up at least. A day is usually how long my patience lasts.
There are probably better stress test approaches out there. The basic idea is to find out if your drive is one that'll die early before putting a load of data you want to keep on it.
I don't do this to ssd as they have lowish write limits anyway, so I just assume any data written to a ssd will vanish at some point. Hard drives in a network share, ssd in dev machines treated as scratch storage.
There are probably better stress test approaches out there. The basic idea is to find out if your drive is one that'll die early before putting a load of data you want to keep on it.
I don't do this to ssd as they have lowish write limits anyway, so I just assume any data written to a ssd will vanish at some point. Hard drives in a network share, ssd in dev machines treated as scratch storage.
Oh wow, that is -bad-. I'm usually being 10-12 and i think i've only had one or two over the last 5 years.
Yes I bought a new WD 1TB a year ago and one day it was damaged beyond repair. I did not do anything with it. Shop technician's diagnosis was "several tiny scratches on the disk". Not sure what that meant, I assume the whole build of the drive is just bad?
I just lost a 4 years worth of my kid's videos. I will never forgive WD.
Edit: I just forgot to add that my old WD HD (bought around 2009-ish) is still working. But reading the contents will take hours. But the point is the quality of WD has gone downhill.
I just lost a 4 years worth of my kid's videos. I will never forgive WD.
Edit: I just forgot to add that my old WD HD (bought around 2009-ish) is still working. But reading the contents will take hours. But the point is the quality of WD has gone downhill.
>I just lost a 4 years worth of my kid's videos. I will never forgive WD.
I'm sorry, but your fate was sealed the moment you decided that one copy was sufficient. Drives can fail randomly, so keeping valuable data on one without redundancy will virtually guarantee that you will lose data at some point in time. Just be thankful that you learned your lesson and it only cost 4 years worth of videos. If it happened 3 years down the line it could easily have been 7 years worth of videos.
I'm sorry, but your fate was sealed the moment you decided that one copy was sufficient. Drives can fail randomly, so keeping valuable data on one without redundancy will virtually guarantee that you will lose data at some point in time. Just be thankful that you learned your lesson and it only cost 4 years worth of videos. If it happened 3 years down the line it could easily have been 7 years worth of videos.
Drives exist in 3 states. About to die, dead and on fire. Plan accordingly.
Just stored 7 years worth of memories in the cloud, they can worry about redundancy!
Ha! I am on iCloud Family max plan and I feel like it will get full 5 years from now.
Different failure mode. Try not to get locked out by Apple. Consider a local copy as well, hard drives are really cheap.
+1. Cloud is ok as a mirror, not as the main copy.
Yeah hard lesson learned indeed. I am slowly working on redundancy with my data, so looking to HN for solutions.
I would strongly recommend signing up for either Backblaze, or using something like iCloud Photos or Google Photos.
My entire 100,000 photos, 20 year library is now in iCloud photos. It's constantly synced and backed up to icloud. Then I have it pulled down in full to a mac that then backs up to backblaze.
Lots of ways to do it, but thats one. :)
For $5-10/month you can get unlimited backup from a machine with Backblaze including your external drives.
My entire 100,000 photos, 20 year library is now in iCloud photos. It's constantly synced and backed up to icloud. Then I have it pulled down in full to a mac that then backs up to backblaze.
Lots of ways to do it, but thats one. :)
For $5-10/month you can get unlimited backup from a machine with Backblaze including your external drives.
Thanks, I am also on iCloud but nearing its limit since I am on iCloud's family plan. I was wondering how did you pull everything to your mac?
You can just add more storage. Of course you could violate TOS and lose access, so still you need to store them in multiple different places.
You should never trust valuable data to a single HD; always make backups.
Personally, I use a portable HD (spinning rust kind, not SSD) to do regular backups of my PC, but the portable HD itself is also backed up onto another portable HD.
Personally, I use a portable HD (spinning rust kind, not SSD) to do regular backups of my PC, but the portable HD itself is also backed up onto another portable HD.
This exact SSD failed on me in a similar way. They also recently had a firmware issue that borked a bunch of drives, and they only released a fix tool for Windows users (I guess macOS/Linux users can pound sand?)
I will say that the warranty service was quite good, but given the below-zero level of trust I have in SanDisk post WD acquisition, the replacement drive is going to be basically fulfilling the role of “paperweight and very oversized temporary USB drive for ephemeral sneakernet transfers”.
Bought a pair of Samsung T7’s to replace it, no issues with them so far.
I will say that the warranty service was quite good, but given the below-zero level of trust I have in SanDisk post WD acquisition, the replacement drive is going to be basically fulfilling the role of “paperweight and very oversized temporary USB drive for ephemeral sneakernet transfers”.
Bought a pair of Samsung T7’s to replace it, no issues with them so far.
When it comes to reliability, simple solutions are usually the best.
I still use spinning rust as my primary back up --- arranged in a redundant array with offset manufacture dates.
I still use spinning rust as my primary back up --- arranged in a redundant array with offset manufacture dates.
>I still use spinning rust as my primary back up --- arranged in a redundant array with offset manufacture dates.
You will still need to set up either BTRFS or ZFS to avoid silent Data corruption which is happening more often than most people know. And you have to set it up to scrub it from time to time. It isn't automatic.
Arguably the cheapest solution that hit the above bar would be Synology DS223J at $180, their newest entry model with BTRFS support ( on a fairly old Linux Kernel, I think it is 5.10 that will never get updated again in the life time of the product ). You will still need to buy 2 HDD in redundant RAID and set it up yourself. And it is quite bulky as well.
Hardly consumer friendly. And that is before 99.9999% on HN jumping on you, telling you need another backup for this NAS, or another off line, third location devices....
It is sad, not only just consumer. Judging by the number of comment here plenty in the tech industry simply uses an additional copy, or external SSD devices as backup. And I can understand why, but there simply isn't a simple, easy to use consumer solution that offer enough security for a reasonable price.
And I have been stating this for more than a decade already. Your Kids's video are worth a lot more than you can imagine. Tried explaining it to your wife some of those video are gone.
You will still need to set up either BTRFS or ZFS to avoid silent Data corruption which is happening more often than most people know. And you have to set it up to scrub it from time to time. It isn't automatic.
Arguably the cheapest solution that hit the above bar would be Synology DS223J at $180, their newest entry model with BTRFS support ( on a fairly old Linux Kernel, I think it is 5.10 that will never get updated again in the life time of the product ). You will still need to buy 2 HDD in redundant RAID and set it up yourself. And it is quite bulky as well.
Hardly consumer friendly. And that is before 99.9999% on HN jumping on you, telling you need another backup for this NAS, or another off line, third location devices....
It is sad, not only just consumer. Judging by the number of comment here plenty in the tech industry simply uses an additional copy, or external SSD devices as backup. And I can understand why, but there simply isn't a simple, easy to use consumer solution that offer enough security for a reasonable price.
And I have been stating this for more than a decade already. Your Kids's video are worth a lot more than you can imagine. Tried explaining it to your wife some of those video are gone.
My favorite trick for this lately has been to use a stripe of mirrored pairs, where each pair is a WD Gold and a WD Red Pro (of matching size).
They're very similar drives (AFAICT, WD Gold is just a Red Pro but with a higher lifetime r/w rating) but it makes me feel a bit better that they're not from too similar of batches without having to actually temporally space out my purchases. I can buy 1 of each in the same cart.
And then for truly unrecoverable data, I back it up to S3 Deep Archive which seems to be one of AWS's only reasonably priced offerings. Don't ask me how much it's cost in egress bandwidth to pull it all back though. I'm hoping if I end up needing it I'll be grateful enough for my foresight to justify paying it.
They're very similar drives (AFAICT, WD Gold is just a Red Pro but with a higher lifetime r/w rating) but it makes me feel a bit better that they're not from too similar of batches without having to actually temporally space out my purchases. I can buy 1 of each in the same cart.
And then for truly unrecoverable data, I back it up to S3 Deep Archive which seems to be one of AWS's only reasonably priced offerings. Don't ask me how much it's cost in egress bandwidth to pull it all back though. I'm hoping if I end up needing it I'll be grateful enough for my foresight to justify paying it.
So they can fail at the same time or did you make sure they are from different batches? Anyway, why use four disks from the same manufacturer???
I'm not sure what exactly you mean - they're different lines of drives. And different SKUs on Amazon. The probability of me getting sibling drives out of that - and that's assuming the only difference between red pros and golds are binning - is infinitesimally small.
in my case i use 2.5" 5TB disks. there is only one model from one manufacturer making those.
i bought one each from two different shops located in different parts of the country 5 years ago. i used them in a mirror raid. one of them died a year ago. then i got two more drives, one of which was second hand slightly used. the price was good and i figured it was a good way to ensure i'd get drives from two very different batches.
Heh, me too, but just a reminder raid is not a backup. raid can keep your data online in the face of some types of hardware failure. It won't protect against mistakes, malice, or disaster. backups will.
What I do is build one good drive array then each machine gets a cheap local drive and can access the good drive via network and I don't have to spend too much putting redundancy in each machine. Now... "looks around embarrassedly" time to work on that backup system.
What I do is build one good drive array then each machine gets a cheap local drive and can access the good drive via network and I don't have to spend too much putting redundancy in each machine. Now... "looks around embarrassedly" time to work on that backup system.
Buy some blu-ray discs. Losing data is soul crushing and while blu-rays are not necessarily cheap they will outlive you and are easy to store/copy.
Too slow --- takes too long to backup.
Pressed discs sure. Consumer grade BD-R discs should only be considered data in transit.
There’s always MDISC, but you’d need a ton of them.
I have this exact drive; when plugged into my desktop PC it instantly wedges the USB root hub, causing USB to not work until I power cycle the entire machine. (Intel, W10)
Yet the same drive works flawlessly in an Intel laptop. I think the m.2 to USB bridge chip inside might be the issue.
Yet the same drive works flawlessly in an Intel laptop. I think the m.2 to USB bridge chip inside might be the issue.
Make sure to read the full article on this one, not just the beginning. I was initially rolling my eyes--yes, loosing data sucks, but all drives can die and there's such a thing as bad luck--but the issue really does appear to be larger than that.
Many years ago we discovered that hard drives can be run in RAID configuration to avoid failure wiping out data.
Bad drive runs happen. They’ve been happing a LOOONG time. If it’s important, dupe or raid.
Otherwise, expect to lose the data. It’s just a matter of time.
Lots of film and audio gear has a double recording option for just this reason.
I’m sorry the author has to relearn this lesson the hard way!
Bad drive runs happen. They’ve been happing a LOOONG time. If it’s important, dupe or raid.
Otherwise, expect to lose the data. It’s just a matter of time.
Lots of film and audio gear has a double recording option for just this reason.
I’m sorry the author has to relearn this lesson the hard way!
Many backups are mandatory - even if a drive never fails it can get lost or stolen.
Even with backups, drive failure rate matters! If you keep n backups, and a drive dies, then you will at least temporarily have n - 1 backups, which is presumably one less than you wanted.
I blame tech journalism like the verge for this failure.
SSD manufacturers are well aware of what their drives should do. But they are also aware that to get sales they need to win benchmarks, and that to do that they cut corners with data integrity.
A benchmark of a drive should involve a heavy workload with a random drive hard and unplanned poweroff every 30 seconds. The drive should never lose a single fsync'ed byte.
The drive should also be put in hot/cold cycling chambers. And in radiation chambers. And journalists should publish which drives meet their obligations despite all the adverse (yet still in spec) conditions they're put through.
I bet no ssd for sale today could survive all 3 tests above for a month and still get okayish performance without data loss.
SSD manufacturers are well aware of what their drives should do. But they are also aware that to get sales they need to win benchmarks, and that to do that they cut corners with data integrity.
A benchmark of a drive should involve a heavy workload with a random drive hard and unplanned poweroff every 30 seconds. The drive should never lose a single fsync'ed byte.
The drive should also be put in hot/cold cycling chambers. And in radiation chambers. And journalists should publish which drives meet their obligations despite all the adverse (yet still in spec) conditions they're put through.
I bet no ssd for sale today could survive all 3 tests above for a month and still get okayish performance without data loss.
…right. Don’t blame the manufacturer for making shitty drives, blame tech review sites for testing those drives with the tools available and reporting the results of those tests. This is some 4D level victim blaming shit. I’m almost impressed at the absurdity of this argument.
Tech review sites put extreme pressure on manufacturers.
When the review sites only review one thing, corners get cut to make that one thing look good.
Any honest manufacturer who did what they thought was best for the customer, not what got them the best reviews, went bankrupt long ago.
When the review sites only review one thing, corners get cut to make that one thing look good.
Any honest manufacturer who did what they thought was best for the customer, not what got them the best reviews, went bankrupt long ago.
Journalists could go even further. Have robustness tests where they drop the drive in salt water, shoot it with a gun, chuck it in a fire etc., And then see how much data can be recovered with data recovery tools.
I did not realize SanDisk used WD. Does anyone have an alternate storage brand to recommend?
WD owns SanDisk, they bought them several years ago
Samsung