Western Digital's 16TB and 18TB drives: EAMR HDDs enter the retail channel(anandtech.com)
anandtech.com
Western Digital's 16TB and 18TB drives: EAMR HDDs enter the retail channel
https://www.anandtech.com/show/15903/western-digitals-16tb-and-18tb-gold-drives-eamr-hdds-enter-the-retail-channel
167 comments
My personal recipe for buying drives: wait until Backblaze has run them for a year and then pick the winner. That gives a pretty safe strategy. I also buy them in small lots to avoid having the same firmware and number of hours on them, and order 3 (cold) spares so I don't have to wait for an order to come in in case a drive dies. Call me paranoid.
My recipe is "Go to Costco, buy the biggest backup drive they have, shuck the drive and put it in the NAS. When one dies, repeat."
I end up with a great variety of drives and firmwares that way.
I'm still waiting on that 2TB drive to die so I can put in a bigger drive and expand the array!
I end up with a great variety of drives and firmwares that way.
I'm still waiting on that 2TB drive to die so I can put in a bigger drive and expand the array!
I’ve heard that many external drives can’t be shucked any longer, as they don’t include proper sata connectors. Have you encountered that?
I have not yet run into that. So far they've all been fine.
The shucking has gotten harder for sure, I need a lot more force to pry them apart now. So much that I worry about damaging the drive.
The shucking has gotten harder for sure, I need a lot more force to pry them apart now. So much that I worry about damaging the drive.
Someone knows the trick. Fun anecdote: I bought a 8TB Seagate from a local retailer at discount because it was an open-box return. It looked fine until I plugged it in and the drive showed up without the usual setup.exe and the drive was partitioned oddly. Okay, go to resize the drive and only see 500GB of space. Defective drive? Maybe that's why it was returned. Then I pull up the SMART details. The drive reported over 10,000 hours on it, a history of read errors, and was manufactured by Toshiba.
I looked over every millimeter of that exterior and couldn't see the slightest mark of a pry tool or even a sticker being out of place. I'm impressed if they were able to open the case and reseal it that cleanly. Although I also theorize that someone is stealing empty cases from the factory that they assemble with "recycled" drives. Either way, it was a professional crime. Not like the lazy fraudsters who just put a literal brick in the box and hope the store employee doesn't look inside.
End of story, I got my money back without problem since the store manager knows me.
I looked over every millimeter of that exterior and couldn't see the slightest mark of a pry tool or even a sticker being out of place. I'm impressed if they were able to open the case and reseal it that cleanly. Although I also theorize that someone is stealing empty cases from the factory that they assemble with "recycled" drives. Either way, it was a professional crime. Not like the lazy fraudsters who just put a literal brick in the box and hope the store employee doesn't look inside.
End of story, I got my money back without problem since the store manager knows me.
Did you tell them what you'd discovered? (If yes, what did they think of the situation? (Did you get any sense where the drive came from?))
Idle thought experiment: maybe the enclosures were cloned. (ie, the originals were 3D scanned + cleaned up + etc.) I wonder how much that would cost end-to-end.
Idle thought experiment: maybe the enclosures were cloned. (ie, the originals were 3D scanned + cleaned up + etc.) I wonder how much that would cost end-to-end.
> Idle thought experiment: maybe the enclosures were cloned. (ie, the originals were 3D scanned + cleaned up + etc.) I wonder how much that would cost end-to-end.
Do you think it's worth the effort of 3d printing, painting, assembling everything, etc just to swap your fake for one real HDD?
Do you think it's worth the effort of 3d printing, painting, assembling everything, etc just to swap your fake for one real HDD?
Like I said, I'm friends with the manager and he was the one who told me about nearby stores being hit with a box-of-bricks return.
Looking at the authentic model I eventually got, it's in three parts with the outer shell a glossy black and and insert that's slate colored with vent panels for passive cooling. And a Seagate logo inset that doubles as an power/activity light. The bottom has small rubber feet and the sticker with model and serial number on it. They didn't have another drive in stock to do a side-by-side comparison with, but I couldn't notice anything that would have tipped me off it was a fake. Even tried prying at the seams to see if it came apart any easier and it was held just as firmly as the real thing.
Looking at the authentic model I eventually got, it's in three parts with the outer shell a glossy black and and insert that's slate colored with vent panels for passive cooling. And a Seagate logo inset that doubles as an power/activity light. The bottom has small rubber feet and the sticker with model and serial number on it. They didn't have another drive in stock to do a side-by-side comparison with, but I couldn't notice anything that would have tipped me off it was a fake. Even tried prying at the seams to see if it came apart any easier and it was held just as firmly as the real thing.
In my experience, only Western Digital portables (2.5") have direct USB connectors.
Costco currently has an 8TB drive for $140, while Amazon has one for $135 with free delivery. And it's a Seagate drive, they don't do so well in Backblaze reports.
I spoiled myself with a 10TB for $150 from Best Buy last year. Now I need another one but I can't stomach a worse deal than that!
Yeah they are always loser drives in the back blaze report, but so far it's been fine. No more than an expected level of loss.
I did that once, and found that even when connected directly via SATA, the drive seemed limited to USB3 speeds...
I guess that's cost reduction? Like, why make a faster drive when it's meant to be accessed only via USB3?
It was one of those Seagate Slim 2TB external drives.
I guess that's cost reduction? Like, why make a faster drive when it's meant to be accessed only via USB3?
It was one of those Seagate Slim 2TB external drives.
USB 3.0 speed limit is 5Gbps. That's 625 MB/s, which is faster than any HDD I'm aware of.
Modern USB 3.2 can do 20 Gbps. That's 2500 MB/s.
Modern USB 3.2 can do 20 Gbps. That's 2500 MB/s.
Not relevant here but a tangent: In USB 2.0 times the efficiency was pretty terrible, you could get ~25 MB/s disk transfer throughput from the 480 Mbit/s signaling rate. It improved a little over the years... I wonder what are the actual speeds you can get from USB 3.0/3.2.
Look at USB 3.2 SSDs and NVMe enclosures. They go over 1000 MB/s for sequential reads.
https://www.anandtech.com/show/14661/usb32-g2-portable-ssds-...
I'm pretty sure they are limited by the USB <--> NVMe controller, not USB itself.
https://www.anandtech.com/show/14661/usb32-g2-portable-ssds-...
I'm pretty sure they are limited by the USB <--> NVMe controller, not USB itself.
I'll check what the speed was again, when I get back home.
Ok, so I got it connected via SATA. I tried the following:
I don't have another HDD on hand to compare since I've upgraded everything to SSDs. Though I don't remember what's normal for HDDs, to be honest 126MiB/s doesn't seem that slow. I thought I remembered something lower.
pv /dev/zero > /mnt/foo
umount /mnt # trying to get rid of any caching of foo.
mount /dev/sda4 /mnt
pv /mnt/foo > /dev/null
pv /dev/sda3 > /dev/null
pv /dev/zero > /dev/sda3
I don't have another HDD on hand to compare since I've upgraded everything to SSDs. Though I don't remember what's normal for HDDs, to be honest 126MiB/s doesn't seem that slow. I thought I remembered something lower.
I think it pays to be paranoid with hard drives. They all fail, it's just a matter of when.
ASIDE: I wish someone had a calculator for which drives were a good buy right now. They get to a certain point and the bank-for-the-buck ratio is for good up to only a certain size, then the prices jump.
ASIDE: I wish someone had a calculator for which drives were a good buy right now. They get to a certain point and the bank-for-the-buck ratio is for good up to only a certain size, then the prices jump.
Something like https://diskprices.com, maybe?
Thanks for that link. Bookmarked because that's going to be really useful when I decide to upgrade my NAS.
Drives get swapped behind the scenes often. Waiting a year isn't reliable.
You should just assume every hard drive has a conservative annualized failure rate of 4%, jumping to 20% after 3 years, and plan your RAID arrays based on that.
You should just assume every hard drive has a conservative annualized failure rate of 4%, jumping to 20% after 3 years, and plan your RAID arrays based on that.
In my experience, buying the “latest and greatest” hard drive is a recipe for disaster. They should put a sticker on the box that says “warning: will likely die within one year.”
Maybe I’ve just gotten unlucky over the years though. Can anyone else confirm/deny that the latest hard drives tend to be unreliable?
(Cue sampling bias...)
EDIT: The article is about an enterprise HD, but I meant to ask a general question about consumer-grade drives. Presumably enterprise drives are much more reliable, because reasons. (It would be fascinating to know those reasons though!)
Maybe I’ve just gotten unlucky over the years though. Can anyone else confirm/deny that the latest hard drives tend to be unreliable?
(Cue sampling bias...)
EDIT: The article is about an enterprise HD, but I meant to ask a general question about consumer-grade drives. Presumably enterprise drives are much more reliable, because reasons. (It would be fascinating to know those reasons though!)
One reason: If you send a flush command to a consumer grade hd, it returns before the flush is done. This makes it look nicer on older HD benchmarking software. Enterprise hds return when the flush is complete, so databases acknowledge a commit only when it was actually succesfull.
I think your wires may be a little crossed. This is unlikely to have ever been true for magnetic drives, but there is a quite significant grain of truth (no pun intended) to it for SSD.
Consumer SSDs may indeed return an early indication that a flush is complete, but whether they are lying depends heavily on your notion of persistence:
- the SSD may return done once the data is buffered in RAM
- or once it has reached a temporary log
- or once it has reached a 'permanent' location on flash
- or once it has recorded the mapping of data<->location to some separate directory
- or once it has updated some separate root pointer pointing to the latest version of the log/mapping structures in some separate special storage
I don't believe any SSD waits until the last step above before returning done. Separately, the SSD:
- may have no electrical power outage protection whatsoever
- may have supercapacitors fitted
- the supercap may guarantee full durability of every acknowledged flush
- or only guarantee atomicity up to some prior acknowledged flush
- or only guarantee that the drive has /some/ data on it in /some/ state on the next reboot
SSDs are insanely complex
Consumer SSDs may indeed return an early indication that a flush is complete, but whether they are lying depends heavily on your notion of persistence:
- the SSD may return done once the data is buffered in RAM
- or once it has reached a temporary log
- or once it has reached a 'permanent' location on flash
- or once it has recorded the mapping of data<->location to some separate directory
- or once it has updated some separate root pointer pointing to the latest version of the log/mapping structures in some separate special storage
I don't believe any SSD waits until the last step above before returning done. Separately, the SSD:
- may have no electrical power outage protection whatsoever
- may have supercapacitors fitted
- the supercap may guarantee full durability of every acknowledged flush
- or only guarantee atomicity up to some prior acknowledged flush
- or only guarantee that the drive has /some/ data on it in /some/ state on the next reboot
SSDs are insanely complex
It is interesting that to remain competitive in benchmarks, SSD vendors have to sell you an incredibly complicated software system alongside their flash chips soldered to a circuit board. They can't fix your filesystem, because you won't switch (and why pay them when your OS comes with one for free). They can't fix your database, because you won't switch (you already pay Amazon a million dollars a month for it). So they have to implement all the complicated parts of a database or filesystem, then try to understand the stream of events that real databases and real filesystems send it, and then restructure them to win the benchmark. All of this with no source code and probably no real test suite, and it all ends up being slower than a database that could just write to the flash chips directly.
I am also "excited" when I think about how the SSD shards blocks across flash chips and adds erasure codes, then people put the SSDs in RAID to shard blocks across SSDs and add erasure codes, then people put a filesystem on top of that that shards blocks across arrays and adds erasure codes, then put an application on top of that that writes to 3 replicas and writes erasure codes to 2 more. Now your "hello world" text file uses 8 gigabytes of storage, but at least you only lose it if there's one bug in the SSD controller, your RAID software, your filesystem, or your application. Or a tornado hits the SSD. Oh well.
HN often contains articles like "abstraction is bad", but even their contrived examples aren't this scary. I'm still pleasantly surprised when I log into my computer in the morning and still have a .bashrc.
I am also "excited" when I think about how the SSD shards blocks across flash chips and adds erasure codes, then people put the SSDs in RAID to shard blocks across SSDs and add erasure codes, then people put a filesystem on top of that that shards blocks across arrays and adds erasure codes, then put an application on top of that that writes to 3 replicas and writes erasure codes to 2 more. Now your "hello world" text file uses 8 gigabytes of storage, but at least you only lose it if there's one bug in the SSD controller, your RAID software, your filesystem, or your application. Or a tornado hits the SSD. Oh well.
HN often contains articles like "abstraction is bad", but even their contrived examples aren't this scary. I'm still pleasantly surprised when I log into my computer in the morning and still have a .bashrc.
I'd been hearing rumors (possibly spread by the mfgs to scare enterprises away from cheap IDE drives) about it during the late 90's and early 2000's.
Then bfitz proved in in 2005, and it turns out SCSI drives sometimes do it too: https://brad.livejournal.com/2116715.html
Then bfitz proved in in 2005, and it turns out SCSI drives sometimes do it too: https://brad.livejournal.com/2116715.html
Oh no, this one came from I think the win98 era. I read about it in some (paper!) computer magazines, I had no or barely internet at the time. SDDs had not yet been invented.
Once one vendor decided to screw us, the others had no choice. It was trivial for benchmarks to work around it, too: Just send more data so the cache becomes irrelevant. The whole episode took a few months at most.
I dont remember if they flat-out ignored the flush, or started a flush but didn't wait for completion.
Once one vendor decided to screw us, the others had no choice. It was trivial for benchmarks to work around it, too: Just send more data so the cache becomes irrelevant. The whole episode took a few months at most.
I dont remember if they flat-out ignored the flush, or started a flush but didn't wait for completion.
How can I find out which drives exactly have synchronous flush and which don't? Is it a widely publicized fact?
> If you send a flush command to a consumer grade hd, it returns before the flush is done.
So essentially its a no-op?
So essentially its a no-op?
Just speculating but perhaps the consumer ones return after the cache is populated and not after the data is durably on disk? That wouldn’t really be a no-op but would be much quicker for small writes at the expense of correctness in a power outage (although in the case of a DB the WAL entries are typically atomic so it would just be a rollback anyway).
Think of it like mongodb thinks of data: it's eventually consistent, maybe.
source? this sounds like an urban myth. there's no way that every single consumer grade hard drive "returns before the flush is done", given that: 1. consumer grade is not well-defined, 2. there are many different "flush commands", and it does not depend on whether a drive is "consumer grade", and 3. if this were the case, then filesystem and database corruption would be rampant, which it is eminently not.
it would be impossible to write a remotely reliable high-performance database or filesystem if flushing were useless. professionals doing formal analyses like https://www.sqlite.org/psow.html would be a complete waste of time, because everyone would be just praying anyways.
it would be impossible to write a remotely reliable high-performance database or filesystem if flushing were useless. professionals doing formal analyses like https://www.sqlite.org/psow.html would be a complete waste of time, because everyone would be just praying anyways.
>> consumer grade is not well-defined
WD has laid out a reasonable standard. Blue/Black/Green are consumer grades. Red is for NAS and Gold is for enterprise.
https://brad.livejournal.com/2116715.html
some enterprise SCSI drives too. This was all back in the late 90's/early 2000's, no idea if it still happens with SATA or SAS drives...
some enterprise SCSI drives too. This was all back in the late 90's/early 2000's, no idea if it still happens with SATA or SAS drives...
sure, and "consumer" filesystems/databases were also pretty bad. ext3 was only released in 2001, and Windows supported FAT system disk since at least Vista. the "standard" OSS database was MySQL until at least 2010, if not later. I argue 2020 is different though. despite the onslaught of mongodb 'engineers', data reliability is far better than it was in 2000.
I've had friends who worked at hard drive manufacturers.
They said companies can very accurately know the lifetime of a drive the day it is manufactured just from the vibration profile.
enterprise > consumer drives.
It also depends on the "customer" buying the drives - certain system manufacturers will get better drives.
I think it's like restaurant owners getting better cuts of meat than you can get.
They said companies can very accurately know the lifetime of a drive the day it is manufactured just from the vibration profile.
enterprise > consumer drives.
It also depends on the "customer" buying the drives - certain system manufacturers will get better drives.
I think it's like restaurant owners getting better cuts of meat than you can get.
> Maybe I’ve just gotten unlucky over the years though.
It's not you, it was WD (and apparently other vendors too).
https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin...
It's not you, it was WD (and apparently other vendors too).
https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin...
Those aren't the "latest and greatest," and they didn't die per se, they're just slow for some workloads (some of those workloads that they're marketed for.)
[deleted]
If you look at the CrashPlan drive reports, that doesn't seem to be their experience (and I don't know of a current, larger, public hard drive reliability report).
It seems that newer models trend with priors from that brand, with some outliers.
It seems that newer models trend with priors from that brand, with some outliers.
Oops. Pretend I said BackBlaze instead of CrashPlan.
Thank you for pointing out BackBlaze!
https://www.backblaze.com/b2/hard-drive-test-data.html
https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1...
https://www.backblaze.com/b2/hard-drive-test-data.html
https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1...
I had a few drives die in rapid succession, but then I realized I wasn't cooling the system properly. That was around 12 years ago, and I'm still using the drives I bought after that today.
Maybe you have a similar problem?
Maybe you have a similar problem?
I stick with things that have been out for a few years. Consumers seem to be just unwilling beta testers of new hardware, later revisions are more reliable. Applies to everything from computers to cars to software heh
Kinda in the phase of buying disks for a $500-ish~ raid 5 NAS setup. The whole recent SMR debacle keeps me a bit careful on buying those disks before extensively checking out all the specs.
Will news like this cause retail price drops anytime soon? Other advice is welcome too :-)
Will news like this cause retail price drops anytime soon? Other advice is welcome too :-)
No. The HDD market is a duopoly.
You’ll find binned versions of these drives in 3 months sold at 40% off in EasyStores and Elements though.
It’s how they get rid of parts that work fine but don’t pass the specs for these enterprise drives.
You’ll find binned versions of these drives in 3 months sold at 40% off in EasyStores and Elements though.
It’s how they get rid of parts that work fine but don’t pass the specs for these enterprise drives.
The drives are filled with helium. What happens when the helium leaks? I assume the drives will slow down or fail instantly.
iPhones use MEMS-based clocks that fail when exposed to helium. Are any servers, NAS devices, or laptops also susceptible to disruption from helium?
iPhones use MEMS-based clocks that fail when exposed to helium. Are any servers, NAS devices, or laptops also susceptible to disruption from helium?
As cptskippy said, the helium is at atmospheric pressure, so a leak would not fail catastrophically. Helium would more or less slowly diffuse. If the leak is small enough, helium could get out but larger molecules would have a harder time. Thus the pressure inside the enclosure would get slightly lower than atmosphere and leaking would slow down.
The iPhone MEMS failed when a rather large amount of pressurized, cryogenic helium was leaked during an MRI machine servicing. The quantity of unpressurized helium in a tiny HDD enclosure is minuscule, and thus the slow diffusion is unlikely to register above background levels, even at very close range.
The iPhone MEMS failed when a rather large amount of pressurized, cryogenic helium was leaked during an MRI machine servicing. The quantity of unpressurized helium in a tiny HDD enclosure is minuscule, and thus the slow diffusion is unlikely to register above background levels, even at very close range.
The pressure differential between the drive and the atmosphere isn't going to be that great, if any, so it's pretty unlikely that a leak will occur.
If the Helium were to leak out then the drive would probably fail much more quickly than it otherwise would have but there isn't nearly enough helium to affect any MEMS devices.
Also consider that these drives are going to be in a system with forced air cooling so any Helium that might escape will be quickly vented.
Additionally, MEMS devices don't fail but rather behave differently while under the influence of helium.
If the Helium were to leak out then the drive would probably fail much more quickly than it otherwise would have but there isn't nearly enough helium to affect any MEMS devices.
Also consider that these drives are going to be in a system with forced air cooling so any Helium that might escape will be quickly vented.
Additionally, MEMS devices don't fail but rather behave differently while under the influence of helium.
What is this thing Anandtech have started doing with the back button, somehow inserting an extra ad page in between getting back to HN?
They’re called bounce or exit ads. Meant to catch people from leaving too quickly, but extremely annoying and filled with clickbait crap and articles from 8 years ago.
Awful. “Check this out before you leave”
And it includes their articles plus sponsored ads about casinos for me.
And it includes their articles plus sponsored ads about casinos for me.
I wonder how many of these hard drives are needed to completely scan the internet every month and create a "time capsule" drive that you can revisit and browse ten years later like a yearbook
Stackoverflow is about 10GB:
https://archive.org/details/stackexchange
English Wikipedia is about 15GB: https://en.wikipedia.org/wiki/Wikipedia:Database_download#En...
Reddit is about 1TB: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...
Common Crawl (May/June 2020):
"It contains 2.75 billion web pages or 255 TiB of uncompressed content". About 50TB compressed.
https://commoncrawl.org/2020/06/may-june-2020-crawl-archive-...
I think 100TB would be plenty. We're not quite there yet but I wouldn't be surprised if you could get "The Internet" on one consumer drive in a few years. This would be a cool product.
English Wikipedia is about 15GB: https://en.wikipedia.org/wiki/Wikipedia:Database_download#En...
Reddit is about 1TB: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...
Common Crawl (May/June 2020):
"It contains 2.75 billion web pages or 255 TiB of uncompressed content". About 50TB compressed.
https://commoncrawl.org/2020/06/may-june-2020-crawl-archive-...
I think 100TB would be plenty. We're not quite there yet but I wouldn't be surprised if you could get "The Internet" on one consumer drive in a few years. This would be a cool product.
Something tells me that 50TB doesnt include video media. There are dozens of chinese pirate streaming sites with more movies and TV than whole of netflix and all its competitors. Then there is YT.
Or pictures.
If you look at that reddit link it's about "every single comment" -- and that's about 1 TB. I'd bet that the pictures from one fairly active if niche sub would exceed that, e.g. r/battlecars or r/breaddit
If you look at that reddit link it's about "every single comment" -- and that's about 1 TB. I'd bet that the pictures from one fairly active if niche sub would exceed that, e.g. r/battlecars or r/breaddit
A fuckload. There are 160M active domains. Say 2% of them are news sites with tens of thousands of active articles, another 2% is e-commerce sites and forums with also tens of thousands of pages, and the rest have like 50 pages on average. So you have something like 154M sites x 50 pages each x let's say 200KB per page that makes it 1.540 TB of data. And for the rest 4% let's assume 20.000 pages on average by 2MB each, that gives another 250.000 TB. And then you add sites like YouTube which could easily triple the above numbers. You'll end-up with a million TBs at least. That's like 100.000 HDDs.
The website that lists all possible Bitcoin private keys would take up all of those drives anyway
With video and other rich media ? You will need hundreds of thousands or even millions of these disks.
YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day - 180,000 disks a year) [1]
If YT is no more than say 1% of the public internet (Facebook is probably storing lot more , most of the data is not public though, Instagram is ) Then you are looking probably at 18 million disks a year without any backups and copies considered .
[1] it is assuming SD quality 220 mb per hour single format and one copy.
YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day - 180,000 disks a year) [1]
If YT is no more than say 1% of the public internet (Facebook is probably storing lot more , most of the data is not public though, Instagram is ) Then you are looking probably at 18 million disks a year without any backups and copies considered .
[1] it is assuming SD quality 220 mb per hour single format and one copy.
> YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day ...) ... assuming SD quality 220 mb per hour single format and one copy.
Nitpick:
Each bigger resolution stacks on top of the previous one, so 8K60fps means storing 8K60 (webm+av1), alongside 4K60 HDR (webm), 4K60 (webm+av1), 4K (webm), 1440p60 HDR (webm), 1440p60 (webm+av1), 1440p (webm), 1080p60 HDR (webm), 1080p60 (webm+h264+av1), 1080p (webm+h264), 720p60 HDR (webm), 720p60 (webm+h264+av1), 720p (webm+h264), 480p60 HDR (webm), 480p (webm+h264+av1), 360p60 HDR (webm), 360p (webm+h264+av1), 240p60 HDR (webm), 240p (webm+h264+av1), 144p60 HDR (webm), and 144p (webm+h264+av1).
And as noted, a given resolution is often stored in multiple video formats, so there's that too.
The data needed to store a single video thus adds up quickly:
It's quite possible YouTube also keeps the original uploads around for a bit as well.
---
It incidentally proved so complicated to correctly type out the format list above manually it was actually easier to figure out the following regex:
Nitpick:
Each bigger resolution stacks on top of the previous one, so 8K60fps means storing 8K60 (webm+av1), alongside 4K60 HDR (webm), 4K60 (webm+av1), 4K (webm), 1440p60 HDR (webm), 1440p60 (webm+av1), 1440p (webm), 1080p60 HDR (webm), 1080p60 (webm+h264+av1), 1080p (webm+h264), 720p60 HDR (webm), 720p60 (webm+h264+av1), 720p (webm+h264), 480p60 HDR (webm), 480p (webm+h264+av1), 360p60 HDR (webm), 360p (webm+h264+av1), 240p60 HDR (webm), 240p (webm+h264+av1), 144p60 HDR (webm), and 144p (webm+h264+av1).
And as noted, a given resolution is often stored in multiple video formats, so there's that too.
The data needed to store a single video thus adds up quickly:
$ youtube-dl -F 1La4QzGeaaQ | grep -o '[0-9\.]\+.iB' | sed 's/MiB/*(1024^2)/;s/GiB/*(1024^3)/;s/.*/(&)/' \
| tr '\n' + | sed 's/\(.*\)+$/scale=3;print(\1)\/(1024^3),"G\n"\n/' | bc -ql
8.423G
It's quite possible YouTube also keeps the original uploads around for a bit as well.
---
It incidentally proved so complicated to correctly type out the format list above manually it was actually easier to figure out the following regex:
$ youtube-dl -F 1La4QzGeaaQ | sed -n '/video/{s/^.\{35\}//;s/ \+[0-9]\+k \+, \+/ (/;s/webm.*\|vp9.*/webm/;s/avc.*/h264/;s/av01.*/av1/;s/4320p/8K/;s/2160p/4K/;p}' \
| sort -hr | sed ':1;H;N;${:2;s/\([^\(]\+\)(\([^\n]\+\)\n\(\1\)(/\1(\2+/;t2;s/\(\n\|$\)/), /g;s/, $//};b1'
The proposed project entails constructing a global-scale shared supercomputer capable of hosting the entire Internet on one platform as an application
https://en.wikipedia.org/wiki/IBM_Kittyhawk
https://en.wikipedia.org/wiki/IBM_Kittyhawk
So, basically where AWS is headed.
I joke, but there's a grain truth there.
I joke, but there's a grain truth there.
It's true, though. IBM misunderstood the implications of cloud computing at the time.
They clearly realized their mistake:
In 2013 researchers used the Kittyhawk project to demonstrate a novel high-performance cloud computing platform by merging a cloud computing environment with a supercomputer.
They clearly realized their mistake:
In 2013 researchers used the Kittyhawk project to demonstrate a novel high-performance cloud computing platform by merging a cloud computing environment with a supercomputer.
Wow... is IBM the Kodak of computing/hosting?
If you actually do this, just make sure your crawler doesn't get caught on some programmatic infinite webpage generator.
Like one of those calendar widgets.
Like one of those calendar widgets.
I suppose one could use the current size of the Internet Archive as a yardstick for how much public-facing stuff there is. Of course in general terms this also depends on your definition of 'the internet' - consider databases and other such stores. Huge numbers.
[deleted]
I remember reading a comment here recently that a few terabytes is enough to store personal history of all web browsing for life (minus media). Trying to recall if that's just an observation or an actual application somewhere. So much content is disappearing, starting to seem like prudent practice.
I've got almost three years of web history with full text locally in compressed 10gib, if you go for images and video it will get bigger though.
> minus media
Well that's not fun. :D
I'm filling up my 8TB SMR drive with archives of my favorite YouTube channels..
Well that's not fun. :D
I'm filling up my 8TB SMR drive with archives of my favorite YouTube channels..
I remember people complaining about the quality of WD drives. These ones claim a MTBF of 2.5M hours, workload rating of 550TB/yr, and a 5 year warranty. Can it be trusted?
Anecdotally, as well as supported by Backblaze, Seagate drives fail more often than WD.
https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1...
I’ve had one Seagate drive fail, and another that would actually return corrupt data even immediately after being written. Thankfully I have ZFS which is how I noticed it in the first place.
I’ve owned more WD drives for longer and I’ve actually never had one fail; at least before they store so little I graveyard them.
https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1...
I’ve had one Seagate drive fail, and another that would actually return corrupt data even immediately after being written. Thankfully I have ZFS which is how I noticed it in the first place.
I’ve owned more WD drives for longer and I’ve actually never had one fail; at least before they store so little I graveyard them.
Same experience here, only seagate drive failures over the past 15 years of owning maybe 30 drives total. WD has never failed. Obviously anecdotal.
The way I read the Backblaze stats is opposite - could you clarify?
* most reliable are HGST drives (model number starts with H, formerly Hitachi, such as DeskStar, UltraStar, etc., now under the WD brand), sub 1% AFR;
* then Seagate (model number starts with S), still pretty reliable at around 1% AFR, with a few exceptions;
* while the original WD (model number starts with W, such as WD RED etc.) drives are quite terrible with AFR from 2% to 5%.
(I'm looking mostly on the long term statistics, spanning at least a year, and older stats regarding WD drives, as they don't use them anymore.)
* most reliable are HGST drives (model number starts with H, formerly Hitachi, such as DeskStar, UltraStar, etc., now under the WD brand), sub 1% AFR;
* then Seagate (model number starts with S), still pretty reliable at around 1% AFR, with a few exceptions;
* while the original WD (model number starts with W, such as WD RED etc.) drives are quite terrible with AFR from 2% to 5%.
(I'm looking mostly on the long term statistics, spanning at least a year, and older stats regarding WD drives, as they don't use them anymore.)
My own anecdote is about the same, except in my ZFS-based NAS, I've had one WD drive fail. It was a WD Green though, so no surprise.
No. It absolutely cannot be trusted.
I buy drives in pairs, and RAID them. If you can't afford to do that with drives the size you need, buy smaller drives and keep less data. It's like being a half-decade behind on the Moore's Law curve. You get used to it and survive.
I buy drives in pairs, and RAID them. If you can't afford to do that with drives the size you need, buy smaller drives and keep less data. It's like being a half-decade behind on the Moore's Law curve. You get used to it and survive.
RAID won't save you from bit rot, and after I added bit rot detection in PhotoStructure, I was surprised to see how many of my files on older spindles had succumbed to minor corruption.
Using a filesystem that supports data scrubbing doesn't require an "enterprise class" budget anymore. After fielding this question from several of my beta users, I wrote it up the why's and how's on my blog: https://photostructure.com/faq/how-do-i-safely-store-files/
Using a filesystem that supports data scrubbing doesn't require an "enterprise class" budget anymore. After fielding this question from several of my beta users, I wrote it up the why's and how's on my blog: https://photostructure.com/faq/how-do-i-safely-store-files/
Yeah. I wish my file system did periodic integrity checks. I should switch to a more modern file system at some point. I'm still on ext4. Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around?
I haven't run into much bit rot yet, at least where it mattered, but I'm guessing I'd see more of it if I went up to bigger drives. For now, my data lives on 4tb.
PhotoStructure sounds neat. Business model may be tough. I wouldn't run a proprietary solution for data management, especially from a small startup. I just wouldn't trust it. I want to know what happens to my data, and to be able to get it out. I understand the claim you'll open source in case of business failure, but there are many other paths (e.g. bought by Oracle, milked for profits, moved to incompetent management, etc).
I would totally would pay for an open solution, if there were a good business model for that, but I'm not sure what that is. I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust).
I'd also consider paying for a split model, where the data management used open formats and auditable, documented open source, while there were UI/UX or search functionality on top of that. That may be what you're doing; I'm not sure. I can see some open source stuff, but I'm not sure what's open and what's closed. Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely.
Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, have an automated way to move to offline storage once a quarter as you recommended, and have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Ideally, I'd also have some way to securely share with people
I haven't run into much bit rot yet, at least where it mattered, but I'm guessing I'd see more of it if I went up to bigger drives. For now, my data lives on 4tb.
PhotoStructure sounds neat. Business model may be tough. I wouldn't run a proprietary solution for data management, especially from a small startup. I just wouldn't trust it. I want to know what happens to my data, and to be able to get it out. I understand the claim you'll open source in case of business failure, but there are many other paths (e.g. bought by Oracle, milked for profits, moved to incompetent management, etc).
I would totally would pay for an open solution, if there were a good business model for that, but I'm not sure what that is. I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust).
I'd also consider paying for a split model, where the data management used open formats and auditable, documented open source, while there were UI/UX or search functionality on top of that. That may be what you're doing; I'm not sure. I can see some open source stuff, but I'm not sure what's open and what's closed. Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely.
Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, have an automated way to move to offline storage once a quarter as you recommended, and have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Ideally, I'd also have some way to securely share with people
XFS don't support checksum for data but only for metadata.
dm-integrity could be used with XFS.
https://www.redhat.com/ja/blog/what-bit-rot-and-how-can-i-de...
https://www.redhat.com/ja/blog/what-bit-rot-and-how-can-i-de...
> Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around?
All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work.
> I want to know what happens to my data, and to be able to get it out.
You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library.
> ... I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust).
That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed.
> I can see some open source stuff, but I'm not sure what's open and what's closed.
The metadata parsing code (exiftool-vendored) and process management (batch-cluster) is open-source. The metadata inference and cleaning, image duplication detection and comparison, and UI are closed-source.
> Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely.
I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting.
> Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything,
borg, tarsnap, and BackBlaze b2 are good tools for this.
> have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes).
Simple SHA collisions are easy, and there are many command-line tools that accomplish this. PhotoStructure does detect SHA-duplicate images, but also downsampled, RAW+JPG, and other variations: https://photostructure.com/faq/what-do-you-mean-by-dedupe/
> Ideally, I'd also have some way to securely share with people
This is coming in a future version of PhotoStructure.
All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work.
> I want to know what happens to my data, and to be able to get it out.
You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library.
> ... I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust).
That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed.
> I can see some open source stuff, but I'm not sure what's open and what's closed.
The metadata parsing code (exiftool-vendored) and process management (batch-cluster) is open-source. The metadata inference and cleaning, image duplication detection and comparison, and UI are closed-source.
> Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely.
I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting.
> Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything,
borg, tarsnap, and BackBlaze b2 are good tools for this.
> have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes).
Simple SHA collisions are easy, and there are many command-line tools that accomplish this. PhotoStructure does detect SHA-duplicate images, but also downsampled, RAW+JPG, and other variations: https://photostructure.com/faq/what-do-you-mean-by-dedupe/
> Ideally, I'd also have some way to securely share with people
This is coming in a future version of PhotoStructure.
> > Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around?
> All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work.
Tech friend (yes you!): Which one do you recommend? I mostly care about avoiding headaches (so reliability, stability, and ease-of-use).
> You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library.
I got that, but my goal is to avoid headaches. If your business goes under, I don't want to lose a couple weekends to data recovery.
I think open source APIs to get at my data would be the level which would give me confidence. If I have access to the same APIs as your UI/UX uses to get at the data, I'd be okay.
> That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed.
I wouldn't look at other failures as ways to discredit an idea. Most failures are due to poor execution. That said, hardware business models are very different from software, and if it's outside of your comfort zone, you'll probably fail to execute too. But my point was:
1) There's a definite business proposition in providing a complete solution to the problem you articulated with very close to the technology you're building. You seem to be on the right track.
2) I'm not sure you've stumbled on the right business model yet.
I want a solution which minimizes headaches, both current and future potential. I'm glad to pay for that.
> I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting.
Well, in part. I think the other pieces are:
1) Reliability. Google can and does shut down accounts regularly. It loses data too. I'm a statistic.
2) Google downscales images and doesn't handle ones from my big cameras very well (it's great for cell phone picks).
3) Storage. I can buy many TB of disk space. Many TB in the cloud is still expensive.
4) Trust. If you build a tool which (a) lines up incentives right (I pay you) (b) has the right legal code (my data is mine, and unless you receive a subpoena, stays mine) (c) has the right technology so I only trust you when I absolutely have to (I can share files with end-to-end encryption, or openly, depending on what I'm doing)
5) Integration. I'd like RAW files and similar to live locally. All I want in the cloud are pics for showing people on my phone, and nice search tools. I want metadata synced, though.
What you're building is a lot more useful than anything else out there, by design, I think.
> borg, tarsnap, and BackBlaze b2 are good tools for this.
Thank you. A few points, though:
1) These do backups. I want synced file management WITH backups, not just backups. If I edit a Word file on my Mac, copy it to my Windows box, edit it again, and look 6 months later, I have a mess. If I then get rid of both machines, and buy a Linux box and copy both files over unto a RAID, I have a mess with duplicates.
2) Tarsnap is $0.25/GB/mo, so $250/TB/month. a 4TB backup would be $12k per year.
Just about the only thing I feel I have well-organized is my code, which lives in git. And a lot of my documents, I author in Markdown or LaTeX, and those live in git too.
Tech friend (yes you!): Which one do you recommend? I mostly care about avoiding headaches (so reliability, stability, and ease-of-use).
> You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library.
I got that, but my goal is to avoid headaches. If your business goes under, I don't want to lose a couple weekends to data recovery.
I think open source APIs to get at my data would be the level which would give me confidence. If I have access to the same APIs as your UI/UX uses to get at the data, I'd be okay.
> That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed.
I wouldn't look at other failures as ways to discredit an idea. Most failures are due to poor execution. That said, hardware business models are very different from software, and if it's outside of your comfort zone, you'll probably fail to execute too. But my point was:
1) There's a definite business proposition in providing a complete solution to the problem you articulated with very close to the technology you're building. You seem to be on the right track.
2) I'm not sure you've stumbled on the right business model yet.
I want a solution which minimizes headaches, both current and future potential. I'm glad to pay for that.
> I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting.
Well, in part. I think the other pieces are:
1) Reliability. Google can and does shut down accounts regularly. It loses data too. I'm a statistic.
2) Google downscales images and doesn't handle ones from my big cameras very well (it's great for cell phone picks).
3) Storage. I can buy many TB of disk space. Many TB in the cloud is still expensive.
4) Trust. If you build a tool which (a) lines up incentives right (I pay you) (b) has the right legal code (my data is mine, and unless you receive a subpoena, stays mine) (c) has the right technology so I only trust you when I absolutely have to (I can share files with end-to-end encryption, or openly, depending on what I'm doing)
5) Integration. I'd like RAW files and similar to live locally. All I want in the cloud are pics for showing people on my phone, and nice search tools. I want metadata synced, though.
What you're building is a lot more useful than anything else out there, by design, I think.
> borg, tarsnap, and BackBlaze b2 are good tools for this.
Thank you. A few points, though:
1) These do backups. I want synced file management WITH backups, not just backups. If I edit a Word file on my Mac, copy it to my Windows box, edit it again, and look 6 months later, I have a mess. If I then get rid of both machines, and buy a Linux box and copy both files over unto a RAID, I have a mess with duplicates.
2) Tarsnap is $0.25/GB/mo, so $250/TB/month. a 4TB backup would be $12k per year.
Just about the only thing I feel I have well-organized is my code, which lives in git. And a lot of my documents, I author in Markdown or LaTeX, and those live in git too.
If you want data integrity I would avoid butterfs at all costs. It deals with it, by not having any.
> RAID won't save you from bit rot
Sure it can. You use mdadm to schedule a monthly full scan of the RAID array, comparing both disks. It's the default in Debian.
You should also use smartmontools to schedule weekly long self tests of the disk, and nightly short self tests. That way you find out about problems when they are still small.
Sure it can. You use mdadm to schedule a monthly full scan of the RAID array, comparing both disks. It's the default in Debian.
You should also use smartmontools to schedule weekly long self tests of the disk, and nightly short self tests. That way you find out about problems when they are still small.
I'm sorry, I should have clarified: as far as I know, mdadm on RAID1 with consumer-grade spinning rust will not be able to save you from bit rot (or at least I haven't seen evidence to that effect). I haven't tried on RAID6 (where there's at least a chance due to the data duplication such that the flipped bit will be "out-voted" by the parity spindles).
Aha! I've had almost exactly that project in my ideas list for years. It sounds like the Right Solution for home photos! I have subscribed to your newsletter.
Excellent! Expect an invite email today or tomorrow. Ping me (there's a chat button on the website) if it doesn't go through, I'll set you up.
Do you buy pairs of identical drives? If you want to be extra paranoid, you can use a pair of different models or at least different batches to reduce the risk that they both fail in a short time window.
I had a RAID-5 array of five identical, same batch and close serial numbers. One went bad. I ordered replacement and procurement blocked the purchase until I got it approved by my manager. I warned it was urgent. They delayed it for 3 days. On day 2, the second drive died.
We had backups and could replay all db transactions up to the minute. Still, the service stopped for more than 24 hours. It took a week to clear the backlog.
We had backups and could replay all db transactions up to the minute. Still, the service stopped for more than 24 hours. It took a week to clear the backlog.
Delayed approval of an urgent 3-figure purchase resulting in a 5-6 figure loss in productivity, customer sat, etc. Great example of penny-wise, pound-foolish!
Buying individual drives to shave a few pennies is pretty foolish to begin with, so why stop there!
Make it an OEM problem.
Make it an OEM problem.
In their defense, we had a bunch of drives coming in in a month. It's just that we ran out of spares faster than expected (we had to increase capacity in a couple servers on short notice).
Indeed. Small powers.
Hopefully they let you keep at least one new drive around as a backup after that experience. If you're not running RAID6 you should at least have one on the shelf, preferably from a different batch.
That was in 2003 or so. IIRC, mdadm didn't have raid6 support back then and the chassis wouldn't support more than 4 devices.
Those were the heroic days.
Those were the heroic days.
Honestly though, 1) the array rebuild might take that long time anyway; 2) the stress of the array rebuilding would probably make the other drive fail anyway.
That's exactly what I do, except I take it a step further and go with different vendors.
With larger drive sizes and failures tending to follow a rather normal distribution w.r.t. mtbf, raid is becoming less of a safeguard these days. There are some good calculators online that will do the math for you, but at these sizes, you're very likely to see the second and even third redundant drive fail before the rebuild is complete.
What your saying is a falsehood, and a persistent rumor started by a bad paper and easily disproved. The individual disk capacity has nothing to do with the drive failure rate.
First, while RAID isn't a "safeguard", the idea that there is a high probability of raid failure during a rebuild has been repeatedly proven to be false as long as proper data hygiene is maintained. For the most part the chance of a second disk falling out during the rebuild is exactly the same as any other day regardless of capacity.
Put another way, if your scrubbing (or just accessing the data) frequently enough to avoid having silent bad sectors the idea that there is a high probability of a drive loss during the day or two it takes to rebuild an array would also result in drive losses on a nearly daily basis.
So unless your constructing a single RAID5 array out of ten thousand plus disks, if the MTBF numbers are anywhere close to realistic, drives in a single array aren't going to be failing at a rate high enough that the chance of a double fault not caused by something symptomatic across the entire line of drives (say a firmware that wipes the drive after X power on hours, in which case no level of redundancy will save you) is a real worry.
So, the rumor is based on the idea that your increasing the RAID rebuild time from a couple hours to days/weeks on end due to the larger capacity. But this is also false, because as density increases so has the sequential throughput of the drives. AKA comparing a ~ten year old 4TB drive doing ~100MB/sec vs a modern 8TB drive doing 200MB/sec. There are some non-linearities due to the number of platters and the rate of TPI vs BPI increases but generally platter count is somewhat constant and TPI and BPI are both increasing. Now if your hitting internal RAID limits because your running modern drives on old controllers which cannot rebuild at the drives peak throughput that is a different problem.
Bottom line, your doing it wrong if RAID5/6 isn't increasing both actual as well as theoretical availability. The MTBF numbers are published to give a baseline on what the expected failure rates are.
First, while RAID isn't a "safeguard", the idea that there is a high probability of raid failure during a rebuild has been repeatedly proven to be false as long as proper data hygiene is maintained. For the most part the chance of a second disk falling out during the rebuild is exactly the same as any other day regardless of capacity.
Put another way, if your scrubbing (or just accessing the data) frequently enough to avoid having silent bad sectors the idea that there is a high probability of a drive loss during the day or two it takes to rebuild an array would also result in drive losses on a nearly daily basis.
So unless your constructing a single RAID5 array out of ten thousand plus disks, if the MTBF numbers are anywhere close to realistic, drives in a single array aren't going to be failing at a rate high enough that the chance of a double fault not caused by something symptomatic across the entire line of drives (say a firmware that wipes the drive after X power on hours, in which case no level of redundancy will save you) is a real worry.
So, the rumor is based on the idea that your increasing the RAID rebuild time from a couple hours to days/weeks on end due to the larger capacity. But this is also false, because as density increases so has the sequential throughput of the drives. AKA comparing a ~ten year old 4TB drive doing ~100MB/sec vs a modern 8TB drive doing 200MB/sec. There are some non-linearities due to the number of platters and the rate of TPI vs BPI increases but generally platter count is somewhat constant and TPI and BPI are both increasing. Now if your hitting internal RAID limits because your running modern drives on old controllers which cannot rebuild at the drives peak throughput that is a different problem.
Bottom line, your doing it wrong if RAID5/6 isn't increasing both actual as well as theoretical availability. The MTBF numbers are published to give a baseline on what the expected failure rates are.
Well if the warranty is any good you don't have to trust them. If the drive fails under expected conditions within a 5 year window, execute the warranty policy and get a replacement.
> and a 5 year warranty.
I would bet a 5 year warranty means they trust their hardware. Still, when you have so much data and you really care about it, you should use a filesystem that prevents bit-rot and can deal with multiple drive failures.
And backups. Never forget the backups.
I would bet a 5 year warranty means they trust their hardware. Still, when you have so much data and you really care about it, you should use a filesystem that prevents bit-rot and can deal with multiple drive failures.
And backups. Never forget the backups.
A 5 year warranty means:
the expected failure rate over 5 years, times
the percentage of customers who claim warranty on those drives, times
the future price of an equivalent drive
is less than their general profit margin on the drive.
Let's say that the price is $600, the marginal cost of a drive is $400, and the expected 5 year failure rate is a whopping 20%, with 50% of buyers claiming warranty.
That allocates $40 of costs over the next five years to warranted replacement, against a current profit of $200.
the expected failure rate over 5 years, times
the percentage of customers who claim warranty on those drives, times
the future price of an equivalent drive
is less than their general profit margin on the drive.
Let's say that the price is $600, the marginal cost of a drive is $400, and the expected 5 year failure rate is a whopping 20%, with 50% of buyers claiming warranty.
That allocates $40 of costs over the next five years to warranted replacement, against a current profit of $200.
The calculation isn’t that simple. It should include the elasticity of sales versus warranty period. For example, if few customers care about warranty periods larger than 3 years, but all customers still would claim a new drive if their drive broke after 3 years, there may be more money in offering 3 years of warranty, even if that means losing some customers.
More is better. Those RAIDs are never satisfied.
Its a pity we don't have anything remotely similar for optical storage. Physical limits and all that.
Its a pity we don't have anything remotely similar for optical storage. Physical limits and all that.
More is not better. Repair time gets slower as the ratio for transport speed to capacity gets lower and lower. This means that more file pieces can be lost during the longer repair window resulting in possible data loss.
Keep in mind that when people calculate lifetimes and repairs they usually assume that drive failures are independent. Given that they are installed at the same time and usually from the same manufacturer and bought at the same time (e.g. same batch), it's not great to assume that all these things are independent.
Keep in mind that when people calculate lifetimes and repairs they usually assume that drive failures are independent. Given that they are installed at the same time and usually from the same manufacturer and bought at the same time (e.g. same batch), it's not great to assume that all these things are independent.
Yeah, what happened to all those 3d optical storage startups that promised us petabyte sized things by now?
Possibilities :
Worked & couldn't gain traction. Capital requirements too great. Eg just a lack of funding. Limited prototypes produced but not mature enough. Tech is quietly smouldering while it is passed from company to company. Slow progress.
Worked & got absorbed. Now in some black ops lab quietly still in development with protoypes already in limited use. A limited commercial version is 5, 10 or 20 years away. The restricted version in use within 2-5 years.
Worked & still in commercial development. Fundamental issues are blocking major progress. Possible confusion over usage priorities is diluting resources, eg they are optimising for one or two use cases and those aren't particularly helping guide development, yet those use cases are also where funding is available. 10-30 years.
Don't work & too unreliable. R&D couldnt get a stable and reliable product due to fundamental issues.
Don't work & unreliable & not enough cash. The prototypes sort of worked but had issues and on top there wasn't enough capital to iron out the issues in the early prototypes.
Other options also exist. The timing started 10 years ago. Should start to see results, if any, from the "worked" options sometime this or next decade if that happened and to whatever extent. A lazy web search still shows some names. A few have quietly disappeared.
Worked & couldn't gain traction. Capital requirements too great. Eg just a lack of funding. Limited prototypes produced but not mature enough. Tech is quietly smouldering while it is passed from company to company. Slow progress.
Worked & got absorbed. Now in some black ops lab quietly still in development with protoypes already in limited use. A limited commercial version is 5, 10 or 20 years away. The restricted version in use within 2-5 years.
Worked & still in commercial development. Fundamental issues are blocking major progress. Possible confusion over usage priorities is diluting resources, eg they are optimising for one or two use cases and those aren't particularly helping guide development, yet those use cases are also where funding is available. 10-30 years.
Don't work & too unreliable. R&D couldnt get a stable and reliable product due to fundamental issues.
Don't work & unreliable & not enough cash. The prototypes sort of worked but had issues and on top there wasn't enough capital to iron out the issues in the early prototypes.
Other options also exist. The timing started 10 years ago. Should start to see results, if any, from the "worked" options sometime this or next decade if that happened and to whatever extent. A lazy web search still shows some names. A few have quietly disappeared.
Working for "Government agencies"? (Assuming they haven't gone broke)
Related:
Continuing Hard Disk Drive Innovation from Western Digital - https://www.youtube.com/watch?v=jPoS94iVC0c
Continuing Hard Disk Drive Innovation from Western Digital - https://www.youtube.com/watch?v=jPoS94iVC0c
Would these be overkill for a personal NAS at home?
Yes. Give it 3-6 months and you’ll see perfectly fine but non-spec-rated HDDs sold as EasyStores, etc sold at 40% off.
It will be on the market soon. They will have to sell those binned parts somehow.
It will be on the market soon. They will have to sell those binned parts somehow.
Here's the way I figure:
1) If you're a student, yes. Store less stuff. 1080 instead of 4k video won't kill you.
2) If you're a software engineer, factor in at least $200 of your time for each upgrade. Bigger drives mean you need to upgrade less often, so no, buying more is not overkill.
1) If you're a student, yes. Store less stuff. 1080 instead of 4k video won't kill you.
2) If you're a software engineer, factor in at least $200 of your time for each upgrade. Bigger drives mean you need to upgrade less often, so no, buying more is not overkill.
Do software engineers really put dollar values on all of their time?
When figuring out build vs buy decisions, it helps anyone to put dollar amounts on their time.
“Should I buy a lawn mower or pay someone to cut my lawn?” You factor in the cost of equipment, maintenance, and your time for both options, then figure out which works for your budget.
If your budget is zero, that also helps make the decisions easier.
“Should I buy a lawn mower or pay someone to cut my lawn?” You factor in the cost of equipment, maintenance, and your time for both options, then figure out which works for your budget.
If your budget is zero, that also helps make the decisions easier.
It's a bit of an alien way of thinking to me because I don't know how to put a value on my free time, as it is not fungible. That time cannot be always be freely exchanged for time doing something else specific, or more productive. I certainly can't monetise that time easily, and if I could its value would probably not be related to my professional salary.
If I always spent money to save myself time in line with how much I am professionaly worth (eg. paying for a lawn mower, never cooking for myself etc.) I think I would quickly be broke.
If I always spent money to save myself time in line with how much I am professionaly worth (eg. paying for a lawn mower, never cooking for myself etc.) I think I would quickly be broke.
The way I used to value my free time was by figuring out how many hours I would have to work to pay for something.
A two hour movie would be one hour of work. Probably worth it.
A trip to Disneyland was about 10 hours of work, but if I got there when the park opened I could do 16 hours of entertainment, so it was probably worth it.
Paying someone to mow the lawn was a few hours of work, but I could mow the lawn myself in an hour. So not worth it.
And so on.
A two hour movie would be one hour of work. Probably worth it.
A trip to Disneyland was about 10 hours of work, but if I got there when the park opened I could do 16 hours of entertainment, so it was probably worth it.
Paying someone to mow the lawn was a few hours of work, but I could mow the lawn myself in an hour. So not worth it.
And so on.
I personally think the entire calculation of how much your free time is worth is pretty stupid.
There are things that I am willing to pay for even if it was cheaper to do it myself. There are also things that I'm willing to do myself even if it is cheaper to buy it.
You only do this calculation because you don't have any free time left so either you put it off until you can or you pay.
There are things that I am willing to pay for even if it was cheaper to do it myself. There are also things that I'm willing to do myself even if it is cheaper to buy it.
You only do this calculation because you don't have any free time left so either you put it off until you can or you pay.
You should probably factor in the enjoyment and experience gained - doing repairs or building something yourself, for example, could be more rewarding than the dollar amount.
I do. That said, the value calculation is complex. It's not just annual salary divided by 2000 hours per year worked. It's more based on things like:
a) If I work harder and/or learn more, my salary and career trajectory will go up. There's an integrated increase over the rest of my life.
b) If I do consulting / startups / etc. on the side, what's that worth?
And you want to factor in risk, as well as not having bank account hit zero (even with risk). So if I'm unemployed (broadly speaking -- including planned unemployment like going to school), the expected value of my time hasn't changed, but the risk profile has, so my spending goes to as close to zero as I can bring it until I'm employed again.
a) If I work harder and/or learn more, my salary and career trajectory will go up. There's an integrated increase over the rest of my life.
b) If I do consulting / startups / etc. on the side, what's that worth?
And you want to factor in risk, as well as not having bank account hit zero (even with risk). So if I'm unemployed (broadly speaking -- including planned unemployment like going to school), the expected value of my time hasn't changed, but the risk profile has, so my spending goes to as close to zero as I can bring it until I'm employed again.
The important bit for NAS is redundancy. If you can afford 2+ of these drives to run in RAID1, RAID5 or RAID10, then go on ahead.
For me personally, my 5TB x 2 RAID1 (ZFS really, so mirrored) configuration with 5TB usable is enough for my personal uses. If I need more space, I'll configure my ZFS with +2 HDDs for another pair worth of space+redundancy.
For me personally, my 5TB x 2 RAID1 (ZFS really, so mirrored) configuration with 5TB usable is enough for my personal uses. If I need more space, I'll configure my ZFS with +2 HDDs for another pair worth of space+redundancy.
To add to that, one will very quickly find that RAID5 really isn't a good idea for big drives if one reads around.
The reason being that for larger drives, resilvering can take so long that secondary failures become probable or—in bad cases—even virtually guaranteed (This is the same probably the WD SMR drives had, but it exists even on normal drives if they get big enough).
For this reason, for modern large drives, most people recommend at least RAID6 (known as RAID-Z2 on ZFS), which means you have two drives worth of parity as compared to RAID5's single one.
Of course, this means that you need at least four drives for it to be worthwhile, and most people realistically end up going for around six disks, otherwise you are spending a lot of your total cost on parity.
The reason being that for larger drives, resilvering can take so long that secondary failures become probable or—in bad cases—even virtually guaranteed (This is the same probably the WD SMR drives had, but it exists even on normal drives if they get big enough).
For this reason, for modern large drives, most people recommend at least RAID6 (known as RAID-Z2 on ZFS), which means you have two drives worth of parity as compared to RAID5's single one.
Of course, this means that you need at least four drives for it to be worthwhile, and most people realistically end up going for around six disks, otherwise you are spending a lot of your total cost on parity.
If you can only afford 2 then I decided against RAID/ZFS. The issue is that "raid is not backup" and I fear user error as much as hardware error so instant mirroring would be just as bad.
For my home bulk storage, I just have two normal 14TB drives and a nightly cron to rsync archive (new files only) from the primary to the backup. This gives me enough of a backup for both user error and hardware failure. It doesn't protect against a house-fire but that is ok.
I should maybe have considered BackBlaze but I can't believe they are going to last long if they continue offer flat-rates for multi-TB attached NAS.
For my home bulk storage, I just have two normal 14TB drives and a nightly cron to rsync archive (new files only) from the primary to the backup. This gives me enough of a backup for both user error and hardware failure. It doesn't protect against a house-fire but that is ok.
I should maybe have considered BackBlaze but I can't believe they are going to last long if they continue offer flat-rates for multi-TB attached NAS.
> If you can only afford 2 then I decided against RAID/ZFS. The issue is that "raid is not backup" and I fear user error as much as hardware error so instant mirroring would be just as bad.
Mirroring is for preventing hardware failures.
ZFS Snapshots are one protection against user-error. Backups are another.
Mirroring is for preventing hardware failures.
ZFS Snapshots are one protection against user-error. Backups are another.
I was tempted by ZFS but it just felt like too much trust to put in something complex as a novice. Just reading about it led me to too many data-loss stories. I have had enough corporate IT teams tell me their RAID went down with total data-loss and their tape drive recovery isn't working to find a lot of virtue in "dumb-as-hell" infrastructure design!
Most of the information surrounding ZFS online is sadly noise, not signal. And most of that noise is "wow so enterprise storage infrastructure", which breaks down into a) Here's All The Complicated Things I'm Gonna Do And How I'm Gonna Do Them and b) I Broke It, Help.
So it seems dizzying and unstable, but the dizzying bit is just nerd ego and hype, and the instability is also just nerd ego meets inevitability. (Now I think about it, it's a bit reminiscent of the abandoned "ABC shall be the ultimate XYZ" software you sometimes find online, and scriptkiddie mentality.)
ZFS is just a reasonable, mostly solid, somewhat idiosyncratic filesystem with nice features. Most of the failure noise online is from people who have no idea what they're doing; there are curiously (good-suspiciously) few public reports of commercial-scale data loss et al.
I've taken to using it as the root filesystem on a couple of Debian boxes. The Debian install instructions are both overblown and also almost hilariously broken, but that's a devops/sysadmin documentation failure, and not ZFS' fault. I spent a few years on Slackware, and have found the ZFS setup experience reminds me of that rock-solid-meets-flying-by-the-seat-of-ones-pants mentality.
So it seems dizzying and unstable, but the dizzying bit is just nerd ego and hype, and the instability is also just nerd ego meets inevitability. (Now I think about it, it's a bit reminiscent of the abandoned "ABC shall be the ultimate XYZ" software you sometimes find online, and scriptkiddie mentality.)
ZFS is just a reasonable, mostly solid, somewhat idiosyncratic filesystem with nice features. Most of the failure noise online is from people who have no idea what they're doing; there are curiously (good-suspiciously) few public reports of commercial-scale data loss et al.
I've taken to using it as the root filesystem on a couple of Debian boxes. The Debian install instructions are both overblown and also almost hilariously broken, but that's a devops/sysadmin documentation failure, and not ZFS' fault. I spent a few years on Slackware, and have found the ZFS setup experience reminds me of that rock-solid-meets-flying-by-the-seat-of-ones-pants mentality.
Spend an hour setting up a ZFS on old hardware. Play with snapshots and manually delete test documents until you are confident on how the feature works.
Highly recommended! One can even learn about, and experiment with, ZFS using files sitting on a disk by setting the files up as loopback devices. For example:
https://dev.to/nikvdp/https-nikvdp-com-post-zfs-part1-31fj
https://dev.to/nikvdp/https-nikvdp-com-post-zfs-part1-31fj
Hmm... that's a good idea.
It may be simpler to play with ZFS on VMs. The VM-disks will be small and inefficient, but "good enough" to prove whether or not the system works.
Not that I've tried any of this before. I happened to have an old computer lying around with ~4 small hard drives that I messed with to learn the ins-and-outs of ZFS the first time I used it. (Shuck hard drives, pull them out of laptops, etc. etc. Its not really that hard to get 4 hard drives to play with, especially if you ask your friends for ancient laptops with 200gb drives or something)
It may be simpler to play with ZFS on VMs. The VM-disks will be small and inefficient, but "good enough" to prove whether or not the system works.
Not that I've tried any of this before. I happened to have an old computer lying around with ~4 small hard drives that I messed with to learn the ins-and-outs of ZFS the first time I used it. (Shuck hard drives, pull them out of laptops, etc. etc. Its not really that hard to get 4 hard drives to play with, especially if you ask your friends for ancient laptops with 200gb drives or something)
ZFS is very robust, especially in a mirrored setup. Compared to your sync setup, with ZFS you could have snapshots, which would let you rewind back to earlier versions of files. This is a good mitigation for ransom ware attacks. You also get bit-rot protection, where you know what you read is what you wrote. Without this protection, if a file gets corrupted on your primary drive, this corrupted file will get synced to your backup drive. ZFS helps with both of these and a mirrored setup would give you a boost for read IO speeds.
Of the many ways storage can get complex, I've found ZFS to be one of the most straightforward.
Of the many ways storage can get complex, I've found ZFS to be one of the most straightforward.
Yeah I've run a home fileserver at home before and it's just not fun continually worrying about updates and having to maintain a bunch of monitoring if some cron job broke or something.
They're not exactly cheap, but I'm another one who went the Synology route. They slap a user-friendly UI ontop of linux mdraid and BTRFS, so you can RAID together a few drives, set up automatic BTRFS snapshots with retention rules (keep 12 hourly snapshots, 7 daily, 3 monthly, etc), set up disk scrubbing schedules etc all just by clicking around.
They're not exactly cheap, but I'm another one who went the Synology route. They slap a user-friendly UI ontop of linux mdraid and BTRFS, so you can RAID together a few drives, set up automatic BTRFS snapshots with retention rules (keep 12 hourly snapshots, 7 daily, 3 monthly, etc), set up disk scrubbing schedules etc all just by clicking around.
Do you have 12TB of data right now? Yes? Then a minimal RAID would need at least two 16TB drives.
The performance specified in the article lists only its sustained transfer rate, with no mention of sustained read speed, sustained write speed, random 4k reads, random 4k writes, or any other useful performance metric. The caches are freaking enormous- 512MB.
Am I to understand that these are SMB disks? The article does not mention SMB at all, which is surprising to me given WD's recent history.
Am I to understand that these are SMB disks? The article does not mention SMB at all, which is surprising to me given WD's recent history.
Whoever has the need to store that much data?
It's really not that hard.
If you're a photographer for example, storage gets filled up VERY VERY quickly. A wedding photographer may do 1000-6000 photos in a weekend, 40-100MB each; which is 40GB - 500GB per event, depending on your workflow.
Video even more so - whether it's an entertainment library, or high quality videos of family, let alone if you do it professionally or semi-pro.
Home labs for anything - virtualization, data science, ML, CI/CD, etc.
My current NAS has 8TB drives in it, but only because of the price curve. I would've preferred 12TB but 8TB is where the price/capacity/performance cusp seems right now.
If you're a photographer for example, storage gets filled up VERY VERY quickly. A wedding photographer may do 1000-6000 photos in a weekend, 40-100MB each; which is 40GB - 500GB per event, depending on your workflow.
Video even more so - whether it's an entertainment library, or high quality videos of family, let alone if you do it professionally or semi-pro.
Home labs for anything - virtualization, data science, ML, CI/CD, etc.
My current NAS has 8TB drives in it, but only because of the price curve. I would've preferred 12TB but 8TB is where the price/capacity/performance cusp seems right now.
Yep, I’m a photographer and I’ve got 4x8TB drives in a NAS and it takes some work to make sure it doesn’t fill up. I like to keep all my wedding raws for a year, and I keep all the raw files I haven’t culled long term.
interesting - what do you think of keeping the best photos on NAS and moving the rest to tape? LTO-8 is ~$10/TB retail and lasts for decades...
Tape also has other overhead... like you need a reader and if you’re operating at the scale where the cost of the tape is a deciding factor, then you probably also would like to have an auto loader, rack mounted gear, etc. Not to mention the different software needed and keeping track of what is on tape (which tape?), vs disk is a lot to manage.
For many, just keeping multiple HDDs around, even if they are slightly more expensive is the simpler option.
For many, just keeping multiple HDDs around, even if they are slightly more expensive is the simpler option.
What's the actual cost of ownership?
I have never found a drive + tapes system that made financial sense, but I'm willing to be shown the error of my GoogleFu ways :-)
Software engineer who deals with large datasets, disk images, etc etc. Throughout the years I've had to have a big "scratch disk" with yet another bigger "archive" to support my work.
For my scratch disks I need at least 1TB locally (preferred 4+), and for my archive I need 16TB easy. I deal with everything from shape files, to GIS, to huge flat-file processing etc etc. I'm the guy who can take 10+ years of data and make business sense of it so storage is a constant headache/need.
To get ahead of someone telling me, "you're doing it wrong": no. No I am not. Just because you don't have the need doesn't mean there's not a market for it.
For my scratch disks I need at least 1TB locally (preferred 4+), and for my archive I need 16TB easy. I deal with everything from shape files, to GIS, to huge flat-file processing etc etc. I'm the guy who can take 10+ years of data and make business sense of it so storage is a constant headache/need.
To get ahead of someone telling me, "you're doing it wrong": no. No I am not. Just because you don't have the need doesn't mean there's not a market for it.
I have something like 5TB of tv shows and movies on a Plex server since I hate swapping in DVDs and Blurays for the stuff I watch more than once and my collection isn't even really that big.
But anyone with a large video library could easily use up a ton of storage. I've filled up my external drive used for Plex, it's upgrade time for me and I'm going for a 4x8TB NAS and hopefully that'll keep me running for at least a few more years.
But anyone with a large video library could easily use up a ton of storage. I've filled up my external drive used for Plex, it's upgrade time for me and I'm going for a 4x8TB NAS and hopefully that'll keep me running for at least a few more years.
Very useful for storing medium/large-scale crawls. Common Crawl is 200TB+ per month (http://commoncrawl.org/connect/blog/). Uncompressed, if compressed it'd probably go through one such disk every 1-2 months. My own crawling operations consume about half a TB each month.
Storing high resolution (1080p/4K) videos is another big spacehog
Storing high resolution (1080p/4K) videos is another big spacehog
Animation studios. I worked for one that produced content for Netflix and they were struggling with space issues. With just over 1PB of storage data, it gets tricky to fit more hardware.
You have to keep archival data in case the studio decides to comes back, not forgetting you have to keep revisions, the raw formats, the rendered formats, the render raws, audio raw, audio mixed.
Television Studio's and Broadcasting Companies all occupy lots storage too.
You have to keep archival data in case the studio decides to comes back, not forgetting you have to keep revisions, the raw formats, the rendered formats, the render raws, audio raw, audio mixed.
Television Studio's and Broadcasting Companies all occupy lots storage too.
Archivists and people who enjoy large collections of data, who are providing a useful service to preserve internet history.
When I bought my first computer it had a 200MB hard disk drive. One of my former professors told me I was nuts, that I would never fill that much disk space.
2002 called and wants its argument back
Anybody who wants to run the latest CoD render pack.
Can't think of a single company that might want to store 18 TB on a single disk? These are enterprise-class products after all.
Working on a security solution right now and I can 100% tell you I can think of multiple companies that need an on-site 18TB datastore, even "small" companies. Also have worked with tons of creative agencies. Video, photo, etc all can take huge amounts of disk to safely store.
Sorry to disagree but your assumption that these are "enterprise-class" products is incorrect. There are lots of business needs that require large on-prem storage solutions without being an "enterprise-type" business.
Edit: downvotes? Really? I'd really love to see someone make a case against this.
Sorry to disagree but your assumption that these are "enterprise-class" products is incorrect. There are lots of business needs that require large on-prem storage solutions without being an "enterprise-type" business.
Edit: downvotes? Really? I'd really love to see someone make a case against this.
>Sorry to disagree but your assumption that these are "enterprise-class" products is incorrect.
Well, it literally says Enterprise Class SATA HDD on the drives.
Well, it literally says Enterprise Class SATA HDD on the drives.
"enterprise class" is a standing term in HDDs for more expensive and more reliable drives, and literally what WD labels those drives. It doesn't mean "small companies never have workloads that need those" (although many might want to wait for cheaper disks of the same size), and the parent didn't suggest so. It seems a bit weird to complain that they worded their answer to restrictively for your liking, instead of just expanding with your experience with specific examples.
Even for home use, if you have a family who wants to keep local backups of multiple devices. A phone, laptop and tablet per person for even two people would eat through a few TBs every few years.
It's definitely a thing. Media is one use case, logging is another.