> Do we know if these drives are the same as you would purchase in a retail outlet
Seagate (for instance) won't sell ANYBODY drives directly, so they force us to get bids from various resellers and distributors. So when we pick the lowest price, I don't think Seagate knows who the drives are going to, but there might be a trick in there somewhere I am not aware of.
Disclaimer: I work at Backblaze, and I was here first.
> I'd worry they would sue for...
If you are referring to Backblaze, we're not going to "sue" anybody for anything.
We (Backblaze) have dealt with a bunch of frivolous lawsuits (and patent trolls) over the years suing us, and OMG we're not going to instigate any lawsuits over some honest person legitimately reporting some issue and being helpful. It isn't going to happen.
Our reputation is important to us. Not just for Backblaze: I'm saying the individuals that founded Backblaze and those people that work now here base our entire existence and careers and the number one marketing efforts at Backblaze are based around we are trying to be "the good guys" and transparent and acting like it. There is no possibly world where we try to suppress a screwup like this through legal means. That would be a PR debacle of epic proportions.
If something went wrong, let's shine a spotlight on that cockroach and figure it out together. I'm not sure the exact drive we are all talking about, but my first guess would be a customer ordered a $189 "USB Restore" all their data shipped to them on an encrypted drive) and we (Backblaze) shipped the customer a USB restore drive and they are subsequently selling it (after copying their restore off of it) on the open market. If it is above 8 TBytes this is absolutely *NOT* the case and we should get to the bottom of it. Without lawyers mucking up the situation.
I think most people assume the SSDs are more expensive, but depending on their application they choose SSDs for the OTHER properties, like SSDs are quieter, and faster, and like you mention use less power.
There was this narrative for a long time that SSDs failed way more often, or beating them up with reads and writes would make them fail while hard drives could take the same number of reads and writes and not fail. So a subset of people chose hard drives for durability regardless of cost. Personally I never saw any evidence of SSDs failing more often than hard drives, and PLENTY of counter evidence. For instance, I wrote a program that unintentionally overwrote the same fixed length 16 byte file in the same location over and over again too often, and it failed more on drives than SSDs. (The failure mode was not whole disk failure, the 16 byte file itself mysteriously stopped being writable on some computers after a long time of working.) I had some friends that worked at Tivo and they said they had the same bug/shortcoming early on. The Tivo would overwrite the same section of disk over and over and it failed LONG before the drive manufacturers said it would fail. The drive manufacturer said they needed to vary the location of where they wrote the data to get anywhere near the published numbers.
I had assumed the file system layer would take care of that kind of management for me, but I guess not so much.
> Capitalism does not reward hard work. Smart work, maybe, but not hard work.
I'd say a person successful/rewarded in a capitalistic society (defined by accruing more wealth than they started with and not just spending down some large family inheritance they were born into) probably has some combination of: luck, smart (intelligent) decisions, hard word, probably has a good "setup" going into the working world like parents didn't abuse them growing up and at least helped pay for their education, etc. Maybe that last one is just a subset of luck, we certainly do not choose our parents.
Sure, you can be missing one of the above list and do Ok, but probably not two. Statistically a smart, hard working person who had the world's crappiest upbringing probably won't be that successful compared with somebody who had all three going for them. But if you look at 20 "success stories" it is extremely rare the person didn't work hard. I think it's an important component of being rewarded in capitalism. It does a disservice to people to say "slacking off or working hard results in identical outcomes".
To support at least half of your point, you cannot be stupid and work hard and be successful. But a smart person who works hard will most likely do better than a smart person who doesn't work hard.
Short summary is there are a few SMART stats that seem to predict failure way more than others, which is probably obvious. But we aren't PhDs in statistics and it isn't our area of focus, so....
I can't comment on their findings, but it's DEFINITELY an interesting thing to study now that we have almost 10 years of these drive stats across a pretty big drive farm.
Disclaimer: I work at Backblaze but more on the backup client side that runs on laptops, not in the datacenter storage side.
> Is there a failure list for write heavy drives?
To be clear about what these drive failure stats are and what they are not: Backblaze runs a data storage service with about 214,000 hard drives in it right now. We don't run any specific tests or induce issues on purpose FOR the drive failure stats, we just report what occurred in our datacenter.
Sometimes readers think we're carefully running a "study", but it's more just what we have experienced as honestly as we can offer it up. If the reads and writes and seeks our drives experience matches your particular application, great! Or maybe it is just interesting to read.
Now, we do save (and publish) all the raw data, and some other awesome people out there have done various analysis on it, which always makes us happy also. You can find the raw data here: https://www.backblaze.com/b2/hard-drive-test-data.html At this point it goes back almost a full decade.
You have to zoom in to see the meaningful information. It can be super surprising to developers and IT people (like us) to see a lot of customers have less data than you might think.
When we started we had no idea what might come of all of this, so it was "stressful". :-) So I welcome another backup client developer to our club, and hope this can help them out.
Disclaimer: I'm the engineer you are commenting on, I just want to straighten out one mis-understanding.
> This quote especially seemed childish and immature to me:
>
> > "And this one makes me actively angry, because both Microsoft and Apple will happily throw away portions of your files and not tell you about it"
>
> No, this is NOT Microsoft's or Apple's fault. It is 100% HIS fault for not understanding what's going on. Even if a file flush is correctly requested
Several times you seem to jump to the assumption that I don't understand fsync and disk flushing and that is the core issue. You aren't understanding what I'm criticizing. Here is an example of what bothers me:
You take a picture at your wedding, and you store it on your hard drive. You like the photo, it means a lot to you, and you use it as the background for your desktop FOR FIVE YEARS. You have rebooted hundreds of times, and it's always the background for your desktop. Then one day 5 years after your wedding, you reboot your laptop, and it seems to take a little longer to boot, and then after you sign into the laptop half the image you use for your desktop background is scrambled. The middle of it looks like dirty snow. You didn't get any reports of any issues from the OS manufacturer, but now one of your photos is corrupted.
This isn't because the software that wrote the photo 5 years earlier forgot to flush the picture to disk. It just isn't. Behind the scenes, as your laptop was booting from an aging drive, it probably encountered some issue, and it went about fixing the problem as best it could - which I have no problem with. My issue is the drive lost some data, and if the OS manufacturer would let you know this occurred you could take useful actions like order a new drive, prepare a restore from a few weeks ago before that issue occurred, etc.
> No, this is NOT Microsoft's or Apple's fault.
It isn't their fault that the drive is going bad, I agree. Drives go bad, that's why we have backups. My issue is the OS manufacturer try to cover up too much, keep too much hidden from the user, and didn't let the user know data loss has occurred (or might have occurred). And yes, I hold them accountable for not telling customers what is going on. It isn't anybody's "fault" that it occurred, but there is a responsibility to let customers know about it so the customer can take appropriate actions so they don't lose data (or more data).
I try to write incredibly paranoid software. Part of the reason is that is the "average" environment the Backblaze client runs in is more unstable than what most software developers are used to. The whole point of backups is to run when the computer is going sideways, it has bad RAM, it's losing disk sectors, or a customer's cat likes sleeping on the keyboard because it's warm, and the fans are clogged with cat fur. And because the family has teenage children that don't know about computer security problems, they download and install unstable junk from all over the internet because why not? That's the environment my software runs in, and I take my job of trying to protect my customer's data very seriously.
Disclaimer: I work at Backblaze so I'm biased and you should keep me honest.
> if you run into big trouble, you can go get the box back for a really fast restore
Backblaze provides this service for our customers! Customers can ask for an 8 TByte encrypted USB attached external hard drive to be prepared in the Backblaze datacenter with all their data beautifully restored on it, then we FedEx this anywhere in the world to them. This is a free service as long as the customer returns the USB drive to Backblaze in a reasonable amount of time (a couple months, and we can work with customers if they need longer). Or customers can pay $189 (which includes the drive, the data, and world wide shipping) and keep the 8 TByte USB drive.
What we often see is that if a customer's laptop is stolen or crashes, they sign into the Backblaze website to download the 3 or 4 individual files they were working on when the laptop was stolen. Let's say that is a term paper due the next day. That way they are back up and running IN SECONDS. Then the customer orders a free USB drive with 8 TBytes of their data which will show up in a few days. They can live without their wedding photos and their music for 3 days, but that term paper has to be handed in.
Yeah, that would work really well. When you look at the "services" control panel in Windows, there are two columns. One column is "Name" of the service, and another column is a longer explanation with the column header "Description". I put a small description in there for bzserv (our service) plus a URL to our company website. I think this is just being polite, customers who don't recognize what "bzserv" is can immediate find more information on it.
Thanks, that was super exiting for us. After 14 years, I claim (and this is controversial) that we're no longer a startup and now we're just a mid-sized publicly traded company. :-)
> process-based firewalls like Little Snitch, because each copy needs to be configured separately
Yeah, that was actually a surprise and unfortunate. What the Mac architect (one of my business partners) and I think is that now that it is nice and stable, we might go down to 1 or 2 bztrans_thread executables, and one bztransmit. That seems like a better tradeoff where we waste much MUCH less disk space, and it is only 3 executables to allowlist in Little Snitch, and it achieves basically what we want now that it's stable and working well.
Originally there were 10 threads MAXIMUM, and we made 10 copies. And each copy was linking with shared libaries so it was only 10 MBytes of disk space which nobody noticed. Then Windows lost their friggin' minds with one of their releases and forced us to link statically which bloated it way up to 5 or 10 MBytes per executable. Then we went to 20 threads maximum and the whole thing was silly. When we went to 100 threads maximum we said "enough" and went to mod 20 for re-using executable names.
By the way, ALL OF THIS could be avoided if Microsoft and Apple provided an API to set the name displayed in Task Manager/Activity Monitor. Maybe that's a security issue, I don't know. But frankly wouldn't it be SUPER TOTALLY USEFUL if chrome displayed the current web page loaded in the process name of each and every chrome process? Then you would know which one to kill when something goes sideways.
I'm not aware of even Windows 10 or Windows 11 being able to create Symlinks on FAT32 and exFAT, but I could be wrong? But it doesn't matter, the code as written is cross platform, it works on Apple Journaled File Systems, APFS, exFAT, etc. We then compile it for Macintosh, compile it for Windows, and compile it for Linux.
We can then spend extra time and carefully detect each filesystem and each platform and then make the optimization if we can. And this is a valid criticism that we have not done this yet. But no matter what we need this general code that will always work FIRST, what the links are is a space optimization to save valuable SSD space when it is possible.
> > Backblaze ships with 21 identical copies of the same executable
>
> This seems needlessly dismissive. I feel like they definitely know that you can execute the same binary multiple times.
I'm the programmer at Backblaze that made the copies on purpose, I wrote some extra code to do this, and it's meant to help us debug certain things. Yes they are identical, the installer only ships with one copy of the executable, the installer then makes the copies on purpose. I get to explain this from time to time. :-)
In Windows when you want to know what is going on behind the scenes, you can bring up Task Manager and look at the different names of the different processes that are running. On the Macintosh this is called Activity Monitor, same sort of thing. The different names for the executables are for different "threads" which have different roles. Backblaze is multi-threaded to get higher performance.
The parent coordination process is called "bztransmit". But when doing the actual transmission it spawns the bztrans_thread01, bztrans_thread02, bztrans_thread03, etc.
So BEFORE I made multiple copies of the executables with different names, a customer would say "bztransmit is hung" or "bztransmit is using up too much memory". There was very little visibility into this. But now that I made multiple copies with different executable names, when the same customer says "bztrans_thread03 is hung". Or they say "bztransmit is using too much memory". We immediately have narrowed down what to look at.
Here is a screenshot showing what "Chrome" looks like to me in Windows, and how it compares to how Backblaze's bztransmit looks like to me in Windows: https://i.imgur.com/KOJHJ9Q.jpg In that screenshot, you can see there is the "main thread", and "worker threads". Meanwhile chrome is just one big list of processes all named the same thing (see the screenshot). I prefer the Backblaze system, but I understand it upsets some customers that prefer the chrome experience.
That's it. It's not some huge mystery.
One question asked here was do we know you can launch the same executable twice? Yes, and we do that. The bztrans_thread05 is launched for thread 05, thread 25, thread 45, thread 65, etc. It's THREAD_NUMBER mod 20. Here is what it looks like to hit 500 Mbits/sec upload speeds, this isn't photoshopped, it's a real screenshot on my development computer: https://i.imgur.com/hthLZvZ.gif
Another question is: why not use hard links or symbolic links? That's the only real optimization possible here, everything else was on purpose. The answer is not an excuse, it's just an explanation if you are curious. The software we develop at Backblaze is cross platform, so what we like to do is make the most general form first that will always work, then if customers complain or we want to refine it we special purpose code in per file system or per platform. The most general thing to do is make full copies. We could then go on to make links on the Mac WHEN POSSIBLE and the equivalent on Windows WHEN POSSIBLE, but it never became a large priority. The reason I can't use one technology is we support several file systems, and not all of them are the same or support the same technology for links.
Every feature we have is the result of prioritizing it over working on other things. Until recently, we did not have a lot of funding or an infinite supply of programmers, so we had to choose what order to implement each feature in. I'm not saying we got all the priorities correct, or that we did things in the correct order. I don't really even think there is one correct order. For example, some individual home user customers prefer saving 180 MBytes of their valuable boot SSD space over me implementing single sign on for our corporate customers. On the other hand some corporate customers DEMANDED single sign on or they wouldn't purchase the product at all. They are both correct, but there is only one of me, so we made some judgement calls and left the multiple copies and worked on single sign on. Some customers were happy, some are miserable.
We do have open client recs for both Windows and Mac programmers, so if you would like to make a good salary, full benefits, and help us out, come join us! :-)
> ... 4 - 6 words ... very conservatively 160 characters is probably going to fit
Gotcha. I'll file a Jira ticket to see if we can easily and quickly raise it to 200 characters (round number). I think we can handle that easily and safely.
> standalone hardware token support (like Yubikeys or Nitrokeys).
I'll run the idea by a few groups and see what people say. For feature requests NOT related to security, traditionally it's kind of a customer voting system. If enough customers want a certain feature it sorts to the top of the priority list and gets done sooner.
For security things it is obviously different, like if a big exploit like Heartbleed is released in the world we drop everything else and focus on that.
This is sort of somewhere in the middle, it's security related so it gets some additional boost up the priority list for that, but it's also a security feature not all customers would choose to use, so it is still affected by whether our sales and support teams are hearing requests for it.
Ha! That's my half second of viral fame right there.
You have no idea how blindly I stumbled into making that webpage. I haven't ridden bicycles more than a handful of times since college, and something possessed me to start taking photo after photo of bicycles without much thought. I was just travelling home from Europe to the USA with a short layover in Amsterdam when I took them. When I got home I stared at this unorganized pile of photos of bicycles and almost deleted them. I honestly thought nobody would ever, in a million years, care about a webpage about bicycles. I blasted that web page out in one continuous stream of tongue in cheek caffeine fueled frenzy, threw it up live, and went to bed.
About 3 or 4 months later my website couldn't serve enough pages to keep up with the load. I called my ISP and they said, "looks legit", I said "not possible, my website is at the ass end of the internet, nobody ever goes there". I bought an analytics package from them just to prove them wrong. All the traffic was to one web page: "Amsterdam Bicycles". A page about bicycles. (sigh)
Totally by accident, without intending to or knowing what I was doing, I stumbled upon several groups of people that are just PASSIONATE about bicycles. First of all, unknown to me there are people that are really, really into bicycles. They have mailing lists, they have websites. Who would have known?
But the group that just floored me was the Dutch people themselves. They are beyond passionate about bicycles, and there is a rich and amazing history behind it. For example, in World War 2 the Germans confiscated all the Dutch bicycles, and there is literally still a saying (in Dutch) that loosely translates to "I want my bicycle back" as an inside joke about still being resentful about this affront. And overlay that with the Dutch actually seem to have a great sense of humor and sense of self, and don't mind it if somebody pokes a little fun at them and they laugh right along. I wasn't disrespectful and mostly just expressed confusion over the situation which was completely honest at the time. So the city of Amsterdam sent me a T-shirt, a bicycle bell, and a bicycle light, and clippings from local newspapers about my silly web page. They were just so over the top great about it.
And the best part was the people from the photos reached out to me, like the woman pedaling here: https://www.ski-epic.com/amsterdam_bicycles/pp9b_amsterdam_b... contacted me. In the caption for that photo I asked "Why is the guy riding, and the woman pedaling?" The answer is this: the owner of the bicycle always pedals. OF COURSE I offered to take the photo down if she wanted and she said, "Are you kidding, I love it!"
> Is there a reason there’s a short limit on how long my Backblaze password can be?
The current limit is 50 characters. There are reasons we don't let it get super long like 2,000 characters but we could easily double or triple the maximum length it if it was requested. Is there a number of characters you have in mind? I found a chart here: https://i.imgur.com/trR2u8g.jpg that says a randomly generated 17 character password would take 93 trillion years to crack. The chart doesn't even go up to 50 characters. Backblaze also supports 2-factor authentication via SMS or Google Authenticator codes which personally I would highly recommend to customers also.
> who knows what kind of encryption you guys are using
Backblaze has two product lines: "Backblaze Backup" where we wrote the app that encrypts data on your laptop before uploading into our storage cloud, and "Backblaze B2" where you use any third party tool (or write your own) so the encryption in that case is all under your control. You can see a list of 3rd party tools here: https://www.backblaze.com/b2/integrations.html (make sure you scroll down). There are little pictures of penguins by the software that supports Linux, little pictures of an apple for software that support Macintosh, and a little Window icon for the software that support Windows.
For "Backblaze Backup", we use a well known design where we use symmetric AES-128 to encrypt the file, but each file is encrypted with a different AES-128 key, and that key and an Initialization Vector (IV) is then encrypted by a 2048 bit RSA public/private key. You can read about the encryption flow we use in this blog post: https://www.backblaze.com/blog/how-to-make-strong-encryption...
I am touched you saved my email and remembered me, and that at the time you didn't respond negatively to the message. Thank you for being a customer.
The only thing I regret is that comment about version number of 2.3.1 and the claim that it would run flawlessly for a decade. Oh geez, please upgrade to 8.0.1, you won't regret it.
Disclaimer: I work at Backblaze on the backup client.
> it looks like two sperm fighting over the egg
In 2017, this was proposed as one of several possible animations in the Backblaze client showing you are uploading files from your laptop to the cloud: https://i.imgur.com/L16xkMj.gifv It is an animated Gif, if you don't see it moving.
When I saw it, I had to send an EXTREMELY uncomfortable workplace email about a very personal activity two people do with something dribbling out. A few minutes later I hear from across the room "Well I CREATED it and now I can't unsee it!"
The decision was made to avoid that animation in favor of something else.
> Do we know if these drives are the same as you would purchase in a retail outlet
Seagate (for instance) won't sell ANYBODY drives directly, so they force us to get bids from various resellers and distributors. So when we pick the lowest price, I don't think Seagate knows who the drives are going to, but there might be a trick in there somewhere I am not aware of.