Ask HN: DigitalOcean Destroyed All My Data, Any Legal Recourse?
I will follow up with a long post documenting timeline etc and all correspondence with them (it's not that much since they never respond). I have been a (paying) DigitalOcean user for more than 2 years for personal projects, and recently convinced a start-up that I work for in Tokyo to move off of Heroku and onto Digital Ocean. Everything fine for 2 months, then all of a sudden I get user reports that the site is down. I check and my account is locked and all it says is to submit a support ticket. No warnings of an issue, not even a mail saying that they turned off instances etc, just silently they lock my account. I submit a ticket and they don't resopnd for a week, I submit another, finally after 5 weeks, yes 5 weeks, I get a response saying that I login from too many locations and the lock is now removed, but they destroyed all instances. How can this even be anything close to standard practice to without warning destroy all instances of a paying user for over 2 years?! All databases everything completely gone forever. I guess this goes to show why you should always not trust 1 cloud provider, but this is just so incredible to me that they can on a whim without warning or justification just destroy all your data, is there any legal recourse against this?
27 comments
Hey folks, I'm the CSO over at DigitalOcean (verification: https://keybase.io/custos). I'm going to look into this so we can figure out what happened here. Garland, I've temporarily set up an email alias which is my HN username [@] digitalocean.com - can you please shoot me some info?
Hi jofe. Without wanting you to comment on specific cases, as a customer paying you $1k per month stories like this terrify me, and it's not the first time I've seen something like this on HN. Can you definitively say whether this kind of thing has ever happened in the past and if it could happen again in the future?
Hey drcongo. Things like this occur at every cloud provider and I would never be comfortable promising you it's not a possibility. What I can tell you is that DO invests very heavily in trust and security (moreso than any company of our size that I've seen in my career). While we are far from perfect, we have a ton of controls in-place to ensure things like this don't happen, especially for customers with your type of track record.
Always happy to chat live with you, just open up a ticket or email me at the address above.
Always happy to chat live with you, just open up a ticket or email me at the address above.
I don't know about other providers, but I don't how the handling of any false positive from flagging is to not give any warning beforehand, destroy all instances, and not even give a courtesy mail after destroying all instances and locking accounts. I had to hear from a user that our site was down...(we were also in the middle of an expensive bio related computation that was 3 days in when all was destroyed)
That's a fair response, and I totally take threatofrain's point below too. I guess a better question would have been: is there anything as a customer I can do to prevent this?
Hey drcongo. Jarland from DigitalOcean here. Truthfully, we would have to talk about individual cases to provide a detailed answer. Being aware of who you are and knowing how the details of these cases compare in relation to you, I would say that you already do everything that you should to prevent being caught up in the kind of experience that has you concerned.
I realize that is vague, but it's a bit of a difficult thing to discuss without exposing private data. If I so much as say "Just don't do X" then I'm effectively saying Client A did X. Tough waters to navigate.
I hope that helps a bit at least.
I realize that is vague, but it's a bit of a difficult thing to discuss without exposing private data. If I so much as say "Just don't do X" then I'm effectively saying Client A did X. Tough waters to navigate.
I hope that helps a bit at least.
Perfect, thanks for taking the time to answer.
Hey drcongo, I'll tell you anything you want to know about my case either public or private.
At least in Tokyo I know plenty of people who are running PG on DO in a docker instances and just doing backups on DO cause its cheaper than something like heroku managed PG so if I was one of them (I kept my data layer off of DO) I would be completely destroyed by this. So I tell everyone I know now at meetups about this experience and so happy to share specific details.
They told me I log on from multiple locations (OMG I have a laptop and work while I travel, busted) and that I had a gmail address on my account.
The most agregious thing is that if their trigger happy "security and trust" flagger flags you, there is no warning where they reach out to you, they just destroy your instances, don't tell you about that either until you logon and are asked to file a ticket if you want your account unlocked...then a month later will unlock it for you but everything nuked.
At least in Tokyo I know plenty of people who are running PG on DO in a docker instances and just doing backups on DO cause its cheaper than something like heroku managed PG so if I was one of them (I kept my data layer off of DO) I would be completely destroyed by this. So I tell everyone I know now at meetups about this experience and so happy to share specific details.
They told me I log on from multiple locations (OMG I have a laptop and work while I travel, busted) and that I had a gmail address on my account.
The most agregious thing is that if their trigger happy "security and trust" flagger flags you, there is no warning where they reach out to you, they just destroy your instances, don't tell you about that either until you logon and are asked to file a ticket if you want your account unlocked...then a month later will unlock it for you but everything nuked.
Yes, backup your data somewhere else.
(1) Gitlab/Github for source code (almost free)
(2) automysqlbackup (or similar) to an S3 bucket for your database (very cheap) using S3FS
(3) Make sure you have a way to reproduce your entire server environment using fabric, ansible, chef, puppet (or Docker images) that is also in source control
If you have PROD code, this is normally called DR (Disaster Recovery)
(1) Gitlab/Github for source code (almost free)
(2) automysqlbackup (or similar) to an S3 bucket for your database (very cheap) using S3FS
(3) Make sure you have a way to reproduce your entire server environment using fabric, ansible, chef, puppet (or Docker images) that is also in source control
If you have PROD code, this is normally called DR (Disaster Recovery)
Hi had heard a few stories like this here too but I figured it was people doing spamming or gambling or something like that. We are a biotech startup and also around $1k per month with our usage, and it was completely shocking.
[deleted]
Thanks for the message, I will send you an email with the details if you want to look into it. I'll gladly add any resolution etc to the write-up I am doing. Thanks.
[deleted]
If the contract (terms of service) you agreed to with the provider says they can nuke your data without notifying you, there may be nothing you can do. If the contract doesn't allow them to do that, you could consult a lawyer to see if this constitutes a breach of contract or negligence. (But if the contract has a forced arbitration clause, you won't be able to sue them.)
In any case, this is a horrible thing for a business to do to a paying customer.
In any case, this is a horrible thing for a business to do to a paying customer.
Sure, you can always sue. I have no idea what your chance of winning is, though.
I want to caution other commenters that many of these stories have turned out to not be what they first appear.
I want to caution other commenters that many of these stories have turned out to not be what they first appear.
There certainly is. You can sue them. I bet you'd lose, though.
In my opinion you lost your data because you didn't make backups.
In my opinion you lost your data because you didn't make backups.
Oh, I did have backups luckily off of DigitalOcean (I will have full details in my post), but I have a huge problem with them marketing how great they are to startups and having a backups product that they sell (and also nuked). Any company that does this is not worth dealing with IMO.
They are great for startups because you can automate everything. You can delete everything and restore from 0 without needing manually run commands.
Treat it as a temporary server that can disappear at any time, have automated backups and streaming database replication, then have an automated restore script that you test periodically.
If the server goes down, restore somewhere else.
Treat it as a temporary server that can disappear at any time, have automated backups and streaming database replication, then have an automated restore script that you test periodically.
If the server goes down, restore somewhere else.
You have to have a data layer stored somewhere, are you saying to not store it on digital ocean? All this is a moot point when they destroy your instances and lock your account...
it's very poor customer service to be sure, not answering the ticket for 5 weeks is ridiculous, deleting your data sucks too, but you can get to a point where that is only a 30 minute inconvenience.
You store the data in a different place to DO, maybe at home or a cheap VPS somewhere else.
You store the data in a different place to DO, maybe at home or a cheap VPS somewhere else.
I'm not a lawyer. The TOSs I've seen, not specifically the digital ocean TOS but in general, usually limit the liability to the amount the service costs. If that's the case you might get credited a month.
But laws are really complicated and I don't even attempt to understand them.
But laws are really complicated and I don't even attempt to understand them.
I am a bit skeptical of this account. Every time I have contacted DigitalOcean support they respond quickly usually within hours.
If this service was successful and profitable, I can't imagine waiting 5 weeks. I'd be in full nerd rage after 1/2 day of being down, and would be spinning up new servers on another provider.
Curious to see the full post though before jumping to more conclusions.
If this service was successful and profitable, I can't imagine waiting 5 weeks. I'd be in full nerd rage after 1/2 day of being down, and would be spinning up new servers on another provider.
Curious to see the full post though before jumping to more conclusions.
Agreed, I have contacted them before over the 2 years and always they responded quickly. They responded w/in 24hrs and said that someone from "security and trust" team would review and get back to me, and that response took 5 weeks. So I guess at the point they already destroyed your instances they assume they lost you as a customer and don't really care anymore to get back to you in a timely manner.
I did go into full nerd rage because I can't believe they would do this to a startup, I had the critical stuff running in <1hr on Linode, but took a day for everything to be rebuilt since I had to redo everything w/out access to backups or images.
I did go into full nerd rage because I can't believe they would do this to a startup, I had the critical stuff running in <1hr on Linode, but took a day for everything to be rebuilt since I had to redo everything w/out access to backups or images.
Talk to your lawyer. I’m not sure what the point of this post is. If it’s a PSA/warning, then you need to provide more proof. If you’re really seeking legal advice from an internet link aggregator, then the best advice you’ll get here is to talk to an actual lawyer.
Sounds incredible to me that anyone would build anything important in the cloud without backups. Seriously, clouds aren't magic, they are just someone else's computer.