Don't use Digital Ocean (DO) for production
79 comments
Don't worry vfulco2, you've reached the front page of HN, the official support portal of Digital Ocean. Your account will be restored within two hours.
Woah, I thought it was just the support portal for Google
Also Facebook / Instagram accounts. And in my case, Stripe too.
Even the estimated time was right. I'm impressed! HN does seem to be the best customer service hub for tech cos.
Thank you. I posted an update, let me know if there is anything else I should do.
HAHAHAHAHA. What a classic.
I've been having this problem a lot recently. First Rackspace, later CloudFlare. Most recently Twitter accused me of being a robot and asked me to verify that I'm a human by posting a government-issued ID even though I rarely use it and I've never posted a comment there and could not have possibly broken any policies. Sorry that this is happening to you. I think tech companies at large are really failing individuals. Even if it's only 1 - 2% of users annually, it adds up across the dozens of services we use day-in, day-out for years and it really isn't good.
It's not 1-2% of users annually. It's a small fraction of that. When it happens people tend to get loud about it though, so it shows up on social media or on HN and similar. Publicly traded companies like DigitalOcean aren't about to wipe out 2% of their paying customers by mistake every year; the problem is small enough relatively speaking that they don't much care (unless the customer can get enough attention to their plight), despite how it impacts the customer that is suffering from their negligence.
*Update* to their credit, they've followed up with me. The github login was routing to the wrong account (a rarely used account) and was caught in their automatic fraud process. They've verified the main account (5 year old) is still intact and given me instructions to get back in.
I am immensely relieved. They've done right by me.
I am immensely relieved. They've done right by me.
So the HN post did work and you’re going back to a platform that treated you badly? If I were you, I’d copy what needs to be copied from there and close all accounts. Encouraging “customer support only through social media outrage” is not a very good choice. If you continue, you wouldn’t be doing right by the other customers of this provider.
Which platform is exempt from this? I have heard similar stories from GCP, AWS, Linode, etc. I would even say: the bigger the platform, the worse the customer support (i.e., Google)
Good to hear!
Did they explain why it took over 48 hours to reply to you?
Did they explain why it took over 48 hours to reply to you?
Something like this has happened before:
DigitalOcean Killed Our Company: https://news.ycombinator.com/item?id=20064169
DigitalOcean Killed Our Company: https://news.ycombinator.com/item?id=20064169
Something like this was reported 8 hours ago[0] I find myself suspicious of this sudden negative press
[0]: https://news.ycombinator.com/item?id=29596138
[0]: https://news.ycombinator.com/item?id=29596138
This is why you should use FOSS as much as possible. So what if digital ocean shuts you down? You have a backed up copy at least once a day of your data at least once a day on off premises backup, right? You just move to another provider and make sure to tweet and HN about how awful DO (or whoever is) so they lose customers and your company keeps moving alone with a 1 day outage.
This is concerning. We were considering a move to DO from Huroku to save quite a few $, but I’m thinking AWS now. I dislike the convoluted mess of a UI that AWS presents, but this happening would be way worse.
I use AWS for a hobby project. It just costs a couple of Euros a month. I am in EU and according to PSD2 every payment would need strong authentication. AWS warns about this but my bank has never required it. I always fear some day they will require the authentication and just these days I will be late with checking mails/accounts, wherever this will turn up and I will notice only when AWS has shut down my server and deleted my data.
I would happily make a 20 or 50 EUR/USD payment in advance to avoid the risk. But AWS is not flexible enough to allow me to do so. I understand in some countries it's possible, but in EU there are no options.
I would happily make a 20 or 50 EUR/USD payment in advance to avoid the risk. But AWS is not flexible enough to allow me to do so. I understand in some countries it's possible, but in EU there are no options.
I always wondered how AWS/Amazon got away without the strong authentication. It turned out that they don't anymore in my case:
> Your card payment ending in XXXX failed when we attempted to charge it for AWS services on account YYYYYYYYYYYY because it requires additional authentication. To complete the payment, please complete your credit card authentication.
Expect an e-mail with subject "Action required – Your AWS account is past due" if you haven't gone through the authentication process for your card.
Luckily they were pretty noisy about it. I received the first mail about it on Dec 2nd and then reminders on Dec 6th, 9th and 13th. At this point I noticed it and made the payment. They didn't shut down any service meanwhile.
> Your card payment ending in XXXX failed when we attempted to charge it for AWS services on account YYYYYYYYYYYY because it requires additional authentication. To complete the payment, please complete your credit card authentication.
Expect an e-mail with subject "Action required – Your AWS account is past due" if you haven't gone through the authentication process for your card.
Luckily they were pretty noisy about it. I received the first mail about it on Dec 2nd and then reminders on Dec 6th, 9th and 13th. At this point I noticed it and made the payment. They didn't shut down any service meanwhile.
It appears to me that they are prepared because they warn about it. But my bank does not follow the rules so the payment goes through without strong authentication.
Good to know that they warn at least 10 days. If that is constant and not dependent on some algorithm. I could not find any information how fast they stop your server and whether data would get locked first or deleted immediately.
Good to know that they warn at least 10 days. If that is constant and not dependent on some algorithm. I could not find any information how fast they stop your server and whether data would get locked first or deleted immediately.
I’m in a similar position at the moment and I’d suggest if you were considering DO, to take a look at Vultr and UpCloud.
I’ve found support response time from both so far to be excellent (UpCloud in particular though their feature set is just slightly behind Vultr and DO currently).
I had been testing the water with DO with a view to hosting some production stuff with them, but I’ve seen a few too many similar stories like this one and decided to get out of DO and rule them out for production stuff for now. Many of their direct competitors seem to provide much better support at lower levels of spend too.
I’ve found support response time from both so far to be excellent (UpCloud in particular though their feature set is just slightly behind Vultr and DO currently).
I had been testing the water with DO with a view to hosting some production stuff with them, but I’ve seen a few too many similar stories like this one and decided to get out of DO and rule them out for production stuff for now. Many of their direct competitors seem to provide much better support at lower levels of spend too.
Digitalocean will automatically lock any account I create, because of an (admitted!) false positive in their fraud software. Literally any IP, any credit card, any email address I use- seconds after taking out a test deposit on the card, the account is locked.
Every time I've messaged support for an account lock, 24 hours later I'm informed it was a false positive and the account is unlocked. Needless to say, this does not inspire confidence that my resources are going to be kept up and running past Kafkaesque fraud monitoring systems
Every time I've messaged support for an account lock, 24 hours later I'm informed it was a false positive and the account is unlocked. Needless to say, this does not inspire confidence that my resources are going to be kept up and running past Kafkaesque fraud monitoring systems
> Every time I've messaged support for an account lock, 24 hours later I'm informed it was a false positive and the account is unlocked.
Sounds like good customer service to me ;) I tried to open an account with Oracle cloud and it rejected my credit card. They never even answered to my messages.
Well, I was not overly convinced I want to do business with Oracle anyway, so now AWS charges the same card every month. Oracle seemed somewhat cheaper for the use case, but nothing that kills me.
Sounds like good customer service to me ;) I tried to open an account with Oracle cloud and it rejected my credit card. They never even answered to my messages.
Well, I was not overly convinced I want to do business with Oracle anyway, so now AWS charges the same card every month. Oracle seemed somewhat cheaper for the use case, but nothing that kills me.
Can confirm. I created an account for a client but I was using a VPN to avoid conflicting with my own personal account.
It was blocked but DO unlocked it once I explained it in an email to them.
It was blocked but DO unlocked it once I explained it in an email to them.
*Update* to their credit, they've followed up with me. The github login was routing to the wrong account (a rarely used account) and was caught in their automatic fraud process. They've verified the main account (5 year old) is still intact and given me instructions to get back in.
I am immensely relieved. They've done right by me. Kudos for a delayed but excellent response.
I am immensely relieved. They've done right by me. Kudos for a delayed but excellent response.
I used to like Digital Ocean. But it seems as though ever since they went IPO all the engineers became rich and absconded to Hawaii. Or they're at least not working. It's Noam Bardin's Law in effect.
I'm looking for the next Digital Ocean. Something still bespoke, with higher cost but better customer support. So far oldschool VPS hosts are looking good.
I'm looking for the next Digital Ocean. Something still bespoke, with higher cost but better customer support. So far oldschool VPS hosts are looking good.
What does HN recommend these days for app hosting? I'm a solo dev without deep pockets. I was looking at Linode and DO, but posts like this have me worried about the later.
YMMV - but I’ve had 2 wordpress blogs on Linode for 14 years now. I’ve spun up (and spun down) 3 different companies on Linode (all monoliths using Postgres, Memcached and Redis). I’ll be with them as long as I can. During this time, I can really only recall 2 times they emailed me out of the blue and said something was wrong with one my hosts. Both times were a couple of hours of downtime to which they credited me the time. I’m sure I’ve had more downtime, but it must not have been too bad because it isn’t readily available from memory.
AWS Lightsail. Linode and DO are not cheaper. Seriously, compare the prices for yourself [0][1][2]. Linode and DO seem to have a reputation of being cheaper options, but they're not, and AWS isn't going to jerk you around with small-hoster issues like Linode and DO have been known to.
[0] https://aws.amazon.com/lightsail/pricing/
[1] https://www.digitalocean.com/pricing
[2] https://www.linode.com/pricing/
[0] https://aws.amazon.com/lightsail/pricing/
[1] https://www.digitalocean.com/pricing
[2] https://www.linode.com/pricing/
I'm a happy Linode user. And I'm a very small hoster. I'm renting a $10/month VPS for quite a few years now, and I've been very impressed with them.
A while back I made a stupid mistake while tired, and I was panicking and couldn't find the feature I needed in the account.
I called Linode at around 11PM CDT, and got a helpful friendly guy on the line who walked me through what I needed. I was amazed that I could get a human on the line in less than a minute after hours, when only paying $10/month.
A while back I made a stupid mistake while tired, and I was panicking and couldn't find the feature I needed in the account.
I called Linode at around 11PM CDT, and got a helpful friendly guy on the line who walked me through what I needed. I was amazed that I could get a human on the line in less than a minute after hours, when only paying $10/month.
I wish I could migrate to Linode... but they don't support VPCs in Europe though.
I've inadvertently done something against Linode's ToS twice and both times they contacted me to get the issue resolved, my account was never locked. So take this as a +1 for them, at least on this front.
What is a "small-hoster issue"?
Things like poor customer service (the OP that we're commenting on right now), limited POPs [0], low service quotas with unpredictable results if you hit them [1], limited services (limited instance types, no long tail of value-add services), requiring manual tickets for things that hyperscalers have fully automated, etc. Things of that nature. These are things that stem directly from being a very small hosting company, and that you won't see at a hyperscaler.
[0] Linode got DDOS'ed literally entirely off the Internet in December 2016. You can't take AWS down with a DDOS. You can take down an individual customer server but you can't take AWS offline with a DDOS.
[1] https://news.ycombinator.com/item?id=20064169
[0] Linode got DDOS'ed literally entirely off the Internet in December 2016. You can't take AWS down with a DDOS. You can take down an individual customer server but you can't take AWS offline with a DDOS.
[1] https://news.ycombinator.com/item?id=20064169
If you don't use any services specific to the vendor. Then practice your backup and restore routines. You will be able to reasonably move to a new cloud provider, on the off chance you get similarly cancelled.
This is probably the most reasonable risk mitigation you can fairly easily do.
I don't think there exists a vendor that's going to be foolproof. Most users aren't going to any have issues with any of the major providers, including DO, Linode, Vultr, Oracle, AWS, Azure, etc...
This is probably the most reasonable risk mitigation you can fairly easily do.
I don't think there exists a vendor that's going to be foolproof. Most users aren't going to any have issues with any of the major providers, including DO, Linode, Vultr, Oracle, AWS, Azure, etc...
I have never had any problems with Hetzner, but I suspect someone who has will reply to this comment.
I've been on Linode for going on 15 years now. I've hosted a myriad of projects for myself and clients there without significant issues. For $corporatejob all of our assets are in AWS or Azure but my own stuff is on Linode.
I had something similar but it was DMCA complaints for a forum I ran. I complied with the deletions but after a handful they just deleted my stuff. I moved to OVH. What a joke.
Don’t use Digital Ocean for anything, not just production. Your account may be locked up and customer service will keep sending boilerplate responses. There is no way I know of to contact a human and find out why the account is locked and what options exist. DO has been heavily understaffed for years, and the mass layoffs in early 2020 likely made things far worse.
I see value in supporting smaller companies (relative to large ones like Amazon, Microsoft or Google), but smaller companies usually tend to have better customer support when things go wrong, and are generally a lot more helpful. With DO, it has the kind of offerings by a smaller company with the kind of support by a large company.
I see value in supporting smaller companies (relative to large ones like Amazon, Microsoft or Google), but smaller companies usually tend to have better customer support when things go wrong, and are generally a lot more helpful. With DO, it has the kind of offerings by a smaller company with the kind of support by a large company.
I tried them after reading here about how their support is excellent and had an interesting one-day experience. Not long after logging in for the first time their site started having some issues, such as the GUI being jumbled because the asset weren't loading correctly. It went down officially a minute later and I didn't have time to hang around until it came back up. I got charged for the things that had enabled themselves or failed to really disable during that mess, so I asked for the money back. The first response gave me a guide to closing my account, so I tried again. The second response was to refund me some loose change (some dollars and cents).
Does anyone know of a similar VPS provider that a) is responsive in situations like the OP's, and b) is not on the UCEPROTECTL3 email blocklist?
I've been happy with DigitalOcean for a while now, but reports like this concern me, and I've also discovered after investigating that the reason MS-hosted email accounts (@hotmail.com, @live.com, etc) reject all emails from my VPS is because the entire provider is on the UCEPROTECT blocklist, and apparently has no interest in changing this.
I've been happy with DigitalOcean for a while now, but reports like this concern me, and I've also discovered after investigating that the reason MS-hosted email accounts (@hotmail.com, @live.com, etc) reject all emails from my VPS is because the entire provider is on the UCEPROTECT blocklist, and apparently has no interest in changing this.
That's very scary, I am building an MVP currently on DO and although I have daily backups of all the data pushed to another cloud provider as a safety, seeing how they closed your account suddenly out of nowhere with no recourse I feel like I should move out of there.
This is insane, if I wouldn't have those backups and the same thing happened to me, I would be screwed. How come they don't contact you beforehand, or at least let you move out your data (keep the account open but cut the droplets from public internet if for whatever reason their algorithm detected nefarious activity on it, or a bogus DMCA, exploits happens, etc).
I wanted to try an alternative to AWS but look like I'll be back to it soon. Does AWS have a similar track record of closing account out of the blue? Is there any other alternative that have better support and would at the very least try to contact the owner before taking drastic action like this?
This is insane, if I wouldn't have those backups and the same thing happened to me, I would be screwed. How come they don't contact you beforehand, or at least let you move out your data (keep the account open but cut the droplets from public internet if for whatever reason their algorithm detected nefarious activity on it, or a bogus DMCA, exploits happens, etc).
I wanted to try an alternative to AWS but look like I'll be back to it soon. Does AWS have a similar track record of closing account out of the blue? Is there any other alternative that have better support and would at the very least try to contact the owner before taking drastic action like this?
This post made me think about the personal DO droplets I have and how I’m backing them up. I’m currently using the built in back up feature…but that doesn’t protect against DO preventing me from accessing my backups.
What does everyone do to easily and automatically back up their side project droplets offsite?
What does everyone do to easily and automatically back up their side project droplets offsite?
I have a daily CRON job, a simple script that backup the database with `pg_dump` and send the file to an S3 bucket (keeping the last X files) and also use `rclone` to sync my files on "Spaces" (DO clone of S3) to an S3 bucket.
It's supposed to be a last resort kind of backup as I also use DO backup solution (managed database) but seeing how they can close an account out of nowhere, it ain't worth nothing.
It's supposed to be a last resort kind of backup as I also use DO backup solution (managed database) but seeing how they can close an account out of nowhere, it ain't worth nothing.
I hope you periodically restore the off-premises backup to check they're working too. Not really fun to discover your backup is broken when you need it the most.
I admit I don't check them often enough, but I did makes sure to verify after this post!
Thanks! That's super helpful!
Daily database dumps to a folder, followed by a rsync (database and users' files) to a 3rd party backup services (rsync.net). I used backblaze b2 in the past, but found that accessing historical backup data is easier on rsync.net. Just a simple sftp into the snapshot directory.
Rclone
Reminder that the cloud is just someone else's computer.
The someone else's computer is not the problem. Most of us take someone else's plane (or more climate-friendly train) and stay at someone else's hotel.
The problem is that customer "service" is handled by algorithms with no chance to reach competent humans who have the power to make reasonable decisions.
The problem is that customer "service" is handled by algorithms with no chance to reach competent humans who have the power to make reasonable decisions.
Cloud providers have a tough business model. The instant you use their compute resource, it's gone forever. But they bill you post-usage, and you can dispute the charges for a very long time. And, there are thousands of simple shell scripts that convert CPU time to cash. All of this makes it very easy to farm cloud providers for money. Not very much money, but free money doesn't have to be efficient.
For work, I run a cloud service like this, and we do indeed have this kind of fraud. Every stolen credit card number under the sun has signed up for an account. People create thousands of emails (and brand new domains) to get free trials. And, when our systems identify them and we shut them down, they whine in all possible support forums. I don't think we've ever made a mistake, but that is likely due to the small scale of our operation.
The only thing that keeps me sane is the extremely poor opsec of the fraudsters. They have pools of thousands of compromised IPs they're using for the command-and-control infrastructure that they install, but they often mistakenly use the same destination crypto wallet (they have a little LD_PRELOAD script they install to try to prevent us from seeing this, but it doesn't work if you don't load dynamic libraries to read the process table), or use a predictable naming scheme, or taint accounts by accessing a few of them from the same IP. (I have a graph that connects the accounts through things like IP address, so this is often the fatal screwup for thousands of accounts all at once.)
I imagine that Digital Ocean has this same problem, but at a million times the volume. So, there are going to be mistakes. It's intrinsically difficult to establish trust at these small scales. DO probably feels pretty uncomfortable giving someone a bunch of compute resources when they haven't even tried billing the credit card. You would feel pretty uncomfortable paying up front. So this is where we land, sometimes you look like fraud and get killed with fire. (As you get bigger, you'll have weekly meetings with your cloud provider, and they probably won't shut you down automatically. But at that small scale, it's tough for everyone!)
For work, I run a cloud service like this, and we do indeed have this kind of fraud. Every stolen credit card number under the sun has signed up for an account. People create thousands of emails (and brand new domains) to get free trials. And, when our systems identify them and we shut them down, they whine in all possible support forums. I don't think we've ever made a mistake, but that is likely due to the small scale of our operation.
The only thing that keeps me sane is the extremely poor opsec of the fraudsters. They have pools of thousands of compromised IPs they're using for the command-and-control infrastructure that they install, but they often mistakenly use the same destination crypto wallet (they have a little LD_PRELOAD script they install to try to prevent us from seeing this, but it doesn't work if you don't load dynamic libraries to read the process table), or use a predictable naming scheme, or taint accounts by accessing a few of them from the same IP. (I have a graph that connects the accounts through things like IP address, so this is often the fatal screwup for thousands of accounts all at once.)
I imagine that Digital Ocean has this same problem, but at a million times the volume. So, there are going to be mistakes. It's intrinsically difficult to establish trust at these small scales. DO probably feels pretty uncomfortable giving someone a bunch of compute resources when they haven't even tried billing the credit card. You would feel pretty uncomfortable paying up front. So this is where we land, sometimes you look like fraud and get killed with fire. (As you get bigger, you'll have weekly meetings with your cloud provider, and they probably won't shut you down automatically. But at that small scale, it's tough for everyone!)
That really doesn't help the honest small customer. I want to buy something, pay the couple of bucks it costs and reliably get the service for my money. If I'd have to pay the couple of bucks a month in advance I would happily do it.
Yeah, being the honest small customer is tough. I find things like "pre-pay" or "use some third-party ID verification service" or "talk to sales" kind of off-putting, but honestly there aren't a lot of great options. (We went with the "talk to sales" route.)
Talk to sales makes no sense in the case of small customers. The sales person costs them more in a hour than I want to pay for my service for a whole month.
That's just not a great analogy because I don't need a hotel for 5 years or a train forever. When you don't own it, and especially when it's a large org, you don't have much leverage when they want to change the terms.
I'm just not interested in the "you will own nothing and like it" society. I don't trust the giant corporations and landlords to do the right thing. So, I own a house, a car, and my data.
Short term, like an Uber (or a plane) it's fine, but it's not a long-term solution unless or until I can rely more.
Giant corporations are certainly a problem, because it means oligopolies if not monopolies. Like Google, Facebook, Amazon in certain areas.
For cloud services to small customers there is luckily still a bit of choice. Unfortunately I don't see that any of them would really be better if things go wrong for some reason or another. Well, if you pay them less than a hourly salary for a whole month, customer service is not really an economic option.
For cloud services to small customers there is luckily still a bit of choice. Unfortunately I don't see that any of them would really be better if things go wrong for some reason or another. Well, if you pay them less than a hourly salary for a whole month, customer service is not really an economic option.
Sooner or later cost cutting becomes a management obsession and quality of service deteriorates - especially with monopolies.
Unless you have the up-front cash, time, and know-how to get your own box in a datacenter, isn't it the most logical option to rent something? The problem seems to be the market for servers is so large now, and incumbents have so many customers, they can afford to kick random people to the curb with no repercussions.
It would be if cloud providers didn't keep changing the deal all the time. Ownership is control and control comes with a premium.
Sure and a bank is just someone else's safe as well, while we are doing the ridiculous reductionism thing
In the same way that Uber is just someone else's car.
Would you be willing to to sell all of your vehicles and sign a time-charter contract with Uber to manage all of your ground transportation for the foreseeable future?
No but I am a single individual. If I was a company that produced widgets I would use FedEx or UPS rather than maintaining my own fleet of trucks, drivers, maintenance staff, etc.
Short answer: No $%^&* way !
I've heard stuff like that for pretty much any hosting company. Are there any companies that don't do that? How do they guarantee it?
It happened similar to us. A payment arrived late and everything was deleted.
I promised to never use DO again, but the low price makes sense for a small project being made by an indie games company, so here we are again.
Luckily for us we added redundancy on payments and alerts, so I hope we won't be in the same case again.
Luckily for us we added redundancy on payments and alerts, so I hope we won't be in the same case again.
This is par for the course with free services, but this can't be legal if you're a paying customer and have signed agreement with them to provide you services in exchange for payment.
This isn't really a complaint and i was lucky that it was my personal stuff.
But somehow a droplet was part of a DDOS and network was booted off ( 1 network spike for 10 minutes). Let's just say it was stressful and very unwelcome at the time.
I also went to mention that i did receive feedback and there was definitely a human from their end involved.
But, I wouldn't want to imagine what would have happened if it were websites of clients. I took more measures by now, but i think the actual culprit was a contractor that did some work for me.
I'm wondering how other providers handle that ( a VPS for example)? Anyone has some insights on how it can be handled better?
But somehow a droplet was part of a DDOS and network was booted off ( 1 network spike for 10 minutes). Let's just say it was stressful and very unwelcome at the time.
I also went to mention that i did receive feedback and there was definitely a human from their end involved.
But, I wouldn't want to imagine what would have happened if it were websites of clients. I took more measures by now, but i think the actual culprit was a contractor that did some work for me.
I'm wondering how other providers handle that ( a VPS for example)? Anyone has some insights on how it can be handled better?
I have made a mistake in a Linode VPS once, and it was coopted into a botnet. When they detected the problem, they took it offline and I received an email from the support. As soon as it was fixed, I emailed them and they put it online again.
> I'm wondering how other providers handle that?
Depending on what you mean by "booted off", any and every reputable provider will do the same. They all have AUPs that allow them to stop routing for your, at a minimum, if you're found to be sourcing traffic that's against their AUP.
That's all normal. You can't expect a company to keep you online when you're causing problems for them and the rest of the internet, and their contracts/AUPs are all worded so they can take action if your systems are found to be a problem.
What can you do to prevent this? Don't let your systems be part of a botnet or get hacked in general. More feasibly, stay up to date on all security issues and take multiple system backups so if you have to restore from a good prior backup, if that's possible.
> I took more measures by now, but i think the actual culprit was a contractor that did some work for me.
Could be, but my guess is not. Botnets survive on having massive numbers of servers in them, and are often automated exploits. More likely, you were running some exploitable software, and it was exploited. Updates need to be run, daemons often need to be restarted after updates because of older code still running in memory, etc. And if you were running some CMS like Wordpress or Drupal, well if you aren't doing it yourself and keeping on top of updates and new versions, you either better have someone on retainer to do it for you or give up and do something else, because you're a perfect target for a botnet at that point because those generally have problems on a regular basis (as well as the problems introduced by different modules they each might support).
Depending on what you mean by "booted off", any and every reputable provider will do the same. They all have AUPs that allow them to stop routing for your, at a minimum, if you're found to be sourcing traffic that's against their AUP.
That's all normal. You can't expect a company to keep you online when you're causing problems for them and the rest of the internet, and their contracts/AUPs are all worded so they can take action if your systems are found to be a problem.
What can you do to prevent this? Don't let your systems be part of a botnet or get hacked in general. More feasibly, stay up to date on all security issues and take multiple system backups so if you have to restore from a good prior backup, if that's possible.
> I took more measures by now, but i think the actual culprit was a contractor that did some work for me.
Could be, but my guess is not. Botnets survive on having massive numbers of servers in them, and are often automated exploits. More likely, you were running some exploitable software, and it was exploited. Updates need to be run, daemons often need to be restarted after updates because of older code still running in memory, etc. And if you were running some CMS like Wordpress or Drupal, well if you aren't doing it yourself and keeping on top of updates and new versions, you either better have someone on retainer to do it for you or give up and do something else, because you're a perfect target for a botnet at that point because those generally have problems on a regular basis (as well as the problems introduced by different modules they each might support).
> booted off
The context is missing without my original sentence. "Network was booted off." Obviously not my account but "network access" and i could only connect through the console through their web dashboard.
> Could be, but my guess is not.
According to the files i found and the zipfiles he left behind. There would be a huge confidence of similar timing and folders involved. He even named the zip similar to the work he did.
I also know how botnets work, backups and security updates. I already took measures as mentioned before.
I never said and/or blamed DO for it. I'm not running sites on it, but some personal stuff as i mentioned ( again).
I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better...
Tbh. I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved. Which is really frustrating to respond too... As you're not talking to me, but a reader of the comments.
The context is missing without my original sentence. "Network was booted off." Obviously not my account but "network access" and i could only connect through the console through their web dashboard.
> Could be, but my guess is not.
According to the files i found and the zipfiles he left behind. There would be a huge confidence of similar timing and folders involved. He even named the zip similar to the work he did.
I also know how botnets work, backups and security updates. I already took measures as mentioned before.
I never said and/or blamed DO for it. I'm not running sites on it, but some personal stuff as i mentioned ( again).
I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better...
Tbh. I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved. Which is really frustrating to respond too... As you're not talking to me, but a reader of the comments.
> The context is missing without my original sentence. "Network was booted off."
I didn't take it as written because it was too ambiguous to know. Often your actual IP or network range might be null routed from upstream providers. Or perhaps DO just turns off routing for the systems in question. Or it could be a poor description of something else.
I work at an ISP. We will null route colocated customer IP addresses if they are causing problems until they resolve them and contact us, or if they are the target of a DDOS. The point of null routing is that other providers won't even send you the traffic anymore, so it neatly solves the problem. Sometimes we have to do it for IPs for our own services if they are the target of a DDOS.
> I never said and/or blamed DO for it.
I'm just letting you know how network providers work. If you cause hard to the network, and all their other customers, whether on purpose or just because you're the person responsible for the resources in question, they will shut you down. That's expected. It's a good thing they do, otherwise none of these providers could even offer useful service because every day there's be an event destroying their network, and your experience every day prior to the problem would have been crappy service.
> I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better...
The correct way to handle your service being part of a botnet is to immediately remove your server from the network, whether they do it or you do. You server is causing active harm to other people on the internet while it's left online in that state. It sucks for you that it goes down, but really, that's how the world works. When something you control is causing harm to the public, it will be dealt with whether you like how it happens or not. The way to deal with that problem is to do everything in your power to prevent if from getting to that state.
Because of that, iit's impossible to be absolutely secure from a network provider shutting down your service. If it's that important, you need to have backup plans, le other services you can can bring up at a moment's notice and swing DNS to. People make good money designing and implementing disaster recovery and business continuity plans like this. That's likely not feasible for a personal site. But usually people understand and can deal with a few hours or even days of downtime for personal sites, while a business might not recover as well from that.
> I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved.
I think perhaps you didn't understand what I was trying to express. What you experienced is normal and the correct and expected way network providers will deal with a problem like that. I tried to express that.
As for whether your consultant installed a botnet on your system, it's possible I thought it unlikely because the amount they could gain from one extra hour of repeat business from you likely dwarfs the gain from a single extra botnet computer, of which botnet networks have thousands. There's just not a lot of incentive for a consultant to put a system they're working on on a botnet. I would rate it as more likely that something the consultant did left the system in a vulnerable state, but it doesn't really matter does it? I wasn't trying to attack you or indicate you screwed up, just indicate that if your system was on a botnet it probably got there like 99.999% of other systems on them, which is some automated work that infects it, adds a control mechanism, and then scans to add other hosts. Or some script kiddie ran some scanner/infector tool and added the control mechanism to join it to a botnet. Because the time to do it manually isn't really worth the one extra system in a lot of cases, I would imagine.
I didn't take it as written because it was too ambiguous to know. Often your actual IP or network range might be null routed from upstream providers. Or perhaps DO just turns off routing for the systems in question. Or it could be a poor description of something else.
I work at an ISP. We will null route colocated customer IP addresses if they are causing problems until they resolve them and contact us, or if they are the target of a DDOS. The point of null routing is that other providers won't even send you the traffic anymore, so it neatly solves the problem. Sometimes we have to do it for IPs for our own services if they are the target of a DDOS.
> I never said and/or blamed DO for it.
I'm just letting you know how network providers work. If you cause hard to the network, and all their other customers, whether on purpose or just because you're the person responsible for the resources in question, they will shut you down. That's expected. It's a good thing they do, otherwise none of these providers could even offer useful service because every day there's be an event destroying their network, and your experience every day prior to the problem would have been crappy service.
> I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better...
The correct way to handle your service being part of a botnet is to immediately remove your server from the network, whether they do it or you do. You server is causing active harm to other people on the internet while it's left online in that state. It sucks for you that it goes down, but really, that's how the world works. When something you control is causing harm to the public, it will be dealt with whether you like how it happens or not. The way to deal with that problem is to do everything in your power to prevent if from getting to that state.
Because of that, iit's impossible to be absolutely secure from a network provider shutting down your service. If it's that important, you need to have backup plans, le other services you can can bring up at a moment's notice and swing DNS to. People make good money designing and implementing disaster recovery and business continuity plans like this. That's likely not feasible for a personal site. But usually people understand and can deal with a few hours or even days of downtime for personal sites, while a business might not recover as well from that.
> I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved.
I think perhaps you didn't understand what I was trying to express. What you experienced is normal and the correct and expected way network providers will deal with a problem like that. I tried to express that.
As for whether your consultant installed a botnet on your system, it's possible I thought it unlikely because the amount they could gain from one extra hour of repeat business from you likely dwarfs the gain from a single extra botnet computer, of which botnet networks have thousands. There's just not a lot of incentive for a consultant to put a system they're working on on a botnet. I would rate it as more likely that something the consultant did left the system in a vulnerable state, but it doesn't really matter does it? I wasn't trying to attack you or indicate you screwed up, just indicate that if your system was on a botnet it probably got there like 99.999% of other systems on them, which is some automated work that infects it, adds a control mechanism, and then scans to add other hosts. Or some script kiddie ran some scanner/infector tool and added the control mechanism to join it to a botnet. Because the time to do it manually isn't really worth the one extra system in a lot of cases, I would imagine.
> I think perhaps you didn't understand what I was trying to express. What you experienced is normal and the correct and expected way network providers will deal with a problem like that. I tried to express that.
As mentioned. I thought that it was normal, i was just curious if there was a better way that's.. less stressful :p
As mentioned. I thought that it was normal, i was just curious if there was a better way that's.. less stressful :p
This could be written about any cloud provider. I for instance have had a much better time using DO than other cloud services but it’s all anecdotal. If your trying to optimize for customer support buy and run your own datacenter.
Long story short, I've had an account with them for roughly 5+ years as I recall, good credit card on file, monthly bill around $20-50, US address. I use a few droplets to run dokku/docker containers for dev and 1 private Google Outline VPN endpoint, only for me. I only ssh into these from non-root and root should be turned off completely (but I'm only a hacker and not a F/T systems person).
Not sure if it matters but I don't do anything nefarious on these droplets-- no webscraping, no botnet creation, no crypto mining. I don't know what else would be considered verboten!?!
Try to log in with github, won't work and asks for email login. Try to login from there, and it asks for credit card info for "authorization" although the card is the one on file and being charged for so many years.
No recourse, and what a crappy way to treat clients even if they are small potatoes in the grand scheme of things. Thankfully I am building a new MVP all on AWS and have my company prod there too running for 5 years. So I don't have to put up with unprofessional dealings and major risks of service interruption.
Assume through no fault of your own that you could be shut out of everything. You have been warned.