Linode DDoS continues – Atlanta down for 16+ hours(status.linode.com)
status.linode.com
Linode DDoS continues – Atlanta down for 16+ hours
http://status.linode.com/incidents/cbbcjnhhpkgm
80 comments
"Je suis Linode" what an awesome sentiment...
I just signed up for a Linode server. Going to use it for part of our offsite backup storage and to monitor some nodes on our primary network.
Huzzah...
I just signed up for a Linode server. Going to use it for part of our offsite backup storage and to monitor some nodes on our primary network.
Huzzah...
Still sticking with Linode. This is definitely not their fault. If anything I blame ISPs for letting it happen. ISP's could just simply filter traffic by filtering forged packets, then dropping traffic at a backbone would be trivial, making most botnets obsolete.
AWS and Google have resources in place for their own benefit, and hosting with them means you get these benefits to some degree. But the issue is that this complicated infrastructure is only reachable by giants(amazon, google, rackspace, etc). AWS probably spends enough money, collectively, on DDoS mitigation and fallback that it could acquire another hosting company for that sum of money.
The to consider the shady practices of who ever is trying to sabotage Linode's business, the likely goal is to direct business elsewhere. So you may end up moving to a hosting company who is shady enough to have caused you a loss of profits for their own profits. Somethings up here, and I'm really curious to find out who did this. I'm not leaving a business because they were a victim of espionage.
AWS and Google have resources in place for their own benefit, and hosting with them means you get these benefits to some degree. But the issue is that this complicated infrastructure is only reachable by giants(amazon, google, rackspace, etc). AWS probably spends enough money, collectively, on DDoS mitigation and fallback that it could acquire another hosting company for that sum of money.
The to consider the shady practices of who ever is trying to sabotage Linode's business, the likely goal is to direct business elsewhere. So you may end up moving to a hosting company who is shady enough to have caused you a loss of profits for their own profits. Somethings up here, and I'm really curious to find out who did this. I'm not leaving a business because they were a victim of espionage.
I love Linode but they've done a poor job communicating with their customers. Compare the Linode Twitter feed, their mentions, and the lack of replies to how Slack handled it: https://medium.com/swlh/slackdown-a-lesson-in-brand-interact....
The Main Twitter account still hasn't announced the Atlanta DoS. https://twitter.com/linode?lang=en
I understand the severity of the issue is different to Slacks', but there should be a bunch of people on the Twitter account replying to people and saving customers. A lot of their customers are angry about lack of communication as much as they are about the downtime.
The Main Twitter account still hasn't announced the Atlanta DoS. https://twitter.com/linode?lang=en
I understand the severity of the issue is different to Slacks', but there should be a bunch of people on the Twitter account replying to people and saving customers. A lot of their customers are angry about lack of communication as much as they are about the downtime.
Although I'm not sure how important it is to announce outages on social media, I'm a customer and I haven't seen an e-mail either warning of an outage or explaining the situation. Generally I think linode is great but taking the tack of "Oh, if they don't notice, we shouldn't point it out to them" is just a bit cowardly.
I love Linode and I agree with you here. It does depend on the situation and I don't think it's necessary to start sending emails for each bit of down time, I mean, if you're savvy enough to run a VPS you should be able to monitor what your running, but since the severity of what's happening at the moment I would have at least expected an email by now explaining what's going on. Did receive my invoice though.. :)
It's basically the same with the status updates, I have to "poll" for status updates rather than to be notified. They have this mobile app (which is pretty useless as it currently sits) which would be a great platform to receive push notifications on, even if it was to let you know that there is a change on the status page.
There's certainly some missed opportunities here, but then again, I assume they're pretty busy at the moment...
It's basically the same with the status updates, I have to "poll" for status updates rather than to be notified. They have this mobile app (which is pretty useless as it currently sits) which would be a great platform to receive push notifications on, even if it was to let you know that there is a change on the status page.
There's certainly some missed opportunities here, but then again, I assume they're pretty busy at the moment...
I agree with this. They're service clearly doesn't include monitoring as a mention commodity, and the monitoring that they provide pro-bono isn't network monitoring. If you want to know that kind of thing, you should have the monitoring in place to do so, or simply subscribe to their RSS.
No VPS provider ever has sent out emails. The only hosting company I ever left was DigitalOcean, because they kept having really silly problems and at the time NY1 was the only datacenter with private networking, which is ridiculously silly to think that your infrastructure communicates over the internet. And the private networking was "experimental" for the longest time, and very limited.
I don't think Linode did anything wrong. And moving to another host, I garuntee you that they will not do this either. Chances are if you move hosts you may feed the shitty business who performed this DDoS.
No VPS provider ever has sent out emails. The only hosting company I ever left was DigitalOcean, because they kept having really silly problems and at the time NY1 was the only datacenter with private networking, which is ridiculously silly to think that your infrastructure communicates over the internet. And the private networking was "experimental" for the longest time, and very limited.
I don't think Linode did anything wrong. And moving to another host, I garuntee you that they will not do this either. Chances are if you move hosts you may feed the shitty business who performed this DDoS.
yes my Atlanta linode just runs a few old non essential services and has been failing on its own randomly over the last through months mainly due to neglect. I assumed it was failing when I got the first few pingdom alerts over Christmas and rebooted it several times.
They should have done a mail out.
They should have done a mail out.
They have a status page[0] that you can subscribe to for email/rss/atom notifications. You could use an IFTTT recipe to send the RSS feed to twitter[1] or even a Slack channel[2], if you wanted.
Personally, I use Uptimerobot[3] to monitor my nodes and post any issues into a Slack channel. This lets me know if it's a problem with my VPS or Linode (or any of the other providers I use).
[0]http://status.linode.com/
[1]https://ifttt.com/connect/feed/twitter
[2]https://ifttt.com/connect/feed/slack
[3]https://uptimerobot.com
Personally, I use Uptimerobot[3] to monitor my nodes and post any issues into a Slack channel. This lets me know if it's a problem with my VPS or Linode (or any of the other providers I use).
[0]http://status.linode.com/
[1]https://ifttt.com/connect/feed/twitter
[2]https://ifttt.com/connect/feed/slack
[3]https://uptimerobot.com
Yes, or a company that isn't providing the service you're paying for could be honest about it and let you know. Especially when it happens to fall on a major holiday. I am moving my account ASAP and probably wouldn't if I had heard from them before I found out myself.
Close.
By making customers aware of the downtime they will have more customers ask for credit. If you suffer 1-2days of downtime and your customer doesn't notice to ask for credit; you win.
It's terrible; but unfortunately cost savings is drilled in from every darned angle.
By making customers aware of the downtime they will have more customers ask for credit. If you suffer 1-2days of downtime and your customer doesn't notice to ask for credit; you win.
It's terrible; but unfortunately cost savings is drilled in from every darned angle.
Pretty much. I've already been billed my full monthly amount, and when asked about it they said its because its automatic and that the credits have not been processed yet...
I agree Linode have mishandled the situation, badly. The lack of information about the ongoing attacks is disconcerting. We have 40+ nodes in London and Frankfurt and are dreading the attacks resuming in those locations. Due to lack of information about the specifics of the attacks, the measures Linode are taking and an explanation why this has happened in the first place, we can't communicate any sound information back to our customers. The only way forward is to switch providers until this quiets down and more details surface.
Out of the top 10 hosting companies, only the 3 largest ones could probably have sustained these attacks, and even those providers would like suffer at least minor outages. The problem wasn't lack of prevention, but scale of attack. There comes a point where the attack, even when mitigated can cause issues.
They were able to mitigate the attacks quickly, but their bandwidth was saturated.
They were able to mitigate the attacks quickly, but their bandwidth was saturated.
I don't disagree and hopefully it didn't come across as blaming Linode for not handling the DDoS more efficiently.
My comment was with regards to their inability to communicate with customers about on-going attacks and what was being done to mitigate them. There has been (after nearing 2 weeks of DDoS) just one statement on the status page by a network engineer. No company statement, no information on the measures taken, no e-mails, no support tickets. Nada. We have customers in the Middle East that are geo-blocked from accessing nodes in London. We had to find this ourselves. [1]
Furthermore their billing system didn't account for all the downtime. Looking at our latest statement, all instances have been at 100% uptime.
[1] http://www.theregister.co.uk/2016/01/04/linode_back_at_last_...
My comment was with regards to their inability to communicate with customers about on-going attacks and what was being done to mitigate them. There has been (after nearing 2 weeks of DDoS) just one statement on the status page by a network engineer. No company statement, no information on the measures taken, no e-mails, no support tickets. Nada. We have customers in the Middle East that are geo-blocked from accessing nodes in London. We had to find this ourselves. [1]
Furthermore their billing system didn't account for all the downtime. Looking at our latest statement, all instances have been at 100% uptime.
[1] http://www.theregister.co.uk/2016/01/04/linode_back_at_last_...
Linode has always been this way. They are not extremely open from a business standpoint, but from what I can see they are fairly transparent from a technical standpoint. They have released a statement about this outage, and past outages. Smaller outages usually don't get public announcements, but often you'll see something on the forums, or you can figure out the problem by opening a ticket.
I've had pretty good communication with Linode and I've always had really good customer service. I don't think Linode is nearly as verbose about their internal business affairs and going-ons as the startup world seems to be, but I think competition has driven many companies to do the same, and the rapid evolution in this market makes it a reasonable effort to constantly keep the public in the loop.
As far as my experience, they've never with held information that I needed to know. But I understand why people are suspicious, and I think this suspicion is paranoia. These attacks are clearly espionage, and if they knew who was doing it, they'd have taken legal action by now.
I've had pretty good communication with Linode and I've always had really good customer service. I don't think Linode is nearly as verbose about their internal business affairs and going-ons as the startup world seems to be, but I think competition has driven many companies to do the same, and the rapid evolution in this market makes it a reasonable effort to constantly keep the public in the loop.
As far as my experience, they've never with held information that I needed to know. But I understand why people are suspicious, and I think this suspicion is paranoia. These attacks are clearly espionage, and if they knew who was doing it, they'd have taken legal action by now.
https://lmddgtfy.net/?q=linode%20status
Twitter is not that relevant to be honest. I assume many customers are not active on Twitter either. I do not see why it is so important to have activity on the account when there is a dedicated page for informing customers of what is going on.
Also, if you look at the medium post, the tweets are mostly sorry/thank you/everything should be fine soon. No actual information on what is going on. How could you put relevant technical details in a tweet? How does a short, uninformative tweet help you more?
edit: also, there is up-to-date technical discussions taking place on the irc channel #linode on oftc
Twitter is not that relevant to be honest. I assume many customers are not active on Twitter either. I do not see why it is so important to have activity on the account when there is a dedicated page for informing customers of what is going on.
Also, if you look at the medium post, the tweets are mostly sorry/thank you/everything should be fine soon. No actual information on what is going on. How could you put relevant technical details in a tweet? How does a short, uninformative tweet help you more?
edit: also, there is up-to-date technical discussions taking place on the irc channel #linode on oftc
I don't disagree with your general premise, but was a LMDDGTFY link (much less a LMGTFY link) necessary there to make the point?
If you look at their Twitter mentions (especially earlier in the day), you will see a ton of angry people. I'm sure lots of people aren't on Twitter, but why not engage with those that are and are feeling angry?
> How does a short uninformative tweet help you more?
The point I've been trying to get across is that a lot of people just want to have their frustrations acknowledged. An honest, real reply from a human (not a bot like lots of airlines use) can make people feel heard and do a lot to restore goodwill.
They can also link to the status page in replies for more technical details.
> How does a short uninformative tweet help you more?
The point I've been trying to get across is that a lot of people just want to have their frustrations acknowledged. An honest, real reply from a human (not a bot like lots of airlines use) can make people feel heard and do a lot to restore goodwill.
They can also link to the status page in replies for more technical details.
I guess there is also apprehension over how much to communicate, lest the info is also uses by the perpetrators.
They have dramatically improved communication in the last couple of days. They got a lot of flack for it in the thread a couple of days ago, and they seem to have responded to it. Updates are on status.linode.com.
I jokingly commented a few days back to a friend of mine that Slack is probably the last company I would bet a billion dollars on, to succeed. To be honest, I was very very wrong - Slack is incredible! Anybody can definitely build a Slack clone - its not an out of the world engineering challenge, but it will never be Slack, it will never have the love that Slack pours into its products and dev APIs. <3 Slack!
Well, the interesting part is that Slack is engaging in underhanded advertisement. I'll make sure to move the hell away from it.
I've not heard of this and am not having luck with searches. Source - details?
I honestly don't know why this comment is being downvoted to hell. I am not an investor in Slack and heck I've been an IRC user all along. I use Slack because my company forces me to. This was a genuine appreciation - I now know better to not praise good things on HN.
Not sure whatever happened to the post about "Less negativity" a few months back.
Not sure whatever happened to the post about "Less negativity" a few months back.
This is just an assumption, but maybe it is because your comment had no relevance to the topic at hand.
I am a (tiny) Linode customer with just one node but for a good number of years. Probably close to 10 now. This is the first outage I have with them. All in all it's a good thing as it made me finally learn how to use EC2 and I now have a backup there. I already shut down the EC2 backup instance and switched back to Linode as they seem to be up now.
I've used Linode increasingly since 2008, now consuming 10 times what I did at the start. I'm preparing to move to AWS today but genuinely hope that Linode comes through with a reasonable explanation of why I can expect this to not be repeated.
Well, in good news, it seems Linode's billing system was unharmed during this outage. ;)
their status page is update after update of the same message, apparently being ad libbed by someone with a thesaurus
I honestly love Linode and am sure they'll come out better as a result of this. But our customers aren't as understanding. Currently we're playing a bit of cat and mouse and with each data centre going down - we're switching our recovery process into gear and restoring to a different VPS (outside Linode). We have linodes on pretty much all locations, but if this continues at this rate, we simply won't have any linodes left there.
It would be very hard to justify going back to Linode afterwards, even with the best intentions to do so. "... So you seriously want us to go back to this hosting provider that caused us all this mess over Christmas / New Year's??"
It would be very hard to justify going back to Linode afterwards, even with the best intentions to do so. "... So you seriously want us to go back to this hosting provider that caused us all this mess over Christmas / New Year's??"
That's exactly what the reaction the attackers want.
However, what are the alternatives? Linode have been dead stable for me the past many years, and delivers what they promise in a transparent way. No overselling of servers. No sudden extra bills.
Linode will come out stronger after this, so it won't be able to happen on this scale again.
The big question is: Who, with a lot of money, would want to hurt Linode's business in this way? This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time.
However, what are the alternatives? Linode have been dead stable for me the past many years, and delivers what they promise in a transparent way. No overselling of servers. No sudden extra bills.
Linode will come out stronger after this, so it won't be able to happen on this scale again.
The big question is: Who, with a lot of money, would want to hurt Linode's business in this way? This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time.
> This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time.
These days such things aren't that hard to come by. See the recent Lizard Squad attacks for example.
These days such things aren't that hard to come by. See the recent Lizard Squad attacks for example.
> Who, with a lot of money, would want to hurt Linode's business in this way? This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time.
Serious question: how much is "a lot of money' in this case? How much does an attack like this cost?
Serious question: how much is "a lot of money' in this case? How much does an attack like this cost?
Unfortunately linode hasn't specified enough to give specifics about that.
From what they have said however, it seems to me like this is generally a very "smart" attack.
Unfortunately, a smart attack can be done much more cheaply in terms of botnet capacity than a dumb volumetric attack (this still combines a volumetric attack, but, in targeted ways).
I have delt with these kinds of attacks (defending them) and they are, unfortunately, much too cheap to pull off. Botnet capacity is dirt cheap, if you're willing to spend the time to find/compromise your own botnet hosts, its effectively free.
From what they have said however, it seems to me like this is generally a very "smart" attack.
Unfortunately, a smart attack can be done much more cheaply in terms of botnet capacity than a dumb volumetric attack (this still combines a volumetric attack, but, in targeted ways).
I have delt with these kinds of attacks (defending them) and they are, unfortunately, much too cheap to pull off. Botnet capacity is dirt cheap, if you're willing to spend the time to find/compromise your own botnet hosts, its effectively free.
Genuine question: What does anyone have to gain from Linode's misfortune? What sort of group would do this?
Maybe the some state actor / mafia drug lord... didn't like one of the customers and put enough effort to shut down the whole service.
Github was attacked by China before I believe for hosting some firewall by-passing software.
Maybe a competitor, but who would bother to go that low? I can't imagine any of the popular ones trying it -- too much risk it will be exposed.
Github was attacked by China before I believe for hosting some firewall by-passing software.
Maybe a competitor, but who would bother to go that low? I can't imagine any of the popular ones trying it -- too much risk it will be exposed.
A competitor seems like a pretty obvious answer.
But the vps market is fairly crowded, I can't imagine what attacking a single competitor would actually accomplish.
But the vps market is fairly crowded, I can't imagine what attacking a single competitor would actually accomplish.
Why would a competitor hate Atlanta in particular? Must be going after something hosted there.
> A competitor seems like a pretty obvious answer.
I don't know... unless Linode has a large presense in a country where this level of play is expected. I have hard time imagining a large US cloud / hosting company comparible in size or customer base with Linode trying it. There is too high a risk, someone will talkt and reveal it.
I don't know... unless Linode has a large presense in a country where this level of play is expected. I have hard time imagining a large US cloud / hosting company comparible in size or customer base with Linode trying it. There is too high a risk, someone will talkt and reveal it.
Usually these are extortion schemes. Pay x bitcoins to this address or else.
It could be a competitor of someone who is using linode for their hosting. Rather than attacking a single server which would be easier to track down, it's probably easier to attack the whole network.
I have been surprised that this sort of approach has not been more common than it has been. It really is a credit to us as a community that this very low hanging fruit has not been exploited more often.
You're crediting the tech community for not exploiting their abilities?
BF Skinner, famous behavioral psychologist, wrote about this in "Beyond Freedom and Dignity" -- that much of positive emotions coming from choosing to do the morally "righteous" action exist because of the capacity for evil.
BF Skinner, famous behavioral psychologist, wrote about this in "Beyond Freedom and Dignity" -- that much of positive emotions coming from choosing to do the morally "righteous" action exist because of the capacity for evil.
>You're crediting the tech community for not exploiting their abilities?
Yes I am :)
It would be easy to engage in morally bad behaviour, yet for the most part we don’t. This is a very good.
Yes I am :)
It would be easy to engage in morally bad behaviour, yet for the most part we don’t. This is a very good.
May not be the case here, but Bitcoin ransoms are become more common.
I'm definitely curious to find out, but feels more like gossip to be honest...
Given the length of time since the first attacks started and the downtime of the most recent attack, the big questions I have are:
1. What are they doing right now?
2. What mistakes were made / Why didn't they realise the risk beforehand? / (how) could this have been avoided?
3. What can they do in future to prevent this?
Given the length of time since the first attacks started and the downtime of the most recent attack, the big questions I have are:
1. What are they doing right now?
2. What mistakes were made / Why didn't they realise the risk beforehand? / (how) could this have been avoided?
3. What can they do in future to prevent this?
I agree, but I am unsure of the come back stronger sentiment. This looks to be a serious attack driven by a knowledgeably adversary. I really feel for linode at this point.
I don't know about dead stable. Fremont?
Linode has been great to us, but we can't risk further outages. We've switched over 20 nodes to Google Cloud for the time being -- thankfully before that 16+ hour outage today in Atlanta. Happy check them out again once the dust settles.
From an armchair, it seems like it's a good idea to distribute a virtual server farm on multiple providers (Linode, AWS, et al). There's even libraries available to abstract away the provider layer, like libcloud. However, IME it's typical to invest in just one provider.
Is anyone currently using libcloud or equiv. and able to share details?
Is anyone currently using libcloud or equiv. and able to share details?
Let's hope the authorities can identify the bad actor(s) in this case -- if they haven't made extortion demands it's hard not to imagine they're a competitor, and it would be really frustrating if they were able to get away with it.
If a competitor, their funders would likely be non-amused at their actions if publicised (even if already aware).
[deleted]
The brutal thing about a DDoS on a web hosting company is that it affects their business in very long lasting ways. If an ecommerce site is down they may lose sales for that day, and a small amount of customers. If a hosting provider is down then they can lose many of their customers for life.
If you look at how interconnected stuff is these days it is really not that far away (if we haven't crossed that point already) were lives will be lost due to crimes like these.
The analogy for me is one of roads. If you block a road on purpose then an ambulance might not be able to reach an accident victim in time. The internet is infrastructure, just like roads and purposefully obstructing it wholesale is doing damage to a large number of parties.
What is sad is that these people get away with this stuff over and over again, it is very rare for DDoS organizers to be caught, rarer still (if it even ever happened) for them to be sentenced.
The analogy for me is one of roads. If you block a road on purpose then an ambulance might not be able to reach an accident victim in time. The internet is infrastructure, just like roads and purposefully obstructing it wholesale is doing damage to a large number of parties.
What is sad is that these people get away with this stuff over and over again, it is very rare for DDoS organizers to be caught, rarer still (if it even ever happened) for them to be sentenced.
Work in IT. Server was stable & online for 189 days before the 25th, knew them for stability.
No notification from them, just a handful of downtime alerts during time with the family. They were completely gone from BGP tables in Newark.
Used backups and moved sites to OVH. Don't know who they pissed off, I suspect another NJ competitor, who is known for taking cheap shots at other VPS companies.
It's a pain in the ass, but at the same time, how is their network so fragile? You would think at least some of the fragile systems being attacked would be firewalled or at least ACL'd off from the public net.
This is what happens when you don't run your own network and rely on other ASN's and uplinks to do the work for you. When it comes to other customers being affected, they will simply null you. Unlike your network ops who would be trying anything they could from OOB to rectify such.
No notification from them, just a handful of downtime alerts during time with the family. They were completely gone from BGP tables in Newark.
Used backups and moved sites to OVH. Don't know who they pissed off, I suspect another NJ competitor, who is known for taking cheap shots at other VPS companies.
It's a pain in the ass, but at the same time, how is their network so fragile? You would think at least some of the fragile systems being attacked would be firewalled or at least ACL'd off from the public net.
This is what happens when you don't run your own network and rely on other ASN's and uplinks to do the work for you. When it comes to other customers being affected, they will simply null you. Unlike your network ops who would be trying anything they could from OOB to rectify such.
What you are saying has been somewhat confirmed in Linode's latest update on the Atlanta outages [1]. I can't help it but to wonder if Linode were prepared or had a plan in place in case of a DDoS? It appears their upstream provider cuts them off completely once an attack starts/resumes and gradually puts them back on. The cycle then repeats.
We are also duplicating in OVH, read good things about their built-in DDoS protection on HN.
[1] http://status.linode.com/incidents/cbbcjnhhpkgm
We are also duplicating in OVH, read good things about their built-in DDoS protection on HN.
[1] http://status.linode.com/incidents/cbbcjnhhpkgm
Wow.
I mentioned FastNetMon to them, but I just read the status update. They're blocking entire continents by communities... Holy shit. This is not some skiddie, this is likely state sponsored or BTC ransom.
Worrisome how the attacker knows so much of their infrastructure, makes me think ex-employee as he knows where to hit their servers, etc.
So glad I replied 'nope' to taking the cheaper SysAdmin position, after hand feeding them how I did mitigation. They asked me how, and were very interested in why. This was a week before this happened.
It's all making sense now. But even FastNetMon couldn't help this, you need a shitload of bandwidth (OVH size) and thousands (hundreds in cases for arbor) of equipment to match.
They need to GRE their /24's from Voxility or some large ass provider, as this is beyond fucked. I just read the status, they're cutting off parts of the internet to VMs. What in actual fuck.
I've worked in cloud for 10 years, and recently left, and will not be going back. Bare metal and OVH FTW. I can understand the 'going above and beyond' during holidays, but the lawyers I work for just want their 'f email online NOW' (direct quote)
I mentioned FastNetMon to them, but I just read the status update. They're blocking entire continents by communities... Holy shit. This is not some skiddie, this is likely state sponsored or BTC ransom.
Worrisome how the attacker knows so much of their infrastructure, makes me think ex-employee as he knows where to hit their servers, etc.
So glad I replied 'nope' to taking the cheaper SysAdmin position, after hand feeding them how I did mitigation. They asked me how, and were very interested in why. This was a week before this happened.
It's all making sense now. But even FastNetMon couldn't help this, you need a shitload of bandwidth (OVH size) and thousands (hundreds in cases for arbor) of equipment to match.
They need to GRE their /24's from Voxility or some large ass provider, as this is beyond fucked. I just read the status, they're cutting off parts of the internet to VMs. What in actual fuck.
I've worked in cloud for 10 years, and recently left, and will not be going back. Bare metal and OVH FTW. I can understand the 'going above and beyond' during holidays, but the lawyers I work for just want their 'f email online NOW' (direct quote)
What kind of VPSs are you guys using with OVH? I had a look at OVH, but to be honest got really confused with too many options to choose from... (not to mention I wasn't sure which site I should sign up to, the .com / .co.uk - is this based on the VPS location in any way?).
Linode clearly wins on simplicity and clarity. I guess under the current circumstances, I'd be willing to compromise simplicity for better availability though.
Linode clearly wins on simplicity and clarity. I guess under the current circumstances, I'd be willing to compromise simplicity for better availability though.
(I work in IT as well. Not affiliated with Linode / OVH)
Linode wins on simplicity, agreed. We are in the same boat, OVH has too many offerings. We are looking at their VPS SSD plans [1]. Last thing we want is to be offline again. As such we are also looking for anti-DDoS which is included in the plans. I intent on spinning up a few nodes to try them out first.
[1] https://www.ovh.ie/vps/vps-ssd.xml
Linode wins on simplicity, agreed. We are in the same boat, OVH has too many offerings. We are looking at their VPS SSD plans [1]. Last thing we want is to be offline again. As such we are also looking for anti-DDoS which is included in the plans. I intent on spinning up a few nodes to try them out first.
[1] https://www.ovh.ie/vps/vps-ssd.xml
Whoever has the most transit wins the mitigation game. You need to take in that traffic, then process it with a shitload of power.
OVH has 3 large datacenter PoPs to absorb attacks and do just that, then push the traffic clean back to your server.
They may blow at support and response times, but once I have a dedicated server from them, their Manager is intuitive enough to get going.
Add the fact I can get 64G server on a brand new E5 chassis with 255 free IPs for VPS of my own, and I've been moving more and more sites there as hosts get arbitrarily hit.
Piss off some competitor or skiddie and you get tested. It's ridiculous, but sadly DDoS mitigation is becoming a must.
Good time to leave being a SysAdmin in cloud and go back to web design full time as I watch a lack of best practices and SPOF take over.
Finally, I backup everything to 2 off-site locations and hope for the best.
OVH has 3 large datacenter PoPs to absorb attacks and do just that, then push the traffic clean back to your server.
They may blow at support and response times, but once I have a dedicated server from them, their Manager is intuitive enough to get going.
Add the fact I can get 64G server on a brand new E5 chassis with 255 free IPs for VPS of my own, and I've been moving more and more sites there as hosts get arbitrarily hit.
Piss off some competitor or skiddie and you get tested. It's ridiculous, but sadly DDoS mitigation is becoming a must.
Good time to leave being a SysAdmin in cloud and go back to web design full time as I watch a lack of best practices and SPOF take over.
Finally, I backup everything to 2 off-site locations and hope for the best.
[deleted]
All OVH servers are in the same data centers regardless which country's domain you buy from.
They have 3 data centers in France and one in Canada, east coast. Status and network map at https://www.ovh.co.uk/community/status/
About the options, I don't know. They updated their products on Q4 2015 and I still didn't have to buy another VPS from them so I didn't investigate.
They have 3 data centers in France and one in Canada, east coast. Status and network map at https://www.ovh.co.uk/community/status/
About the options, I don't know. They updated their products on Q4 2015 and I still didn't have to buy another VPS from them so I didn't investigate.
[deleted]
This story fall from #2 on HN to page #102 in seconds.
Is it because of only 38 upvotes vs 43 comments? The story is just 3 hours old.
@HN / dang: What's going on with HN sorting algorithm? The #3 story on HN is 1 hour old and has just 9 upvotes and 2 comments: "Churchill and His Money, or Lack of It": https://news.ycombinator.com/item?id=10825575
Screenshot: http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png
Is it because of only 38 upvotes vs 43 comments? The story is just 3 hours old.
@HN / dang: What's going on with HN sorting algorithm? The #3 story on HN is 1 hour old and has just 9 upvotes and 2 comments: "Churchill and His Money, or Lack of It": https://news.ycombinator.com/item?id=10825575
Screenshot: http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png
So I'm not the only one who saw this quickly disappear.
Couldn't find it on the 2nd or 3rd page, very odd.
Couldn't find it on the 2nd or 3rd page, very odd.
It set off the flamewar detector. Looks like an edge case in the algorithm.
Btw you should email [email protected] instead of asking us questions here. It's pretty random whether or not we see the latter.
Btw you should email [email protected] instead of asking us questions here. It's pretty random whether or not we see the latter.
In this moment I can access to my linode and my website is operational. News about status of attack?
I'm a Linode customer. I have been calling on a regular basis because I have a hosted service for my customers (thousands of them) that has been all but dead for 24 hours now. If I hear 'we are working on it' one more time, I may lose my mind. My customers are suffering severely, and I'm losing thousands of dollars as we speak. They keep telling me they are working with their upstream provider and that it's out of their hands. I'm not paying their upstream provider. I don't have a service agreement with their upstream provider. I'm paying them based on the service agreement I have with them. This is either a completely new level of DDoS or they are just completely incompetent in their way of handling it. In any case, I believe that Linode is going to suffer greatly for this in terms of lost customers. They're going to lose me by the end of next week, that's for certain.
>> This is either a completely new level of DDoS or they are just completely incompetent in their way of handling it.
My take is that it's somewhere in the middle. This type of outage has in the past, and will potentially in future, hit providers at all price points regardless of their SLAs and guarantees.
In my experience (and other comments indicate) Linode are genearlly very reliable. This isn't a mickey mouse, dirt-cheap VPS operation. The communication hasn't been awesome but by keeping an eye on status.linode.com I personally have felt reasonably well informed. If you trust they're working on it and doing the best they can.
Other commenters suggest this is a fairly sophisticated, deliberate and sustained attack. They're putting significant resources toward it. It also seems whoever is behind it has potentially gained inside knowledge of their network topology.
Based that understand, I'm not wasting my time calling them or sitting around hitting F5. I'm working to improve my systems' architecture for resiliency in this type of situation. That involves geo-distributed, multi-site redundancy and fail-over.
My advice: be optimistic and proactive.
My take is that it's somewhere in the middle. This type of outage has in the past, and will potentially in future, hit providers at all price points regardless of their SLAs and guarantees.
In my experience (and other comments indicate) Linode are genearlly very reliable. This isn't a mickey mouse, dirt-cheap VPS operation. The communication hasn't been awesome but by keeping an eye on status.linode.com I personally have felt reasonably well informed. If you trust they're working on it and doing the best they can.
Other commenters suggest this is a fairly sophisticated, deliberate and sustained attack. They're putting significant resources toward it. It also seems whoever is behind it has potentially gained inside knowledge of their network topology.
Based that understand, I'm not wasting my time calling them or sitting around hitting F5. I'm working to improve my systems' architecture for resiliency in this type of situation. That involves geo-distributed, multi-site redundancy and fail-over.
My advice: be optimistic and proactive.
Why do you not have a disaster recovery plan that allows you to restore services to your customers in case of outage or other emergency?
I'm asking sincerely. I wouldn't be able to sleep at night if I had a SaaS product with thousands of customers and no way to restore service if my primary provider went down for an extended period of time.
I'm asking sincerely. I wouldn't be able to sleep at night if I had a SaaS product with thousands of customers and no way to restore service if my primary provider went down for an extended period of time.
[deleted]
What SLA do you have with them? I'm pretty sure it is the one where they will give a credit for the time they are down, I'm also pretty sure that your hosting on max a $90 Linode so that would be like $6.
There are SLA's available from hosts that will give the whole month ,1/4 or even year back if they miss even a single month's 99.99%
This is past a joke now this whole period of downtime but still Linode is not a big dollar hosting outfit, need a plan B for when a region or all of Linode goes down.
There are SLA's available from hosts that will give the whole month ,1/4 or even year back if they miss even a single month's 99.99%
This is past a joke now this whole period of downtime but still Linode is not a big dollar hosting outfit, need a plan B for when a region or all of Linode goes down.
Every business endeavor has associated risks, which can be mitigated in a variety of ways for a variety of costs.
Offloading the responsibility for continuation of your business to Linode (or any other data center provider) is unfair. A history of uptime, verbal promises, or fancy SLA terms should never be interpreted to mean that disasters won't happen. A ten day long DDOS is a disaster and in this case a man-made disaster.
Using Linode (or another provider) instead of building your own data center is more cost effective, but it means you are no longer in direct control of your infrastructure (decreased costs, increased risk).
Designing your application to span multiple availability zones (data centers) can mitigate single points of failure within a single vendor but is more expensive than operating in a single zone.
Designing your application to span multiple vendors can mitigate single vendor failures (or changes in offerings from a single vendor) but is even more expensive.
And still there are ways to mitigate these costs, business interruption insurance can help cover the costs for moving to a new data center or vendor in case of a disaster (such as hiring staff, overtime, etc.). Lost profits can be covered by business interruption insurance also.
Of course it is expensive to operate any business in a hostile environment. A seaside restaurant better be prepared to weather a hurricane. I wonder how much money has been spent on security cameras, guards, metal detectors, and so on since 9/11? The increasing occurrence of targeted DDOS (and other types) of attacks is the physical equivalent of an increasingly hostile environment and is going to be associated with higher costs.
In the longer term, I think we need to find ways to get law enforcement better suited to deal with these problems but ultimately I think we'll need to radically change the way we handle network operations and the technical foundations of the network such as content centric networking (https://en.wikipedia.org/wiki/Content_centric_networking)
Offloading the responsibility for continuation of your business to Linode (or any other data center provider) is unfair. A history of uptime, verbal promises, or fancy SLA terms should never be interpreted to mean that disasters won't happen. A ten day long DDOS is a disaster and in this case a man-made disaster.
Using Linode (or another provider) instead of building your own data center is more cost effective, but it means you are no longer in direct control of your infrastructure (decreased costs, increased risk).
Designing your application to span multiple availability zones (data centers) can mitigate single points of failure within a single vendor but is more expensive than operating in a single zone.
Designing your application to span multiple vendors can mitigate single vendor failures (or changes in offerings from a single vendor) but is even more expensive.
And still there are ways to mitigate these costs, business interruption insurance can help cover the costs for moving to a new data center or vendor in case of a disaster (such as hiring staff, overtime, etc.). Lost profits can be covered by business interruption insurance also.
Of course it is expensive to operate any business in a hostile environment. A seaside restaurant better be prepared to weather a hurricane. I wonder how much money has been spent on security cameras, guards, metal detectors, and so on since 9/11? The increasing occurrence of targeted DDOS (and other types) of attacks is the physical equivalent of an increasingly hostile environment and is going to be associated with higher costs.
In the longer term, I think we need to find ways to get law enforcement better suited to deal with these problems but ultimately I think we'll need to radically change the way we handle network operations and the technical foundations of the network such as content centric networking (https://en.wikipedia.org/wiki/Content_centric_networking)
This is nothing less than cyber-terrorism, and I hope the FBI is involved.
A problem I see with people throwing Linode under the bus is that only the mega hosting providers (AWS/Google) have the resources to mitigate such attacks. I wound hope that the industry can find a solution out there that allows smaller players (Linode, Digital Ocean, etc) to run a hosting business without the threat of DDoS.
If everyone moves to the big players, it is a loss for us all: feature-wise, quality-wise, security-wise, cost-wise.