Cloudwatch just stopped triggering our Lambdas(twitter.com)
twitter.com
Cloudwatch just stopped triggering our Lambdas
https://twitter.com/bugaralife/status/1331604999802122240
28 comments
This seems like a larger system outage, we are seeing issues with Kinesis and Cloudwatch.
Cloudwatch, cloudwatch events, and kinesis events in us-east-1 have been down for about 2 hours now
For folks who go in on managed services because self-hosted has the danger of an outage - how does this factor? I self-host lots of my company services and frequently catch heck from my peers for it: if your "x" goes down it costs more than the managed. (It's more than just cost tho)
Seems like folk factor managed as if it was zero-downtime game and that self-hosted is an huge burden.
Maybe these monitors aren't critical but what about when it is?
Seems like folk factor managed as if it was zero-downtime game and that self-hosted is an huge burden.
Maybe these monitors aren't critical but what about when it is?
In some sense I suppose if AWS goes down in some fashion it's kind of like an act of nature. There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix.
There is also (and especially if your a small shop) the thing where if your self hosted stuff goes down, you have a crappy time fixing it, over weekend, holiday, etc, until its done.
If AWS goes down, thousands of other people are working on it frantically. And most likely, many of them are smarter/better than you.
If AWS goes down, thousands of other people are working on it frantically. And most likely, many of them are smarter/better than you.
So this just happened to a friend of mine who used managed.com, which was down for a week last week because of a ransomware attack. Didn't really get mentioned here, but that was a lot of websites.
Instead of stressing about getting his sites back up, he was stressing that he'd never get any of his data or code back and that he couldn't contact his 100,000s of users because he didn't even have a recent backup of the database with all of the emails.
Even his DNS settings went down as those sort of sites push you to making them the nameservers.
I ended up telling him about wayback machine just so he could get a homepage set up.
He'd relied on their claims that they managed his backups for him, but when they went down, it took down his ability to even access the backups, and they wouldn't do anything, I assume in case those backups were infected too.
So in fact if you're blindly relying on an external provider, it's even more risky than hosting yourself. At least the stuff you self host you are almost pushed to back up to something like S3, and both won't go down.
Instead of stressing about getting his sites back up, he was stressing that he'd never get any of his data or code back and that he couldn't contact his 100,000s of users because he didn't even have a recent backup of the database with all of the emails.
Even his DNS settings went down as those sort of sites push you to making them the nameservers.
I ended up telling him about wayback machine just so he could get a homepage set up.
He'd relied on their claims that they managed his backups for him, but when they went down, it took down his ability to even access the backups, and they wouldn't do anything, I assume in case those backups were infected too.
So in fact if you're blindly relying on an external provider, it's even more risky than hosting yourself. At least the stuff you self host you are almost pushed to back up to something like S3, and both won't go down.
The entire premise of claiming to have backups is that you are the fork who ensures multiple copies have no commonalities, no overlap, in what keeps them healthy.
You can delegate if you aren't interested in DIY and getting in the trenches, but you still need to apply the original premise by being a fork and delegating to multiple separate entities...
Otherwise, you can't claim to have backups, despite the fact that the one basket you put your eggs in can legitimately claim to have backups.
You can delegate if you aren't interested in DIY and getting in the trenches, but you still need to apply the original premise by being a fork and delegating to multiple separate entities...
Otherwise, you can't claim to have backups, despite the fact that the one basket you put your eggs in can legitimately claim to have backups.
Single tenancy systems are simpler and avoid neighbourhood noise and safety limits.
In other words we're in the "Nobody gets fired for choosing managed AWS" paradigm.
Still if I cause P2/P1 issue I still need to explain to the board of top 50 fortune 500 company. Had to do it once its no fun at all.
Basically. If you go down you're probably in the company of half the fortune 500.
And of course AWS is a legitimately capable and useful platform that does remove a lot "ops" although it probably frontloads a lot of "config" in return for that.
And of course AWS is a legitimately capable and useful platform that does remove a lot "ops" although it probably frontloads a lot of "config" in return for that.
It's not just diffusion of responsibility. It's the fact that — assuming you're a B2B SaaS — when you're down, the customers depending on your service are probably down too, since they're very likely to also be dependent on the same service that knocked you offline.
It's like a city-wide power outage, or a flood in a city that's in a flood-basin. The city's entire economy will "pause." If you're a B2B business affected by the outage/flood, you don't have to worry about losing sales/customers, as your customers won't even be trying to make any requests of you, since their capacity to serve requests to their customers is knocked out, and without that, they won't be making any requests of you. In fact, as long as they're down, they may not even notice that you're down!
It's like a city-wide power outage, or a flood in a city that's in a flood-basin. The city's entire economy will "pause." If you're a B2B business affected by the outage/flood, you don't have to worry about losing sales/customers, as your customers won't even be trying to make any requests of you, since their capacity to serve requests to their customers is knocked out, and without that, they won't be making any requests of you. In fact, as long as they're down, they may not even notice that you're down!
>There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix.
I feel like this statement perfectly embodies the problems that people face when choosing a closed source OS vs Linux/BSD. When stuff doesn't work on Windows there's pretty much nothing you can do so you have to just accept it and pretend that you wanted it to not work this way. As opposed to Linux where you can make it work your way, but it costs a lot of time to do that.
I feel like this statement perfectly embodies the problems that people face when choosing a closed source OS vs Linux/BSD. When stuff doesn't work on Windows there's pretty much nothing you can do so you have to just accept it and pretend that you wanted it to not work this way. As opposed to Linux where you can make it work your way, but it costs a lot of time to do that.
a) Don't deploy prod to us-east-1
b) Don't stick to a single region/ AZ. Cross-region/AZ failures are quite rare
c) If AWS goes down a huge portion of the internet is fucked. Customer expectations are totally different. When 90% of the internet isn't loading images no one's coming to you to complain.
d) The fire is elsewhere. Someone else is on it. At worst we spend some time trying to work around it, at best we work on other things while we wait.
b) Don't stick to a single region/ AZ. Cross-region/AZ failures are quite rare
c) If AWS goes down a huge portion of the internet is fucked. Customer expectations are totally different. When 90% of the internet isn't loading images no one's coming to you to complain.
d) The fire is elsewhere. Someone else is on it. At worst we spend some time trying to work around it, at best we work on other things while we wait.
>> For folks who go in on managed services because self-hosted has the danger of an outage
Seems like an overly simplistic and narrow view of the tradeoffs. The mainstream cloud services providers operate at very high levels of availability, but they nonetheless still have the "danger of an outage." There are many reasons why managed infrastructure may make sense for an organization, from ease of scaling to the completeness of the available solution space and cost efficiencies, but "absolutely avoiding any chance of downtime" isn't one of them.
Seems like an overly simplistic and narrow view of the tradeoffs. The mainstream cloud services providers operate at very high levels of availability, but they nonetheless still have the "danger of an outage." There are many reasons why managed infrastructure may make sense for an organization, from ease of scaling to the completeness of the available solution space and cost efficiencies, but "absolutely avoiding any chance of downtime" isn't one of them.
If I'm using a managed service, it's more that I am expecting to get better reliability and less work on our part than it is expecting perfect reliability and no work on our part, both of which are a pipe dream no matter which route you go.
if big companies go down. the ops guy can blame it on the hosting provider esepcially if its amazon. no place can guarantee 100% uptime but if you pick some joe crabs hosting you will be blamed but if you pick amazon or azure the company will be like oh well shit happens.
From their own status page:
"We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in errors for Kinesis in the US-EAST-1 Region. It's not posted on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD."
"We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in errors for Kinesis in the US-EAST-1 Region. It's not posted on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD."
> "It's not posted on SHD as the issue has impacted our ability to post there."
At the risk of stating the obvious...why host your status page of a service on something that depends on that service?!
At the risk of stating the obvious...why host your status page of a service on something that depends on that service?!
Had Service Unavailable for about 10 mins on Cloudwatch events. Just corrected itself for me at least. us-east-1
It has been disappointing to see how many people on HN have business-critical production deployments in a single AZ. I thought we learned not to trust any one AWS location years ago when Netflix learned for us.
I recently joined an org that has all their eggs in the us-east-1 basket. I was warned as much by the devops lead that it was at least a partially conscious decision that the cost of multi-region would outweigh the risk. I think I'm going to start questioning that point of view.
This is affecting all of us-east-1, not just a single availability zone.
Over the past few years, it's become easier to host an application in multiple regions, but it's still a heavy lift, requiring a lot of thought not just from site reliability engineers, but application architecture.
Over the past few years, it's become easier to host an application in multiple regions, but it's still a heavy lift, requiring a lot of thought not just from site reliability engineers, but application architecture.
AWS status board is reporting a major outage in Kinesis for us-east-1. Causing big problems for us right now.
Is anyone else getting "Capacity unavailable" when trying to add tasks in Fargate?
[deleted]