Preliminary Incident Statement(status.cloud.google.com)
status.cloud.google.com
Preliminary Incident Statement
https://status.cloud.google.com/incident/zall/20013
90 comments
(Disclaimer, I worked at Google over a decade ago, for about half a decade; as an SRE for about 2 of those years)
Yes, it absolutely makes sense. You never know when a typo in a config file or bug in your job-management code will go bonkers and try to take over all resources. Same think as disk quotas on computers with tons of users: you want to limit the damage that a user can accidentally (or not...) do.
Good quota systems saved my team's bacon way back when a few times, when fewer people were at the company; I can only imagine how useful they are at what, 10-20x the size?
Yes, it absolutely makes sense. You never know when a typo in a config file or bug in your job-management code will go bonkers and try to take over all resources. Same think as disk quotas on computers with tons of users: you want to limit the damage that a user can accidentally (or not...) do.
Good quota systems saved my team's bacon way back when a few times, when fewer people were at the company; I can only imagine how useful they are at what, 10-20x the size?
In particular you cannot provide an internal SLA on write requests to low-level storage solutions if you can't guarantee that you actually have the spare disks.
The SLA would have to be something like "we guarantee 99.999% write availability unless team A or team B or team C makes a mistake", so now you have to look up what team A and B and C are up to and what do they guarantee. Instead of the more sensible "we guarantee 99.999% write availability unless you are out of quota".
The SLA would have to be something like "we guarantee 99.999% write availability unless team A or team B or team C makes a mistake", so now you have to look up what team A and B and C are up to and what do they guarantee. Instead of the more sensible "we guarantee 99.999% write availability unless you are out of quota".
Exactly. At some point, someone somewhere will have a bad day / sneeze on a keyboard at just the perfectly wrong time -- at Google's scale, that's statistically a daily (hourly?) occurrence.
Quota systems for the win -- if only as a shared agreement on who can do how much damage / take how much resources before they're automatically stopped and more resources must be justified through some review process (likely involving budgetary concerns).
Quota systems for the win -- if only as a shared agreement on who can do how much damage / take how much resources before they're automatically stopped and more resources must be justified through some review process (likely involving budgetary concerns).
Shouldn't there be a soft limit that start sending out warnings before hitting the hard one ?
Unless some service started eating the quota so fast that it reached both limits in quick succession, the upper limit should never be reached.
I can't comment on the current incident, as I've been gone for more than a decade.
But my recollection is that most SRE teams would have had exactly such things -- either as really crazy borgcfg (sorry, I mean kubectl) code, or as a borgmon (sorry, prometheus) alert.
The former were a real PITA to work with (borgcfg hadn't been designed with that in mind), and the latter were only reactive (and thus unable to warn about things until they were fast becoming a real problem).
So most likely those safeguards were somehow disabled or made ineffective by the exact circumstances of the problem, possibly the speed at which things developed.
But my recollection is that most SRE teams would have had exactly such things -- either as really crazy borgcfg (sorry, I mean kubectl) code, or as a borgmon (sorry, prometheus) alert.
The former were a real PITA to work with (borgcfg hadn't been designed with that in mind), and the latter were only reactive (and thus unable to warn about things until they were fast becoming a real problem).
So most likely those safeguards were somehow disabled or made ineffective by the exact circumstances of the problem, possibly the speed at which things developed.
From the article capacity was reduced
You're right.
Still, a better approach could have been adopted, like blocking a change that would cause some service to go over quota, or have the automatic system change the quotas gradually, thus giving time for alerts and human intervention
I'm sure that this Google outage was the combination of several independent issues. Perhaps there are the kind of safeguards you mentioned in place, but they returned an incorrect view of the world because of a bug exposed by a network partition? The point is that with huge distributed systems the things take them down (at least once they're mature) are typically the result of several failures that compound and interact in a way that wasn't foreseen. IMO it's more likely that the suggestions you made (blocks and/or automation) are already in place but failed in some more complex way.
Source: worked at a FAANG for a decade, saw many big incidents. Almost all were the confluence of several smaller issues that, had they happened alone, wouldn't have been newsworthy.
Source: worked at a FAANG for a decade, saw many big incidents. Almost all were the confluence of several smaller issues that, had they happened alone, wouldn't have been newsworthy.
Well here’s the thing though. Google are removing quotas from things like appengine. How can it be simultaneously helpful to have quotas but at the same time google wants to remove them?
I think you're conflating very different things. Quotas may be removed if you've got a virtually unlimited amount of a certain resource (and even there they can be useful to prevent overbilling), but disk quotas are necessary because disk space is not unlimited. Without those, a noisy neighbour can break your application.
byecomputer(4)
Yes, because in a large organisation where your internal services are just "out there" for other teams to use, and you might not personally know the people who are using your service, you basically need to protect yourself/your service as much as you would for any external service. It also helps your service know how much it needs to scale and size itself if a team gets in touch and says "Hey we are making Service X - the default your service offers is 100 QPS but we think we will need 1000 QPS - can you support that?".
In a small org where you know/roughly-know who is doing stuff then it is probably less of a concern as you can go walk over to their desk and have a chat about what they are doing or let them know they're sending too much traffic etc. This is harder to do with an org with 10s of thousands of engineers spread across the globe and in different timezones - you might suddenly get huge amounts of traffic at 3am from a team on the other side of the planet who just launched a feature you had zero idea about, and they're hammering you with 100x the QPS you usually get.
In a small org where you know/roughly-know who is doing stuff then it is probably less of a concern as you can go walk over to their desk and have a chat about what they are doing or let them know they're sending too much traffic etc. This is harder to do with an org with 10s of thousands of engineers spread across the globe and in different timezones - you might suddenly get huge amounts of traffic at 3am from a team on the other side of the planet who just launched a feature you had zero idea about, and they're hammering you with 100x the QPS you usually get.
[deleted]
Consider: my team owns a public-facing website for AWS and relies on a number of back-end services to support it. We occasionally face DoS attacks. And sometimes, they can get past our various security measures to prevent them.
By everyone else having quotas on how often I can call their systems, they protect every other client they have. They protect customers who are using anything other than my website. Otherwise, someone could take down all of AWS by spamming just one system. I want them to throttle me, quota me, because I want to protect other customers if I get compromised.
And even if it's one internal system calling another, it's better to have protection at every layer than to try to only have it on the outer layer.
This can burn us, of course. We're always growing, and quotas need to be managed (automatically or manually). It all has to be done carefully.
Edit: as always, I speak for myself and not for AWS in any official capacity. I'm mabbo, not jeffbar :)
By everyone else having quotas on how often I can call their systems, they protect every other client they have. They protect customers who are using anything other than my website. Otherwise, someone could take down all of AWS by spamming just one system. I want them to throttle me, quota me, because I want to protect other customers if I get compromised.
And even if it's one internal system calling another, it's better to have protection at every layer than to try to only have it on the outer layer.
This can burn us, of course. We're always growing, and quotas need to be managed (automatically or manually). It all has to be done carefully.
Edit: as always, I speak for myself and not for AWS in any official capacity. I'm mabbo, not jeffbar :)
Definitely. One thing that grieved me about the attempts at microservice architectures I had to work on in my previous jobs was the complete lack of administration; in theory, a microservice can be called by any other service, but what we never implemented was a kind of authentication or logging (e.g. access keys), to track which other services used the one. It quickly became an untransparent mess.
Yes,
usually developers (and I'm one of those) don't think about how to size correctly the cloud. You could incur in costs you don't really need.
Although in my company we generate an alert and a report so that people are aware and can change instead of letting a system doing it automatically for you.
To be fair, if they didn't use it due to concerns about availability, then that would suggest they needed more work on the service.
I imagine once they knew what the issue was, they would have been able to fix it as i'm pretty sure that service doesn't need to be authorised by itself to be configured ;)
The trouble I imagine was getting visiblity of what the issue was...
I imagine once they knew what the issue was, they would have been able to fix it as i'm pretty sure that service doesn't need to be authorised by itself to be configured ;)
The trouble I imagine was getting visiblity of what the issue was...
> I imagine once they knew what the issue was, they would have been able to fix it as i'm pretty sure that service doesn't need to be authorised by itself to be configured ;)
You'd be surprised at the circular dependencies that creep up at Google scale...
(I used to work there and we had last resort secure access mechanisms, but I wouldn't be surprised if the normal front door wound up shut pretty hard for this one)
You'd be surprised at the circular dependencies that creep up at Google scale...
(I used to work there and we had last resort secure access mechanisms, but I wouldn't be surprised if the normal front door wound up shut pretty hard for this one)
That sounds exactly like the new autopilot system they describe in their USENIX paper.
Talk: https://eventsonair.withgoogle.com/events/autopilot-research...
Paper: https://research.google/pubs/pub49174/
Talk: https://eventsonair.withgoogle.com/events/autopilot-research...
Paper: https://research.google/pubs/pub49174/
Googler, opinions are my own.
The paper and the system that caused this incident are different. Google has a ton of different automated systems for maintaining production.
The paper you linked is for a system called autopilot, which can scale a specific jobs memory and CPU usage up or down depending on historical load of that job. Think of it as having different instance sizes on GCP or AWS, then you have a tool that monitors how much actual CPU and memory your job uses on the instance, in will upsize or downsize your machine depending on load over time.
The system that broke based on the incident report above had to do with quota that a given production system could use as a maximum. Similar resource management automation, but very different systems.
So one is about fine-tuning, while the other is about preventing excessive resource usage.
The paper and the system that caused this incident are different. Google has a ton of different automated systems for maintaining production.
The paper you linked is for a system called autopilot, which can scale a specific jobs memory and CPU usage up or down depending on historical load of that job. Think of it as having different instance sizes on GCP or AWS, then you have a tool that monitors how much actual CPU and memory your job uses on the instance, in will upsize or downsize your machine depending on load over time.
The system that broke based on the incident report above had to do with quota that a given production system could use as a maximum. Similar resource management automation, but very different systems.
So one is about fine-tuning, while the other is about preventing excessive resource usage.
I guess at least someone's buying the largest Google One storage plan next year...
And now we know why they are limiting their free space offerings! ;)
What does this mean? I don't see any mention of Google One or storage on the page.
I guess I'm confused like @raybb. Why would a service outage make it more likely that someone will opt in to a storage plan? I wouldn't want to jump into a service that's been recently demonstrated to be unavailable for extended periods of time.
I think the parent is jokingly suggesting that google's auth service is going to be buying a largest google one storage plan so they don't run out of quota.
Many of our internal users and tools experienced similar errors
automated quota management system
At least it's not just us pleb users at the mercy of their automated systems. I wonder if someone internally appealed the decision and got told their review had been carefully considered and subsequently declined.
At least it's not just us pleb users at the mercy of their automated systems. I wonder if someone internally appealed the decision and got told their review had been carefully considered and subsequently declined.
Record-breaking regulatory fines? Baked into the budget. Picking up the phone? Unfeasible, can't be done.
Yeah was thinking the same.
"But picking up the phone doesn't scale"
Me: looks at size of fines paid in last 10 years.
"But picking up the phone doesn't scale"
Me: looks at size of fines paid in last 10 years.
Also fine with payrolling a call center that schedules haircutting appointments and restaurant reservations when Google Duplex shorts out—but clearly that's a temporary stopgap for a new technology, unlike the quite finished Google Play robot which bans your app and then makes you guess where and what exactly the problem was.
The fines seem to scale just... fine.
One a cost of doing business, the other a line item to be “optimised”.
I wonder if we'll ever get to see the post-mortem of specifically the status page saying that everything's fine when not everything's fine. That's the one that interests me most.
When AWS had their major outage a few weeks ago, comments indicated that the status pages are updated manually at these companies. It's heavily influenced by contractual and internal political factors.
Building incentives to not truthfully report into those contracts with uptime guarantees like the WHO and pandemic bonds.
Unfortunately, much of the world is built like this. Those who draw attention to failures get blamed, not rewarded (even from a young age: "he who smelt it, dealt it").
Did you miss the "Additional Details" section?
> Many of our internal users and tools experienced similar errors, which added delays to our outage external communication.
Also:
> We will publish an analysis of this incident once we have completed our internal investigation.
> Many of our internal users and tools experienced similar errors, which added delays to our outage external communication.
Also:
> We will publish an analysis of this incident once we have completed our internal investigation.
This post would better be named like "Google Outage Preliminary Incident Statement". I had no idea what I'm gonna see before I read the link.
It's suggested in the HN guidelines to keep the original title whenever possible and not to editorialize.
But I agree, without prior knowledge of yesterday's outtake it's hard to guess what's behind that title.
It’s not really editorializing to add detail of what a vague thing is referring to.
Editorializing would be changing it to something like “Breaking the silence after a catastrophic outage caused by our gross incompetence”.
Editorializing would be changing it to something like “Breaking the silence after a catastrophic outage caused by our gross incompetence”.
Hah yes.
Or what was that one we had the other day?
"The death of Google"
Or what was that one we had the other day?
"The death of Google"
It's ok to use a subtitle or a representative phrase from the article body, if that better expresses what the submission is about. The main thing we're trying to avoid is people cherry-picking some detail to put in the title or using it to put their own spin on a story.
Since "Preliminary Incident Statement" is basically a subtitle in the OP, I've put that in the title field above.
Since "Preliminary Incident Statement" is basically a subtitle in the OP, I've put that in the title field above.
Mod policy is to replace any useful or informative submitter-provided title with the title of the linked page, so the result would have been the same after a few hours.
Well we could see its google's website
Preliminary Incident Statement for google outage
"Don't split us the google company! We are tightly coupled, things will go wrong, your precious gmail account will fail more"
The root cause was an issue in our automated quota management system which reduced capacity for Google's central identity management system, causing it to return errors globally.
.... so much for limiting Blast Radius
.... so much for limiting Blast Radius
On the flipside, isn't this just a case where you are bound by the CAP theorem? If you want your central identity management database to be consistent and available, as I imagine Google do, it can't be parallel?
Note IAM is also global in AWS, perhaps the only global service other than those that are effectively global by requiring you to use us-east-1
If all of your services make use of one service, maybe that should be treated as very special case...
What do you mean by blast radius?
The area (in this case, number of services or regions) impacted by an outage.
The fact one service had an issue. And not only did it take said service down globally, it also impacted dozens of other services...globally.
A good infra architecture has the “blast radius” of any issue confined to only part of your infra fleet. Avoiding the global outage.
Think of it like a navy ship. When a mussel breaches the hull, the ship is designed to contain the leak in a single section. Avoiding the entire ship sinking. Similar pattern is desired in your service and compute infrastructure.
The outage google just faced is equivalent to a single missel taking down an entire navy fleet.
A good infra architecture has the “blast radius” of any issue confined to only part of your infra fleet. Avoiding the global outage.
Think of it like a navy ship. When a mussel breaches the hull, the ship is designed to contain the leak in a single section. Avoiding the entire ship sinking. Similar pattern is desired in your service and compute infrastructure.
The outage google just faced is equivalent to a single missel taking down an entire navy fleet.
Petition to have Blast Radius Awards per year among top companies pls
I love this analogy, thanks for visuals.
> The outage google just faced is equivalent to a single missel taking down an entire navy fleet.
I really thought you were going to say "is equivalent to a single missile taking down an entire ship" - but your extreme analogy made even more sense in this context.
> The outage google just faced is equivalent to a single missel taking down an entire navy fleet.
I really thought you were going to say "is equivalent to a single missile taking down an entire ship" - but your extreme analogy made even more sense in this context.
How could one design a system that would not fail when something as critical and cross-cutting as capacity or identity management failed?
You can limit the damage though; in theory there’s no reason why authentication needs to be centralised. You could split the backend up into multiple authentication services and “bucket” the auth requests per email address or something. So essentially make a bunch of different authentication services that can fail independently. I’m sure they have good reasons for not doing that though.
For example youtube should have defaulted to the logged out experiance. No one forced them to use the same identity platform for completly unrelated services.
Gmail dose raise hard issues but people logged in should not have needed to recontact a auth service.
[deleted]
[deleted]
[deleted]
Curious how Spotify and other enterprise GCP users have mitigated GCPs repeated global outages. This isn’t their first nor will it be their last.
Services hosted on GCP didn't lose any traffic during this incident. This was a control plane outage, not a data plane outage. That means that if you needed to go in and adjust something to resolve your own outage, you were in trouble, but if your services were healthy, nothing bad happened.
As the time of the outage increases, service health probably trends to zero without being able to manage the service... but for a few hours, it's not a disaster. Usually.
As the time of the outage increases, service health probably trends to zero without being able to manage the service... but for a few hours, it's not a disaster. Usually.
The Google Cloud was almost unaffected, during the outage you could not log into the cloud console or use tools like gcloud - but the services itself (vms, gke, pubsub, etc) kept working throughout the outage.
That's not correct, you wouldn't be able to connect these services after restart. E.g. you wouldn't be able to auth to Cloud SQL or Datastore. Stopping development
Even Google!
>Even Google!
The time when Google was special is long gone.
The time when Google was special is long gone.
How so?
Long live the flame bait
Am I the only one reading: "someone launched a set replica 0 in prod"? :D
just kidding.
just kidding.
I'm curious, as I might learn something from interesting asnwers to the above questions.