Partial Cloudflare outage on October 25, 2022(blog.cloudflare.com)
blog.cloudflare.com
Partial Cloudflare outage on October 25, 2022
https://blog.cloudflare.com/partial-cloudflare-outage-on-october-25-2022/
60 comments
Would love to learn more about why a tracing function mutates the request (clears headers). That seems like foot gun that would be impossible for anyone not intimately familiar with the function's implementation to avoid.
Internal retro is going to be digging into that question (amongst others). I imagine we’ll update the post with further details.
Mutating the headers might make sense if it needs to inject a trace ID into the headers. That doesn't explain why it would clear all headers though.
I suspect it'll be a trivial programming error that passed through tests that didn't test whether it leaves other headers unaffected as well as code review.
Related:
Cloudflare CDN Partial Outage - https://news.ycombinator.com/item?id=33333332 - Oct 2022 (54 comments)
Cloudflare CDN Partial Outage - https://news.ycombinator.com/item?id=33333332 - Oct 2022 (54 comments)
The incident report is very light on details. For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers or how they'd contain it in the future, except for that they'd need to "fail fast" in this situation.
(You can obviously catch it with a blue-green deployment, which they do mention, but at that point some small portion of traffic has already been affected.)
(You can obviously catch it with a blue-green deployment, which they do mention, but at that point some small portion of traffic has already been affected.)
This incident report was out within hours of the failure. The team is now doing an internal retro to look at decision speed, rollback speed, and the circumstances around that function having a side effect. We'll update the post with more detail.
> We'll update the post with more detail.
Just a small suggestion: there's nothing in the post (unless I missed something) that mentions or even implies that this is not the "final report". You could have a small paragraph at the end saying "we might update this post further once we have more details", or even better, have a bullet point list of the things your team is still looking at. That way, readers of the blog post would know it might be worth it to come back later and read the post again.
Just a small suggestion: there's nothing in the post (unless I missed something) that mentions or even implies that this is not the "final report". You could have a small paragraph at the end saying "we might update this post further once we have more details", or even better, have a bullet point list of the things your team is still looking at. That way, readers of the blog post would know it might be worth it to come back later and read the post again.
> For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers
They probably don't need to at all, but I can think or several ways it might unintentionally happen.
Perhaps someone chose to implement the modification as “separate headers & body, modify headers, put the request back together” and something went wrong that made the putting-back-together fail quietly (so the now malformed request passes to the next part without an exception being raised) or not get called at all in some circumstances. Alternately a badly coded routine to remove a header, perhaps the instrumentation information after the point in the processes it is no longer useful, maybe removed more than it should.
How such things get past a code review without someone suggesting a less error-prone method might be a question that gets asked in the internal investigation. I assume somewhere like CF has many checks between initial code modifications and things dropping into production, unless they've really taken “fail fast” to heart!
They probably don't need to at all, but I can think or several ways it might unintentionally happen.
Perhaps someone chose to implement the modification as “separate headers & body, modify headers, put the request back together” and something went wrong that made the putting-back-together fail quietly (so the now malformed request passes to the next part without an exception being raised) or not get called at all in some circumstances. Alternately a badly coded routine to remove a header, perhaps the instrumentation information after the point in the processes it is no longer useful, maybe removed more than it should.
How such things get past a code review without someone suggesting a less error-prone method might be a question that gets asked in the internal investigation. I assume somewhere like CF has many checks between initial code modifications and things dropping into production, unless they've really taken “fail fast” to heart!
Yes they probably can fill in plenty of detail, but overall I was impressed with the incident report on it's own.. I don't think it's super common for companies' their size to release a timeline/report on the fkup unless they're coerced to.
It seems to me that the rollout process is flawed.
A change should be rolled out to a small fraction of nodes and then monitored for an extended period. 5% of requests returning errors would easily be spotable. Only when a new release has run stable on that portion for some time should it proceed to a bigger subset. You probably want to do this in several steps at the size of CF.
We also learn that it took customer reports to get the investigations rolling but 5xx errors are easily monitored so it points at internal monitoring being lacking even though it's hard for me to believe that they don't have an eye on this already.
It's not the first time that a deploy has brought CloudFlare (partially) down. From the timeline we see there's several hours between incident investigations starting and the rollout being stopped. The rollback should be the first thing considered even before looking at what the actual issue is.
Ideally you have someone sitting next to a red rollback button during a rollout whose only job is it to have an eye on all automatic and customer error reports. :)
A change should be rolled out to a small fraction of nodes and then monitored for an extended period. 5% of requests returning errors would easily be spotable. Only when a new release has run stable on that portion for some time should it proceed to a bigger subset. You probably want to do this in several steps at the size of CF.
We also learn that it took customer reports to get the investigations rolling but 5xx errors are easily monitored so it points at internal monitoring being lacking even though it's hard for me to believe that they don't have an eye on this already.
It's not the first time that a deploy has brought CloudFlare (partially) down. From the timeline we see there's several hours between incident investigations starting and the rollout being stopped. The rollback should be the first thing considered even before looking at what the actual issue is.
Ideally you have someone sitting next to a red rollback button during a rollout whose only job is it to have an eye on all automatic and customer error reports. :)
They did a canary deploy in a data center that didn't have hierarchical caching enabled, so they missed this code path.
87 sounds large
The 503 graph appears to show a steep drop between 17:38 and and 17:45, which looks like the "rollback" speed to me.
I think they likely drained many of the lower tier caches/pops/edges after which roll back speed doesn't matter very much since the machines being rolled back likely weren't servicing traffic.
> large data centers acting as Upper tier for many customers.
To me this sounds like a very awkwardly worded way to say "we shifted traffic".
Clarification on whether traffic was shifted or tiers were drained would be nice.
2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.
2022-10-25 17:28: Peak impact with approximately 5% of all HTTP requests resulting in an error with status code 530.
2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.
2022-10-25 18:04: Rollback is complete in all Upper Tiers.
2022-10-25 18:30: Rollback is complete.
The 503 graph appears to show a steep drop between 17:38 and and 17:45, which looks like the "rollback" speed to me.
I think they likely drained many of the lower tier caches/pops/edges after which roll back speed doesn't matter very much since the machines being rolled back likely weren't servicing traffic.
> large data centers acting as Upper tier for many customers.
To me this sounds like a very awkwardly worded way to say "we shifted traffic".
Clarification on whether traffic was shifted or tiers were drained would be nice.
Additionally, this was a code release based outage(?). It seems like a graph annotated with releases (or a/b testing changes) would have made this outage somewhat trivial to identify.
How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?
2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors.
2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.
How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?
I dunno; they say
> At first glance, the errors looked like they were caused by a different system that had started a release some time before.
And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.
> At first glance, the errors looked like they were caused by a different system that had started a release some time before.
And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.
Is this why Apple's iMessage was down for a bit - are they maybe using CloudFlare?
[deleted]
Unfortunately, my sites are still not ok. I'm getting HTTP 409 errors on some of the domains that CNAME to my Cloudflare domains. I've looked everywhere and I still cannot get this resolved.
Can you email me (jgc) details?
Surprised it both took them so long to decide to rollback and that the rollback lasted so long.
> [..] and that the rollback lasted so long
That does not surprise me. There are companies working with CI/CD systems whose tests easily take more than an hour. And until they are completed, they cannot start releasing (enforced via system permissions). This is a good thing, but the possibility of an emergency rollback should be considered beforehand and maybe (it always depends) exceptions made for such purposes.
And sometimes it makes more sense to revert slowly to make sure it doesn't get worse.
That does not surprise me. There are companies working with CI/CD systems whose tests easily take more than an hour. And until they are completed, they cannot start releasing (enforced via system permissions). This is a good thing, but the possibility of an emergency rollback should be considered beforehand and maybe (it always depends) exceptions made for such purposes.
And sometimes it makes more sense to revert slowly to make sure it doesn't get worse.
Yeah,
> Once identified we expedited a rollback which completed in 87 minutes.
I appreciate that it's a massive infrastructure spanning the entire globe, but... 87 minutes to revert a change? I wouldn't want the job of fixing it, but that doesn't seem good enough when the impact is that bad.
> Once identified we expedited a rollback which completed in 87 minutes.
I appreciate that it's a massive infrastructure spanning the entire globe, but... 87 minutes to revert a change? I wouldn't want the job of fixing it, but that doesn't seem good enough when the impact is that bad.
I don't know how cloudflare is configured, but doing anything fast to caches is generally a really bad idea. Caches often protect upstream data stores from overload. If an upstream data store becomes overloaded with too much traffic, then all requests that use that data store will start to slow down. As requests slow down, they will either timeout and fail resulting in major error spikes or they will start to consume all the resources of the service making requests.
So if any caches were cleared in the process of rolling back, a slow rollback makes perfect sense to me.
I would expect a company like cloudflare would have previous revisions of software on their machines that could be reverted to near instantly.
Another thing to consider is what capacity they run at and how long a service takes to initialize. If you can only take 10% of your machines out at any given moment and each one takes a while to initialize, you don't really have much choice in the matter.
Lastly, if you look at the 530 graph, it seems that the drop in 530's was almost instant.
> 2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.
This leads me to believe they shifted traffic away from the low tiers, which enabled them to roll back as slowly as they like at the cost of a likely minor increase in latency.
So if any caches were cleared in the process of rolling back, a slow rollback makes perfect sense to me.
I would expect a company like cloudflare would have previous revisions of software on their machines that could be reverted to near instantly.
Another thing to consider is what capacity they run at and how long a service takes to initialize. If you can only take 10% of your machines out at any given moment and each one takes a while to initialize, you don't really have much choice in the matter.
Lastly, if you look at the 530 graph, it seems that the drop in 530's was almost instant.
> 2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.
This leads me to believe they shifted traffic away from the low tiers, which enabled them to roll back as slowly as they like at the cost of a likely minor increase in latency.
What happens if a rollback somehow makes things worse? Even emergency rollbacks need a gradual rollout so you’ve got a chance to catch any new issues you didn’t expect.
Ah, good point - I was reading it as 87 minutes for all of it, but it could easily be 1 minute per datacenter or whatever, and intentionally incremental, which would make a lot more sense.
> 87 minutes to revert a change?
If their processes require a suite of tests be run on staging systems before the revert hits production, that may take considerable time depending on how much is potentially affected.
Also the 87 minutes might be for everything, perhaps it was a rolling update: update some nodes, test, update the next batch, repeat. This might mean that a good chunk of the infrastructure was updated for sooner and traffic could be diverted through those parts (presumably those nodes are over-specified, or just easily scaled, to be able to take a glut of extra load - as they would need to in more normal circumstances in order to contend with waxing & waning traffic patterns).
Perhaps it is even the case that the errant update didn't get out to all areas before the problem became apparent, so they were not rolling back over the whole network. Again this may offer the opportunity to divert load to non-affected areas while the others are rolled back.
So while the full revert took 87 minutes, things may have been up and running properly (except perhaps a little slower than usual) for most users/processes much quicker than that.
If their processes require a suite of tests be run on staging systems before the revert hits production, that may take considerable time depending on how much is potentially affected.
Also the 87 minutes might be for everything, perhaps it was a rolling update: update some nodes, test, update the next batch, repeat. This might mean that a good chunk of the infrastructure was updated for sooner and traffic could be diverted through those parts (presumably those nodes are over-specified, or just easily scaled, to be able to take a glut of extra load - as they would need to in more normal circumstances in order to contend with waxing & waning traffic patterns).
Perhaps it is even the case that the errant update didn't get out to all areas before the problem became apparent, so they were not rolling back over the whole network. Again this may offer the opportunity to divert load to non-affected areas while the others are rolled back.
So while the full revert took 87 minutes, things may have been up and running properly (except perhaps a little slower than usual) for most users/processes much quicker than that.
As an outsider without any knowledge of the company infra:
1) I wonder if they so many layers of CI/CD and testing, that the whole pipeline, even rollbacks probably re-run their tests which takes time. Are CI/CD runners the problem or it is something like network/DNS thing?
2) What if someone inside Cloudflare glibly yells "Why can't we just rsync our code to all data centers in less than a minute?" What would be the pushback this person would get? Why is it a stupid suggestion? Answering that would reveal some crazy layers of abstractions that get piled up in large companies.
1) I wonder if they so many layers of CI/CD and testing, that the whole pipeline, even rollbacks probably re-run their tests which takes time. Are CI/CD runners the problem or it is something like network/DNS thing?
2) What if someone inside Cloudflare glibly yells "Why can't we just rsync our code to all data centers in less than a minute?" What would be the pushback this person would get? Why is it a stupid suggestion? Answering that would reveal some crazy layers of abstractions that get piled up in large companies.
In all fairness: If nothing else, a way to quickly deploy or revert is also a way to quickly break everything.
Related: anyone know the easiest free/self hosted alternative to cloudflare tunnels?
I'm struggling to understand exactly what it does (via my employer I'm an admin of a CloudFlare account and we use some features outside DNS and proxy-CDN, but far from all of them) but... I wanna say Wireguard?
Of course, anything else is probably gonna have worse UX, because it won't have all the infra pre-built and auto-configured for you.
Also you'll need at least one publicly-accessible server to be the Internet-facing side of it, and that part would necessarily not be free, unless you can get by on some free-tier VM somewhere.
Of course, anything else is probably gonna have worse UX, because it won't have all the infra pre-built and auto-configured for you.
Also you'll need at least one publicly-accessible server to be the Internet-facing side of it, and that part would necessarily not be free, unless you can get by on some free-tier VM somewhere.
[deleted]
5% is it because you inject traces in 5% of requests?
INeedMoreRam(1)
A bit of an aside, but maybe not.
I live on the same block as the Cloudfare headquarters. They have a big common space that has open windows to the street. They have some big projectors that play what are basically Screensavers on the wall in this common space, visible from the street. For months now one of these big wall sized projections has had in large white letters "this copy of office is not legit, please contact Microsoft to purchase a legit license of windows " (I paraphrase a bit). This seems like something they should address, no?
I live on the same block as the Cloudfare headquarters. They have a big common space that has open windows to the street. They have some big projectors that play what are basically Screensavers on the wall in this common space, visible from the street. For months now one of these big wall sized projections has had in large white letters "this copy of office is not legit, please contact Microsoft to purchase a legit license of windows " (I paraphrase a bit). This seems like something they should address, no?
I strongly doubt they don't have enough office licenses. What's probably happening is no one can be bothered to fix the user hostile copy protection.
I bet cracked versions of Office don't display that message, so it's an indication that this one is actually legit :)
I bet cracked versions of Office don't display that message, so it's an indication that this one is actually legit :)
It doesn't surprise me that they are heavy on the marketing and light on the detail. Also fits with every outage they've had large enough for others to notice, the root cause is always some embarrassingly simple cock-up that only an intern would make.
I'm replying to my own comment to note that I'm getting downvoted for calling out a big company for what seems like software theft. That seems odd as I'm seemingly not the one in the wrong, I'm just sharing a simple observation.
I doubt you're getting downvoted for that. I suspect you're getting downvoted for your comment having nothing to do with the partial outage. And then going on to argue that you're doing the right thing by posting here and spending effort commenting about your comment rather than telling someone at Cloudflare that you spotted something bad.
I did tell someone, the problem persisted for months after. I told the guy that sits next to the lava lamps.
OK. You've edited your comment to say you told the night guard. Sorry that they didn't communicate this to someone. I've now spoken to the San Francisco office team despite the late hour and it'll be addressed.
You’re free to start a Tell HN or write a blog post and post the link here for people to discuss. But hijacking of unrelated threads is not welcome on HN. Check guidelines.
Thank you, I understand, I will not do so in the future.
Either of these courses of action seem like even more of a public shame, which was not my intent.
Either of these courses of action seem like even more of a public shame, which was not my intent.
It might be “software theft” but it could be a simple mistake with the license, which can happen if you reinstall or change hardware or something like that, and they need MS support to fix it.
Have you ever considered contacting them instead of public shaming?
It's not my job. They know this is an issue, it's literally written on their walls. If they wanted to address it they would have.
Edited to add - I did, the night guard didn't seem to care or even know who to bring the issue to.
Edited to add - I did, the night guard didn't seem to care or even know who to bring the issue to.
Well, I didn't know about it because I am in Lisbon and can't see that far. I've ask the office team about it.
[deleted]
> Once identified we expedited a rollback which completed in 87 minutes
Roll forwards and back should usually be done slowly and carefully.
However, during a major outage, there is a good reason to do the rollback very fast. Ie. Within 1 minute. Fast rollbacks end any outage sooner. And if your service is already down, you have nothing to lose and everything to gain.
That provides an interesting infrastructure problem. Fast rollbacks probably involve restarting a large number of servers, having a large number of cold caches, etc. You can find new bugs that only rear their heads during a fast rollback. Things like docker registries getting overloaded, kubernetes control plane being overloaded, other services becoming unhealthy due to a large influx of requests, etc.
The traditional approach is 'dont take risks, don't do a fast rollback'. But that can double or more the length of your downtime. Especially if you roll back the wrong thing first.
I'd therefore encourage preparing ahead of time for fast rollbacks. The dev/test environment should automatically do 'big bang' rollbacks every week. Production rollbacks should even be practiced from time to time, getting faster and faster each time, keeping an eye on metrics to look for instability.
Roll forwards and back should usually be done slowly and carefully.
However, during a major outage, there is a good reason to do the rollback very fast. Ie. Within 1 minute. Fast rollbacks end any outage sooner. And if your service is already down, you have nothing to lose and everything to gain.
That provides an interesting infrastructure problem. Fast rollbacks probably involve restarting a large number of servers, having a large number of cold caches, etc. You can find new bugs that only rear their heads during a fast rollback. Things like docker registries getting overloaded, kubernetes control plane being overloaded, other services becoming unhealthy due to a large influx of requests, etc.
The traditional approach is 'dont take risks, don't do a fast rollback'. But that can double or more the length of your downtime. Especially if you roll back the wrong thing first.
I'd therefore encourage preparing ahead of time for fast rollbacks. The dev/test environment should automatically do 'big bang' rollbacks every week. Production rollbacks should even be practiced from time to time, getting faster and faster each time, keeping an eye on metrics to look for instability.
> And if your service is already down, you have nothing to lose and everything to gain.
They had a 5% request failure rate at the peak of this outage. So a botched rollback could definitely do more harm than good if the impact isn’t 100% understood.
They had a 5% request failure rate at the peak of this outage. So a botched rollback could definitely do more harm than good if the impact isn’t 100% understood.
But there is a good chance that was a 100% request failure rate for a specific service. And when you have 100% of requests failing for a service, you ought to be thinking about doing a fast rollback for that service as long as you have confidence that such a fast rollback won't take other infrastructure down.
Exactly.
There have been YouTube outages that were exacerbated precisely because the SRE teams explicitly trained for the disaster scenario where everything is hard down and needs to be brought back as quickly as possible.
> The impact lasted for almost six hours in total.
And faster monitoring, they detected the issue after 4.5 hours and took one to fix
And faster monitoring, they detected the issue after 4.5 hours and took one to fix
Another chaos monkey case? Just do a fast rollback every day your D10 rolls a 1.