Piercing the Cloud Armor – The 8KB Bypass in Google Cloud Platform WAF(kloudle.com)
kloudle.com
Piercing the Cloud Armor – The 8KB Bypass in Google Cloud Platform WAF
https://kloudle.com/blog/piercing-the-cloud-armor-the-8kb-bypass-in-google-cloud-platform-waf
35 comments
It’s a belt and suspender system — make it a bit harder to pierce and attackers may go elsewhere.
And coding as well, I keep seeing APIs that I would easily exploit with Postman, because devs code as if the JavaScript requests were the only ones in the world doing the API calls, in a well behaved manner.
For my team, the WAF wasn’t about protecting us but rather reducing the amount of junk we saw coming from the public internet to our service. We’d get lots of weird payloads that would cause Nginx or Rails to throw an error even though they weren’t causing any breaches.
The WAF took away at least 99% of our random nighttime pages from the bad payloads. Best investment we made for oncall quality of life.
The WAF took away at least 99% of our random nighttime pages from the bad payloads. Best investment we made for oncall quality of life.
WAFs are security theater. False positives and false sense of security.
People say "security in depths", but WAFs add nothing that don't 100% _have_ to be mitigated down the stack - this 8K limit is just another part of the joke.
People say "security in depths", but WAFs add nothing that don't 100% _have_ to be mitigated down the stack - this 8K limit is just another part of the joke.
A WAF is to buy you time to fix your application. It isn't a bulletproof situation.
For example if you discover a vulnerability in your code that is being exploited you can push a WAF rule to block the easiest ways to trigger it. Or if you use a common library like log4j you can rely on rulesets that are updated for you. Likely before your smaller team has time to react.
In that time you should be rushing to fix your application or update dependencies and deploy the new version.
For example if you discover a vulnerability in your code that is being exploited you can push a WAF rule to block the easiest ways to trigger it. Or if you use a common library like log4j you can rely on rulesets that are updated for you. Likely before your smaller team has time to react.
In that time you should be rushing to fix your application or update dependencies and deploy the new version.
WAF’s intention is to catch common attacks. I was also skeptical about it until I saw Azure application gateway WAF’s default policies to fend off log4j attacks.
My take is now to configure WAF at a generic level with minimal impact on application functionalities. And yes, the value of WAF’s is overrated. Cause with proper development guidelines like using ORM instead of writing plain SQL queries, you can throw away half of the default policies which revolve around SQL injections.
For large organizations, having a WAF does add some value. With security it’s all about adding layers of protection wherever possible.
My take is now to configure WAF at a generic level with minimal impact on application functionalities. And yes, the value of WAF’s is overrated. Cause with proper development guidelines like using ORM instead of writing plain SQL queries, you can throw away half of the default policies which revolve around SQL injections.
For large organizations, having a WAF does add some value. With security it’s all about adding layers of protection wherever possible.
You hardly need to use ORM! Just use prepared statements and bind your parameters and no SQL injection is possible. And faster than plain SQL or using an ORM!
Every time someone posts about a WAF bypass on HN, someone like you comes along and talks about how WAFs are nothing but security theatre. Simply not true. Are they perfect? Absolutely not. I'll simply repost one of my previous comments on this topic:
You deploy a WAF as part of a defense in depth strategy, with one of the best use-cases being situations where you have legacy web systems that nobody is maintaining. Additionally, you can get TLS upscaling, easy HTTP rewrite capabilities, DDoS protection, and other granular controls with some SaaS offerings. So while it's true that a WAF won't stop a determined attacker, there are certainly benefits to operating them, particularly in large enterprise environments.Some web application firewalls have better telemetry, monitoring, notification, etc. than the services behind them, especially for unsophisticated attacks. Also, any attacker filtered is one less connection wasting resources on the hosts running your actual services, leaving more CPU and memory to handle legitimate requests on application servers.
None of them defend against everything, and there is an important nugget in your post: “defense in depth.” You should also filter requests that make it through the WAF. But, getting rid of even 90% of bogus requests before they hit your app servers seems worthwhile.
None of them defend against everything, and there is an important nugget in your post: “defense in depth.” You should also filter requests that make it through the WAF. But, getting rid of even 90% of bogus requests before they hit your app servers seems worthwhile.
Those are all nice to have, but that's called a load balancer. It should also have rate limiting and DOS protection, and I use the AWS WAF for those things, and don't really care that they're in a separate feature and that that feature is misnamed.
For every defense in depth, there's an org that's put up a WAF rule in front of an unpatched application and called it good, only yo be owned days later. False sense of security is the thing to stress here, because there is no depth.
For every defense in depth, there's an org that's put up a WAF rule in front of an unpatched application and called it good, only yo be owned days later. False sense of security is the thing to stress here, because there is no depth.
Hard disagree on this, they certainly shouldn't be your only defence but I think for the vast majority of sites they are essential.
Most site on the internet are either run by a tiny team with no dedicated security expert or no tech team at all (someone asked a friend/freelancer to put up a Wordpress site). They aren’t applying security patches and subscribing to the developer mailing lists. A WAF service (like CloudFlare) ensures that older, unpatched, infrastructure is protected. Yes, they should be patching their deployment, but try asking a small business to pay for their web guy to spend a few fours a month ensuring everything is secure. Not going to happen.
From my perspective as someone with a good understanding of web security. I run a WAF service on my sites so that if there is a zero day on part of my stack We are protected immediately before we have time to patch the component, which I will do. That is invaluable to me and my business.
Finally developers do make mistakes and accidentally introduce vulnerabilities. A WAF helps to ensure against that, although you should still be auditing your code.
Most site on the internet are either run by a tiny team with no dedicated security expert or no tech team at all (someone asked a friend/freelancer to put up a Wordpress site). They aren’t applying security patches and subscribing to the developer mailing lists. A WAF service (like CloudFlare) ensures that older, unpatched, infrastructure is protected. Yes, they should be patching their deployment, but try asking a small business to pay for their web guy to spend a few fours a month ensuring everything is secure. Not going to happen.
From my perspective as someone with a good understanding of web security. I run a WAF service on my sites so that if there is a zero day on part of my stack We are protected immediately before we have time to patch the component, which I will do. That is invaluable to me and my business.
Finally developers do make mistakes and accidentally introduce vulnerabilities. A WAF helps to ensure against that, although you should still be auditing your code.
> A WAF service (like CloudFlare) ensures that older, unpatched, infrastructure is protected
It ensures that they're protected against a trivial and fairly outdated set of threats. You might say that's better than nothing, but questionable protection leading to a sense of security is more dangerous than having nothing and knowing it (also, false positives).
> We are protected immediately
Look at the recent log4j vulnerabilities. Hosted WAFs didn't "immediately" address the issue (though they were fairly quick). However, to the best of my knowledge they were unable to fully mitigate the threat. For example, if you look at CloudFlare's wording on rule 2c5413e155db4365befe0df160ba67d7:
> In addition to the above rules we have also released a fourth rule that will protect against a much wider range of attacks at the cost of a higher false positive rate. For that reason we have made it available but not set it to BLOCK by default
It's clear that, by default, their updated rules didn't fully mitigate the issue. Also, it IS NOT clear, whether the extra rule did fully mitigate the issue.
Fundamentally, having a WAF in the face of log4j did nothing for you. You still have to patch everything, you still had to try and figure out if you'd been compromised, and you still had to update/alert your clients/customers.
It ensures that they're protected against a trivial and fairly outdated set of threats. You might say that's better than nothing, but questionable protection leading to a sense of security is more dangerous than having nothing and knowing it (also, false positives).
> We are protected immediately
Look at the recent log4j vulnerabilities. Hosted WAFs didn't "immediately" address the issue (though they were fairly quick). However, to the best of my knowledge they were unable to fully mitigate the threat. For example, if you look at CloudFlare's wording on rule 2c5413e155db4365befe0df160ba67d7:
> In addition to the above rules we have also released a fourth rule that will protect against a much wider range of attacks at the cost of a higher false positive rate. For that reason we have made it available but not set it to BLOCK by default
It's clear that, by default, their updated rules didn't fully mitigate the issue. Also, it IS NOT clear, whether the extra rule did fully mitigate the issue.
Fundamentally, having a WAF in the face of log4j did nothing for you. You still have to patch everything, you still had to try and figure out if you'd been compromised, and you still had to update/alert your clients/customers.
> It ensures that they're protected against a trivial and fairly outdated set of threats
Isn’t a WAF getting access to pre-public intelligence and updating their rules accordingly? I was under the impression that major vendors like Cloudflare and Google were able to block zero-days before the public announcements.
Isn’t a WAF getting access to pre-public intelligence and updating their rules accordingly? I was under the impression that major vendors like Cloudflare and Google were able to block zero-days before the public announcements.
> It ensures that they're protected against a trivial and fairly outdated set of threats. You might say that's better than nothing, but questionable protection leading to a sense of security is more dangerous than having nothing and knowing it (also, false positives).
Rules for remotely exploitable vulnerabilities like log4j are typically crafted within hours of a CVE being released, and deployed as quickly as possible to minimize false positives. I'm not sure I'd consider such threats "outdated".
> However, to the best of my knowledge they were unable to fully mitigate the threat ... Fundamentally, having a WAF in the face of log4j did nothing for you.
If you're only able to mitigate 9X% of a rapidly evolving zero-day and not 100%, should you not use a WAF at all? Security is about reducing risk, and raising the cost for attackers.
Cloudflare tracked and responded to active exploitation attempts[1] in the wild with updated rules, while imploring customers to patch, e.g., from our second CVE (CVE-2021-45046) post[2] discussing additional rules we deployed:
"This vulnerability is actively being exploited and anyone using Log4J should update to version 2.16.0 as soon as possible, even if you have previously updated to 2.15.0. The latest version can be found on the Log4J download page."
> You still have to patch everything, you still had to try and figure out if you'd been compromised, and you still had to update/alert your clients/customers.
Yes, of course. Nobody is arguing otherwise. Such is the nature of these massive vulnerabilities that pop up every few years.
[1] - https://blog.cloudflare.com/exploitation-of-cve-2021-44228-b...
[2] - https://blog.cloudflare.com/protection-against-cve-2021-4504...
Rules for remotely exploitable vulnerabilities like log4j are typically crafted within hours of a CVE being released, and deployed as quickly as possible to minimize false positives. I'm not sure I'd consider such threats "outdated".
> However, to the best of my knowledge they were unable to fully mitigate the threat ... Fundamentally, having a WAF in the face of log4j did nothing for you.
If you're only able to mitigate 9X% of a rapidly evolving zero-day and not 100%, should you not use a WAF at all? Security is about reducing risk, and raising the cost for attackers.
Cloudflare tracked and responded to active exploitation attempts[1] in the wild with updated rules, while imploring customers to patch, e.g., from our second CVE (CVE-2021-45046) post[2] discussing additional rules we deployed:
"This vulnerability is actively being exploited and anyone using Log4J should update to version 2.16.0 as soon as possible, even if you have previously updated to 2.15.0. The latest version can be found on the Log4J download page."
> You still have to patch everything, you still had to try and figure out if you'd been compromised, and you still had to update/alert your clients/customers.
Yes, of course. Nobody is arguing otherwise. Such is the nature of these massive vulnerabilities that pop up every few years.
[1] - https://blog.cloudflare.com/exploitation-of-cve-2021-44228-b...
[2] - https://blog.cloudflare.com/protection-against-cve-2021-4504...
99% of malicious traffic, in my experience, comes from random untargeted vulnerability scans.
WAFs absolutely do help with these, if only by lowering load on actual servers.
Think of a WAF like the thinnest possible veil over your code. You wouldn't go out into public wearing the thinnest possible veil as your only piece of clothing, would you?
So make sure you bake in security through your code, and then when you are done, you can afford to make a decision about whether you need to put the thinnest possible veil over that.
So make sure you bake in security through your code, and then when you are done, you can afford to make a decision about whether you need to put the thinnest possible veil over that.
I mean, most of the cloud is theater that your managers want to see. Setting up queues, distributed storage (s3 compatible), etc is incredibly easy these days. There’s very little reason for using the cloud, unless you really need the elasticity and I can only imagine a very few verticals that actually need it. Everyone else just “thinks” they need it, the reality is there elasticity requirements are on the order of 5%-10% of a baseline, which they’d save far more with bare metal all the way.
Don’t get me wrong, serverless is pretty amazing until you get to a certain scale. The cloud really has that going for it.
Don’t get me wrong, serverless is pretty amazing until you get to a certain scale. The cloud really has that going for it.
Solution seems funky. Attackers could just omit the Content-Length header
If you omit the Content-Length header from an HTTP request, the server will not process the request body.
See: https://reqbin.com/Article/ContentLength
"If the value of the Content-Length header is zero, or if neither the Transfer-Encoding header nor the Content-Length header is specified, then the message has no body. "
See: https://reqbin.com/Article/ContentLength
"If the value of the Content-Length header is zero, or if neither the Transfer-Encoding header nor the Content-Length header is specified, then the message has no body. "
Maybe Cloud Armor calculates en sets it?
AWS WAF has the same limitation. It’s a trade-off between being able to be useful enough in a pinch while keeping the added latency low enough to not have an impact.
Whether that’s a good trade-off I’m not sure. Folks know this is a thing so if someone really wants to get past it they will. But it’s proven effective enough at neutralising a lot of bots and other annoying traffic whenever I’ve resorted to using it.
Whether that’s a good trade-off I’m not sure. Folks know this is a thing so if someone really wants to get past it they will. But it’s proven effective enough at neutralising a lot of bots and other annoying traffic whenever I’ve resorted to using it.
Agreed. It's not just AWS, however. Azure, Cloudflare, and other WAF providers have a similar trade-off.
It's fair to say that they are waf for the mass! Basic modsec provides much better security...
AWS has the same bypass (documented). The WAF provided by cloud providers is more to protect their infrastructure from noise than clients. Unfortunately this type of restrictions are never evaluated. You can really tell if a WAF provider is serious when reviewing the whitelisting. (i.e. not just a on/off).
AWS has the same bypass (documented). The WAF provided by cloud providers is more to protect their infrastructure from noise than clients. Unfortunately this type of restrictions are never evaluated. You can really tell if a WAF provider is serious when reviewing the whitelisting. (i.e. not just a on/off).
Why only POST? Query params can be "hidden" in a similar way too, is there a reason GET requests are not vulnerable? Url length limit maybe?
Hi, article author here. Cloud Armor will drop requests without regard to the size of the request body for HTTP PUT or PATCH request (i.e., the payload won't have to padded with 8192 bytes, like in the case of POST requests.) Of course, for an attacker to successfully exploit this, the underlying application would have to be configured to accept and process PUT/PATCH requests.
As for query parameters in GET requests, I'm not entirely sure about Cloud Armor's limits there. I'll check and get back to you.
As for query parameters in GET requests, I'm not entirely sure about Cloud Armor's limits there. I'll check and get back to you.
https://docs.aws.amazon.com/waf/latest/developerguide/limits...
Pad your POST query by 8k and you are through!
Pad your POST query by 8k and you are through!
Thank you for the clarification!
GET doesn’t typically have a body, and many services ignore it even if you include one. An API I once worked on simply truncated the body of all GET requests at the front door, because we knew we’d never use it. Less to worry about.
Sorry if I wasn't clear enough, I was talking about query params in the URL, not request body.
I guess this limitation is on the size of the request body and including that doesn’t make sense for GET requests?
Are the header rules discarded as well when the post payload is>8kb?
Don’t know about google but in AWS the equivalent rule only looks at the first 8k of the body. There’s a separate rule for the header and another rule to block requests with header size exceeding n KB. If you allowed large headers then you could prevent the header rules too, but IIRC by default it doesn’t. Besides most web infrastructure would barf at huge headers…
Your application has to be made secure, the WAF cannot do it for you.
At best a third party managed WAF provides speed for sticking plaster to be put over a newly discovered vulnerability, but as all WAFs can essentially be thought of as advanced string parsing without the context of what your app thinks of as good or bad, then there will always be some false negatives that affect your app.
Security engineering cannot be purchased as an afterthought, it needs to be something your engineers practise as commonly as observability and testing.