Kubernetes ingress-nginx controller vulnerability(armosec.io)
armosec.io
Kubernetes ingress-nginx controller vulnerability
https://www.armosec.io/blog/new-kubernetes-high-severity-vulnerability-alert-cve-2021-25742
51 comments
For whatever it’s worth my startup began life as a multi tenant kubernetes provider. We have a ton of custom controls - and for example do not allow this “custom snippet” directive at all. We spent a very long time locking down as much of the api surface as possible.
I have war stories up the wazoo - at one point we had 16k random internet users on a single cluster!
I have war stories up the wazoo - at one point we had 16k random internet users on a single cluster!
What's the actual vulnerability?
The issue links to itself for "more details"…
(Also, yeah, it'd be nice if the URL of the OP had just been the Github issue. It is just as informative (or uninformative), and isn't an advertisement…)
The issue links to itself for "more details"…
(Also, yeah, it'd be nice if the URL of the OP had just been the Github issue. It is just as informative (or uninformative), and isn't an advertisement…)
Although this particular vulnerability is all about ingress-nginx and has nothing to do with Kubernetes itself, the current design of the ingress API sort of encouraged it.
The reason that an ingress controller need permission to read all secrets across all namespaces in the first place, is because ingress objects can specify their own certs (stored in namespaced secret objects) within their namespaces, hence the ingress controller needs permission to read those secrets and load the certs into the reverse proxy (in case of ingress-nginx, it is nginx). This is such a scary permission to grant that cluster admins have to hand over all the secrets in the cluster to the ingress controller and hope it does the right thing.
Now, ingress-nginx actually allows you to specify a default secret to be used by all ingresses it manages, and that would be the only secret it needs access to. I am not sure how many users out there are willing to dig into thousands lines of its manifest yaml files to tweak its rbac settings.
The reason that an ingress controller need permission to read all secrets across all namespaces in the first place, is because ingress objects can specify their own certs (stored in namespaced secret objects) within their namespaces, hence the ingress controller needs permission to read those secrets and load the certs into the reverse proxy (in case of ingress-nginx, it is nginx). This is such a scary permission to grant that cluster admins have to hand over all the secrets in the cluster to the ingress controller and hope it does the right thing.
Now, ingress-nginx actually allows you to specify a default secret to be used by all ingresses it manages, and that would be the only secret it needs access to. I am not sure how many users out there are willing to dig into thousands lines of its manifest yaml files to tweak its rbac settings.
Can we fix the title to include ingress-nginx instead?
The upstream URL seems more clear as well: https://github.com/kubernetes/ingress-nginx/issues/7837
The upstream URL seems more clear as well: https://github.com/kubernetes/ingress-nginx/issues/7837
Agreed, I'd prefer a GitHub issue over an advertisement. The HN user who posted this works for the company that wrote this post, and that person's title is "VP Marketing & Business Development".
The issue contains a comment where they're also advertising their service, so I think that using the GitHub issue would be better, and would be a good compromise for them since they'll still have their ad.
The issue contains a comment where they're also advertising their service, so I think that using the GitHub issue would be better, and would be a good compromise for them since they'll still have their ad.
The problem is in intersection of nginx-ingress and kubernetes. Since ingress controller has access to secrets from all the namespaces (which is kubernetes side of the story) the nginx implementation with snippets added by users (the nginx contribution) may expose these secrets.
The post also points to an open source tool that helps people to check if they are vulnerable, whether they want to get to the bottom of the issue or not.
I know nearly nothing about ARMO, and I have nothing against you, the company you founded and are CTO of (ARMO), or ARMO providing an open source tool. I appreciate that you have made this work open source.
Still, some of us on hackernews dislike advertising. I 100% agree with everything you've written in this comment, but as I mentioned in the comment you're replying to, the GitHub issue also contains an ad for the same tool - it just doesn't make it front and center, allows for any competitors to make their own comments on an open platform, and it feels less biased. Less advertisey.
Instead of "install our product", it could have been something like "run this thing specific to this exact vulnerability that does the absolute bare minimum. Oh, btw, if you liked that, you should check out our product that does much more, kubescape"
But as it stands now, it just feels different
Still, some of us on hackernews dislike advertising. I 100% agree with everything you've written in this comment, but as I mentioned in the comment you're replying to, the GitHub issue also contains an ad for the same tool - it just doesn't make it front and center, allows for any competitors to make their own comments on an open platform, and it feels less biased. Less advertisey.
Instead of "install our product", it could have been something like "run this thing specific to this exact vulnerability that does the absolute bare minimum. Oh, btw, if you liked that, you should check out our product that does much more, kubescape"
But as it stands now, it just feels different
why? does ingress-nginx run outside of k8s?
You can run a kubernetes cluster without the nginx-ingress.
calling it "Kubernetes ingress-nginx controller" seems reasonable
Ok, we've put that in the title above. Thanks!
"... a user who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster." I'm not a Kubernetes guy, so I'm curious, how often is there a user with only those permissions? Is it common to have a user that can create/update and then doesn't already have some kind of other access to everything else? I don't know much about Kubernetes permissions.
(I also just learned that the word Kubernetes is in the default Chrome spell checker thing, which was sure helpful in writing this comment!)
(I also just learned that the word Kubernetes is in the default Chrome spell checker thing, which was sure helpful in writing this comment!)
A user can have many permissions in his/her own namespace without compromising the other tenants. This sounds like it would allow any user with Ingress create/update access in any namespace to compromise secrets across all namespaces.
It is unclear whether this is a problem if you're running namespaced ingress-controllers though. This comment [0] in the bug report says:
[0] https://github.com/kubernetes/ingress-nginx/issues/7837#issu...
there's definitely an attack path that gets the ingress-nginx service account token, which has list rights on secrets at a cluster level (so allowing for all secret values to be retrieved).
[0] https://github.com/kubernetes/ingress-nginx/issues/7837#issu...
> You'd need get permissions for that.
I'm afraid not. HTTP GET on a collection endpoint (which is the operation represented by the list verb) returns the full object content.
https://kubernetes.io/docs/reference/access-authn-authz/auth...
I'm afraid not. HTTP GET on a collection endpoint (which is the operation represented by the list verb) returns the full object content.
https://kubernetes.io/docs/reference/access-authn-authz/auth...
Interesting, thanks for the reference. This is at best surprising, at worst sloppy security design IMO.
> "... a user who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster."
To this point, handling of secrets in the Kubernetes ecosystem always seemed like a mess to me. Both `vault` and AWS SecretsManager/KMS in combination with IAM allow me to finely partition access to secrets by instance or developer identity whereas in the Kubernetes world one expects "secrets" to be handled by a small core group and machines seem to have mostly unfettered access.
Specifically[1],
>> Caution:
>> Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.
>> In order to safely use Secrets, take at least the following steps:
>> * Enable Encryption at Rest for Secrets[2].
>> * Enable or configure RBAC rules that restrict reading data in Secrets (including via indirect means).[3]
>> * Where appropriate, also use mechanisms such as RBAC to limit which principals are allowed to create new Secrets or replace existing ones.
When you dig deeper, both [2] and [3] look like they are someone's job security plan. No wonder it was developed at a place which employs people just to keep them off the market.
Especially since you really don't gain much unless you also have a KMS Provider[4].
No wonder most K8 setups I've seen still have BASE64 encoded secrets in some YAML file.
There is also no shortage of tutorial-spam -- so unless you happen to be lucky enough to apprentice with people who know what they are doing, it is hard to gain a solid understanding and move confidently.
[1]: https://kubernetes.io/docs/concepts/configuration/secret/
[2]: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...
[3]: https://kubernetes.io/docs/reference/access-authn-authz/auth...
[4]: https://kubernetes.io/docs/tasks/administer-cluster/kms-prov...
To this point, handling of secrets in the Kubernetes ecosystem always seemed like a mess to me. Both `vault` and AWS SecretsManager/KMS in combination with IAM allow me to finely partition access to secrets by instance or developer identity whereas in the Kubernetes world one expects "secrets" to be handled by a small core group and machines seem to have mostly unfettered access.
Specifically[1],
>> Caution:
>> Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.
>> In order to safely use Secrets, take at least the following steps:
>> * Enable Encryption at Rest for Secrets[2].
>> * Enable or configure RBAC rules that restrict reading data in Secrets (including via indirect means).[3]
>> * Where appropriate, also use mechanisms such as RBAC to limit which principals are allowed to create new Secrets or replace existing ones.
When you dig deeper, both [2] and [3] look like they are someone's job security plan. No wonder it was developed at a place which employs people just to keep them off the market.
Especially since you really don't gain much unless you also have a KMS Provider[4].
No wonder most K8 setups I've seen still have BASE64 encoded secrets in some YAML file.
There is also no shortage of tutorial-spam -- so unless you happen to be lucky enough to apprentice with people who know what they are doing, it is hard to gain a solid understanding and move confidently.
[1]: https://kubernetes.io/docs/concepts/configuration/secret/
[2]: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...
[3]: https://kubernetes.io/docs/reference/access-authn-authz/auth...
[4]: https://kubernetes.io/docs/tasks/administer-cluster/kms-prov...
You can use Kubescape (open source) to check if you are vulnerable. https://github.com/armosec/kubescape
Are they for real with that curl? Normalize better security practices!
I think it's contrary to currently-recommended ops practice (clearly imperative rather than nominally declarative), but how is it bad security practice? If you don't trust the origin, it's not like saving to a temporary file first is going to help you. Unless you're suggesting that everyone should always do their own code review and then compile from source...
> but how is it bad security practice?
Companies / organizations / team members go rogue sometimes. We've seen this even recently with e.g. kicad, freenode, the timezone database, etc.
Just because you trust the source doesn't mean you should trust all of the scripts they tell you to run. Even if it's a good-faith script, you have no idea if it's making assumptions about your system that are not true and opening you up to sidechannel attacks and the like.
Curl to a file first, inspect the script, consider it within the context of your own system, then run it if you deem it's safe.
Companies / organizations / team members go rogue sometimes. We've seen this even recently with e.g. kicad, freenode, the timezone database, etc.
Just because you trust the source doesn't mean you should trust all of the scripts they tell you to run. Even if it's a good-faith script, you have no idea if it's making assumptions about your system that are not true and opening you up to sidechannel attacks and the like.
Curl to a file first, inspect the script, consider it within the context of your own system, then run it if you deem it's safe.
> Curl to a file first, inspect the script, consider it within the context of your own system, then run it if you deem it's safe.
But you can still do that if you want? It is not like this is a hidden executable or something.
But you can still do that if you want? It is not like this is a hidden executable or something.
Sure, but as OP put it, "Normalize better security practices!".
Here's some discussion about that. But to put it simply, security is done in layers. Checking something simple like a script before running it is relatively easy and can catch low-effort malicious attempts. Sure, it won't protect against an advanced persistent threat doing a supply chain attack, but you're probably much more likely to be hit with low effort attacks that could be avoided by just not being careless.
https://news.ycombinator.com/item?id=17636032
https://news.ycombinator.com/item?id=17636032
> If you don't trust the origin, it's not like saving to a temporary file first is going to help you.
Huh? The whole point is to review the the code before you run it.
If it seems like many people _don't_, well frankly, that's their problem.
Encouraging bad practise is not the right direction.
Huh? The whole point is to review the the code before you run it.
If it seems like many people _don't_, well frankly, that's their problem.
Encouraging bad practise is not the right direction.
At minimum you could download software that is at least signed by someone you trust. Rather than streaming arbitrary commands from a source that the extent of the authentication is that their CDN's TLS certs matches their domain.
Saving to a file first means you can at least run it in a test environment first, then be confident you’re running the same thing when moving into production.
Yeah I think everyone feels icky about 'curl https://.. | sh' but to what degree is doing so any more insecure than other "normalized" software distribution practices?
See also:
https://twitter.com/moyix/status/1451318133021675520
You can check for this vulnerability using the existing tooling any k8s admin already has on their systems by necessity. It's always foolish to install unknown software and security professionals should never advise that.
> unknown software
Looking at the blog URL and header bar, and the script URL, this is pretty clearly a company blog recommending to use the company's own product. I hardly think that context counts as "unknown".
Looking at the blog URL and header bar, and the script URL, this is pretty clearly a company blog recommending to use the company's own product. I hardly think that context counts as "unknown".
Lets say (for example) that was published on a wordpress site, and the admins for whatever reason didn't secure it properly.
The article in question which _today_ looks all legit and points to a nice working script, might tomorrow be pointing to someone else's script that's a lot less legit.
And yes, in this imaginary scenario that wordpress install leads to a host of other problems for the company.
No need to make them your problems too though. ;)
The article in question which _today_ looks all legit and points to a nice working script, might tomorrow be pointing to someone else's script that's a lot less legit.
And yes, in this imaginary scenario that wordpress install leads to a host of other problems for the company.
No need to make them your problems too though. ;)
I will argue until the day I die: curl | bash is actually more secure than most traditional software delivery mechanisms. Slightly, but meaningfully.
With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run. If its on Github or a similar site, I can see in plain language the exact organization who published it (I wouldn't implicitly trust whoever "armosec" is, but I would trust, for example, "kubernetes", "microsoft", or "google" e.g.).
Alternatives?
- For something like this, to generate a report on whether you're affected? Maybe just a copy-paste bit of code? Isn't that the same thing, but less convenient?
- apt-get install something. Who published it? Does apt have security scanning (no lol)? Where's the source code for that package (scattered to the wind no doubt)? Does it have reproducible builds so I can be certain the code I eventually find is what I install (lol no. but bash scripts do!)
- Let's talk about `snap` for a second; Did you know that there's a `snap install aws-cli` package available [1], published by "Amazon Web Services", with a verified checkmark. Sounds great. Except, for years, no one on the actual aws-cli team had any idea where it came from [2]. They didn't authorize it, and they don't maintain it. It turns out, it was probably the result of a hackathon project by one AWS engineer. "Better security practices" for sure!
The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content. Legit risky, good to educate about; don't curl | bash from sites you don't recognize. But this is a GitHub link!
I would agree that package managers are superior from a security angle, if most package managers had strong security review processes, on both the package content and the publisher. The problem is: they don't. They're all lacking in this regard, from apt to flatpak to the Apple App Store. The most sophisticated is probably the Play Store, but binary command line distribution channels are nowhere close to having reproducible builds, verified Real ID developers, static analysis on distributed binaries, etc. Package managers are literally just "curl | bash" with some steps added. By and large, anyone can publish anything under any name; educating people that they're "safe" is actually a massive disservice.
[1] https://snapcraft.io/aws-cli
[2] https://github.com/aws/aws-cli/issues/5142
With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run. If its on Github or a similar site, I can see in plain language the exact organization who published it (I wouldn't implicitly trust whoever "armosec" is, but I would trust, for example, "kubernetes", "microsoft", or "google" e.g.).
Alternatives?
- For something like this, to generate a report on whether you're affected? Maybe just a copy-paste bit of code? Isn't that the same thing, but less convenient?
- apt-get install something. Who published it? Does apt have security scanning (no lol)? Where's the source code for that package (scattered to the wind no doubt)? Does it have reproducible builds so I can be certain the code I eventually find is what I install (lol no. but bash scripts do!)
- Let's talk about `snap` for a second; Did you know that there's a `snap install aws-cli` package available [1], published by "Amazon Web Services", with a verified checkmark. Sounds great. Except, for years, no one on the actual aws-cli team had any idea where it came from [2]. They didn't authorize it, and they don't maintain it. It turns out, it was probably the result of a hackathon project by one AWS engineer. "Better security practices" for sure!
The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content. Legit risky, good to educate about; don't curl | bash from sites you don't recognize. But this is a GitHub link!
I would agree that package managers are superior from a security angle, if most package managers had strong security review processes, on both the package content and the publisher. The problem is: they don't. They're all lacking in this regard, from apt to flatpak to the Apple App Store. The most sophisticated is probably the Play Store, but binary command line distribution channels are nowhere close to having reproducible builds, verified Real ID developers, static analysis on distributed binaries, etc. Package managers are literally just "curl | bash" with some steps added. By and large, anyone can publish anything under any name; educating people that they're "safe" is actually a massive disservice.
[1] https://snapcraft.io/aws-cli
[2] https://github.com/aws/aws-cli/issues/5142
> The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content
This is the reason why you shouldn't do curl ... | bash, you should do bash -c "$(curl ...)"
The former is truly vulnerable, for the exact reason you mention, while the latter isn't.
> With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run
Not exactly, see http://thejh.net/misc/website-terminal-copy-paste
More savvy users might have set their shell up in a way that mitigates this risk, and some users will, as you mentioned, copy and paste URL, but go ahead and try that with the command in the link I provided - note that if you copy just the git URL, it only copies that bit, and then you can paste it into your browser, and not realize that there's a malicious bit before it
Of course, copying and pasting a command involving apt/dnf/etc is also vulnerable to that same attack :)
This is the reason why you shouldn't do curl ... | bash, you should do bash -c "$(curl ...)"
The former is truly vulnerable, for the exact reason you mention, while the latter isn't.
> With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run
Not exactly, see http://thejh.net/misc/website-terminal-copy-paste
More savvy users might have set their shell up in a way that mitigates this risk, and some users will, as you mentioned, copy and paste URL, but go ahead and try that with the command in the link I provided - note that if you copy just the git URL, it only copies that bit, and then you can paste it into your browser, and not realize that there's a malicious bit before it
Of course, copying and pasting a command involving apt/dnf/etc is also vulnerable to that same attack :)
'bash -c $(curl )' doesn't mitigate a website's ability to detect whether you're downloading a file via curl versus accessing it in a browser (e.g. if (headers.USER_AGENT.includes("curl")) {}).
It does mitigate a different issue whereby the website can detect if you're piping the output of 'curl' directly to bash.
The safest option is definitely --> curl to file, audit what you downloaded locally, not in a browser, then bash -c that downloaded file.
But, again, this only really mitigates attacks from sketchy websites. If the website is trustworthy; both the website you're getting the curl | bash download string from, and the website the script is hosted at, such as Github, where 90% of these scripts are hosted; its not really mitigating a legitimate threat.
The copy-paste thing is interesting, but frankly: That's an argument for why the web is insecure, and that insecurity impacts every single method of software distribution which touches it. Copy-pasting a 'sudo apt-get install' line is the same class of risk here as copy-pasting a 'curl | bash' or 'bash -c $(curl)' line; you could swap the script to a new URL, or you could have them install a malicious package from apt, or even just nuke the apt-get install part and just do a curl | bash anyway. Definitely something that's great to know and educate about, but its not the strongest argument against curl | bash.
It does mitigate a different issue whereby the website can detect if you're piping the output of 'curl' directly to bash.
The safest option is definitely --> curl to file, audit what you downloaded locally, not in a browser, then bash -c that downloaded file.
But, again, this only really mitigates attacks from sketchy websites. If the website is trustworthy; both the website you're getting the curl | bash download string from, and the website the script is hosted at, such as Github, where 90% of these scripts are hosted; its not really mitigating a legitimate threat.
The copy-paste thing is interesting, but frankly: That's an argument for why the web is insecure, and that insecurity impacts every single method of software distribution which touches it. Copy-pasting a 'sudo apt-get install' line is the same class of risk here as copy-pasting a 'curl | bash' or 'bash -c $(curl)' line; you could swap the script to a new URL, or you could have them install a malicious package from apt, or even just nuke the apt-get install part and just do a curl | bash anyway. Definitely something that's great to know and educate about, but its not the strongest argument against curl | bash.
I agree 100% with that comment :) Including the first part:
> 'bash -c $(curl )' doesn't mitigate a website's ability to detect whether you're downloading a file via curl versus accessing it in a browser
You're totally right, I forgot about this one! It's wild how many tricks there are in the security world
> 'bash -c $(curl )' doesn't mitigate a website's ability to detect whether you're downloading a file via curl versus accessing it in a browser
You're totally right, I forgot about this one! It's wild how many tricks there are in the security world
The people downvoting you did so while waiting on npm install to run.
>I can copy it, plug it into my browser, and inspect with ease the shell script that will run
If the source is some shady site, it's possible to serve two different versions: one for curl/wget users and one for browsers.
If the source is some shady site, it's possible to serve two different versions: one for curl/wget users and one for browsers.
[deleted]
I see you can also download it from Git and try it out, probably just tried to make it easier for people.
> Hey Everyone, here's a high severity vulnerability report, to check for it, run the folliowing: curl -shttps://raw.githubusercontent.com/noideawhatyourerunning/ins... |/bin/bash
.... oh man
.... oh man
Honestly, this kind of made my blood boil, and just points out how difficult security can be.
Here is presumably a security company, in their own blog post outlining a high-severity vulnerability alert, advocating that you run untrusted code from some rando GitHub repo. It also outlines my fear about dealing with some "security consultants", because on the one hand they outline all of the things you need to "box check" for some security audit, but at the same time leave you less secure because they've just opened another giant gaping attack surface for your company - this is exactly what happened with the SolarWinds attack.
Here is presumably a security company, in their own blog post outlining a high-severity vulnerability alert, advocating that you run untrusted code from some rando GitHub repo. It also outlines my fear about dealing with some "security consultants", because on the one hand they outline all of the things you need to "box check" for some security audit, but at the same time leave you less secure because they've just opened another giant gaping attack surface for your company - this is exactly what happened with the SolarWinds attack.
[deleted]
Don't forget to add a "sudo" in there for extra goodness. :D
It does that in the script for you :)
Not only that, they don't do any checksum checks either, and seem to not even publish checksums.
While I understand the concern with curl | bash, this method is used in many different open source product installations. The sh file is coming from github - pretty trustful source. You can always watch it in the browser (github will not trick you into different version) but you can also curl it or just download the source from the repository (just half a page above).
You can also always review and rebuild the entire tool - another beauty of the open source.
But most people just want to use the tool as fast and as simple as it can be. I guess there are options for any possible taste.
[deleted]
[deleted]
The only fix so far is removing the feature.
>According to the publication, multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.
Ouch! I guess this means lots of secret/password rotations for k8s admins coming soon. I've always wondered if there are public Kubernetes multi-tenant setups in the wild rather than just ones with admins and developer roles/namespaces.