Twelve-factor app anno 2022(xenitab.github.io)
xenitab.github.io
Twelve-factor app anno 2022
https://xenitab.github.io/blog/2022/02/23/12factor/
35 comments
While theoretically valid, in practice, nothing beats env vars and I have yet to see problems. If it leaks, you can change it anyway. If somebody is on the server, env vars are the least of your problems (also they are session wise and not saved locally).
You should always use both - keep passwords in secret manager and during build communicate them to relevant parts of app via env vars. That way, you can easily reproduce it locally without secret manager available or slowing you down (maybe you need VPN to access it?, GitLab access to secrets etc.). Also, you can't easily change them for testing purposes or store variations of them - perhaps you mix several production with several testing environments. Contrary to that, you can keep dozen of them and easily change them in a single file while developing.
Thee are bunch of tools to hide leaking environment vars too by masking them.
So, no, this is not obsolete. Maybe it doesn't make sense in some scenarios but such cases shouldn't be the norm.
You should always use both - keep passwords in secret manager and during build communicate them to relevant parts of app via env vars. That way, you can easily reproduce it locally without secret manager available or slowing you down (maybe you need VPN to access it?, GitLab access to secrets etc.). Also, you can't easily change them for testing purposes or store variations of them - perhaps you mix several production with several testing environments. Contrary to that, you can keep dozen of them and easily change them in a single file while developing.
Thee are bunch of tools to hide leaking environment vars too by masking them.
So, no, this is not obsolete. Maybe it doesn't make sense in some scenarios but such cases shouldn't be the norm.
This is straight-up bad advice.
Shuttling secrets via env vars expose you to strictly more risk vectors than through properly permissioned files on disk. Point blank, no argument. It's not acceptable to use env vars for secrets.
Shuttling secrets via env vars expose you to strictly more risk vectors than through properly permissioned files on disk. Point blank, no argument. It's not acceptable to use env vars for secrets.
I wouldn't want to change the usage of env vars, they're ergonomic and only the current user or root can see the env vars of a running process which makes it easy for us to architect secure usage patterns.
But... it'd be cool if we had better support for avoiding accidental leaks. E.g. a way to identify another process as my user that ends up reading /proc/<pid>/environ unexpectedly. You could model that with SELinux but it could be a bear to own and not worth it for outside of large/important/high-value target companies. I suppose that's an apt description of SELinux generally...
Maybe we could have a "light" version at the code level, maybe use the type system to throw an exception if you ever try to .toString() a secret. I'm trying to think how i would do that in Java without binding someone to an awkward type hierarchy (yuk!). This is not (currently) valid Java:
I think the way to do this in Java today would need to be via a dedicated class, i don't think i could use a composable interface. I mean that's fine-ish but i feel allergic to constraining other developers to my
But... it'd be cool if we had better support for avoiding accidental leaks. E.g. a way to identify another process as my user that ends up reading /proc/<pid>/environ unexpectedly. You could model that with SELinux but it could be a bear to own and not worth it for outside of large/important/high-value target companies. I suppose that's an apt description of SELinux generally...
Maybe we could have a "light" version at the code level, maybe use the type system to throw an exception if you ever try to .toString() a secret. I'm trying to think how i would do that in Java without binding someone to an awkward type hierarchy (yuk!). This is not (currently) valid Java:
interface Secret {
default sealed String toString() {
throw new IllegalAccessException()
}
}
I think the way to do this in Java today would need to be via a dedicated class, i don't think i could use a composable interface. I mean that's fine-ish but i feel allergic to constraining other developers to my
abstract class Secret { ... }
Its way more probable that it will leak some other way, like in logs IMO. Who knows what can other process running on the same machine do - it can change stuff in memory of another process so preventing env vars leakage sounds like non issue compared to that. Sure, there are more easily available tools OTB on most systems to do that but somebody needs to run them anyway...
If your machine is compromised, it’s only a matter of time until they get the credentials anyway. Whether you have more time than them is ultimately the question.
I don’t think the vulnerability you mention here is “worth working around” for most apps, where you can trust the hardware and OS you’re running on. For those that can’t trust the OS and hardware they are running on, this may be legitimate. Even then, if you can’t trust that, what are you doing running software on it?
I don’t think the vulnerability you mention here is “worth working around” for most apps, where you can trust the hardware and OS you’re running on. For those that can’t trust the OS and hardware they are running on, this may be legitimate. Even then, if you can’t trust that, what are you doing running software on it?
My bigger concern with credentials in environment variables is that environment variables are broadly accessed through may different frameworks, libraries, etc. If you can convince a library to give you an arbitrary (or all) environment variables, your credentials are leaked. The developer of that particular library may have never even considered that environment variables might contain sensitive data.
PHP's phpinfo() function comes to mind immediately: https://www.php.net/manual/en/function.phpinfo.php
PHP's phpinfo() function comes to mind immediately: https://www.php.net/manual/en/function.phpinfo.php
If your threat model is executing untrusted 3rd party code, then you should be vetting them to ensure they're not reading your environment data or anything else nefarious. Otherwise they've already accomplished remote code execution. Why would they need to read your env when they can open a backdoor?
> Why would they need to read your env when they can open a backdoor?
It's a good question.
But consider, containers are (in modern infra) transient and usually sit in a constrained security context where access to other resources is pretty limited. Meanwhile the login credentials and security tokens are often long-lived and (usually) give broad access to a wide security context.
If you give me a back door to a container, the first thing I would be trying to do is use it to level up my credentials before the container dies. One of the first things I would try is dumping the environment!
But if the secrets were learned via an ephemeral file or through a temporary network connection, and even better if they were exchanged for time limited or single use type tokens, I am really left with both a challenging task to dig them out and / or something of limited value anyway.
It's a good question.
But consider, containers are (in modern infra) transient and usually sit in a constrained security context where access to other resources is pretty limited. Meanwhile the login credentials and security tokens are often long-lived and (usually) give broad access to a wide security context.
If you give me a back door to a container, the first thing I would be trying to do is use it to level up my credentials before the container dies. One of the first things I would try is dumping the environment!
But if the secrets were learned via an ephemeral file or through a temporary network connection, and even better if they were exchanged for time limited or single use type tokens, I am really left with both a challenging task to dig them out and / or something of limited value anyway.
It used to be not-impossible to find public phpinfo.php pages out there put up for debugging purposes. Hopefully people have smartened up to that, but it’s an example of how the data could be exposed without remote code execution.
In any case, defense in depth seems like reason alone to do this.
In any case, defense in depth seems like reason alone to do this.
I've seen credentials appear in logs from env vars. Logs tend to replicate to a few different systems and are usually less locked down than the app they came from. An attacker could get lucky with logs that live in a less-secure storage bucket or monitoring system.
> I've seen credentials appear in logs from env vars.
Then the blame is on the logging system configuration, not the env vars. Like you sanitize sensitive information out of logs, you should sanitize and not expose environment variables in your logs.
Then the blame is on the logging system configuration, not the env vars. Like you sanitize sensitive information out of logs, you should sanitize and not expose environment variables in your logs.
When I’m feeling especially paranoid e.g. regarding a key used for financial transactions, then I’ll compartmentalise the runtime using it with the absolute minimum surface area, talk to that over a message queue or similar async construct, and audit the wazoo out of all usage.
But also note, in such a case the key probably isn’t coming from an environment variable, more likely is a subkey generated by a HSM.
But also note, in such a case the key probably isn’t coming from an environment variable, more likely is a subkey generated by a HSM.
I’m a huge fan of the berglas approach where you pass “location” of the secrets in env vars and the app loads them by itself.
This allows you to keep permission grants separate, and you can safely send “.env” files and (gasp) save them in version control.
You can change where and how you store and encrypt them, rotate keys etc. all transparently.
This allows you to keep permission grants separate, and you can safely send “.env” files and (gasp) save them in version control.
You can change where and how you store and encrypt them, rotate keys etc. all transparently.
Am I understanding correctly that everyone with read access to your version control also has your production secrets?
Not the secrets themselves, but their location, eg /opt/secrets/some-database.sqlite3
I'm not parent commenter, but that's how I've done it before. The file on disk has the appropriate permissions etc.
I'm sure there are issues with this, so if anyone can enlighten me I'd be interested.
I'm not parent commenter, but that's how I've done it before. The file on disk has the appropriate permissions etc.
I'm sure there are issues with this, so if anyone can enlighten me I'd be interested.
That's a unix access problem. It's a little harder to do on containers, but the UNIX way is having the secrets file root-readable only, your daemon spawns as root, reads the configurations and drops down to a less privileged UID.
I guess it's about to check in just .env file, not files that contain secrets.
This is already in the article:
> This factor is now obsolete in one respect. As much as possible, secrets (passwords, private keys, et c) should be stored using a secrets management system such as Hashicorp Vault or Azure Key Vault. Particularly where we can rely on the infrastructure to authenticate the calling process (e.g. via a Kubernetes service account) access to secrets will not directly require credentials. The existence of the secret is under version control, but the actual secret content is immaterial.
> This factor is now obsolete in one respect. As much as possible, secrets (passwords, private keys, et c) should be stored using a secrets management system such as Hashicorp Vault or Azure Key Vault. Particularly where we can rely on the infrastructure to authenticate the calling process (e.g. via a Kubernetes service account) access to secrets will not directly require credentials. The existence of the secret is under version control, but the actual secret content is immaterial.
They are also exposed to anyone who can read from /proc/PID/environ, which means a simple directory traversal vulnerability can translate to database credentials.
This is effectively saying that they're exposed to root or the owner of the PID, which is the same story as storing secrets in files. Root also has other ways to get secrets out of remote processes.
My problem is with owner-of-PID, if you have directory traversal as root... good luck I guess.
You can store secrets in a process belonging to a different PID and get them with some kind of RPC.
You can store secrets in a process belonging to a different PID and get them with some kind of RPC.
Now that process has to get its secret from somewhere. At some point, you're just recapitulating the design of Hashicorp Vault. But even if you're using Vault, you're most likely injecting secrets as environment variables.
Belonging to a different user*
It can get it from a file that isn't readable by the user that runs remote-accessible code.
It can get it from a file that isn't readable by the user that runs remote-accessible code.
One approach I like (tell me if I'm mistaken!) is to pass things in the environment but read and remove them immediately on startup. This prevents leaking your secrets accidentally if you spawn another process etc., and has the advantage that you can use a secret store in production but raw env vars in development.
I was curious what alternative is suggested besides using env variables. From the linked article:
> Our solution thus has been to use a configuration file which is managed by the server's configuration management software (chef in our case). This way we can store secret keys outside of the code repository, and manage them using the same tool used to configure our servers. This solution, while not perfect, doesn't expose our secret keys to child processes and requires explicit access, which also communicates how developers should work with it.
> Our solution thus has been to use a configuration file which is managed by the server's configuration management software (chef in our case). This way we can store secret keys outside of the code repository, and manage them using the same tool used to configure our servers. This solution, while not perfect, doesn't expose our secret keys to child processes and requires explicit access, which also communicates how developers should work with it.
I mean, pick your poison. If you store config secrets in files, those files are targets (and you have to provision them). If you store them in environment variables, you can inject them with something like Vault. There's no perfect answer here.
What do you think about secret volumes like kubernetes does?
https://kubernetes.io/docs/concepts/configuration/secret/#co...
The existence of "best practices" that are so detailed can become a real pain at times. While they provide a good checklist of potential issue to take care with, there are plenty of cases where a specific situations would be greatly improved by some changes (while still addressing the checklist) but the mid-experience developers will dogmatically drag down any architecture into the same rigid structure.
I honestly don't know how so often now smart and somewhat experienced people don't have any confidence in their abilities over some random list on the internet, especially since when I forced them to think for themselves they do come up with improved approaches.
I honestly don't know how so often now smart and somewhat experienced people don't have any confidence in their abilities over some random list on the internet, especially since when I forced them to think for themselves they do come up with improved approaches.
Personally, I've seen much more of the opposite - clever devs shrugging off good advice without understanding it, just thinking "it doesn't apply to me"
But I think you're right about the more mid-experience developers. It would be nice if things like the 12factor specification weren't driven by Heroku's desire to make people feel like their approach is right, but rather, had a more nuanced approach like "these are our guidelines, we're confident in most cases they should be followed. If you fully understand a guideline and have a reason (driven by the bottom line, not subjective tastes, and not mentioned by us) to avoid following that guideline, do not follow it"
But I think you're right about the more mid-experience developers. It would be nice if things like the 12factor specification weren't driven by Heroku's desire to make people feel like their approach is right, but rather, had a more nuanced approach like "these are our guidelines, we're confident in most cases they should be followed. If you fully understand a guideline and have a reason (driven by the bottom line, not subjective tastes, and not mentioned by us) to avoid following that guideline, do not follow it"
Is there a conventional solution for authn/authz that just works for new systems and services?
Authorization is often very complex and depends quite a bit on the project domain. It's hard to see how any one size fits all solution could work.
> Note that environment variables are not suitable for passing secrets (such as passwords, key material, …) to service processes. Environment variables set for a unit are exposed to unprivileged clients via D-Bus IPC, and generally not understood as being data that requires protection. Moreover, environment variables are propagated down the process tree, including across security boundaries (such as setuid/setgid executables), and hence might leak to processes that should not have access to the secret data.
The former reason is systemd-specific, but the latter isn’t. There are some more reasons in https://movingfast.io/articles/environment-variables-conside....