Ask HN: Best Practices for storing customer secrets?
15 comments
IMO this depends on the strength of your org's ops function (which will usually correlate with scale of your org), and the responsibility you hold for the customer's data.
if your org does not have a strong ops function yet, and you're in scrappy startup phase: just use your language or framework's best practices, standard encryption tools. do not try to create your own encryption scheme. let your app take two keys; `CURRENT_ENCRYPTION_KEY` and `PREVIOUS_ENCRYPTION_KEY`, obviously named however you want. when encrypting data, always encrypt with `CURRENT_ENCRYPTION_KEY`. when decrypting data, first attempt with `CURRENT_ENCRYPTION_KEY`, then attempt with `PREVIOUS_ENCRYPTION_KEY` if it is set. if `PREVIOUS_ENCRYPTION_KEY` was successful, re-encrypt the unencrypted value with `CURRENT_ENCRYPTION_KEY` and save it back to the database. this gives you backward compatible deploys. you'll then still want to write and run a script that will run through and manually re-encrypt all the old data, or else you might rotate a second time and lose access to any data encoded with the first key that had not yet been rotated. rotate your key regularly by copying `CURRENT_ENCRYPTION_KEY` to `PREVIOUS_ENCRYPTION_KEY`, and putting a new key into `CURRENT_ENCRYPTION_KEY`. the appropriate value of "regularly" depends on a lot of variables, and only you can determine what is appropriate for your situation.
this is table stakes level security; realistically if your DB is compromised, your encryption key probably is too, because they probably got in through your application which holds the key in memory. this just prevents "oops I accidentally copied the DB somewhere and it leaked" (which, at most startups, I would say is the most likely leak).
if you have, or when you get to the point that you have, a competent ops org, just use HashiCorp Vault.
if your org does not have a strong ops function yet, and you're in scrappy startup phase: just use your language or framework's best practices, standard encryption tools. do not try to create your own encryption scheme. let your app take two keys; `CURRENT_ENCRYPTION_KEY` and `PREVIOUS_ENCRYPTION_KEY`, obviously named however you want. when encrypting data, always encrypt with `CURRENT_ENCRYPTION_KEY`. when decrypting data, first attempt with `CURRENT_ENCRYPTION_KEY`, then attempt with `PREVIOUS_ENCRYPTION_KEY` if it is set. if `PREVIOUS_ENCRYPTION_KEY` was successful, re-encrypt the unencrypted value with `CURRENT_ENCRYPTION_KEY` and save it back to the database. this gives you backward compatible deploys. you'll then still want to write and run a script that will run through and manually re-encrypt all the old data, or else you might rotate a second time and lose access to any data encoded with the first key that had not yet been rotated. rotate your key regularly by copying `CURRENT_ENCRYPTION_KEY` to `PREVIOUS_ENCRYPTION_KEY`, and putting a new key into `CURRENT_ENCRYPTION_KEY`. the appropriate value of "regularly" depends on a lot of variables, and only you can determine what is appropriate for your situation.
this is table stakes level security; realistically if your DB is compromised, your encryption key probably is too, because they probably got in through your application which holds the key in memory. this just prevents "oops I accidentally copied the DB somewhere and it leaked" (which, at most startups, I would say is the most likely leak).
if you have, or when you get to the point that you have, a competent ops org, just use HashiCorp Vault.
> this is table stakes level security; realistically if your DB is compromised, your encryption key probably is too, because they probably got in through your application which holds the key in memory. this just prevents "oops I accidentally copied the DB somewhere and it leaked".
Good point. If the attacker gains access to e.g. a web service that needs to access the stored secrets, they will have encryption keys and DB access.
> if you have, or when you get to the point that you have, a competent ops org, just use HashiCorp Vault.
I watched a video about Vault, but I don't see how it would help. Attacker gains access to the web service which can access Vault -> Attacker downloads all API keys from Vault. Or is there something I'm missing?
Good point. If the attacker gains access to e.g. a web service that needs to access the stored secrets, they will have encryption keys and DB access.
> if you have, or when you get to the point that you have, a competent ops org, just use HashiCorp Vault.
I watched a video about Vault, but I don't see how it would help. Attacker gains access to the web service which can access Vault -> Attacker downloads all API keys from Vault. Or is there something I'm missing?
Sorry for stream of consciousness here;
at the end of the day if the code that handles the sensitive secret is compromised, you’re leaking secrets
one of the big ways Vault can help is by separating reads and writes. the web UI that stores a secret, exposed to internet, only receives a token that can write that secret for that customer, and only that customer. that service cannot get tokens that allow the code to read secrets. the background jobs, that aren't exposed to internet, do have the ability to generate scoped tokens to read.
it also helps you mitigate risk by shortening the lifespan of tokens that can access this data. the app container/lambda/process has a Vault token that is only valid for X seconds (whatever you want it to be). This can make it a lot more difficult for an attacker to do anything useful. First they find an exploit, then they try to do something with it. If their token/access is removed every 10 seconds, that makes it a hell of a lot harder to get anywhere once they get in
Vault also increases the discoverability of a compromise by letting you log all accesses to the secrets. this helps manage the aftermath of the compromise by having more certainty in which customers have been impacted etc
It’s all basically risk mitigation. If you have data you need to use legitimately, it’s possible for someone to get it illegitimately. Limit the scope of access they can get with one break in and the length of time they can do anything once they’re in. compartmentalize systems to create defense in depth
Disclaimer: I am not a security expert, but have managed this stuff at startups too small to hire one yet
at the end of the day if the code that handles the sensitive secret is compromised, you’re leaking secrets
one of the big ways Vault can help is by separating reads and writes. the web UI that stores a secret, exposed to internet, only receives a token that can write that secret for that customer, and only that customer. that service cannot get tokens that allow the code to read secrets. the background jobs, that aren't exposed to internet, do have the ability to generate scoped tokens to read.
it also helps you mitigate risk by shortening the lifespan of tokens that can access this data. the app container/lambda/process has a Vault token that is only valid for X seconds (whatever you want it to be). This can make it a lot more difficult for an attacker to do anything useful. First they find an exploit, then they try to do something with it. If their token/access is removed every 10 seconds, that makes it a hell of a lot harder to get anywhere once they get in
Vault also increases the discoverability of a compromise by letting you log all accesses to the secrets. this helps manage the aftermath of the compromise by having more certainty in which customers have been impacted etc
It’s all basically risk mitigation. If you have data you need to use legitimately, it’s possible for someone to get it illegitimately. Limit the scope of access they can get with one break in and the length of time they can do anything once they’re in. compartmentalize systems to create defense in depth
Disclaimer: I am not a security expert, but have managed this stuff at startups too small to hire one yet
I have to be annoying, but - if you have a token that is only valid for X seconds - you still need a token to renew the expiring token.
I have the feeling that damage control is the only option:
1) Secrets store is on different credentials
2) Decryption key is only known outside of secrets storage
3) There is a maximum number of different credentials that can be queried per day (adjustable over time)
I have the feeling that damage control is the only option:
1) Secrets store is on different credentials
2) Decryption key is only known outside of secrets storage
3) There is a maximum number of different credentials that can be queried per day (adjustable over time)
Yeah you do, but you compartmentalize that with your orchestration (hence strong ops). With HashiCorp Nomad for example you might setup a parameterized job. When Nomad receives a job to do X for customer Y, it allocates a container with a short lived token. Nomad is the system with the longer living token that lets it generate short lived tokens for short lived workloads, that are themselves containerized to add a layer of security for a compromise. And so on.
Abstract that a little bit; the system that generates the short lived token ideally would not be the same as the system that is using it
Turtles all the way down
Abstract that a little bit; the system that generates the short lived token ideally would not be the same as the system that is using it
Turtles all the way down
Thank you, very useful!
We solve this problem by storing an identifier of encryption key itself along with each encrypted record. The key is loaded on demand assuming the viewer has access.
The encryption key identifier is essentially a timestamp of when the key was generated. We have a process that periodically re-encrypts all data encrypted with the old keys and then purges them. This effectively removes the secrets from our backups because after long enough there is no key to decrypt the data.
This is an acceptable trade off for our use case as the sensitive data is generally not required to be kept very long.
The encryption key identifier is essentially a timestamp of when the key was generated. We have a process that periodically re-encrypts all data encrypted with the old keys and then purges them. This effectively removes the secrets from our backups because after long enough there is no key to decrypt the data.
This is an acceptable trade off for our use case as the sensitive data is generally not required to be kept very long.
Be aware that there are some encryption algorithms where decryption with the wrong key won't fail but lead to a wrong result (rot13 being the simplest example, there are some modes in AES with this behavior too).
A potential improvement could be to use a different key per customer, where the key mapping (customer -> key) is encrypted with the application key, this process simplifies key-rotation and prevent that a leaked key gets access to all the secrets.
The key mapping schema could be handled with postgres security definer functions to avoid dumping the schema from code.
A potential improvement could be to use a different key per customer, where the key mapping (customer -> key) is encrypted with the application key, this process simplifies key-rotation and prevent that a leaked key gets access to all the secrets.
The key mapping schema could be handled with postgres security definer functions to avoid dumping the schema from code.
> realistically if your DB is compromised, your encryption key probably is too, because they probably got in through your application which holds the key in memory.
Not necessarily. It can reside in memory that isn’t readable by your web application (e.g. in a different process, in the OS kernel, in Apple’s Secure Enclave or ARMs TrustZone))
If your hardware supports something like that, you should seriously consider using it.
Not necessarily. It can reside in memory that isn’t readable by your web application (e.g. in a different process, in the OS kernel, in Apple’s Secure Enclave or ARMs TrustZone))
If your hardware supports something like that, you should seriously consider using it.
Without more context it is hard to say.
For example how often are you using their API Key? Who initiates using the key (e.g. customer, yourselves on schedule, yourselves but manually)?
The password cheatsheet linked elsewhere isn't directly relevant, because an API Key needs to be stored in a recoverable way, unlike a password. A more relevant cheatsheet might be this one (Cryptographic Storage):
https://cheatsheetseries.owasp.org/cheatsheets/Cryptographic...
Without more context or information I am making a lot of assumptions: My gut "go to" is store an entire SQLite single-file database in your regular database (binary/file column), and take advantage of SEE[0] AES-256 (OFB), SQLCipher, or one of the free/open source extension alternatives. Then encrypt/decrypt it using something derived securely from the logged-in user's password (not otherwise stored in the database) OR a mix of something user specific held in your database (e.g. UUID) + something secret stored in environmental variables/on servers. The result is that database compromise ALONE isn't enough to steal the user's secrets, you'd need server + database, or the logged-in user's original password.
I realize a whole SQLite database for secret storage might seem excessive, but it is very scalable and self-documenting as you add additional secrets or elements to existing secrets. Most commercial database offerings offer encryption at rest, and a few offer table-specific encryption, but in my experience utilizing some of those implementations is actually MORE complex than what I'm describing here, even if you'd expect the inverse.
[0] https://sqlite.org/com/see.html
For example how often are you using their API Key? Who initiates using the key (e.g. customer, yourselves on schedule, yourselves but manually)?
The password cheatsheet linked elsewhere isn't directly relevant, because an API Key needs to be stored in a recoverable way, unlike a password. A more relevant cheatsheet might be this one (Cryptographic Storage):
https://cheatsheetseries.owasp.org/cheatsheets/Cryptographic...
Without more context or information I am making a lot of assumptions: My gut "go to" is store an entire SQLite single-file database in your regular database (binary/file column), and take advantage of SEE[0] AES-256 (OFB), SQLCipher, or one of the free/open source extension alternatives. Then encrypt/decrypt it using something derived securely from the logged-in user's password (not otherwise stored in the database) OR a mix of something user specific held in your database (e.g. UUID) + something secret stored in environmental variables/on servers. The result is that database compromise ALONE isn't enough to steal the user's secrets, you'd need server + database, or the logged-in user's original password.
I realize a whole SQLite database for secret storage might seem excessive, but it is very scalable and self-documenting as you add additional secrets or elements to existing secrets. Most commercial database offerings offer encryption at rest, and a few offer table-specific encryption, but in my experience utilizing some of those implementations is actually MORE complex than what I'm describing here, even if you'd expect the inverse.
[0] https://sqlite.org/com/see.html
How sensitive are the secrets? How often will you be rotating them?
Are the secrets sensitive enough to encrypt them at rest?
Keeping the lock (the encrypted secret) and the key (the decryption key) in two separate places makes it slightly harder for an attacker to recover the plaintext secret, but also means you need to take the necessary precautions to not leak the key accidentally.
Sometimes, we can't even trust our system to be secure enough to prevent the key from becoming compromised, so Hardware Security Modules (HSMs)[1] became a thing, something with, presumably, a smaller attack surface that holds the key and can be used to decrypt the secret.
[1]: https://en.wikipedia.org/wiki/Hardware_security_module
Are the secrets sensitive enough to encrypt them at rest?
Keeping the lock (the encrypted secret) and the key (the decryption key) in two separate places makes it slightly harder for an attacker to recover the plaintext secret, but also means you need to take the necessary precautions to not leak the key accidentally.
Sometimes, we can't even trust our system to be secure enough to prevent the key from becoming compromised, so Hardware Security Modules (HSMs)[1] became a thing, something with, presumably, a smaller attack surface that holds the key and can be used to decrypt the secret.
[1]: https://en.wikipedia.org/wiki/Hardware_security_module
OWASP cheat sheet on password storage. Can be used for API key storage.
https://cheatsheetseries.owasp.org/cheatsheets/Password_Stor...
https://cheatsheetseries.owasp.org/cheatsheets/Password_Stor...
Thanks, but this is about password hashing. I would like to know about storing third party customer secrets, like API keys, in the most secure way possible.
Nice website though.
Nice website though.
The best practice is to leave the secrets on the customers systems and use asymmetric encryption (using FIDO2 or mTLS) to authenticate customers.
Are there any security measures that can be implemented in case an attacker gains access to the database?