Secure Your REST API(stormpath.com)
stormpath.com
Secure Your REST API
https://www.stormpath.com/blog/secure-your-rest-api-right-way
79 comments
So let's say I have an iOS or Android app that needs to talk to my REST API. I have my user login in the app, which does Basic Auth over HTTPS.
Once they're authenticated, I generate a token and send it to them. They store it, and use that token from here on out (HTTPS only).
Perhaps I expire it after some time (hours or days, if I don't want my user to have to login all the time).
That's it? :D Seems like I'm missing something...
Once they're authenticated, I generate a token and send it to them. They store it, and use that token from here on out (HTTPS only).
Perhaps I expire it after some time (hours or days, if I don't want my user to have to login all the time).
That's it? :D Seems like I'm missing something...
> Semantic mismatches between API endpoints and web UI endpoints (ie, semantic differences between API auth and cookie-based authentication) are a classic, pervasive, hard- to- eradicate source of serious security flaws. Don't make things any more complicated than they need to be.
Could you explain that one in more detail? I don't think I understand it.
Could you explain that one in more detail? I don't think I understand it.
You have one code path that authenticates web users through forms and sets cookies.
You have another code path that authenticates API callers.
You have code scattered through your whole application that makes authorization checks based on which user you're authenticated on.
Inconsistencies between the first two code paths often break that code.
You have another code path that authenticates API callers.
You have code scattered through your whole application that makes authorization checks based on which user you're authenticated on.
Inconsistencies between the first two code paths often break that code.
Wow, I don't think I've ever seen tptacek gray in a thread about security. I don't know why, you are absolutely right.
One neat variant on this is how GitHub uses HTTP Basic auth to give you an OAuth token... http://developer.github.com/v3/#authentication
One neat variant on this is how GitHub uses HTTP Basic auth to give you an OAuth token... http://developer.github.com/v3/#authentication
If you don't need delegation, there are a number of simple but secure methods: HTTPS + token via Basic auth seems quite popular.
What simple but secure methods are there where delegation is required? Is there something simpler than OAuth or AWS style signed HMAC?
What simple but secure methods are there where delegation is required? Is there something simpler than OAuth or AWS style signed HMAC?
The one true solution is client authenticated TLS. For the life of me I can't understand why 25 years later certificate management is still such a mess.
We don't need some perfect universally recognized root trust system to get started. Why doesn't the sign up process for authenticated API access routinely include the issuance of a certificate signed by the API owner?
Unlike for interactive users there's no expectation that a customer will be accessing an API from some random computer where he might not have access to his certificate store.
We don't need some perfect universally recognized root trust system to get started. Why doesn't the sign up process for authenticated API access routinely include the issuance of a certificate signed by the API owner?
Unlike for interactive users there's no expectation that a customer will be accessing an API from some random computer where he might not have access to his certificate store.
The user interface in browsers for client certificates is appalling.
The API support in client SSL libraries for managing multiple certificates, for applications with multiple API affiliations, is shaky.
I like TLS client authentication a lot, but it's hard to make it work.
The API support in client SSL libraries for managing multiple certificates, for applications with multiple API affiliations, is shaky.
I like TLS client authentication a lot, but it's hard to make it work.
I agree it's awful to use, but what I don't understand is why no one has bothered to improve the tools.
It's one of those areas where the underlying tool (either a library like openssl or NSS, or an OS feature like SSPI) could do the hard work in one place and make it simple for downstream libraries to wrap the functionality.
In other, somewhat analogous, domains that happened, but for whatever reason not in this case.
It's one of those areas where the underlying tool (either a library like openssl or NSS, or an OS feature like SSPI) could do the hard work in one place and make it simple for downstream libraries to wrap the functionality.
In other, somewhat analogous, domains that happened, but for whatever reason not in this case.
Look at the OpenSSL libraries for php regarding certificates. Its very close to unusable, you cannot get or set some pretty basic information like SANs.
Do you think there would be value in accepting HTTP requests and then automatically and immediately expiring every credential that ever gets passed over plain HTTP?
With an API, I think you're better off just not accepting HTTP connections to begin with. This is library design, so, principle of least astonishment. Break hard and fast.
[deleted]
Even if you use HTTPS, aren't you vulnerable to lazy devs who put the password in the URL?
If I use https://username:[email protected]/, doesn't that URL show up in server logs all over the internet?
If I use https://username:[email protected]/, doesn't that URL show up in server logs all over the internet?
Don't ever use passwords for API authentication. It's an API, not a browser. The users of an API are other programs, not people. Issue single-purpose random credentials.
I didn't think this was something I had to point out about API authentication, but apparently it is.
I didn't think this was something I had to point out about API authentication, but apparently it is.
By "don't ever use passwords" you mean, "don't let users set their own passwords," right?
Obviously, you're still using a password if you use HTTP Basic Auth.
Obviously, you're still using a password if you use HTTP Basic Auth.
[deleted]
You shouldn't ever use username/password pairs for API authentication. If the user ever changed their password, then their API calls would immediately fail!
This is one of many benefits of using multiple (revokable) API Keys.
This is one of many benefits of using multiple (revokable) API Keys.
No. The UA strips those and puts them in an Auth header.
Depends on the UA (curl?)
OK, so it'll either add an auth header or not know what to do and refuse to do anything. Either way the password is not sent in the request url
Actually, I mused have screwed up last night. The version of curl I have does support it and does put it into the Auth header.
% curl --trace-ascii /dev/stdout http://jimktrains:[email protected]
== Info: About to connect() to news.ycombinator.com port 80 (#0)
== Info: Trying 184.172.10.74... == Info: connected
== Info: Server auth using Basic with user 'jimktrains'
=> Send header, 223 bytes (0xdf)
0000: GET / HTTP/1.1
0010: Authorization: Basic amlta3RyYWluczpwYXNzd29yZA==
.......
Even curl does the right thing for basic auth
Please correct me if I'm wrong.
HTTPS will encrypt the URL, but the DNS lookup is in clear. So the "username:[email protected]" part will be sniffable/loggable.
http://stackoverflow.com/questions/499591/are-https-urls-enc...
http://stackoverflow.com/questions/499591/are-https-urls-enc...
That's incorrect. The URI library will parse the URL into a hostname, username, and password. Only the hostname is sent to the DNS server.
The username and password are sent in the "Authorization" HTTP header, which will be encrypted.
The username and password are sent in the "Authorization" HTTP header, which will be encrypted.
Shouldn't HTTP Digest still be preferred, so your server doesn't need to see the passwords?
That's not how HTTP Digest auth works, and you should never, ever be using "passwords" for API authentication.
Are you saying the server sees the password in HTTP Digest authentication?
How else would the server know what to check your response against? If you don't want to give the server a password, try SRP.
... but please don't use SRP for your API authentication.
Well, in this case (having gone back and actually checked the docs rather than relying on recollection roughly a decade old) you could store MD5(username:realm:password) rather than password, though that doesn't buy you all that much in a case like this where making the password a long, random string that's not reused is not only good practice but also easy and so likely to be general practice.
Stipulating all the other points you've made, exactly what is the purpose of hiding from a server a long random string generated by that server and useful only to that server?
Preventing someone else from getting that and using it on the server, especially if the service can rack up charges.
If you have the hash, you can log into the site (just not using the typical libraries), so it really doesn't add a lot of security. That's why I was wrong :-P
Yeah, I got there, just was confused about the digest thing.
I also wouldn't put too much value in a security article served on page using mixed HTTP and HTTPS. This shows a distinct lack of regard for and understanding of security.
[deleted]
Please stop using the term UUID when you mean 'random alphanumeric string'. Because UUIDs have a standardized format (it's not just a random string): http://en.wikipedia.org/wiki/Universally_unique_identifier
UUID v4 is, in fact, random and "alphanumeric" in the sense that it's hex.
That being said, I have a few other issues with their wording as well. They should just say "we have a custom HMAC-based authentication scheme for our REST API". Also, it took me about 3 days to realize HMAC over SSL/TLS is about as secure and easy as you can get for most any language -- If you can send HTTP requests, you can probably do HMAC. You can add further safety by making expiring private keys for HMAC and other things, although my use cases are based on long running (weeklong+) batch computations, and not end users. (i.e. initial distribution of an expiring private key for HMAC over SSL, reauthentication schemes, etc...
That being said, I have a few other issues with their wording as well. They should just say "we have a custom HMAC-based authentication scheme for our REST API". Also, it took me about 3 days to realize HMAC over SSL/TLS is about as secure and easy as you can get for most any language -- If you can send HTTP requests, you can probably do HMAC. You can add further safety by making expiring private keys for HMAC and other things, although my use cases are based on long running (weeklong+) batch computations, and not end users. (i.e. initial distribution of an expiring private key for HMAC over SSL, reauthentication schemes, etc...
It's not just random, it's a standard requiring some hex digits to be non-random (4, 8, 9, a, b). Otherwise it won't validate as a UUID4.
A UUID is first and foremost a 128 bit number, irrespective of its text encoding.
Its 'canonical' form uses HEX-only encoding. A 'Url62' can be another encoding. 'Url62' wouldn't be a canonical encoding, but it's still a 128 bit UUID number.
Its 'canonical' form uses HEX-only encoding. A 'Url62' can be another encoding. 'Url62' wouldn't be a canonical encoding, but it's still a 128 bit UUID number.
> Best practices say to encrypt your passwords in the database to limit a potential data breach. This increases overhead for each request when authenticating a user. Unique API keys authentication skips the hashing step and therefore speeds up your calls.
Wait, why can you can skip the hashing step and still be secure? Because hashing is only neccesary if you call it a 'password', but not if you use it in the same way but call it an 'api key' instead?
I guess it depends on the purpose for hashing. If it's just about 'data breaches', then maybe it doesn't matter if your api keys get out... because they at least won't grant access to any _other_ systems, since they weren't manually entered by users and re-used accross systems. Is that what you're thinking?
But don't you still want to avoid a data breach like this for your _own_ service?
And, I think, isn't the other reason hashed passwords are used, to make it harder to do a brute force attack? Ie, it's quite intentional and deliberate that it increases request overhead. And doesn't this still apply to an api key, possibly even MORE so when you have a single api key instead of a username/pass combo?
Wait, why can you can skip the hashing step and still be secure? Because hashing is only neccesary if you call it a 'password', but not if you use it in the same way but call it an 'api key' instead?
I guess it depends on the purpose for hashing. If it's just about 'data breaches', then maybe it doesn't matter if your api keys get out... because they at least won't grant access to any _other_ systems, since they weren't manually entered by users and re-used accross systems. Is that what you're thinking?
But don't you still want to avoid a data breach like this for your _own_ service?
And, I think, isn't the other reason hashed passwords are used, to make it harder to do a brute force attack? Ie, it's quite intentional and deliberate that it increases request overhead. And doesn't this still apply to an api key, possibly even MORE so when you have a single api key instead of a username/pass combo?
If you lose the database for your own service, it does not matter how you authenticate to that service. Attackers have already fatally bypassed your credential system. At the same time, API credentials should by definition be single-use.
> At the same time, API credentials should by definition be single-use.
Could you spell this one out a little bit more? Do you mean only a single session should be able to use an API credential?
Could you spell this one out a little bit more? Do you mean only a single session should be able to use an API credential?
I mean the credential should only be relevant to the service, never shared across multiple services, because the API generates it for you.
HMAC authentication requires both the client and the server to have a shared secret (or more
likely a derived key based on a shared secret). The secret cannot be saved as a one-way hash
(as might be common for a password). So you couldn't use BCrypt or SCrypt to hash the shared
(or derived) secret since the server would never be able to acquire the value to calculate the
same digest.
You can still encrypt the secret, e.g. using AES 256 bit encryption with secure random Initialization Vectors and rolling keys. This too is not easily 'brute forceable', but is very fast to decrypt compared to a BCrypt comparison (key storage should also be in a separate location than the main data store).
You can still encrypt the secret, e.g. using AES 256 bit encryption with secure random Initialization Vectors and rolling keys. This too is not easily 'brute forceable', but is very fast to decrypt compared to a BCrypt comparison (key storage should also be in a separate location than the main data store).
What are your thoughts on Amazon like security scheme? As far as there are no third party apps involved, I think OAuth is an overkill.
What I mean by Amazon like securtiy is described in this article http://www.thebuzzmedia.com/designing-a-secure-rest-api-with...
What I mean by Amazon like securtiy is described in this article http://www.thebuzzmedia.com/designing-a-secure-rest-api-with...
Stormpath's custom scheme is very similar to Amazon's.
But per the blog article, you'd only want to do this if you are willing to support client libraries/sdks that implement it as well. No one wants to spend the time to implement non-standard custom HMAC algorithms.
But per the blog article, you'd only want to do this if you are willing to support client libraries/sdks that implement it as well. No one wants to spend the time to implement non-standard custom HMAC algorithms.
We tried this with Pagify http://pagify.io And I think supporting client libraries was not an issue, since we had to provide SDK one way or the other.
Totally agree. The key here is that you're doing the work to implement the algorithms, not your customers. If they had to do it, they probably just wouldn't use it.
Thanks for the pointer, I was looking for a sane explanation of the Amazon security and signature algorithm.
I have to disagree with the other comments here regarding the client library. I think that given the precedence of Amazon API, given that people understand how to sign APIs like amazon, this method will be accepted even without a client library.
Being a Java / Scala dev, I prefer that an API provider allow me to select the HTTP client libraries to use and prevent from forcing me to use a specific library & version via SDK transitive dependencies.
I have to disagree with the other comments here regarding the client library. I think that given the precedence of Amazon API, given that people understand how to sign APIs like amazon, this method will be accepted even without a client library.
Being a Java / Scala dev, I prefer that an API provider allow me to select the HTTP client libraries to use and prevent from forcing me to use a specific library & version via SDK transitive dependencies.
Does anyone know of a good reason not to use TLS with client-side certificates (and you as the CA) for API authentication?
It can become a pain to manage from the server side (issuing new ones &c), but technically they're pretty nice. The company I used to work for used them for their api.
In my humble opinion, security through obscurity via UUIDS rather than sequential integers is not great advice. It merely masks the real problem.
This isn't an instance of "security through obscurity". This is more akin to suggesting people use non dictionary passwords.
http://en.wikipedia.org/wiki/Security_through_obscurity
http://en.wikipedia.org/wiki/Security_through_obscurity
Obscurity can be a fine part of being secure. You have to think carefully about what happens when it fails, and realize that automated bots will tend to find things humans would consider obscure, but don't let the usually-good heuristic become a straightjacket.
Obscurity of the secret is a necessary part of being secure :-P
Security-through-obscurity refers to the notion that your algorithms are a meaningful part of the key.
Security-through-obscurity refers to the notion that your algorithms are a meaningful part of the key.
Shoots down Basic auth without SSL, without mentioning Digest auth, weird.
I'm not sure I've ever seen a scenario where digest authentication was a win.
That's a good suggestion - will see if we can add a para about it. We use digest authentication, fwiw.
There are many types of digest authentication - OAuth1.0a and Amazon's and Stormpath's custom schemes are examples. Browser-specific digest authentication wasn't covered however since the article was about REST APIs and most REST clients are not browsers.
I'm pretty sure he means RFC 2617 Digest authentication. There's nothing browser-specific about it.
I gathered as much. But in practice, how often do you see RFC 2617 Digest authc used in non-browser scenarios? (I'm genuinely curious. I haven't seen it used much at all outside of web browsers, so I'm curious what others may have come across).
I've written Atom Publishing Protocol servers that use it.
It's not badly-suited for non-browser tasks (although yes, SSL and Basic is much simpler - if you don't mind paying for the certificate). It's unusual, but it's pretty unusual to use it (or Basic) for web browsers these days, too.
[deleted]
Didn't oAuth2 support signed requests via MAC token on request?
I expected something very different from this article based on the headline. Specifically, I completely disagree with the "leave it to industry standards" approach, as that doesn't help people understand what they should do, and more importantly, why.
Request/response protocols (well, many things) really break down into 5 top-level categories (some sources will say the 6th is Audit):
- Authentication
- Authorization
- Integrity
- Confidentiality
- Non-repudiation
It's a far more interesting exercise to walk through what you would get from each solution. Basic-Auth over TLS, actually gets you quite a ways towards that goal (specifically, C-I-A (authentication)). Where that, and notably HMAC, fall over is non-repudiation because they're based on a shared-secret model; admittedly HMAC keys are better than passwords because you're not sending the secret on every request, but asymmetric crypto is preferred. Authorization that the server system does is really out of scope in all of these protocols, so Basic-Auth over TLS doesn't really impact that one. It can be as simple as "caller = owner," or as full-featured as a security policy language [1] (full disclosure: I am the original author of [1]).
OAuth really doesn't differ that much besides specifically solving the delegated access and website SSO problem(s); but IMO it does so with an overly baroque protocol that has too many parts. The "long pole" of setting up such a system (that is, allowing 3rd party sites to act on behalf of my site's users) isn't the specifics of what my REST api looks like, but really it's all the "governance" of user decisions, and more-over, key management (in all these cases, key management is generally the hardest or almost hardest problem).
While it can be debated whether it was right or wrong, we (Joyent) released an open-source spec to solve straight up authentication of REST requests using SSH keys [2]. At the end of the day, the user signs the Date header of requests with their private key (which by definition the server has never seen), and all requests must be over TLS. Disregarding Authorization, this scheme gives you Confidentiality (TLS), Integrity (TLS), Authentication (Signature), Non-Repudiation (asymmetric signature), and adds a "poor man's nonce," assuming you disallow requests where the clock skew of the date header is too large. And lastly, SSH solves a lot of key management problems for humans. Note: I didn't drop that reference to advocate for our specification here, but rather the security process you should think about when evaluating whether a protocol is secure.
[1] http://docs.aws.amazon.com/IAM/latest/UserGuide/policy-refer...
[2] https://github.com/joyent/node-http-signature/blob/master/ht...
PS
Mutual-Auth SSL/TLS is a royal PITA, and is basically guaranteed to cause you grief. The client compatibility matrix might as well be considered an NP hard problem to assure yourself coverage, and failure modes of the different browsers/SDKs all differ. As a REST API should have maximum accessibility to clients (i.e., don't wed yourself to any one language/sdk), this is pretty much a non-starter.
* edited: copy/paste formatting
Request/response protocols (well, many things) really break down into 5 top-level categories (some sources will say the 6th is Audit):
- Authentication
- Authorization
- Integrity
- Confidentiality
- Non-repudiation
It's a far more interesting exercise to walk through what you would get from each solution. Basic-Auth over TLS, actually gets you quite a ways towards that goal (specifically, C-I-A (authentication)). Where that, and notably HMAC, fall over is non-repudiation because they're based on a shared-secret model; admittedly HMAC keys are better than passwords because you're not sending the secret on every request, but asymmetric crypto is preferred. Authorization that the server system does is really out of scope in all of these protocols, so Basic-Auth over TLS doesn't really impact that one. It can be as simple as "caller = owner," or as full-featured as a security policy language [1] (full disclosure: I am the original author of [1]).
OAuth really doesn't differ that much besides specifically solving the delegated access and website SSO problem(s); but IMO it does so with an overly baroque protocol that has too many parts. The "long pole" of setting up such a system (that is, allowing 3rd party sites to act on behalf of my site's users) isn't the specifics of what my REST api looks like, but really it's all the "governance" of user decisions, and more-over, key management (in all these cases, key management is generally the hardest or almost hardest problem).
While it can be debated whether it was right or wrong, we (Joyent) released an open-source spec to solve straight up authentication of REST requests using SSH keys [2]. At the end of the day, the user signs the Date header of requests with their private key (which by definition the server has never seen), and all requests must be over TLS. Disregarding Authorization, this scheme gives you Confidentiality (TLS), Integrity (TLS), Authentication (Signature), Non-Repudiation (asymmetric signature), and adds a "poor man's nonce," assuming you disallow requests where the clock skew of the date header is too large. And lastly, SSH solves a lot of key management problems for humans. Note: I didn't drop that reference to advocate for our specification here, but rather the security process you should think about when evaluating whether a protocol is secure.
[1] http://docs.aws.amazon.com/IAM/latest/UserGuide/policy-refer...
[2] https://github.com/joyent/node-http-signature/blob/master/ht...
PS
Mutual-Auth SSL/TLS is a royal PITA, and is basically guaranteed to cause you grief. The client compatibility matrix might as well be considered an NP hard problem to assure yourself coverage, and failure modes of the different browsers/SDKs all differ. As a REST API should have maximum accessibility to clients (i.e., don't wed yourself to any one language/sdk), this is pretty much a non-starter.
* edited: copy/paste formatting
> Mutual-Auth SSL/TLS is a royal PITA, and is basically guaranteed to cause you grief. The client compatibility matrix might as well be considered an NP hard problem to assure yourself coverage, and failure modes of the different browsers/SDKs all differ. As a REST API should have maximum accessibility to clients (i.e., don't wed yourself to any one language/sdk), this is pretty much a non-starter.
Given the inconsistent support for mutual-auth TLS, which along with its direct ancestors has existed for a decade and a half, what makes you think they'll be widespread and correct support for SAS?
Given the inconsistent support for mutual-auth TLS, which along with its direct ancestors has existed for a decade and a half, what makes you think they'll be widespread and correct support for SAS?
If by SAS you mean SSH, then really it's that there's a much smaller client surface area (really, openssh), and the fallback is always "use a PKCS#8 private key", which is much more uniform than SSL. SSL mixes in x509, and protocol, and UX.
That said, you don't have to agree with it. My macro point was really assessing these things against basic security threat modeling, not whether or not you agree with our choices of using SSH.
That said, you don't have to agree with it. My macro point was really assessing these things against basic security threat modeling, not whether or not you agree with our choices of using SSH.
Well it's more than SSH. The client library needs to to implement the headers and do the date signing. Sure it's easier than TLS, but the libraries suffers more from a lack of dedicated interest than insane complexity I would think.
But I do agree with your basic point that too many people stop at authentication, instead of considering the full range of concerns.
But I do agree with your basic point that too many people stop at authentication, instead of considering the full range of concerns.
How does using bare api keys over TLS compare to these suggestions? It's less secure but is it still a recommended option?
The advantage to OAuth over basic-auth is that you can delegate the credentials, which is to say you can set up your system so that users can give a limited-use credential to a 3rd party application to perform API calls on their behalf.
Some very large services have abused OAuth to "delegate" credentials to mobile devices, which has set up the expectation among developers that OAuth is the "sophisticated" way of doing all-around credential management of any sort. Not so. If you don't have delegation to third parties, don't use OAuth.
Most applications do not need delegation.
If you need delegation, there are simpler ways to do it than OAuth that won't meaningfully sacrifice the security of your controls. At the same time, using an OAuth solution you don't fully understand (for instance, using OAuth through a high-level library that hides the details from you) can damage the integrity of your whole application by creating new classes of mistakes for you to make.
Some very large services have abused OAuth to "delegate" credentials to mobile devices, which has set up the expectation among developers that OAuth is the "sophisticated" way of doing all-around credential management of any sort. Not so. If you don't have delegation to third parties, don't use OAuth.
Most applications do not need delegation.
If you need delegation, there are simpler ways to do it than OAuth that won't meaningfully sacrifice the security of your controls. At the same time, using an OAuth solution you don't fully understand (for instance, using OAuth through a high-level library that hides the details from you) can damage the integrity of your whole application by creating new classes of mistakes for you to make.
[deleted]
[Late addition:
* Do not use passwords as API authentication. The user of an API is a computer program, not a human. Issue single-purpose random credentials for API access.]
* Make sure that your API is accessible only over HTTPS; test the API endpoint to ensure requests aren't honored over unencrypted HTTP.
* Use the simplest API authentication mechanism that (a) works and (b) you understand. You should probably degrade the capabilities of your API before you try to adopt an authentication system that you don't completely, fully, absolutely grok.
This means, if you don't need delegation right away (for instance, if you don't have 3rd party applications making calls to your API on behalf of your users), don't bother with anything like OAuth.
Contrary to the implications in this article, HTTP Basic Auth is not less secure than OAuth if your endpoints all require HTTPS. If you allow HTTP calls to an API, you have bigger problems than your auth token scheme.
Semantic mismatches between API endpoints and web UI endpoints (ie, semantic differences between API auth and cookie-based authentication) are a classic, pervasive, hard- to- eradicate source of serious security flaws. Don't make things any more complicated than they need to be.
[Late edit: I didn't think I needed to make the first point, but I clearly did need to. Sorry.]