Notifying administrators about unhashed password storage(cloud.google.com)
cloud.google.com
Notifying administrators about unhashed password storage
https://cloud.google.com/blog/products/g-suite/notifying-administrators-about-unhashed-password-storage
119 comments
Doesn't really explain that the "scrambling" is deterministic which is an important feature. In colloquial use "scramble" means to randomize, for example "scramble the code to your gym locker."
When you scramble your locker combo, is really is random. When you scramble your password with a hash function, it may look random, but it's really not.
Also just for fun, here's a creative stab at my own explanation.
As every dog knows, each dog has a unique smell. If you have smelled the dog before, then you can use its smell to recognize who it is, but without that past experience, nothing in the smell itself tells you who it might be.
The hash function creates the smell from your password, which is the dog. A website does't save a photo of your dog, just the smell. So if a hacker breaks in, they might smell your password but they can't see it. And when you login to the website, it makes sure your password smells right before it lets you in.
When you scramble your locker combo, is really is random. When you scramble your password with a hash function, it may look random, but it's really not.
Also just for fun, here's a creative stab at my own explanation.
As every dog knows, each dog has a unique smell. If you have smelled the dog before, then you can use its smell to recognize who it is, but without that past experience, nothing in the smell itself tells you who it might be.
The hash function creates the smell from your password, which is the dog. A website does't save a photo of your dog, just the smell. So if a hacker breaks in, they might smell your password but they can't see it. And when you login to the website, it makes sure your password smells right before it lets you in.
> Also just for fun, here's a creative stab at my own explanation.
I know you said "just for fun", but I really think we need to do better than these types of explanations.
I remember many years ago, I was wondering what the heck hyper-threading was, and I came across a video that compared it to a bowl of cereal. Even though you only have one mouth, you'll be able to eat the cereal more quickly if you use both your hands, instead of just one.
I watched this video, and felt gratified, and closed the browser window... and then suddenly realized that I still had no idea what hyper-threading was, only that I should use both my hands if I'm ever in a cereal-eating competition.
A better explanation would have gone something like, "CPUs solve math problems more quickly than we can give them problems to solve. Hyper-threading queues up additional sets of problems that the CPU can switch to when it would otherwise be doing nothing."
Analogies are useful tools, but usually only after we have some understanding of the real version.
I know you said "just for fun", but I really think we need to do better than these types of explanations.
I remember many years ago, I was wondering what the heck hyper-threading was, and I came across a video that compared it to a bowl of cereal. Even though you only have one mouth, you'll be able to eat the cereal more quickly if you use both your hands, instead of just one.
I watched this video, and felt gratified, and closed the browser window... and then suddenly realized that I still had no idea what hyper-threading was, only that I should use both my hands if I'm ever in a cereal-eating competition.
A better explanation would have gone something like, "CPUs solve math problems more quickly than we can give them problems to solve. Hyper-threading queues up additional sets of problems that the CPU can switch to when it would otherwise be doing nothing."
Analogies are useful tools, but usually only after we have some understanding of the real version.
Analogies are great to memorize the content. They don't explain the content because they're not 1:1 translatable.
Does that mean that analogies themselves are an analogy of hash functions?
(not serious; hope it is not against the guidelines)
More seriously analogies are better for a bigger picture of a complex system, as they effectively describe mode of interactions. The dog/smell analogy was not bad but you have to also add a role for the other actor in the system (here I would say: the third party, and the database)
(not serious; hope it is not against the guidelines)
More seriously analogies are better for a bigger picture of a complex system, as they effectively describe mode of interactions. The dog/smell analogy was not bad but you have to also add a role for the other actor in the system (here I would say: the third party, and the database)
It says that it scrambles it "the same way" when it checks it. Isn't that clear enough?
The scrambling is modulated by a random nonce which is stored together with the hash. So you are scrambling the original randomly; you then scramble the password-to-be-checked the same "way" (ie. with the same nonce) to compare hashes.
The scrambling is modulated by a random nonce which is stored together with the hash. So you are scrambling the original randomly; you then scramble the password-to-be-checked the same "way" (ie. with the same nonce) to compare hashes.
I can’t believe you think “scrambles the same way” is misleading and “smells right” isn’t.
It’s sort of implied when they explained they do the same hashing and ensure the hashes match though.
I fully understand password hashing and I still had trouble following your analogy. I think the key is using as little jargon as possible and explaining the actual concepts plainly.
>"Hashing" a password is a method of scrambling it that will produce the same scramble every time you do so. However two different passwords will never produce the same scramble.
Could be better, but emphasizes the deterministic nature of hashing but also avoids tangential details and gotchas like collisions which aren't relevant here.
>"Hashing" a password is a method of scrambling it that will produce the same scramble every time you do so. However two different passwords will never produce the same scramble.
Could be better, but emphasizes the deterministic nature of hashing but also avoids tangential details and gotchas like collisions which aren't relevant here.
I don't think explaining determinism is necessary or even helpful here. I showed the post to my girlfriend and it made perfect sense to her. Explaining that sometimes doing the same thing in the same way results on different results unless you're careful only muddies the message.
The explanation is clear that they scramble the password in the same way to get the same result.
The explanation is clear that they scramble the password in the same way to get the same result.
The other thing they could address is that more than one password can map to the same hash. But it would not improve the explanation.
There’s no need to confuse things with metaphors as anyone with elementary understanding of maths should be able to understand it:
A cryptographic hash function is an algorithm that:
Given an input, it returns a very large number.
Given a bit-for-bit identical input it returns exactly the same number.
If the input changes by one bit, it _should_ give an extremely different number, so different that it appears to be random.
The probability of two different inputs producing the same output is extremely low, and the better the hash function, the lower this probability is.
A cryptographic hash function is an algorithm that:
Given an input, it returns a very large number.
Given a bit-for-bit identical input it returns exactly the same number.
If the input changes by one bit, it _should_ give an extremely different number, so different that it appears to be random.
The probability of two different inputs producing the same output is extremely low, and the better the hash function, the lower this probability is.
Funny thing, human scrambling is not actually random (it's a shuffle not a randomization) and neither is a coin toss.
When asked to produce a random password, people gravitate towards nonrepetition and that dependent on medium they use to type the password in. This results in a highly skewed blue noise statistics.
People aren't random even when they try to be.
People aren't random even when they try to be.
Wait, a coin toss isn’t random?
It's nearly impossible to unscramble the hash if it is random.
It's trivial to unscramble hashed dictionary words with 123! or 2019! appended to the end.
Google should somehow note that distinction as it is important.
It's trivial to unscramble hashed dictionary words with 123! or 2019! appended to the end.
Google should somehow note that distinction as it is important.
Technically, you're guessing the password, you're not unscrambling it (which, afaik is not just "nearly impossible", but it is impossible by design for modern hash functions).
Being able to make millions (or hundreds of millions or even billions) of guesses each second makes successful guessing more feasible.
Being able to make millions (or hundreds of millions or even billions) of guesses each second makes successful guessing more feasible.
It's worth pointing out that while modern hash functions are believed to be infeasible to unscramble (technically, to require exponential time), none have been proven so. In fact, proving the mere existence of such a function would prove P!=NP.
Wouldn’t it prove the opposite namely that P equals NP?
No, if you can prove that a secure hash function exists, then you have proved that P != NP. (This is what alexbecker said.)
If you can prove P = NP, then you have proved that no secure hash functions exist. (This is the contrapositive.)
If you can prove P = NP, then you have proved that no secure hash functions exist. (This is the contrapositive.)
For convenient definitions of "secure" :-). "Unscrambling" a particular function could be in P but still effectively intractable.
(For a fun angle not related to complexity, the algorithm itself could be so large that it wouldn't fit in the observable universe without breaking the Bekenstein bound.)
And it has been a long time since I did proof theory, but is there the possibility of a non-constructive proof but no actual examples no matter how long you keep enumerating algorithms? Or a non-constructive proof, but no constructive proof? In that case there could be a "perfect heuristic"...
(For a fun angle not related to complexity, the algorithm itself could be so large that it wouldn't fit in the observable universe without breaking the Bekenstein bound.)
And it has been a long time since I did proof theory, but is there the possibility of a non-constructive proof but no actual examples no matter how long you keep enumerating algorithms? Or a non-constructive proof, but no constructive proof? In that case there could be a "perfect heuristic"...
Oops, this is what I get for not reading carefully :) But this is interesting: it implies, if I'm not mistaken, it implies that we haven't ever proven that a function takes exponential time, just that we haven't found any algorithms for certain problems faster than exponential time. I hadn't considered P != NP this way before...
No, we know that P != EXP_TIME, by the time hierarchy theorem - i.e. we know some problems (and we do have concrete examples) take exponential time. What we don't know is that there exists a problem that requires exponential time (or otherwise greater than polynomial) where we can create a proof that a answer is correct which is checkable in polynomial time.
https://en.wikipedia.org/wiki/Time_hierarchy_theorem
https://en.wikipedia.org/wiki/Time_hierarchy_theorem
I think you mean... P = NP. Which is very unlikely as we could do away with mathematicians if that were true.
Seems unlikely, but not proven. That's the point. I dunno about 'do away with mathematicians'.
Yeah, I just thought that showing how to solve a best-case exponential function in polynomial kind was the sort of thing that would prove P = NP (I dictated this before, so NP came out as MP)
There's one thing I've never understood about this "guessing" technique to find the plain text of a known hash value. Wouldn't the sticker also need to know exactly what hashing function had been used as well as how many times applied (i.e. what the "stretching" factor was)?
Because without that knowledge what would the attacker run their guesses through? And wouldn't that information be hard to get unless you had the full source code along with all relevant configuration?
Because without that knowledge what would the attacker run their guesses through? And wouldn't that information be hard to get unless you had the full source code along with all relevant configuration?
One of the assumptions in designing cryptographic systems (Kerckhoff's Principle [1]) is that the attacker knows everything about the system and how it is implemented, except for the password itself. While you could hide the implementation details, that would mean that nobody could verify how strong your system is.
[1] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle
[1] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle
You have to test the password against the real login feature, otherwise you wouldn’t know if you guessed it. The hashing function can be widely known and attackers using brute force guesses would still be blocked by a security feature that requires a second factor (e.g. email) after a certain number of failed attempts.
As an attacker you're not trying to guess the hashed password and even if you had it you still would not be able to use it to log in. Hashing functions that are fit for real-world use are pure functions and their input cannot be determined by looking at their output, which renders rainbow tables (a list of plain text input and their corresponding hashed output) useless to attackers.
So knowing the hash function is useless and even if you also manage to access the database and get the hashed password you still need to run through a brute force technique to find which input gets that output. If the original password is itself a long random set of characters stored with Keychain or Onepass you’re looking at potentially trillions of attempts before you get it right. Hashing functions are written to be slow and computationally expensive so you’re going to incur a lot of energy costs by the time you get it. Also, you’ll be dead by then. When brute force works it’s because the password has no failure restrictions and is something short and simple like a mother’s maiden name with a zip code at the end.
As an attacker you're not trying to guess the hashed password and even if you had it you still would not be able to use it to log in. Hashing functions that are fit for real-world use are pure functions and their input cannot be determined by looking at their output, which renders rainbow tables (a list of plain text input and their corresponding hashed output) useless to attackers.
So knowing the hash function is useless and even if you also manage to access the database and get the hashed password you still need to run through a brute force technique to find which input gets that output. If the original password is itself a long random set of characters stored with Keychain or Onepass you’re looking at potentially trillions of attempts before you get it right. Hashing functions are written to be slow and computationally expensive so you’re going to incur a lot of energy costs by the time you get it. Also, you’ll be dead by then. When brute force works it’s because the password has no failure restrictions and is something short and simple like a mother’s maiden name with a zip code at the end.
Since you know your own password on the system you can use that to figure out the hashing mechanism. Also, the hashing strategy and salt is often part of the stored hash.
This technique is referred to as a chosen plaintext attack.
You can make a few simplifying guesses. 1. Google uses a lot of go so its likely they use bcrypt, so try that first.
2. Most people arent security experts so dont bother w/ stretching, just do one hash.
Those 2 assumptions right there get you what you want 90% of the time.
Those 2 assumptions right there get you what you want 90% of the time.
The "key" of a "key derivation function" typically stores all that information. For example, see the format for scrypt: https://github.com/Tarsnap/scrypt/blob/master/FORMAT
[deleted]
IMO they would have ideally used "key derivation function" instead of "hash function". It could lead those who know enough to be dangerous to think that safely storing passwords is a simple `sha512($password)` away.
You’re kidding, right? It’s supposed to be a simple explanation.
The ordinary response to notifications like these is, to put it bluntly and accurately, a rage-mob. But here you have a corrective, a case where that doesn't make sense. Because Google operates what is considered, by an overwhelming majority of expert opinion, one of the 3 best security teams in the industry, likely exceeding in so me ways the elite of some major world governments. And they can't reliably promise, at least not in 2019, never to accidentally durably log passwords. If they can't, who else can? What are we to do with this new data point?
The issue here is meaningful, and it's useful to have a reminder that accidentally retaining plaintext passwords is a hazard of building customer identity features.
But I think it's at least equally useful to get the level set on what engineering at scale can reasonably promise today.
correction: in the world -> in the industry
The issue here is meaningful, and it's useful to have a reminder that accidentally retaining plaintext passwords is a hazard of building customer identity features.
But I think it's at least equally useful to get the level set on what engineering at scale can reasonably promise today.
correction: in the world -> in the industry
Great stuff to wake up to.
We've got five affected accounts.
So far the G-Suite experience has been underwhelming to say the least. Crap interface and less fine grained control over document access than even Google drive offers for free, gmail UI just plain sucks over IMAP and so on.
And now this. As much as there is to like about this email in terms of transparency it is also very interesting for what it does not say: Apparently they can't determine with certainty whether or not the accounts were accessed.
"legacy functionality that enabled customer Domain Admins to view password"
That functionality should have never existed to begin with.
"primarily impacted system generated or admin generated passwords intended for one-time use"
Note the weasel word 'primarily', either it did or it did not potentially affect all passwords.
"We have reviewed the login information for the account(s) and have found no evidence that the unhashed passwords were misused."
No evidence does not mean it did not happen, so there apparently is a chance that even if they did not find evidence that it did happen and that their audit trails for reviewing that log data are not such that they can guarantee that nobody had access to it or viewed it.
"an internal system that logged account signup information for diagnostic purposes, also inadvertently logged the administrator’s account password in our encrypted systems in an unhashed format."
This suggests some pretty major process failures, this change was apparently found after it had already been pushed to production without review or with a review that did not catch this pretty basic mistake.
Having the best security team in the world is great but if you then have stuff like this happening you have to wonder about the processes around deployment, which are just as important as having a great security team to begin with.
Damned if you do and damned if you don't, a small company is better off relying on the likes of Google for their secure storage of mail and documents, but at the same time that's also not perfect and apparently includes some random strangers potentially having access to all of that which is something that never happened to us in the last decade or so to the best of my knowledge.
We've got five affected accounts.
So far the G-Suite experience has been underwhelming to say the least. Crap interface and less fine grained control over document access than even Google drive offers for free, gmail UI just plain sucks over IMAP and so on.
And now this. As much as there is to like about this email in terms of transparency it is also very interesting for what it does not say: Apparently they can't determine with certainty whether or not the accounts were accessed.
"legacy functionality that enabled customer Domain Admins to view password"
That functionality should have never existed to begin with.
"primarily impacted system generated or admin generated passwords intended for one-time use"
Note the weasel word 'primarily', either it did or it did not potentially affect all passwords.
"We have reviewed the login information for the account(s) and have found no evidence that the unhashed passwords were misused."
No evidence does not mean it did not happen, so there apparently is a chance that even if they did not find evidence that it did happen and that their audit trails for reviewing that log data are not such that they can guarantee that nobody had access to it or viewed it.
"an internal system that logged account signup information for diagnostic purposes, also inadvertently logged the administrator’s account password in our encrypted systems in an unhashed format."
This suggests some pretty major process failures, this change was apparently found after it had already been pushed to production without review or with a review that did not catch this pretty basic mistake.
Having the best security team in the world is great but if you then have stuff like this happening you have to wonder about the processes around deployment, which are just as important as having a great security team to begin with.
Damned if you do and damned if you don't, a small company is better off relying on the likes of Google for their secure storage of mail and documents, but at the same time that's also not perfect and apparently includes some random strangers potentially having access to all of that which is something that never happened to us in the last decade or so to the best of my knowledge.
You might be aware that in Google not every application team works together with the security teams. They are supposed to work with them. That is the best practice and that helps with security compliance and review. They often do. But it is also sometimes easy to forget doing so if the team is not doing their due diligence.
It is possible that this team did not work with the security team even if it is a highly unlikely scenario. The likely scenario is that this team did work with a security team and they were aware they were supposed to hash the passwords but they made a mistake during the implementation.
I think what is being underappreciated here is that very very smart application developers can have little to no idea about security best practices. I can say this confidently from my direct experience of working with Googlers.
It is possible that this team did not work with the security team even if it is a highly unlikely scenario. The likely scenario is that this team did work with a security team and they were aware they were supposed to hash the passwords but they made a mistake during the implementation.
I think what is being underappreciated here is that very very smart application developers can have little to no idea about security best practices. I can say this confidently from my direct experience of working with Googlers.
Who else would you put in that league?
He's mentioned before that Google, Microsoft, and Apple are in a league of their own with regards to security.
To be honest, as far as I'm concerned there is no reason why Google or anyone should ever even have access to the plaintext passwords, let alone log them.
There Is no reason why we can't have 2 rounds of hashing, one client side and one server side. This way even if Google is malicious, it cannot know the actual password.
There Is no reason why we can't have 2 rounds of hashing, one client side and one server side. This way even if Google is malicious, it cannot know the actual password.
If you do a round of hashing on the client side, whatever is sent to the server becomes the actual secret needed to log in to the service.
The only improvement is if you share your password between websites/services, which you shouldn't do. And a malicious service can change the javascript to exfiltrate the input password any time you log in.
The only improvement is if you share your password between websites/services, which you shouldn't do. And a malicious service can change the javascript to exfiltrate the input password any time you log in.
There are two classes of algorithms you can use to solve that problem: zero-knowledge proofs and Password Authenticated Key Exchanges (PAKE):
In the algorithm I’m most familiar with (SRP6a) the plaintext password never leaves the client machine either on initial signup or on later password-based authentication, and what does transit the wire can be used neither to recover the password nor to replay the login attempt. You cannot break into the account just by passively watching the messages go back and forth.
https://en.m.wikipedia.org/wiki/Salted_Challenge_Response_Au...
One nice thing that falls out of a PAKE algorithm, as opposed to a zero-knowledge proof, is that you get a cryptographic key on both sides (thus the “key exchange” part of the acronym) that the peers can use with a symmetric encryption algorithm to communicate with each other. This then lets you construct things like TLS-SRP, which is a variant of TLS (ne SSL) that uses SRP to get the symmetric key instead of X.509 certificates and an entirely different key exchange algorithm:
https://en.m.wikipedia.org/wiki/Zero-knowledge_proof
https://en.m.wikipedia.org/wiki/Password-authenticated_key_agreement
In the algorithm I’m most familiar with (SRP6a) the plaintext password never leaves the client machine either on initial signup or on later password-based authentication, and what does transit the wire can be used neither to recover the password nor to replay the login attempt. You cannot break into the account just by passively watching the messages go back and forth.
https://en.m.wikipedia.org/wiki/Secure_Remote_Password_protocol
https://en.m.wikipedia.org/wiki/Salted_Challenge_Response_Au...
One nice thing that falls out of a PAKE algorithm, as opposed to a zero-knowledge proof, is that you get a cryptographic key on both sides (thus the “key exchange” part of the acronym) that the peers can use with a symmetric encryption algorithm to communicate with each other. This then lets you construct things like TLS-SRP, which is a variant of TLS (ne SSL) that uses SRP to get the symmetric key instead of X.509 certificates and an entirely different key exchange algorithm:
https://en.m.wikipedia.org/wiki/TLS-SRP
There is a better way: use an oracle-like device whose secret key is unavailable.
See (shameful plug) http://thesybil.net
Yes, it is academic but it should be everywhere.
I improved it to perform client-side hashing and encryption but have had not the time to update the docs.
See (shameful plug) http://thesybil.net
Yes, it is academic but it should be everywhere.
I improved it to perform client-side hashing and encryption but have had not the time to update the docs.
I can't connect, the domain does not resolve.
But sure, there are good solutions to this, like SCRAM. Unfortunately, there is not much point when the authentication code is controlled by the server (eg. JS served by a server)
But sure, there are good solutions to this, like SCRAM. Unfortunately, there is not much point when the authentication code is controlled by the server (eg. JS served by a server)
Sorry, I always forget the order: http://thesibyl.net
So worst case scenario it does nothing, and best case scenario it protects against a common security risk? Seems to me it should be standard...
Worth noting that the security team at Google that you are likely referring to (P0 I assume?) has nothing to do with G Suite. Google has many security teams, and I’ve seen no indication that the security team whose scope includes G Suite is particularly exceptional, if they even have one. Some teams at Google don’t work with any security team at all.
I am not referring specifically to P0. The work P0 does is barely tangentially related to GSuite.
For one of the world's best security teams, "rumor" has it that their password hashing scheme is very far from best practices...
What do you mean?
That data point is meaningless by itself, everyone makes mistakes. Most of those people have nothing like the scale for making mistakes that Google does
My domain of 800+ users had one unhashed user. It was an account I created for testing something then suspended over two years ago. It never completed the first logon. I have many newer users that have not completed a first logon, yet their passwords are (apparently) hashed.
They said users created through admin console are affected. The might explain my nearly zero count. I've used GAM to automate creation for a very long time. We are a high school school so we churn about 1/4 of our user count every year.
They said users created through admin console are affected. The might explain my nearly zero count. I've used GAM to automate creation for a very long time. We are a high school school so we churn about 1/4 of our user count every year.
I had the same thing happen, only one email. An account for my sister, actually. Turns out she has never logged-in either.
Same here: One user!
the email I got said it was primarily accounts with one-time use and generated passwords were affected. I'm really confused as to why this isn't in the blog.
> due to legacy functionality that enabled customer Domain Admins to view passwords, some of your users’ passwords were stored in our encrypted systems in an unhashed format. This primarily impacted system generated or admin generated passwords intended for one-time use.
Sure enough, the accounts listed in the email as affected were never signed in and still needed a password reset.
The email: https://i.judge.sh/Twilight/iRcF0e7D.png
> due to legacy functionality that enabled customer Domain Admins to view passwords, some of your users’ passwords were stored in our encrypted systems in an unhashed format. This primarily impacted system generated or admin generated passwords intended for one-time use.
Sure enough, the accounts listed in the email as affected were never signed in and still needed a password reset.
The email: https://i.judge.sh/Twilight/iRcF0e7D.png
Same, the address in the notification email had never been signed in.
They sure liked repeating "secure encrypted infrastructure".
I was a little confused about the timeline of this as they mentioned implementing some functionality back in 2005. If I understand this right the specific issue of storing some (unsure as to what percentage or quantity) users passwords unhashed:
I was a little confused about the timeline of this as they mentioned implementing some functionality back in 2005. If I understand this right the specific issue of storing some (unsure as to what percentage or quantity) users passwords unhashed:
- started in January of this year
- is now fixed
- were stored for up to 14 days in this state
- would only effect new years signing up (based on it being part of the signup flow)
My interpretation was that unhashed passwords were stored for all users whose passwords were manually set by an administrator, from 2005 until some time very recently.
Looks like two issues were discovered.
1. All passwords set by admins for other accounts since 2005 were impacted. 2. Also all passwords starting Jan 2019 were also stored un-hashed for 14 days.
1. All passwords set by admins for other accounts since 2005 were impacted. 2. Also all passwords starting Jan 2019 were also stored un-hashed for 14 days.
I have no inside info on this incident and haven't worked for Google in over 4 years, but the description of the 14-day 2019-only issue sounds like it would have affected only a subset of users whose data they included in a sample for troubleshooting purposes. The sample rate they use is probably far below 100%.
(Once again: this is speculation based on general firsthand knowledge about how Google operates, but I know nothing specific to this incident.)
(Once again: this is speculation based on general firsthand knowledge about how Google operates, but I know nothing specific to this incident.)
Yes, of course. Not all but a different subset of passwords.
> They sure liked repeating "secure encrypted infrastructure".
Their audience is people who know those words and know that they want them, but might not know what they mean. Using technically accurate details when addressing people who don't understand the vocabulary is generally a recipe for misery.
Their audience is people who know those words and know that they want them, but might not know what they mean. Using technically accurate details when addressing people who don't understand the vocabulary is generally a recipe for misery.
Just had a thread yesterday where people raged at the very idea companies and organizations still use poor hashing algos.
Here we have google storing in plaintext by accident.
Here we have google storing in plaintext by accident.
It can simultaneously be true that nobody should do this and that even the best people can make mistakes.
I am reminded of the discourse on hypocrisy in The Diamond Age. That we fail does not mean we are insincere in our aspirations to do good.
I doubt Google would store passwords unhashed in a database - after all you check against the hashed record. This sounds like good old "accidentally logged sensitive data" issue.
I've always been intrigued as to why passwords are not hashed and salted client side before sending (and then hashed again on the server side). The usual argument against it is that SSL is better - but I don't see why it shouldn't be standard to do both. It would prevent issues like this (Malicious admins or compromised server) from ever occurring.
Assuming the salt is unique to the website it also solves the problem of users using the same password on many different websites - this would no longer be an issue.
It would also mean as an end user we can have a look at our login POST request and confirm our password is not sent and therefore cannot be stored in plaintext.
EDIT: I see the other discussion and understand the hash now becomes the password so if a hacker obtains the hash they now can login. I'm advocating for it to be hashed server side also, protecting the user from cases just like this one.
Assuming the salt is unique to the website it also solves the problem of users using the same password on many different websites - this would no longer be an issue.
It would also mean as an end user we can have a look at our login POST request and confirm our password is not sent and therefore cannot be stored in plaintext.
EDIT: I see the other discussion and understand the hash now becomes the password so if a hacker obtains the hash they now can login. I'm advocating for it to be hashed server side also, protecting the user from cases just like this one.
This is essentially the same as using a password manager since your browser (and all your devices) would have to know about your username/salt pairs.
Yeah very good comparison. And password managers are by all means the best way to go to ensure privacy. So I don't see why people argue against client side hashing.
I think they only say it's not perfect.
Strange... Didn't realize G Suite and Cloud Platform were well coupled, but it appears they must be.
I recently opened two of what I though were new 'consumer' Google accounts to be dedicated to managing Google Cloud Platform resources. On both accounts I'v also setup Identity & Access Management with an organization and additional sub-user-accounts.
Both of those parent account received this email warning about five hours ago...
Subject: [ACTION REQUIRED] Important Information about your G Suite Account
Google Customer Alert Dear G Suite Administrator, We are writing to inform you that between January 13, 2019 and May 9, 2019, an internal system that logged account signup information for diagnostic purposes, inadvertently stored one of your user account passwords in our encrypted systems in an unhashed format. This impacted the user account password provided during the initial account signup process. The log information was retained for 14 days following the signup process, and then was deleted according to our normal retention policies. We have reviewed the login information for the account and have found no evidence that the unhashed password was misused. The following is the user account impacted in your domain(s): [@] Google Planned Action: for your security, starting tomorrow Wednesday, May 22, 2019 PT we will force a password change unless it has already been changed prior to that time. Our password update methodology is as follows: • We will terminate the impacted user’s session and prompt the user to change their password at their next login. • In addition, starting Wednesday, May 29, 2019 PT we will reset the password for the user if they have not yet selected a new password or have not had a password reset. This user will need to follow your organization’s password recovery process. However, Super Admins will not be impacted. For information on password recovery options please refer to the following [Help Center Article]. For further questions please contact [Google Support] and reference issue number ###. Sincerely, The G Suite Team
I recently opened two of what I though were new 'consumer' Google accounts to be dedicated to managing Google Cloud Platform resources. On both accounts I'v also setup Identity & Access Management with an organization and additional sub-user-accounts.
Both of those parent account received this email warning about five hours ago...
Subject: [ACTION REQUIRED] Important Information about your G Suite Account
Google Customer Alert Dear G Suite Administrator, We are writing to inform you that between January 13, 2019 and May 9, 2019, an internal system that logged account signup information for diagnostic purposes, inadvertently stored one of your user account passwords in our encrypted systems in an unhashed format. This impacted the user account password provided during the initial account signup process. The log information was retained for 14 days following the signup process, and then was deleted according to our normal retention policies. We have reviewed the login information for the account and have found no evidence that the unhashed password was misused. The following is the user account impacted in your domain(s): [@] Google Planned Action: for your security, starting tomorrow Wednesday, May 22, 2019 PT we will force a password change unless it has already been changed prior to that time. Our password update methodology is as follows: • We will terminate the impacted user’s session and prompt the user to change their password at their next login. • In addition, starting Wednesday, May 29, 2019 PT we will reset the password for the user if they have not yet selected a new password or have not had a password reset. This user will need to follow your organization’s password recovery process. However, Super Admins will not be impacted. For information on password recovery options please refer to the following [Help Center Article]. For further questions please contact [Google Support] and reference issue number ###. Sincerely, The G Suite Team
In addition, as we were troubleshooting new G Suite customer sign-up flows, we discovered that starting in January 2019 we had inadvertently stored a subset of unhashed passwords in our secure encrypted infrastructure. These passwords were stored for a maximum of 14 days.
Is this another way of saying they stored cleartext passwords? Interesting they choose to use "uncashed" instead of "cleartext"
Is this another way of saying they stored cleartext passwords? Interesting they choose to use "uncashed" instead of "cleartext"
They are probably not saying "cleartext" because the data was still encrypted, just not hashed.
Thank you Google (AKA fk you Facebook).
would this violate GDPR?
Wow, TIL that strong passwords are actually a part of GDPR (I was aware of the policies around deleting data, being clear about the data that you're storing, but not about passwords).
That's actually great that industry standards are being codified with actual deterrents for failing to strongly store secure passwords.
> If passwords are being used, they should not be accessible and should be securely stored to avoid them being visible to unauthorized individuals. Some sort of standard such as encryption or an equivalent should be used to protect the passwords.
quote from some random blog [0]
[0] https://www.compliancejunction.com/gdpr-password-requirement...
That's actually great that industry standards are being codified with actual deterrents for failing to strongly store secure passwords.
> If passwords are being used, they should not be accessible and should be securely stored to avoid them being visible to unauthorized individuals. Some sort of standard such as encryption or an equivalent should be used to protect the passwords.
quote from some random blog [0]
[0] https://www.compliancejunction.com/gdpr-password-requirement...
> That's actually great that industry standards are being codified with actual deterrents for failing to strongly store secure passwords.
What would be better is for us folks to do this to ourselves first. If self regulation works well, the government will never need to step in. The GDPR seems to be pretty good, but it's not unimaginable that some assholes in office who are looking to get reelected respond to a high-profile data breach by making a shitty set of regulations to govern all software developers in the country for all time.
W.R.T. enforcing compliance, having a licensing board that oversees this would provide incentive to developers to not cut corners, and would also provide a way to resist management that is intent on releasing a shitty product quickly in the hope of making a quick buck before it all crashes down / padding their resume.
What would be better is for us folks to do this to ourselves first. If self regulation works well, the government will never need to step in. The GDPR seems to be pretty good, but it's not unimaginable that some assholes in office who are looking to get reelected respond to a high-profile data breach by making a shitty set of regulations to govern all software developers in the country for all time.
W.R.T. enforcing compliance, having a licensing board that oversees this would provide incentive to developers to not cut corners, and would also provide a way to resist management that is intent on releasing a shitty product quickly in the hope of making a quick buck before it all crashes down / padding their resume.
So, in terms of self-enforcement, I have pretty much every poorly built PHP webapp from 1996-2014 as evidence that this won't happen. People are bad at security, including big people with lots to lose (see... I don't know, any high profile breech ever) so regulating this is quite necessary. Sometimes it'll even empower developers to push back on deadlines with the comment that cutting security corners will violate laws.
I don't think this is a practical solution at all. What would a licensing board certification entail? Thousands of certifications already exist, and we don't take any of them seriously. Why would the licensing board have any power? We'd need software engineers to form a union or have government regulate that companies hire certified engineers. Either way, there needs to be some enforcement mechanism.
Shifting the onus on security away from the companies and onto the developers also seems like a bad idea. With GDPR, there is a financial incentive for companies to use good practices. With a licensing board, companies will care far less if Joe Q Developer might lose his license. Why would they care unless there is financial or legal incentive? I just don't see how this would work without getting back to government regulation. I'll definitely reconsider my view point if you have a solution to this hurdle.
Shifting the onus on security away from the companies and onto the developers also seems like a bad idea. With GDPR, there is a financial incentive for companies to use good practices. With a licensing board, companies will care far less if Joe Q Developer might lose his license. Why would they care unless there is financial or legal incentive? I just don't see how this would work without getting back to government regulation. I'll definitely reconsider my view point if you have a solution to this hurdle.
Yeah, I disagree. Most companies would just not sign up to the licensing board. Also, from a practical standpoint, imagine trying to license every site on the internet!
I've never built a website that doesn't securely store passwords. I've definitely _used_ websites that do though [0]. Up until now my only choice is to stop using any of these sites and just hope they magically fix their crap.
It's obvious which would be a better motivator for any of these sites, either receiving an email stating "what you're doing is wrong, and unsecure" or one stating "what you're doing is wrong and unsecure and, by the way, you could be fined €50,000" [1]
[0] https://plaintextoffenders.com/
[1] https://www.gamingtechlaw.com/2019/04/first-gdpr-fine-italy....
I've never built a website that doesn't securely store passwords. I've definitely _used_ websites that do though [0]. Up until now my only choice is to stop using any of these sites and just hope they magically fix their crap.
It's obvious which would be a better motivator for any of these sites, either receiving an email stating "what you're doing is wrong, and unsecure" or one stating "what you're doing is wrong and unsecure and, by the way, you could be fined €50,000" [1]
[0] https://plaintextoffenders.com/
[1] https://www.gamingtechlaw.com/2019/04/first-gdpr-fine-italy....
As long as data is private, I don't think so. Although, it is pointless to discuss that since keeping passwords in plaintext is, well, :)
Probably not, because the passwords were encrypted (with a google-owned key)
I'm not a GDPR expert, but I've done a fair bit of work making old systems GDPR compliant. With my understanding of GDPR, yes, this is a violation since part of GDPR requires you keep customer data secure using "appropriate technical and organizational measures". I would argue that this violates the "appropriate technical measure" part.
Again, I'm not a lawyer or expert in this domain, so I could be wrong.
Again, I'm not a lawyer or expert in this domain, so I could be wrong.
They seem to be hammering home the point that the passwords were always stored on ‘secure encrypted infrastructure’ but the data was not hashed, so anyone with access to googles ‘secure encrypted infrastructure’ could read the data.
The encryption is most likely enough to be within GDPR compliance.
The encryption is most likely enough to be within GDPR compliance.
>The encryption is most likely enough to be within GDPR compliance. //
Why do you think that, allowing staff to read plaintext passwords is contrary to standard security practice; companies are expected to make reasonable effort to secure PII and allowing staff to read your password doesn't appear to be "reasonable effort" by even the casualist of readings.
I don't think the EU courts are that stupid.
FWIW I don't think there is a case here particularly, as it appears to be a genuine error and being fixed.
Why do you think that, allowing staff to read plaintext passwords is contrary to standard security practice; companies are expected to make reasonable effort to secure PII and allowing staff to read your password doesn't appear to be "reasonable effort" by even the casualist of readings.
I don't think the EU courts are that stupid.
FWIW I don't think there is a case here particularly, as it appears to be a genuine error and being fixed.
I would be surprised if a bug constituted a violation. The intent was clearly to have them be hashed. Once the bug was detected it was quickly patched and appropriate incidence response taken. That wouldn't violate other security standards, so I'd be very surprised if it violated GDPR.
I believe a bug would still cause them to be in violation, but it likely means there would not be any punishment for being in violation. GDPR has a lot of leeway for companies that are making an effort to be in compliance, but have failed for one reason or another. In this case, I would assume the punishment for the bug would just be "fix the bug".
[deleted]
I mean hedgqw is my mother's maiden name and 23328 was my first pet, this isn't a very strong hash
It's a shame that password hashing
isn't standardized.
There should be a standard password box in html in which you can just call a function to pull out a salted hash of a password. The attributes of the box could include options for password restrictions such as a minimum of 8 chars, etc.
The fact that developers still need to write custom hashing algorithms in javascript is kind of ridiculous .
edit: nevermind, I just realized this wouldn't work
There should be a standard password box in html in which you can just call a function to pull out a salted hash of a password. The attributes of the box could include options for password restrictions such as a minimum of 8 chars, etc.
The fact that developers still need to write custom hashing algorithms in javascript is kind of ridiculous .
edit: nevermind, I just realized this wouldn't work
How would this work when you have to store the hash on a server? A hypothetical client-side hasher doing this means that an attacker would just...send the hash, acquired through nefarious means. Just as they'd send the password today. Server-side, that hash would be the password and that itself would need to be hashed before storage.
I don't know why you're talking about "custom hashing algorithms", in JavaScript or elsewhere; you pull a bcrypt/scrypt/Argon2 library and use it.
I don't know why you're talking about "custom hashing algorithms", in JavaScript or elsewhere; you pull a bcrypt/scrypt/Argon2 library and use it.
Yea you're right. But what about this, the client uses the password as a seed to create a public/private key pair, and sends the public key to the server. If it is during password creation, the server stores this public key in relation to the user, otherwise verifies it. Then the server sends a challenge* and the client uses the private key to sign it and return a response.
* actually the server would send the challenge ahead of time along with serving the web page
* actually the server would send the challenge ahead of time along with serving the web page
That sounds like PAKE. I'm not qualified to opine on that; it's above my pay grade. I can see a lot of implementation issues, though, if you wanted to make it a browser component outside the control of the developer.
Perhaps you'd be interested in reading about Secure Remote Password (SRP): https://en.wikipedia.org/wiki/Secure_Remote_Password_protoco...
Copying the "layman's terms" from that page for your convenience...
In layman's terms, during SRP (or any other PAKE protocol) authentication, one party (the "client" or "user") demonstrates to another party (the "server") that they know the password, without sending the password itself nor any other information from which the password can be derived. The password never leaves the client and is unknown to the server.
Copying the "layman's terms" from that page for your convenience...
In layman's terms, during SRP (or any other PAKE protocol) authentication, one party (the "client" or "user") demonstrates to another party (the "server") that they know the password, without sending the password itself nor any other information from which the password can be derived. The password never leaves the client and is unknown to the server.
Thanks, I thought there should be something like this.
There is no client-side security. If you hash on the client side then that hash becomes the password and you have plain text password storage.
One important reason we hash passwords is protect what is essentially a user secret (one which the user may use across many different services) from prying DBAs. In that respect client-side hashing (with salting) makes perfect sense.
The other major benefit of server-side hashing, to what you're correctly surmising, is that it prevents certain privilege escalations. A read-only database dump (from e.g. SQL injection in to a SELECT query) can otherwise be used to gain read-write access by simply copying dumped passwords from the database back in to the login form. The thing is, server-side hashing doesn't actually prevent this attack, it merely slows it down. It also doesn't help at all with weak passwords.
In any case, to address your concern you can also simply run the client-side hashed value (which could and should be produced by something like Argon2) through a single round of SHA2.
The other major benefit of server-side hashing, to what you're correctly surmising, is that it prevents certain privilege escalations. A read-only database dump (from e.g. SQL injection in to a SELECT query) can otherwise be used to gain read-write access by simply copying dumped passwords from the database back in to the login form. The thing is, server-side hashing doesn't actually prevent this attack, it merely slows it down. It also doesn't help at all with weak passwords.
In any case, to address your concern you can also simply run the client-side hashed value (which could and should be produced by something like Argon2) through a single round of SHA2.
There are ways to improve security via encrypting the password clientside with a nonce and avoiding passing raw text over the request to the server (which should be https not http) but I believe it isn't considered necessary above the inherent security in most circumstances.
You could just hash the hash that the client sends, no? Then you never know their password and it's still stored securely.
If the client always hashes the password in the same way (no nonce or usage of encryption) then the password essentially becomes the hashed string with some extra clientside code you're burdened with forever. And since that same hashing will need to be done whenever they log in the hashing algorithm needs to be in the client side code sent to everyone... meaning an attacker would have a bit of obscurity in their way and just need to regenerate a rainbow table based off of your hashing algorithm to test common/weak passwords.
It's not about the security, it's about preventing side-channel leaks of the plain password (e.g. incorrectly sanitized logs or buffer overflows). Having a data breach of your site-specific hash is much better than leaking the user's password that an attacker could use on any website.
Your site specific hash, if leaked into logs and generated using a fixed hashing algorithm, would just be a step of obscurity, a rainbow table generated according to that hashing algorithm would be able to extract the original typed password pretty reliably for use in compromising other accounts held by the individual on other sites.
There are ways to actually add security above SSL but they require a per-session nonce. The most secure versions of these operate on some sort of out-of-band time-scrambling algorithm that's issued to users on an individual basis (usually a dongle) which is basically a nonce that's been pre-arranged a few years in advance, these can (if done correctly) provide all the security of on-demand generated nonces and physical keys combined.
Fixed hashing algorithms add such a negligible amount of security that I'd prefer to avoid them due to the tech debt their existence incurs outweighing their advantage.
There are ways to actually add security above SSL but they require a per-session nonce. The most secure versions of these operate on some sort of out-of-band time-scrambling algorithm that's issued to users on an individual basis (usually a dongle) which is basically a nonce that's been pre-arranged a few years in advance, these can (if done correctly) provide all the security of on-demand generated nonces and physical keys combined.
Fixed hashing algorithms add such a negligible amount of security that I'd prefer to avoid them due to the tech debt their existence incurs outweighing their advantage.
Their password would have to already be in a rainbow table, and at that point nothing can help them. But I agree it's kind of pointless, I was just arguing that there is a benefit even if negligible. If you want real security, WebAuthn is the future.
I can’t think of any situation where password hashing should be happening on the client - that would eliminate a large part of the benefit.
The company not having any access to the plaintext password would already be a huge benefit. (many people are giving random companies access credentials to their e-mail, as it stands) Although that would be hard to enforce.
They would only be able to screw users by allowing access to their own service if they were forced to provide a salt and only be able to get the hash.
But at html level, everything is possible and this would be hardly possible to enforce, aside from informing the user that they are typing a password into unsecure password field (hard to do, not to be fakeable, too).
It would still be hackable, by bad actors by using a salt of a service they want to get pw for, but that would at least be detectable client side.
They would only be able to screw users by allowing access to their own service if they were forced to provide a salt and only be able to get the hash.
<input type=password salt=687161>
But at html level, everything is possible and this would be hardly possible to enforce, aside from informing the user that they are typing a password into unsecure password field (hard to do, not to be fakeable, too).
It would still be hackable, by bad actors by using a salt of a service they want to get pw for, but that would at least be detectable client side.
If the salt is static, the hash becomes the password. Nobody does this.
If the hash is dynamic, the password has to be stored in plaintext. Kerberos does something like this.
Neither of these are better than the current state of the art. Read up on SRP for a public-key based system that does something like what you want.
Third party data access should go through OAuth, not by sharing passwords to services.
If the hash is dynamic, the password has to be stored in plaintext. Kerberos does something like this.
Neither of these are better than the current state of the art. Read up on SRP for a public-key based system that does something like what you want.
Third party data access should go through OAuth, not by sharing passwords to services.
People share passwords, and use e-mail address to create accounts on all kinds of websites.
Anything that would prevent the website's server from getting the plaintext password would be good. Salted hash is obviously not a plaintext password and can't be stuffed around to other services.
Anything that would prevent the website's server from getting the plaintext password would be good. Salted hash is obviously not a plaintext password and can't be stuffed around to other services.
Thankfully people have realized this in settings like PHP leading them to drop the old suggested password hashing method... which at this point I can't recall, but it involved setting up salts manually and making sure you had a strong IV... which no common web devs would do.
Now you've just got to call these two:
https://www.php.net/manual/en/function.password-hash.php
https://www.php.net/manual/en/function.password-verify.php
and if you really want to tweak things you can, but the simple route is moderately secure
Now you've just got to call these two:
https://www.php.net/manual/en/function.password-hash.php
https://www.php.net/manual/en/function.password-verify.php
and if you really want to tweak things you can, but the simple route is moderately secure
Its my understanding that the hashing should not be done client side, as that would expose the secret salt that is being used and reduce security?
Password salts aren't secret. They are stored alongside a password hash because they're necessary to reconstruct the hash from the plaintext. If you look at a bcrypt string, for example, the salt's encoded in it.
The problem is instead that now that the client-side hash is actually the password.
(edit: some people use a "pepper" at the application level and apply it to all passwords, which might be kind of what you're thinking of? Buy you don't need to do this with modern key derivation algorithms. You can if you want, it just doesn't really matter much.)
The problem is instead that now that the client-side hash is actually the password.
(edit: some people use a "pepper" at the application level and apply it to all passwords, which might be kind of what you're thinking of? Buy you don't need to do this with modern key derivation algorithms. You can if you want, it just doesn't really matter much.)
That's not why (the salt does need to be secret secret), but it is true that the hashing should not be done client side. It's more because it doesn't actually accomplish a whole lot. The hash basically just becomes the password, so you'd need to hash it again on the server to get the same level of security.
Mere hashing indeed wouldn't be useful, but there's SCRAM, and it's standardized as one of HTTP authentication mechanisms (see RFC 7804).
Edit: though perhaps it's worth noting that it won't force a server to store a password securely, but would facilitate it.
Edit: though perhaps it's worth noting that it won't force a server to store a password securely, but would facilitate it.
This issue only affect those account resetted by admin.
Time to introduce a new Postgre data type. (:troll:)
-----
> If you have a Google account, Google’s core sign-in system is designed not to know your password. How, then, can we verify your password when you sign in to your Google account again? The answer lies in a bit of cryptography: when you set your password, instead of remembering the exact characters of the password, we scramble it with a “hash function”, so it becomes something like “72i32hedgqw23328”, and that’s what we store with your username. Both are then also encrypted before being saved to disk. The next time you try to sign in, we again scramble your password the same way. If it matches the stored string then you must have typed the correct password, so your sign-in can proceed.
> The effectiveness of the hash function lies in its one-way nature: it is simple to scramble your password, but nearly impossible to unscramble it. So, if someone should obtain the scrambled password, they won’t be able to recover your real password. The downside of password hashing is that if you forget your password, we cannot show you what it was; there’s nothing we can do other than reset it to a temporary password (valid one time only) and then require you to pick a new one.