Zomato security breach: 17M email addresses and hashed passwords leaked(blog.zomato.com)
blog.zomato.com
Zomato security breach: 17M email addresses and hashed passwords leaked
http://blog.zomato.com/post/160791675411/security-notice
73 comments
If they are simply hashed, rainbow tables can be used to crack them. Password must be salted preferably using bcrypt or something equivalent.
A rainbow table is not necessary to crack passwords, and is not able to crack all passwords (even if not salted).
And even if salted, a regular (non-rainbow table) attack will crack some passwords.
And even if salted, a regular (non-rainbow table) attack will crack some passwords.
They state thatthey weresaltedin the blog post, however, they do not specify through what method.
Couple of things are pretty apparent from the post:
1. Why would compromised developer account cause such a hack? Unless they don't have proper data control in place for production access.
2. The line about how "payment" related information is under PCI DSS compliant shows how these guys are running their business. I frequently see people, who are confronted about not following some form of compliance or hack, argue back on "segregation" of compliance. They say - X data needs (or was) to be compliant while Y doesn't. Sure not everything needed to be PCI DSS compliant but why not have better practices?
3. What about PII (Personally identifiable information) data they collect? Isn't that supposed to be rigorously protected? It seems they believe only payment and passwords are of use to hackers.
2. The line about how "payment" related information is under PCI DSS compliant shows how these guys are running their business. I frequently see people, who are confronted about not following some form of compliance or hack, argue back on "segregation" of compliance. They say - X data needs (or was) to be compliant while Y doesn't. Sure not everything needed to be PCI DSS compliant but why not have better practices?
3. What about PII (Personally identifiable information) data they collect? Isn't that supposed to be rigorously protected? It seems they believe only payment and passwords are of use to hackers.
sigh I really wish these kind of notifications became a lot clearer about what actually happened and what the user is supposed to do.
Their message is: (Ignoring the update on top):
- Para. 1: Marketing bullshit
- Para. 2: Short blurb about what happened
- Para. 3: Incomprehensible techno-babble for the average user of their service, ending with an important action point (however, I suspect most users' eyes would have glazed over by now)
- Para. 4: A random note, again with important info in the tail.
- Para. 5: Something about what they have done that will cause user impact, mixed with some more about their internal remediation, and the suspected cause.
It goes on, but I'm already bored.
Instead they should clarify their format:
- There has been an incident that resulted in an unknown 3rd party getting hold of your email address for this site, and a scrambled version of your password. This may impact you in the following ways:
-- There is a strong possibility that the thieves will be able to unscramble your password.
--- If you use the same password here as you do with other websites, you should change your password on all those websites immediately See the end of this notice for more assistance and information.
-- We will automatically change your password for our site, and this means you will have to [insert process]
- We are taking the following steps to ensure this isn't going to happen again:
-- Step 1
-- Step 2
- We have reported this incident to the relevant authorities.
- For the next x months we will contract with service y to track if any of the stolen data will surface, and will inform you if this happens.
- We recommend that you use a password manager. please see [link] for a handy guide on how to choose and get started with a password manager.
- You can always reach us via [channels] to discuss this issue further. If any part of this message isn't clear, please click [here] to chat with one of our support staff.
- Apology.
- More background details, etc.
Their message is: (Ignoring the update on top):
- Para. 1: Marketing bullshit
- Para. 2: Short blurb about what happened
- Para. 3: Incomprehensible techno-babble for the average user of their service, ending with an important action point (however, I suspect most users' eyes would have glazed over by now)
- Para. 4: A random note, again with important info in the tail.
- Para. 5: Something about what they have done that will cause user impact, mixed with some more about their internal remediation, and the suspected cause.
It goes on, but I'm already bored.
Instead they should clarify their format:
- There has been an incident that resulted in an unknown 3rd party getting hold of your email address for this site, and a scrambled version of your password. This may impact you in the following ways:
-- There is a strong possibility that the thieves will be able to unscramble your password.
--- If you use the same password here as you do with other websites, you should change your password on all those websites immediately See the end of this notice for more assistance and information.
-- We will automatically change your password for our site, and this means you will have to [insert process]
- We are taking the following steps to ensure this isn't going to happen again:
-- Step 1
-- Step 2
- We have reported this incident to the relevant authorities.
- For the next x months we will contract with service y to track if any of the stolen data will surface, and will inform you if this happens.
- We recommend that you use a password manager. please see [link] for a handy guide on how to choose and get started with a password manager.
- You can always reach us via [channels] to discuss this issue further. If any part of this message isn't clear, please click [here] to chat with one of our support staff.
- Apology.
- More background details, etc.
They used a 2 alpha character salt with md5 as the hash algorithm. Thats pretty bad
Whenever a company says "one way hashing algorithm" without specifying the algorithm, we should all assume it's MD5.
It was md5 with a 2 alpha character salt
One of the key issues with this type of leak is where people use the same password in multiple sites (check out the data in https://haveibeenpwned.com as an example). They should recommend resetting the passwords they used at Zomato in any other apps/sites.
"A layer of authorisation will be added for internal teams having access to this data to avoid the possibility of any human breach." -- First of all it is not hacked. Data is just stolen by someone internally. Above statement proves that. And they are not a startup.
Lack of which security measures might lead to this?
In the post they mention the probable cause is a developer account which got compromised. One of the most important rules of most security frameworks (including PCI DSS) is segregating access between development and production: developers are not supposed to be able to access production data. I suppose this rule was not followed here - or an alternative mitigating control (which can be applied instead of the rule above) was not good enough to prevent this.
Zomato is almost 10 years old, hardly a "startup".
It won't be appropriate to paint every Indian company with the same brush, but I knew one that was in a professional spamming business who slyly marketed themselves as an "email marketing company". The business model was to —
1) Lure employees of big companies (especially e-commerce ones) with cash and buy email dumps of their users. I don't think email security was something many companies were vigilant about. 2) Send email campaigns for their clients, which included other e-commerce companies, colleges, and travel agencies.
The business, as I remember, was quite profitable. I can't say if something like this is at work here but there's a healthy black market of buying email lists.
1) Lure employees of big companies (especially e-commerce ones) with cash and buy email dumps of their users. I don't think email security was something many companies were vigilant about. 2) Send email campaigns for their clients, which included other e-commerce companies, colleges, and travel agencies.
The business, as I remember, was quite profitable. I can't say if something like this is at work here but there's a healthy black market of buying email lists.
Someone who owned and operated a globally blacklisted spam company came to power in Delhi, ironically on an "anti-corruption" message. Ethics aren't part of contemporary social contracts in India.
See "Somnath (Spamnath ?) Bharati".
See "Somnath (Spamnath ?) Bharati".
Come on, please don't single him out. Who will pay attention to the drama queen?
> I can't say if something like this is at work here but there's a healthy black market of buying email lists.
There's a healthy 100% legitimate (legally) market too. They correlate data from all sorts of retailers with things like your bank account, email address, etc.
Pay cash for anything you'd be embarrassed about someone finding out.
There's a healthy 100% legitimate (legally) market too. They correlate data from all sorts of retailers with things like your bank account, email address, etc.
Pay cash for anything you'd be embarrassed about someone finding out.
I can relate to that. I know multiple firms in NCR who operate on a model like that. Mass email campaigns, promotions and spam - undertaken from offshore servers hosted on shady ISPs based on data like this. I can bet this data has been made available to them and it has been already put in use.
They should mention if the passwords were salted or not. Just saying the passwords were hashed isnt sufficient.
They do mention it:
"We hash passwords with a one-way hashing algorithm, with multiple hashing iterations and individual salt per password."
"We hash passwords with a one-way hashing algorithm, with multiple hashing iterations and individual salt per password."
So is my password safe?
Probably? The hacker is working with them and claims to have deleted all of the info already. read their updated article, its an interesting development
Maybe, if they are being truthful. Hopefully you don't share it with another account on another site.
> Hopefully you don't share it with another account on another site
Bluntly, for a user of a site like this in 2017 there is simply no excuse for sharing passwords. In ops we used to say "If it's not backed up, it doesn't exist" - and I think a modern corollary could be "if your password isn't unique per site, then no, it's not safe". If you reuse passwords, your security is only as good as the weakest site you used it on, and you are playing with fire.
Just go and buy 1password (or similar) and use it.
Bluntly, for a user of a site like this in 2017 there is simply no excuse for sharing passwords. In ops we used to say "If it's not backed up, it doesn't exist" - and I think a modern corollary could be "if your password isn't unique per site, then no, it's not safe". If you reuse passwords, your security is only as good as the weakest site you used it on, and you are playing with fire.
Just go and buy 1password (or similar) and use it.
> "with multiple hashing iterations and individual salt per password."
If your username is...
If your username is...
[deleted]
The post was updated with this information. The initially posted version did not include information about iterations and salts.
> we encourage you to change your password for any other services where you are using the same password
Based on that statement it sure doesn't sound like the PWs were salted (or at least properly salted).
Based on that statement it sure doesn't sound like the PWs were salted (or at least properly salted).
So can someone explain to me why the downvotes? If the salts are unique to the user wouldn't it be nearly impossible for the hacker to use the hashed pw on other sites? (I'm not a web dev but to my understanding that's how it works)
Even if salts are unique to a single user, there's still a possibility where someone determined can get the password.
That's because the user specific salts were probably in the same db. This will make it difficult since a brute force attack must be done for every single password. But it is still not impossible.
That's because the user specific salts were probably in the same db. This will make it difficult since a brute force attack must be done for every single password. But it is still not impossible.
> If the salts are unique to the user wouldn't it be nearly impossible for the hacker to use the hashed pw on other sites
Even if the salts aren't unique, can you explain how can they use the hashed password anywhere?
If I give you a hashed password and the salt used, what exactly can you do with it? (apart from brute forcing)
Even if the salts aren't unique, can you explain how can they use the hashed password anywhere?
If I give you a hashed password and the salt used, what exactly can you do with it? (apart from brute forcing)
For instance, you can make a list of the top 100,000 passwords, plus another 1000-per-user variations (based on username or email or whatever). Now run them against all users. You'll get some successes, and now those users are at risk.
A password DB breach, salted or not, allows an offline, non-rate-limited, attack on those passwords.
A password DB breach, salted or not, allows an offline, non-rate-limited, attack on those passwords.
[deleted]
That'll teach them to outsource development to Africans making a third the wage!
If you provided more information and facts (if it's really true) on this matter besides emotional accusation, you'd get upvotes instead of downvotes.
Irrelevant, since he's correct. I've actually known an ex-Zomato employee who's said the same thing, this is the first time I've heard strangers mention it though.
Upvotes do not determine if information is true or not, and there's plenty of people out there who aren't the least bit concerned with whether or not HN upvotes them.
Tangentially, I don't think outsourcing is really that big of a deal. LOTS of companies do it, and it's easy to get bitten by it, but it's also easy to check over what they bring back, as well. I'm not trying to make a case for excusing this incident, as they could have very easily dedicated some dev time to quality check the code (if it's the code that they outsourced in the first place, which we don't know for sure or not) so there's really no excuse for it. People in the states outsource to india, people in india outsource to africa or south america, people in africa probably have some place they outsource to, maybe china or something. It's an economical oddity, not a moral dilemma, imo.
Upvotes do not determine if information is true or not, and there's plenty of people out there who aren't the least bit concerned with whether or not HN upvotes them.
Tangentially, I don't think outsourcing is really that big of a deal. LOTS of companies do it, and it's easy to get bitten by it, but it's also easy to check over what they bring back, as well. I'm not trying to make a case for excusing this incident, as they could have very easily dedicated some dev time to quality check the code (if it's the code that they outsourced in the first place, which we don't know for sure or not) so there's really no excuse for it. People in the states outsource to india, people in india outsource to africa or south america, people in africa probably have some place they outsource to, maybe china or something. It's an economical oddity, not a moral dilemma, imo.
people in africa probably have some place they outsource to, maybe china or something
Companies in South Africa outsource to India quite often. Africa is huge, and hard to generalise about.
Companies in South Africa outsource to India quite often. Africa is huge, and hard to generalise about.
> Irrelevant, since he's correct.
Purely coincidental, it was a joke like we normally hear about outsourcing to India.
Purely coincidental, it was a joke like we normally hear about outsourcing to India.
[deleted]
Such notifications are becoming routine.
Seems to me that you need to design your systems around the question of "how can the problem be minimised when our systems get hacked".
Seems to me that you need to design your systems around the question of "how can the problem be minimised when our systems get hacked".
Many people came to the same conclusion during the WannaCry malware attack. I'd suggest taking a look at those threads fpr related discussion.
"We really need to invest in technical security" - Everyone after WannaCry
"We really need to invest in technical security" - Everyone after Heartbleed
If the end-user doesn't notice it, it's barely budgeted for by the business bods.
We're condemned to repeat history, much to my own chagrin.
"We really need to invest in technical security" - Everyone after Heartbleed
If the end-user doesn't notice it, it's barely budgeted for by the business bods.
We're condemned to repeat history, much to my own chagrin.
I did not know that Zomato was Indian.. nor am I quite sure why it matters that they are? Breaches are common everywhere and at least these guys (apparently) salt, etc.
Puts the 17m into a more appropriate scale, imo.
> at least these guys (apparently) salt, etc
It seems the salt was stored in the database itself.
https://twitter.com/sajal_thomas/status/865134165288599553
It seems the salt was stored in the database itself.
https://twitter.com/sajal_thomas/status/865134165288599553
I don't see anything in that Twitter thread indicating where the salt was stored, but salts are actually supposed to be stored next to the password hash. They purely try to prevent precomputation of password hashes by introducing a random value per password, and that works just fine by putting them next to the hash.
Today, of course you shouldn't be manually maintaining salts and instead use a proper PBKDF (which indeed uses a salt, which is stored alongside the hash).
Today, of course you shouldn't be manually maintaining salts and instead use a proper PBKDF (which indeed uses a salt, which is stored alongside the hash).
Yes, a little bit of salt on your Zomato makes it much more palatable.
We need password managers and use unique 20 digit password (or max allowed) for each login. If it's online, we need to assume that it will be hacked in time.
Frankly this is to be expected when "developers" working on these projects tend to be script kiddies more concerned with UX/design rather than solid architecture engineering. When backends are thrown together as an afterthought in node, piled in bunch of 3rd party services and things just hacked together ... well this is exactly results one would expect.
Is there a correlation between the size and experience of a company and it's engineering team or technology choices and the likelihood of it being hacked? Tiny startups get hacked, and billion dollar corporations get hacked, and businesses of every size and shape in between get hacked. 'solid architecture engineering' might be a defence (although even the best engineered backend has very human weakpoints), but whether or not a business has a pretty UI and a node based backend is hardly a signal of how good the engineering might be.
I agree that technology choices are not the best signal to use.
They're one of a variety of primitive proxies for engineering culture, depth of talent and "process maturity". IMHO there is a weak correlation with team size, but a strong small team with good culture can of course outperform a large team with worse organisational culture.
As you scale a team you get more capability, but there are more bases to cover and more value to protect. Some points along the appsec spectrum of practices and culture:
At very small team sizes, reviews of pull requests with some minimum number approvers, plus a culture of responding to comments and enough expertise on the team that someone recognises basic issues like "we shouldn't store passwords like that" or "maybe we shouldnt chuck a url parameter straight into window.location here".
Small-medium, some level of technical leadership and design going into features with security as at least a consideration. Maybe a budget for limited code reviews / pentesting now and then or someone on the team with a constructive focus on security. Some automation with mostly free tools. Issue tracking for security defects and sane priorities.
Further along / large: employees who conduct security reviews (e.g, devs cc on prs and some spelunking), pentest and work with developers during the design phase. Plug tools like static analysers, scanners / runtime analyses into ci & test envs. Conduct threat modelling. Run regular training for developers. Have devs who work on code safety (like wrapping unsafe libs or fixing bugs in key dependencies). Use an artifact repository and stay on top of bugs in dependencies. Maintain internal checklists and a security knowledge base. Sane metrics and reporting (this is surprisingly hard). Conducting 360's where you track down the root cause of bugs and see where you can stop it happening again. Processes for dealing with bug reports (security@). Bug bounties. And more.
Although all sizes of company experience a variety of incidents, where you see problems is when engineering maturity isn't commensurate with the value of the data / transactions / assets / brand.
They're one of a variety of primitive proxies for engineering culture, depth of talent and "process maturity". IMHO there is a weak correlation with team size, but a strong small team with good culture can of course outperform a large team with worse organisational culture.
As you scale a team you get more capability, but there are more bases to cover and more value to protect. Some points along the appsec spectrum of practices and culture:
At very small team sizes, reviews of pull requests with some minimum number approvers, plus a culture of responding to comments and enough expertise on the team that someone recognises basic issues like "we shouldn't store passwords like that" or "maybe we shouldnt chuck a url parameter straight into window.location here".
Small-medium, some level of technical leadership and design going into features with security as at least a consideration. Maybe a budget for limited code reviews / pentesting now and then or someone on the team with a constructive focus on security. Some automation with mostly free tools. Issue tracking for security defects and sane priorities.
Further along / large: employees who conduct security reviews (e.g, devs cc on prs and some spelunking), pentest and work with developers during the design phase. Plug tools like static analysers, scanners / runtime analyses into ci & test envs. Conduct threat modelling. Run regular training for developers. Have devs who work on code safety (like wrapping unsafe libs or fixing bugs in key dependencies). Use an artifact repository and stay on top of bugs in dependencies. Maintain internal checklists and a security knowledge base. Sane metrics and reporting (this is surprisingly hard). Conducting 360's where you track down the root cause of bugs and see where you can stop it happening again. Processes for dealing with bug reports (security@). Bug bounties. And more.
Although all sizes of company experience a variety of incidents, where you see problems is when engineering maturity isn't commensurate with the value of the data / transactions / assets / brand.
I had this discussion with my boss the other day, his position was that as engineers we should be able to write our own security, we're not "just" web developers. My position was that as an engineer I might be able to, but I should know that if I do, it's a huge liability - and there are plenty of examples of the best of companies that did it wrong.
I'm not convinced that you read the article properly. You're making projections about their infrastructure without having all of the facts.
You should read the reddit AMA done by the Zomato CEO, it'll give nice insight into how the company works :)
Heh, thanks for that. It's two years old, but certainly worth reading: https://www.reddit.com/r/india/comments/2yo614/hi_im_deepind...
> Do you really expect a good developer to be able to be in the zone for 12 hours straight?
> Why would we contribute to open source if we are barely able to ship what we need to ship to keep our users happy.
> Do you really expect a good developer to be able to be in the zone for 12 hours straight?
> Why would we contribute to open source if we are barely able to ship what we need to ship to keep our users happy.
Exactly the line I was wanting to say, but did not want to sound too negative
I'm curious about how the hack occurred.
With the recent Hipchat hack, it is unclear if the login server was compromised or the database dump was obtained some other way.
If the Hipchat webserver that handles sign-in attempts was the one compromised, which is what I've been lead to believe, then it doesn't matter how the passwords were hashed. The plain-texts would be exposed in the majority of use cases. Sure, the DB data could also get stolen triggering reporting requirements in organizations with controls, but that may be less of a concern at that point.
With the recent Hipchat hack, it is unclear if the login server was compromised or the database dump was obtained some other way.
If the Hipchat webserver that handles sign-in attempts was the one compromised, which is what I've been lead to believe, then it doesn't matter how the passwords were hashed. The plain-texts would be exposed in the majority of use cases. Sure, the DB data could also get stolen triggering reporting requirements in organizations with controls, but that may be less of a concern at that point.
Development environments with access to live data. Tasty!
Are the hashed passwords salted? If not, then the claim that the "password cannot be converted/decrypted back to plain text" is not defensible. Either way, including such an unclear statement in a security notice is troubling.
Did you even read the post? It clearly says: "We hash passwords with a one-way hashing algorithm, with multiple hashing iterations and individual salt per password."
The article has been been silently edited since original posting time. I just checked the version I read in the morning and indeed, it never said anything about salting.
The original quote is:
>The reason you’re reading this blog post is because of a recent discovery by our security team - about 17 million user records from our database were stolen. The stolen information has user email addresses and hashed passwords.
?The hashed password cannot be converted/decrypted back to plain text - so the sanctity of your password is intact in case you use the same password for other services. But if you are paranoid about security like us, we encourage you to change your password for any other services where you are using the same password.
The original quote is:
>The reason you’re reading this blog post is because of a recent discovery by our security team - about 17 million user records from our database were stolen. The stolen information has user email addresses and hashed passwords.
?The hashed password cannot be converted/decrypted back to plain text - so the sanctity of your password is intact in case you use the same password for other services. But if you are paranoid about security like us, we encourage you to change your password for any other services where you are using the same password.
Why do they say this then:
> We however, strongly advise you to change your password for any other services where you are using the same password.
> We however, strongly advise you to change your password for any other services where you are using the same password.
It is standard practice to include such a statement. If they said you were safe based on their practices, then it turned out there was a bug in their code or a library that exposed you, that would be really really bad.
Salting is irrelevant to making such a statement. For instance, some users probably have their password set to "zomato1234" or "passw0rd" or (anything in the next 10000+ most common password patterns). Salted or not, it's very feasible to perform that kind of analysis on the leaked data, find weak passwords, then use them with that user's account on other sites.
What? Hashed passwords can definitely be cracked for a significant percent of users.