Why do we never hear of Google security problems?
35 comments
Up until late 2013, the connection between Google data centers were all in plaintext, which allowed State sponsored attackers to acquire pretty much every piece of data that Google had. Source codes, customer data, internal data, everything.
[0] https://www.washingtonpost.com/world/national-security/nsa-i...
[1] https://www.washingtonpost.com/world/national-security/googl...
[0] https://www.washingtonpost.com/world/national-security/nsa-i...
[1] https://www.washingtonpost.com/world/national-security/googl...
Yes, the infamous "NSA smiling face": http://www.slate.com/blogs/future_tense/2013/10/30/nsa_smile...
(I don't have access to links you've posted).
(I don't have access to links you've posted).
Insider anecdotes: security bugs were considered P0s when I was at Google, i.e. "drop everything and fix it immediately", where "everything" also included things like dinner and sleep (if a security bug was reported in the middle of the night, the on-call SRE had authority to roll back to the last known good version and have engineering fix it in the morning and go through the normal review process). That created a pretty strong incentive not to introduce security bugs.
There was also a strong defense-in-depth culture, where we'd strive to avoid introducing security bugs in the JS, but the frameworks would also defend against them, and the GFEs (load balancers) would themselves detect intrusion attempts, as would the webservers, which would be architected so that bad data would be rejected & logged rather than triggering undefined behavior, and so on. So if a bug was discovered, it usually wouldn't be exploitable, and the whole time the hacker is trying to exploit it they have Google's security team working against them (and potentially calling up their ISP saying "The user with IP address xxx.xxx.xxx.xxx is trying to hack us, could you kick them off the Internet so we don't have to block your whole IP range?")
There was also a strong defense-in-depth culture, where we'd strive to avoid introducing security bugs in the JS, but the frameworks would also defend against them, and the GFEs (load balancers) would themselves detect intrusion attempts, as would the webservers, which would be architected so that bad data would be rejected & logged rather than triggering undefined behavior, and so on. So if a bug was discovered, it usually wouldn't be exploitable, and the whole time the hacker is trying to exploit it they have Google's security team working against them (and potentially calling up their ISP saying "The user with IP address xxx.xxx.xxx.xxx is trying to hack us, could you kick them off the Internet so we don't have to block your whole IP range?")
Out of interest why have an on call SRE when you have Google employees around the world. It's a reasonable time somewhere? The advantage is having someone who is more 'with it' when the problem arises.
I can guess the answer - maybe it is a trust issue, they want to minimize the number of people with rollback privileges.
I can guess the answer - maybe it is a trust issue, they want to minimize the number of people with rollback privileges.
Partially trust, but also because SRE is a specialized role with specialized tools and a specialized skillset. (And a slightly different mindset from regular engineers - SREs are very attuned to "what can go wrong?", while engineers are supposed to be attuned to "what can go right?".) You need to be more conversant with the high-level details of how the system operates (eg. which binaries are run, what are their resource requirements, what are common failure modes, what can cause cascading failures, when is everything deployed), and much less conversant with the details of how a particular piece of code runs, since the default action when it breaks is just "shut it off and let the engineer debug". Plus SREs are paid more (because they have on-call hours), so it wouldn't really save Google any money to have any every engineer act as a potential SRE.
That said, when I was there many teams of smaller, riskier, or less critical services actually did have engineers carry the pager, sometimes with reduced SLAs (and an interlock built into the code so that an outage in their service wouldn't eg. take down websearch).
That said, when I was there many teams of smaller, riskier, or less critical services actually did have engineers carry the pager, sometimes with reduced SLAs (and an interlock built into the code so that an outage in their service wouldn't eg. take down websearch).
For what it's worth, and as someone who worked at both places, Facebook does these things pretty much the same way as Google does.
They give security researchers rewards for reporting vulnerabilities properly. I hear their program is pretty decent compared to a lot of others. It only costs them about $3 million a year so it works out supper cheap for them too.
https://www.google.com/about/appsecurity/reward-program/
They also seem to keep a wall of fame for bug hunters.
https://bughunter.withgoogle.com/rank/hof
https://www.google.com/about/appsecurity/reward-program/
They also seem to keep a wall of fame for bug hunters.
https://bughunter.withgoogle.com/rank/hof
Facebook give out a minimum of $500 for bugs and issues they deem worthy of rewarding. Last year they paid more than $880,000, and as of January this year they had paid a total amount of $6,300,000 to the people participating in their white hat program.
https://www.facebook.com/whitehat https://www.facebook.com/notes/facebook-bug-bounty/2017-high...
https://www.facebook.com/whitehat https://www.facebook.com/notes/facebook-bug-bounty/2017-high...
I worked at both Google and Facebook. They both have very strong security teams and security procedures, but at the same time have a very high profile and a lot of hackers targeting them. Both companies have security failures frequently, but most of those failures are small enough that you wouldn't know about them unless you went looking.
I think the biggest difference is that Facebook stores much more sensitive personal data, and stores more of it in a complicated product with a large surface area of possible vulnerabilities. In the search engine part of Google, the vast majority of data isn't even private. There's no point to hacking into Google search. Gmail does contain personal data, but it has far less surface area than the Facebook product. With Facebook, almost every new feature involves your personal data in some way, so all of those features have some security risk.
There is also a related selection bias. If a service gets hacked, you might or might not hear about it. But if there is something like the communications of celebrities that are now made public, or the social media account of a celebrity that a hacker posted a message to, it's just much more exciting news than your average vulnerability. The nature of Facebook's product means that Facebook security issues often have this kind of highly-visible outcome.
I think the biggest difference is that Facebook stores much more sensitive personal data, and stores more of it in a complicated product with a large surface area of possible vulnerabilities. In the search engine part of Google, the vast majority of data isn't even private. There's no point to hacking into Google search. Gmail does contain personal data, but it has far less surface area than the Facebook product. With Facebook, almost every new feature involves your personal data in some way, so all of those features have some security risk.
There is also a related selection bias. If a service gets hacked, you might or might not hear about it. But if there is something like the communications of celebrities that are now made public, or the social media account of a celebrity that a hacker posted a message to, it's just much more exciting news than your average vulnerability. The nature of Facebook's product means that Facebook security issues often have this kind of highly-visible outcome.
Disclaimer I don't have Facebook anymore, since like 2015 so I don't know which data they require to register an account.
I would disagree with you that Facebook stores more sensitive data. I mean just look at all the services Google offers, which might have way more critical data and lots of more data in general.
If someone would just have access to one of my co-workers Google account he would've access to his phone numbers, addresses, names (contacts); access to all his private appointment's (calendar); mail; and maybe even critical documents and other files (gdrive + docs, ...); All of his searches, even voice commands (Google now); music, book, movie, game - preferences (Google play store) and not to forget social media (G+ and YouTube).
The difference to Facebook is, that on social media you choose what you want to display in public, or to a certain group. (Verification methods are an exception here) When you use your private phone/Computer, you expect that the information will stay private until you share it somewhere.
Of course it's hard to generalize that, everyone uses these services differently
TL;Dr I think a Google account linked to an android phone of a "normal user" would be more valuable than a "normal" Facebook account.
I would disagree with you that Facebook stores more sensitive data. I mean just look at all the services Google offers, which might have way more critical data and lots of more data in general.
If someone would just have access to one of my co-workers Google account he would've access to his phone numbers, addresses, names (contacts); access to all his private appointment's (calendar); mail; and maybe even critical documents and other files (gdrive + docs, ...); All of his searches, even voice commands (Google now); music, book, movie, game - preferences (Google play store) and not to forget social media (G+ and YouTube).
The difference to Facebook is, that on social media you choose what you want to display in public, or to a certain group. (Verification methods are an exception here) When you use your private phone/Computer, you expect that the information will stay private until you share it somewhere.
Of course it's hard to generalize that, everyone uses these services differently
TL;Dr I think a Google account linked to an android phone of a "normal user" would be more valuable than a "normal" Facebook account.
Many of the security flaws of other companies are actually exposed by Google's Project Zero. The charitable view would be that they put more effort to fix bugs that they have found, the cynical take would be that they extend their own deadlines indefinitely. This would of course explain only some classes of bugs.
One story I do remember though was that the introduction of Google Buzz leaked your contact list to all of your contacts.
One story I do remember though was that the introduction of Google Buzz leaked your contact list to all of your contacts.
Anecdata - I was using Google docs a few years ago and was shown a complete stranger's document. This was on my own PC that no one else had used or logged into. I immediately logged out and in again and I saw my stuff again. So maybe they do have security issues sometimes.
Similar thing happened to me. On two occasions, I logged into my Gmail but was shown someone else inbox. This was several years ago and there were lots of reports on the net about similar incidents which I annoyingly can't find now. Google would never admit it and instead claimed PEBKAC, but I believe they had a bug that affected their single-sign on system.
PEBKAC. That is Hilarious. So what ... we accidentally typed a strangers username and password. Or it was a dream.
Oh I see they mean Google's chairs and keyboards. That makes sense.
Oh I see they mean Google's chairs and keyboards. That makes sense.
Similar. I found a spreadsheet in my docs that I had nothing to do with.
My only thought was that it was extremely rare hash collision.
My only thought was that it was extremely rare hash collision.
Was it private? I often see things in there and have no idea how they got there, but it seems they get added when you open a URL to someone else's public document, even if you don't let it fully load.
That's definitely a breach of security, but it's a bit different, too. It's not (so far as we know) exploitable. That is, an attacker couldn't use it to get all the documents stored on Google docs. They could, maybe, trigger it to see one random other document. That's still a security breach, but it's not in the same league as exploitable attacks.
I am not sure of the extent because I am not a security researcher and I didn't want to try to exploit it more as an 'amateur' lest I do something illegal. But it is possible there was an exploit at the time.
The Chinese hacked them and stole their source code and was caught tracking dissidents emails.
Any links for this absurd claim?
What's absurd about the claim? I thought it was widely known that government-sponsored hacking was one of the major reasons Google pulled out of China back in 2010.
[deleted]
They had that engineer who was able to stalk people. So, at the time, a failure of their internal security/privacy policies.
A few years back a Google engineer was spying on underage teens' private conversations: http://gawker.com/5637234/gcreep-google-engineer-stalked-tee...
Not security problems per se, but interesting:
Google.com registered for a minute by someone else: http://time.com/4199940/google-domain-sanmay-ved/
Government backdoor (not just Google): https://en.wikipedia.org/wiki/PRISM_(surveillance_program)
Google DNS: https://techcrunch.com/2018/04/24/myetherwallet-hit-by-dns-a...
Google.com registered for a minute by someone else: http://time.com/4199940/google-domain-sanmay-ved/
Government backdoor (not just Google): https://en.wikipedia.org/wiki/PRISM_(surveillance_program)
Google DNS: https://techcrunch.com/2018/04/24/myetherwallet-hit-by-dns-a...
Not so long ago a large amount of Google documents were indexed by Yandex search engine, including very sensitive personal data and became publicly accessible via search results. Not really Google mistake because documents had public access, but I personally think Google should have added no-index by default to all documents if a document is only available to those who have a link to it.
Many hacks these days involve social engineering and phishing. Google's blanket use of security keys by its employees has pretty much stopped phishing.
And their lack of customer support solves the social engineering problem too.
I do remember about a flaw that let anyone remove/deindex any website from Google(1).
Not sure how big it would be compared to Facebook's issues though as no user data was comprised.
(1) https://www.davidnaylor.co.uk/google-webmasters-flaw-allows-...
Not sure how big it would be compared to Facebook's issues though as no user data was comprised.
(1) https://www.davidnaylor.co.uk/google-webmasters-flaw-allows-...
Pretty sure that they were hacked.
Can't find the link now but someone in Japan/China Google was hacked by a good old PDF zero day.
Can't find the link now but someone in Japan/China Google was hacked by a good old PDF zero day.
The Big-G dont roll like that ! /s :)
We only hear about breaches when they get user data.
The Cambridge analytica thing wasn't really a breach IMO.
1 large breach vs 0 over however many years does not seem statistically significant.
The Cambridge analytica thing wasn't really a breach IMO.
1 large breach vs 0 over however many years does not seem statistically significant.
A company that big has an almost 100% chance of making a security mistake at least once, so why don't we hear about it?
Yes, I am implying that they maybe use their internet search dominion to censor these news (among other things), but I am open to changing my mind. What do you guys think?