Improving our account management policies to better support customers(cloudplatform.googleblog.com)
cloudplatform.googleblog.com
Improving our account management policies to better support customers
https://cloudplatform.googleblog.com/2018/07/improving-our-account-management-policies-to-better-support-customers.html
91 comments
I no longer recommend Google Cloud after one of these insta-bans. The previous week, I had run a (literal) multi-million event on GC for Nike (which went fine, no complaints) and I realized immediately that there was literally nothing stopping Google from insta-banning an account in the future in the middle of a Nike event. No warning, no recourse, just "Fuck off, customer." These are live events, I simply cannot risk being shut off for "fraud". Seriously, fuck these guys.
My trust level for Google: ZERO. I can't risk my business on that level of stupidity.
My trust level for Google: ZERO. I can't risk my business on that level of stupidity.
I realize that you may never come back, but "insta-bans" are extremely unlikely in any case, but even less so if you take any of the actions in the post (all for free):
- Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
- Provide one or more billing admins to your account.
- Provide a secondary payment method in case there are problems charging your primary method.
- Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards.
To be clear, if you do the last one (which you easily qualify for), you are under contract with us. In that case, the phrase "there was literally nothing stopping Google from insta-banning an account in the future in the middle of a Nike event" would be completely incorrect.
- Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
- Provide one or more billing admins to your account.
- Provide a secondary payment method in case there are problems charging your primary method.
- Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards.
To be clear, if you do the last one (which you easily qualify for), you are under contract with us. In that case, the phrase "there was literally nothing stopping Google from insta-banning an account in the future in the middle of a Nike event" would be completely incorrect.
This can happen with any provider, either due to policy, human error, malicious actor, ...
The correct answer here is to use multiple providers and establish proper relationship with them.
As an Amazon seller, I have to say this is an order of magnitude better response from Google than I'd ever expect from Amazon. Admittedly low standards.
I've been locked out at times from my seller account because they wanted to verify a 10 euro purchase on a related account, and blocked the account for fraud until I faxed over a bank statement. A different time, I signed up for AWS, they couldn't charge the card for some reason, and so the account got blocked for fraud and I couldn't log into the seller account.
I've been locked out at times from my seller account because they wanted to verify a 10 euro purchase on a related account, and blocked the account for fraud until I faxed over a bank statement. A different time, I signed up for AWS, they couldn't charge the card for some reason, and so the account got blocked for fraud and I couldn't log into the seller account.
This is a fantastic insight, it crystallizes a sense of foreboding about Google I have had for several years now since a friend got locked out of his GMail account by Google and had no redress. Dealing with Google is like dealing with an insect colony in contrast to Amazon or Apple support. Other large tech firms like Atlassian offer excellent support as well. I think hguhghuff is right: they don't seem to want a real conversation with their customers. No human touch.
Atlassian is not a "large" tech firm. By any size metric it is 1% of the others you mentioned.
Fair point they are in the top 100 but not the top 50 Software/SaaS firms. They are at about a 600M run rate so I don't think of them as a "small" firm but but they not the behemoths that Apple, Google, Amazon, Facebook are.
>It’s googles attitude to customers that is the ingrained and deep problem. Google has built into its DNA from the beginning that it does not want to be in contact with you, that you cannot get emergency attention, and that you are probably wrong.
Of all the tech companies I've interacted with I've never dealt with one that has come off as uniquely elitist and tone-deaf as Google.
Steve Yegge (former Googler) makes some very good points on this.
https://medium.com/@steve.yegge/why-i-left-google-to-join-gr...
Of all the tech companies I've interacted with I've never dealt with one that has come off as uniquely elitist and tone-deaf as Google.
Steve Yegge (former Googler) makes some very good points on this.
https://medium.com/@steve.yegge/why-i-left-google-to-join-gr...
To be clear: we're fixing the algorithms, adding extra humans, and showing customers how to take action on their own. We never want to let what happened here happen again, and we are definitely listening to any and all suggestions.
Tl;Dr We screwed up. We're sorry.
Disclosure: I work at Google on Google Cloud.
Tl;Dr We screwed up. We're sorry.
Disclosure: I work at Google on Google Cloud.
>We never want to let what happened here happen again
This is basically the problem. You could say "this will never happen again", but instead it's "we don't want this to happen" as if it's somehow out of GCP's control.
As long as you will still delete accounts that you suspect of fraud, based on some opaque definition of suspecting fraud, "we don't want this to happen" is the best you can do, and that's not very reassuring.
This is basically the problem. You could say "this will never happen again", but instead it's "we don't want this to happen" as if it's somehow out of GCP's control.
As long as you will still delete accounts that you suspect of fraud, based on some opaque definition of suspecting fraud, "we don't want this to happen" is the best you can do, and that's not very reassuring.
> As long as you will still delete accounts that you suspect of fraud, based on some opaque definition of suspecting fraud, "we don't want this to happen" is the best you can do, and that's not very reassuring.
Having worked in fraud detection, this is true for every provider you use. The adversarial nature of fraud detection means making _your definition of fraud_ public is not an option because people will use it as a loss function for their fraud.
I would just make sure I don't rely on a single provider and minimize single points of failure.
Having worked in fraud detection, this is true for every provider you use. The adversarial nature of fraud detection means making _your definition of fraud_ public is not an option because people will use it as a loss function for their fraud.
I would just make sure I don't rely on a single provider and minimize single points of failure.
i understand why fraud detection mechanisms need to be opaque. i don't understand why the consequence has to be "we will delete your account"
In the context of cloud instances, accounts allocate resources. If they never deleted fraudulent accounts, they couldn't release the allocated resources, which is a huge problem. Imagine bad guy x who steals 100k credit cards, then uses a botnet to spin up VMs on 100k fraudulent accounts; if each VM has 10gb of data on it, you've just frozen 1 petabyte. Say there's a 90 day appeals process; those resources can't be deleted or reclaimed for 90 days, which is already tremendously expensive. If that expanded to forever, you would just have to close up shop.
In reality bad guys operate at even larger scales than this, doing all kinds of things. Depending on what company you are, you might not even be able to pull the compute resources away from them right away; so in addition to burning disk alloc they're now burning compute units. It depends on the product too -- what if they spin up pubsub instances and leave messages in the queue? You might have to freeze those queues so that if they're valid customers they don't lose data. The more offerings you have, the more potential loss.
But yeah, they don't delete accounts right away -- there is indeed a grace period, and they do keep all your stuff (I suspect legislation may have something to do with it too). Last I remember it goes to appeal process (human review), and your data can't be deleted / resources reclaimed for quite a while. So it's not zero recourse for the user. Sounds like they're adding even more humans into the loop to ensure they don't unfortunately flag and then delete valid users. But it's non-trivial.
In reality bad guys operate at even larger scales than this, doing all kinds of things. Depending on what company you are, you might not even be able to pull the compute resources away from them right away; so in addition to burning disk alloc they're now burning compute units. It depends on the product too -- what if they spin up pubsub instances and leave messages in the queue? You might have to freeze those queues so that if they're valid customers they don't lose data. The more offerings you have, the more potential loss.
But yeah, they don't delete accounts right away -- there is indeed a grace period, and they do keep all your stuff (I suspect legislation may have something to do with it too). Last I remember it goes to appeal process (human review), and your data can't be deleted / resources reclaimed for quite a while. So it's not zero recourse for the user. Sounds like they're adding even more humans into the loop to ensure they don't unfortunately flag and then delete valid users. But it's non-trivial.
Sorry, but if a company has such a big trouble with fraud it's absolutely doesn't mean it has any right to destoy someone else business. Especially considering that as you noted criminals usually tend to work on scale and they usually won't bother with registering any kind of legal entity or doing some manual confirmation, phone talk, etc.
And what more important: there is plenty of services that will do lot's of check in advance and ask your passport / card / face photo and such, but they never ever stop your servers or destoy your data unless you do something extremely bad. And if it's come to company account it's even less likely to get in any trougle since most of services take work with companies serious.
Yet Google just like everyone around have extremely easy sign up process that all criminals are happy about, but can blow you in a face at any time. So they first create a problem to make marketing easier and then try to compensate for it by increasing risk of fault for thrir legit customers.
And what more important: there is plenty of services that will do lot's of check in advance and ask your passport / card / face photo and such, but they never ever stop your servers or destoy your data unless you do something extremely bad. And if it's come to company account it's even less likely to get in any trougle since most of services take work with companies serious.
Yet Google just like everyone around have extremely easy sign up process that all criminals are happy about, but can blow you in a face at any time. So they first create a problem to make marketing easier and then try to compensate for it by increasing risk of fault for thrir legit customers.
> Yet Google just like everyone around have extremely easy sign up process that all criminals are happy about, but can blow you in a face at any time. So they first create a problem to make marketing easier and then try to compensate for it by increasing risk of fault for thrir legit customers.
That's a good point. What's interesting is that Google's capabilities mean that they are able to allocate you the resources you ask for post-signup pretty much instantly, whereas AWS doesn't (they actually are just unable to allocate resources so quickly). This turns out to be a nice property if that n day spinup time window is enough to detect some bad guys and take them down in the interim, before they've caused massive damage to your infra / cost to your org. It's an O(1) improvement though. Bad guys will still just wait you out and then cause damage, so it's not like you're actually less likely to have your account frozen at AWS/Azure vs Google.
That's a good point. What's interesting is that Google's capabilities mean that they are able to allocate you the resources you ask for post-signup pretty much instantly, whereas AWS doesn't (they actually are just unable to allocate resources so quickly). This turns out to be a nice property if that n day spinup time window is enough to detect some bad guys and take them down in the interim, before they've caused massive damage to your infra / cost to your org. It's an O(1) improvement though. Bad guys will still just wait you out and then cause damage, so it's not like you're actually less likely to have your account frozen at AWS/Azure vs Google.
In your extreme example of 100k valid stolen credit cards being burned simultaneously in one attack, you've got 1PB which is only $25k/mo online, $7k/mo coldline, retail, so even less wholesale or on tape backup. That's hardly noticeable on the bottom line.
Fwiw speaking from experience 100k is a toy problem and the "fake accounts freezing out resources" problem is _very_ noticeable to the bottom line. The scale of attacks they face are orders of magnitude higher. The 10gb figure is also contrived.
If you don't freeze the _bandwidth and compute_ resources you're even more fucked, because those have far worse scaling characteristics. So big company X taking immediate action to take down services make a lot of sense in that light. They don't do it in every situation though e.g borderline cases that require human review.
Fraud fighting is costly.
If you don't freeze the _bandwidth and compute_ resources you're even more fucked, because those have far worse scaling characteristics. So big company X taking immediate action to take down services make a lot of sense in that light. They don't do it in every situation though e.g borderline cases that require human review.
Fraud fighting is costly.
It is a violation of the gdpr to not the user why he was suspended, sharing any data about why with the user, not allowing him to correct the data if they are wrong or not allowing decisions made by algorithms to be appealed for human review. Missing any of those and you might be subject to fines, although this only applies to human customers.
The customer has been informed.
You'd trust then more if they told you an obvious lie: "we'll never make a mistake in the future"?
no, i'd trust them more if they told me they removed the decision point where this mistake was made. i don't want "we'll never make this mistake again" any more than i want "we will try really hard not to make this mistake again".
what i want to hear is "we can't make this mistake again".
what i want to hear is "we can't make this mistake again".
Then you should have have press release that says “We screwed up, we’re sorry, and we won’t delete your accounts any more”, instead of all this many-lawyered garbage about “account management policies”, words that no one is interested in.
This post has 7 comments and 11 points after an hour. Compare that to the post that led to this outcome. If your trying to reassure the market that you don’t delete a companies entire back end infrastructure then yes this meets some internal management butt covering goal, but does not tell the market definitively that google cloud no longer deletes your entire computer systems.
This post has 7 comments and 11 points after an hour. Compare that to the post that led to this outcome. If your trying to reassure the market that you don’t delete a companies entire back end infrastructure then yes this meets some internal management butt covering goal, but does not tell the market definitively that google cloud no longer deletes your entire computer systems.
While we debated that exact thing for a long time, the reality was that we knew customers would want to dig deeper and understand HOW we wouldn't let it happen again. We tried to do that here. We tried not to make it sound too lawyerly, but I'll definitely take your feedback internally.
Disclosure: I work at Google on Google Cloud.
Disclosure: I work at Google on Google Cloud.
Amazon does not delete your account as far as I know until some extreme has been reached.
You guys need to understand the scale of distrust this issue raises.
If you want to fix it, give up on putting lawyers and spin doctors up front.
Instead, give this disastrous issue the response it deserves and get the CEO and Chairman of google to make s loud, clear and focused apology and say loudly that we do not delete your entire computer systems over account payment issues until defcon 3 is reached.
Why the F would I build my SAAS business on google cloud without that message being yelled repeatedly and loudly and clear from the top level of the company, as much so Google understands it as the market understands it.
And by the way, I pay ALL my cloud accounts on credit card. I note carefully that at google this means I have a higher chance of having my entire computer systems deleted than if I use some other payment mechanism. Your policy here appears to me to be using systems deletion add leverage to get me off credit card payment. All of which says to me that google still just doesn’t get it.
You guys need to understand the scale of distrust this issue raises.
If you want to fix it, give up on putting lawyers and spin doctors up front.
Instead, give this disastrous issue the response it deserves and get the CEO and Chairman of google to make s loud, clear and focused apology and say loudly that we do not delete your entire computer systems over account payment issues until defcon 3 is reached.
Why the F would I build my SAAS business on google cloud without that message being yelled repeatedly and loudly and clear from the top level of the company, as much so Google understands it as the market understands it.
And by the way, I pay ALL my cloud accounts on credit card. I note carefully that at google this means I have a higher chance of having my entire computer systems deleted than if I use some other payment mechanism. Your policy here appears to me to be using systems deletion add leverage to get me off credit card payment. All of which says to me that google still just doesn’t get it.
Amazon locked out a seller's account because they changed their Kindle's profile name to "Daddy" (in Chinese). Amazon asked for the seller's ID with that new name, and no amount of support escalation was enough to fix it[0]. It took an absurd amount of bad PR for them to come out and fix it.
Google is bad, but Amazon is really just as bad. They both have absurd policies with no recourse, that work in the 80% case and fail very badly in the 20% left.
[0] https://www.reddit.com/r/amazon/comments/5gvgdl/using_a_amaz...
Google is bad, but Amazon is really just as bad. They both have absurd policies with no recourse, that work in the 80% case and fail very badly in the 20% left.
[0] https://www.reddit.com/r/amazon/comments/5gvgdl/using_a_amaz...
Amazon Web Services is essentially unrelated as a business from Amazon.com. This is even less relevant than someone trying to compare the policies of some random end-user service like Google Photos with Google Cloud Platform. I personally had serious issues (not account suspension problems, but an inability to get things like basic tax paperwork to correct addresses) with Amazon Flexible Payments (which was part of some seemingly unrelated-to-everyone division), but I count absolutely none of them against AWS (which wasn't involved in running the service).
I once signed up for AWS, they couldn't verify the card on initial signup and the entire account got blocked, including a login to a seller account that was doing 6 figures/month.
Amazon has a big issue in that all their services are linked to a single Amazon account, and if that account is blocked you can be prevented from accessing entirely unrelated services. In that case, maybe it was ok to suspend a brand new AWS account until accounts could be verified, but the linked accounts that were also blocked should have been looked at manually because of their volume.
Amazon has a big issue in that all their services are linked to a single Amazon account, and if that account is blocked you can be prevented from accessing entirely unrelated services. In that case, maybe it was ok to suspend a brand new AWS account until accounts could be verified, but the linked accounts that were also blocked should have been looked at manually because of their volume.
Yep I’ve also had absurd cross Amazon-AWS payment issues they were unable to fix for months. Their ultimate solution “uhhh make a new account.”
> Google is bad, but Amazon is really just as bad.
I think the main problem is that fraud is real problem and if you don't want your account to ever be considered fraudulent, then you should take ample steps to make sure that it is not.
There are countless stories of people being careless with their keys causing tons of crypto-mining servers to launch.
There is no win for cloud providers here. Either you get PR smashed for charging careless customers, pay tons of money to wave off their mistake, or get ridiculed for trying to block customers whom you think have been hacked.
I think the main problem is that fraud is real problem and if you don't want your account to ever be considered fraudulent, then you should take ample steps to make sure that it is not.
There are countless stories of people being careless with their keys causing tons of crypto-mining servers to launch.
There is no win for cloud providers here. Either you get PR smashed for charging careless customers, pay tons of money to wave off their mistake, or get ridiculed for trying to block customers whom you think have been hacked.
Or just throttle the account and contact the customer by phone and help them resolve the issue.
If you can’t provide that level of support you don’t deserve my business.
If you can’t provide that level of support you don’t deserve my business.
I received this a few months ago on a demo account I was setting up. Here's the mail. (It was my personal account and cc not my corporate one) - I don't even have a fax machine.
---------
Greetings from Amazon Web Services,
We were unable to validate important details about your Amazon Web Services (AWS) Account. Your AWS account has been suspended. While your account is suspended, you will not be able to log in to the AWS portal or access AWS resources.
If you do not respond to us within 3 days of this email (by Saturday, April 28, 2018), your AWS Account, and any data will be terminated.
At your earliest convenience, please send us a copy of a current bill (utility bill, phone bill or similar), showing your name and address. Please provide for both the credit card holder and account holder if different.
IMPORTANT: When you send us your information, please include your email address, so we are able to process your fax as soon as possible. We can be reached via secure fax line at +1-206-922-5831 and 866-437-9072.
Once you have sent the requested documents via fax, please contact us at [email protected] and include the following details:
- The billing address and phone number of the credit card on the AWS account - The billing phone number of the credit card on the AWS account - Business name and phone number (if applicable) - The URL for your website (if applicable) - A contact phone number where you can be reached should we need additional information - Potential business/personal expectations for using Amazon Web Services
If you have any additional information you feel will expedite this process, please feel free to include that as well.
We apologize for any inconvenience this may have caused you and appreciate your patience with our security measures.
Thank you for your immediate attention to this matter.
Sincerely,
Amazon Web Services Team
---------
Greetings from Amazon Web Services,
We were unable to validate important details about your Amazon Web Services (AWS) Account. Your AWS account has been suspended. While your account is suspended, you will not be able to log in to the AWS portal or access AWS resources.
If you do not respond to us within 3 days of this email (by Saturday, April 28, 2018), your AWS Account, and any data will be terminated.
At your earliest convenience, please send us a copy of a current bill (utility bill, phone bill or similar), showing your name and address. Please provide for both the credit card holder and account holder if different.
IMPORTANT: When you send us your information, please include your email address, so we are able to process your fax as soon as possible. We can be reached via secure fax line at +1-206-922-5831 and 866-437-9072.
Once you have sent the requested documents via fax, please contact us at [email protected] and include the following details:
- The billing address and phone number of the credit card on the AWS account - The billing phone number of the credit card on the AWS account - Business name and phone number (if applicable) - The URL for your website (if applicable) - A contact phone number where you can be reached should we need additional information - Potential business/personal expectations for using Amazon Web Services
If you have any additional information you feel will expedite this process, please feel free to include that as well.
We apologize for any inconvenience this may have caused you and appreciate your patience with our security measures.
Thank you for your immediate attention to this matter.
Sincerely,
Amazon Web Services Team
You work at google; surely you should google how to fax something without a fax machine? Or were you just unwilling to trust those services? (which is understandable, you're sending pretty personal data).
And in general, I find it difficult to buy your "I don't own a fax machine argument". I don't own a photocopier, but that's a small blocker, no?
And in general, I find it difficult to buy your "I don't own a fax machine argument". I don't own a photocopier, but that's a small blocker, no?
I'll be honest, that comment was snark. I apologize.
That's an equivalently bad e-mail from AWS, but it's a single datum, out of context.
Actually, you did provide some context, which is that it was a "demo" account on a personal credit card. This, at the very least, implies much less spending than in the blog post that kicked this all off.
More importantly, what's missing is the human/support response and attitude from AWS, which is the OC's primary concern.
Anecdotally, from the comments I've seen on HN, AWS customers (at least beyond a certain size) seem generally pleased with the availability of human judgment during support issues, especially so when paying for that support.
Actually, you did provide some context, which is that it was a "demo" account on a personal credit card. This, at the very least, implies much less spending than in the blog post that kicked this all off.
More importantly, what's missing is the human/support response and attitude from AWS, which is the OC's primary concern.
Anecdotally, from the comments I've seen on HN, AWS customers (at least beyond a certain size) seem generally pleased with the availability of human judgment during support issues, especially so when paying for that support.
I know, I guess the funny part is the opposite was true with this account on AWS. The GCP issue (as detailed in the original medium post) was a lot of spending on what appeared to be a new account (again, this was OUR mistake). The reason the AWS one frustrated me so much was there was effectively ZERO spend.
In our case, we're adding a second human in the loop (we already had one) to double check this. Customers can also add their own human to reach out to by connecting to a sales person (it's free and extremely low pressure!)
In our case, we're adding a second human in the loop (we already had one) to double check this. Customers can also add their own human to reach out to by connecting to a sales person (it's free and extremely low pressure!)
As stated in the original medium post, the issue was they only had one credit card on file and no contact at the company. If they had either a second credit card or moved to offline billing or had a contact with a phone number (all free steps), they would not have been suspended.
One problem I have had with dealing with Google (not for this particular service) is that they set up mechanisms where adding a "backup" creates a new single point of failure rather than adding redundancy. If you can't access one particular backup method of verification, your primary and other backup are of no use. It's a trap! And quite Kafkaesque.
While this might not strictly apply to some other particular situation, my experience would lead me not to ever trust Google when they say "do this seemingly logical thing because it will make you safer".
While this might not strictly apply to some other particular situation, my experience would lead me not to ever trust Google when they say "do this seemingly logical thing because it will make you safer".
I'm pretty sure that's undisputed, but it also doesn't address the concern of what happens once a customer finds themselves in that situation, whatever the reason.
Furthermore, this response could be construed as reinforcing the OC's precise concern regarding Google's attitude:
> that you are probably wrong.
Furthermore, this response could be construed as reinforcing the OC's precise concern regarding Google's attitude:
> that you are probably wrong.
I hope we can convince you all that that is no longer our attitude, but I know that only time will tell.
As I mentioned above, in this case, WE were absolutely wrong, and the blog post details all the ways we're fixing it.
As I mentioned above, in this case, WE were absolutely wrong, and the blog post details all the ways we're fixing it.
I suspect that part of what has people worried and continuing to complain about Google's attitude and "tone-deafness" is that your responses, which are seen as being representative of GCP, based on your revealing your affiliation, suggest you're not quite absorbing the meaning of the actual concern/complaint.
You did admit error and apologize in the blog post (perhaps not explicitly or early enough in the text for some people, but that's a separate issue), and you specified how you were going to fix it. It may take time for the reputational effects of those fixes to permeate.
That's not the issue.
The issue is that, here, you have, effectively, said, "but the customer could have..".
Even the implication that the customer might share in the blame, no matter how objectively true it may be, takes credibility away from the sincerity of an apology and can easily be seen as evidence of the persistence of same, old attitude. Given that the statement happened after the apology, further time is unlikely to make a difference.
You did admit error and apologize in the blog post (perhaps not explicitly or early enough in the text for some people, but that's a separate issue), and you specified how you were going to fix it. It may take time for the reputational effects of those fixes to permeate.
That's not the issue.
The issue is that, here, you have, effectively, said, "but the customer could have..".
Even the implication that the customer might share in the blame, no matter how objectively true it may be, takes credibility away from the sincerity of an apology and can easily be seen as evidence of the persistence of same, old attitude. Given that the statement happened after the apology, further time is unlikely to make a difference.
Shoot, ok, let me try again.
1) This was, in NO WAY, the customer's fault. At all. Not even a little bit. They did stuff that was totally normal for a customer to do.
To remedy us not accidentally flagging totally normal stuff as fraud again, we talked about 7 steps we were taking (that was the first 7 steps in the blog).
2) IF, and only IF, you'd like to take ADDITIONAL steps to avoid us accidentally flagging you, we also offered three steps (at the end of the article). However, these were totally optional.
Hope that helps.
1) This was, in NO WAY, the customer's fault. At all. Not even a little bit. They did stuff that was totally normal for a customer to do.
To remedy us not accidentally flagging totally normal stuff as fraud again, we talked about 7 steps we were taking (that was the first 7 steps in the blog).
2) IF, and only IF, you'd like to take ADDITIONAL steps to avoid us accidentally flagging you, we also offered three steps (at the end of the article). However, these were totally optional.
Hope that helps.
[deleted]
Maybe the language in setting all those up should be very upfront with the benefits they provide. Or even proactive emails and large dialogs about doing that. Perhaps it should be like those sites heavily suggesting 2FA if you've been a customer long enough. If there is a concern about noise, maybe the messaging for that should pop up in the GUI in a non-ignoreable way and start on accounts with a sizable spend or have existed for some time with activity.
Let me break down exactly why I think the language in what appears to be a lawyer-driven PR pot-bashing is so terrible, because if your plan was to go back to internal G+ with "someone on HN said a lawyer wrote it" little will probably change.
> Established GCP customers...
> Online customers with established payment history...
What exactly qualifies as being "established"? In the first example, additional stipulations around billing arrangements are imposed, but in the latter it remains unclear.
> ...review for flagged fraud accounts...
> ...event that fraud or account compromise activity is detected...
You gotta stop using the f-bomb like that. This piece is supposed to be a follow on to a story of a customer who apparently was conducting completely non-fraudulent activity within their account. Your copy should, like your culture, assume "innocent until proven guilty". Saying something is "flagged for fraud" in a PR piece breaks that rule.
> We will provide ways for customers to pre-verify their accounts with us if they desire...
Sounds very passive-aggressively optional. Nor does it tell me what I actually gain here, or even a hint of what might be involved for verification or how potentially simple or easy it may or may not be - do I need my counsel to post via registered service on company letterhead confirmation of my organisation's official trading details, or do I click an link in an email?
> There are also steps that customers can take to help us protect their accounts including...
The word "protect" is suitable if you were a bank, and you wanted to encourage customer behaviour to prevent them being victims to fraudsters. But you're not, so really this language is suggesting customers need to "protect" themselves from your algorithms and automated systems, which is kind of inappropriate to be saying.
> Established GCP customers...
> Online customers with established payment history...
What exactly qualifies as being "established"? In the first example, additional stipulations around billing arrangements are imposed, but in the latter it remains unclear.
> ...review for flagged fraud accounts...
> ...event that fraud or account compromise activity is detected...
You gotta stop using the f-bomb like that. This piece is supposed to be a follow on to a story of a customer who apparently was conducting completely non-fraudulent activity within their account. Your copy should, like your culture, assume "innocent until proven guilty". Saying something is "flagged for fraud" in a PR piece breaks that rule.
> We will provide ways for customers to pre-verify their accounts with us if they desire...
Sounds very passive-aggressively optional. Nor does it tell me what I actually gain here, or even a hint of what might be involved for verification or how potentially simple or easy it may or may not be - do I need my counsel to post via registered service on company letterhead confirmation of my organisation's official trading details, or do I click an link in an email?
> There are also steps that customers can take to help us protect their accounts including...
The word "protect" is suitable if you were a bank, and you wanted to encourage customer behaviour to prevent them being victims to fraudsters. But you're not, so really this language is suggesting customers need to "protect" themselves from your algorithms and automated systems, which is kind of inappropriate to be saying.
I suspect they'll treat it as a secret since telling you the magic formula for established accounts will compromise their anti-abuse regime. It is the same reason there are no docs about how to become a high-reputation email origin from Gmail's perspective. It's a trade secret.
> they'll treat it as a secret since telling you the magic formula for established accounts will compromise their anti-abuse regime
This is accurate.
>> Online customers with established payment history...
> What exactly qualifies as being "established"? In the first example, additional stipulations around billing arrangements are imposed, but in the latter it remains unclear.
They can't tell you. Bad actors will even binary search the parameters of your fraud detection (source: I worked in fraud detection). Minimizing the information feedback loop to the baddies is super important to avoid becoming a loss function they can optimize against. This is true of every fraud department.
Abuse detection is cat-and-mouse. Say as a provider I track your true IP through proxies by embedding some flash that calls home. You figure it out and block flash or do some clever sandboxing. I work with proxy providers to build blacklists. You somehow get the blacklist and just avoid all proxies on it. Etc etc.
This is accurate.
>> Online customers with established payment history...
> What exactly qualifies as being "established"? In the first example, additional stipulations around billing arrangements are imposed, but in the latter it remains unclear.
They can't tell you. Bad actors will even binary search the parameters of your fraud detection (source: I worked in fraud detection). Minimizing the information feedback loop to the baddies is super important to avoid becoming a loss function they can optimize against. This is true of every fraud department.
Abuse detection is cat-and-mouse. Say as a provider I track your true IP through proxies by embedding some flash that calls home. You figure it out and block flash or do some clever sandboxing. I work with proxy providers to build blacklists. You somehow get the blacklist and just avoid all proxies on it. Etc etc.
Fraud detection needs to be balanced against the massive amounts of bad will being generated. It seems like anti-fraud has taken over. If Google keeps this up they aren't going to have many real customers to "protect".
Yeah, false positives are expensive. They aren't trying to make these mistakes :(
Being "established" requires any of the following (from the original post) - all free:
- Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
- Provide one or more billing admins to your account.
- Provide a secondary payment method in case there are problems charging your primary method.
- Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards.
- Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
- Provide one or more billing admins to your account.
- Provide a secondary payment method in case there are problems charging your primary method.
- Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards.
> While we debated that exact thing for a long time, the reality was that we knew customers would want to dig deeper and understand HOW we wouldn't let it happen again.
One saying I’m sorry isn’t an amends. Neither is offering a contrition-free remedy for past bad acts. Amends requires both contrition and a remedy.
For example, inserting the following (or anything similar) after the first sentence would have demonstrated some contrition: “We have truly been humbled by our prior lack of insight, and we are very sorry for what happened to any customers affected in any way by our shortcomings in this area. We realize we need to regain your trust and with that in mind, have thoughtfully examined all the feedback we received.”
Learning you all actually thought it through and consciously chose to save face does not inspire confidence. Nor does the underlying subtext that customers big enough to pay by check and get an account rep get a free pass, while little bootstrapped guys like me can still have the hammer dropped on them, albeit with slightly more scrutiny than last week.
One saying I’m sorry isn’t an amends. Neither is offering a contrition-free remedy for past bad acts. Amends requires both contrition and a remedy.
For example, inserting the following (or anything similar) after the first sentence would have demonstrated some contrition: “We have truly been humbled by our prior lack of insight, and we are very sorry for what happened to any customers affected in any way by our shortcomings in this area. We realize we need to regain your trust and with that in mind, have thoughtfully examined all the feedback we received.”
Learning you all actually thought it through and consciously chose to save face does not inspire confidence. Nor does the underlying subtext that customers big enough to pay by check and get an account rep get a free pass, while little bootstrapped guys like me can still have the hammer dropped on them, albeit with slightly more scrutiny than last week.
I understand, and, though it's not in the original post, I will say again.
We screwed up. We are really sorry. This was a confluence of unexpected events that slipped through our systems (which included both humans and algorithms). We know it will take a long time to rebuild the trust we've burned, and we are committed to doing it.
In addition, as I said earlier, the options to ensure that you have additional protection (WHICH YOU SHOULDN'T HAVE TO DO, I'M JUST OFFERING IT) are at no cost:
- Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
- Provide one or more billing admins to your account.
- Provide a secondary payment method in case there are problems charging your primary method.
- Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards (this is much lower than you would generally think)
We screwed up. We are really sorry. This was a confluence of unexpected events that slipped through our systems (which included both humans and algorithms). We know it will take a long time to rebuild the trust we've burned, and we are committed to doing it.
In addition, as I said earlier, the options to ensure that you have additional protection (WHICH YOU SHOULDN'T HAVE TO DO, I'M JUST OFFERING IT) are at no cost:
- Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
- Provide one or more billing admins to your account.
- Provide a secondary payment method in case there are problems charging your primary method.
- Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards (this is much lower than you would generally think)
Thanks for the replies. Despite the incessant whining that can happen on HN, I assure you that there's many here who appreciate the response. I'm sure it was a big deal for your team.
While you're right people will want an actual plan on how things are going to change, they also want a genuine expression of contrition. As hguhghuff noted, deleting someone's business is a serious, serious issue.
The _only_ admissions of guilt or wrongdoing was were these lines:
"Here’s what we are doing to be as good as our word, and provide a more careful, accurate, thoughtful and empathetic account management experience for our GCP customers."
"We sincerely apologize to all our customers who’ve been concerned or had to go through a service reinstatement. Please keep the feedback coming, we’ll work to continue to earn your trust every day."
Which, if you yourself were scared that your livelihood could be wiped away, would you think that was a sufficient apology?
It doesn't help that Google tends to lack a human touch when it comes to its image. But here, now, you all have an opportunity to change that. I know it sounds "politics" or "public relations" or whatever, but all we need to know is that you fully understand the fear GCP customers felt and what that means for us.
While you're right people will want an actual plan on how things are going to change, they also want a genuine expression of contrition. As hguhghuff noted, deleting someone's business is a serious, serious issue.
The _only_ admissions of guilt or wrongdoing was were these lines:
"Here’s what we are doing to be as good as our word, and provide a more careful, accurate, thoughtful and empathetic account management experience for our GCP customers."
"We sincerely apologize to all our customers who’ve been concerned or had to go through a service reinstatement. Please keep the feedback coming, we’ll work to continue to earn your trust every day."
Which, if you yourself were scared that your livelihood could be wiped away, would you think that was a sufficient apology?
It doesn't help that Google tends to lack a human touch when it comes to its image. But here, now, you all have an opportunity to change that. I know it sounds "politics" or "public relations" or whatever, but all we need to know is that you fully understand the fear GCP customers felt and what that means for us.
The worst that happened to me on amazon when I had credit card issues is that I could no longer make changes in the developer console and instead was forced to contact amazon accounts to resolve the issue.
I could still see everything and all my systems continued running.
I could still see everything and all my systems continued running.
As you can see above, I was given three days on AWS to fax (!!!) proof of identity or my account would be deleted.
> We tried not to make it sound too lawyerly
Who's we? Did you have a part in writing this? What does "not make it sound too lawyerly" even mean? This press release screams "we made a mistake but we're going to use paragraph after paragraph to ensure we don't actually say that". I have a hard time believing that the people who make up the Google Cloud team don't recognize this.
Who's we? Did you have a part in writing this? What does "not make it sound too lawyerly" even mean? This press release screams "we made a mistake but we're going to use paragraph after paragraph to ensure we don't actually say that". I have a hard time believing that the people who make up the Google Cloud team don't recognize this.
You have a culture problem.
Maybe because you are not a nice company, and you are selling people's private information to ad networks?
That's one possible reason z another is that you probably have huge egos everywhere internally, and it's kind of tricky to actually care about customers then.
Maybe because you are not a nice company, and you are selling people's private information to ad networks?
That's one possible reason z another is that you probably have huge egos everywhere internally, and it's kind of tricky to actually care about customers then.
Why didn't the blog post have a link to the customer's blog post that is referenced in the first paragraph?
It’s odd that people think Google should keep their business running even when “minor credit card issues” result in them not being paid. If you don’t pay the meter your car gets towed. Simple.
On the other hand, parking meters don't choose to reject your money halfway through your paid time and then immediately dispatch robot tow trucks.
Kudos to Google for taking feedback and working internally to make things better. Too often do people expect every process at a company to work perfectly so it's great to see that the community has done a good job of providing feedback here and Google using it to improve.
I tend to stick with vendors who are open to improvement rather than expecting perfection up front. There's always something that can be improved.
I tend to stick with vendors who are open to improvement rather than expecting perfection up front. There's always something that can be improved.
One of the big problems is companies with "too many" customers not having enough humans to deal with them interactively. It's probably possible, but it always seems like a side-job or afterthought, often shipped to low-wage countries, and hidden behind a wall of 'support documents' that hardly cover the many cases you might need to get in touch with a vendor.
Google being Google (and this probably goes for many tech leaders) probably thought that using an algorithm and some documentation would be enough, even if it means about 1% of customers slips through the cracks and loses all of their Google-stuff. The problem is that in the real world this generates enough backlash that people don't want or trust your services anymore.
While a cloud and even a more classical MSP is supposed to take what they are good at and make it available at a reasonable price (so you can focus on what you are good at instead of everything you might ever need to know to function) at this point you need a second service to backup your first service just to be sure you don't lose everything over some stupid automation you cannot influence. It's like an older HN post (week or 2 ago) about someone being transferred between systems during a company merger and the system deciding that he is fired. No human could influence it, everything was revoked and removed automatically and security just acted on what the computer was saying, and escorted him out. They had to wait until the automated process was done before he could re-enroll in the system and continue to work. That, and GC (and to an extent others like PayPal, Amazon, Apple, MS) are examples of trying to put nuances in human interactions into algorithms and failing, have bad real-world effects.
Google being Google (and this probably goes for many tech leaders) probably thought that using an algorithm and some documentation would be enough, even if it means about 1% of customers slips through the cracks and loses all of their Google-stuff. The problem is that in the real world this generates enough backlash that people don't want or trust your services anymore.
While a cloud and even a more classical MSP is supposed to take what they are good at and make it available at a reasonable price (so you can focus on what you are good at instead of everything you might ever need to know to function) at this point you need a second service to backup your first service just to be sure you don't lose everything over some stupid automation you cannot influence. It's like an older HN post (week or 2 ago) about someone being transferred between systems during a company merger and the system deciding that he is fired. No human could influence it, everything was revoked and removed automatically and security just acted on what the computer was saying, and escorted him out. They had to wait until the automated process was done before he could re-enroll in the system and continue to work. That, and GC (and to an extent others like PayPal, Amazon, Apple, MS) are examples of trying to put nuances in human interactions into algorithms and failing, have bad real-world effects.
To be clear, there was always a human in the loop here. We've added a second one, just to be sure.
That's good to hear! What is the general ratio between humans-servicing-humans and humans-needing-service?
A note on offline billing, it seems like this is only available for people paying $2.5k/mo and up. I tried to find out more information about it two weeks ago and got this response:
*
Hello Daniel,
I'm <Sales Rep> and you have reached the Google Cloud Platform sales development team. I understand that you're looking for an Invoice Billing.
Prior to applying for monthly invoicing, please review the following minimum requirements to determine if you are eligible to apply. These requirements include, but are not limited to:
1. Being registered as a business for a minimum of one year.
2. Spending a minimum of $2,500 a month for the last 3 months.
You can also look through the Cloud Billing Help Center which should hopefully provide answers to your questions. You can also contact the billing team through the Cloud Console Billing Support Request Form in the event you need more assistance.
Thank you.
*
Hello Daniel,
I'm <Sales Rep> and you have reached the Google Cloud Platform sales development team. I understand that you're looking for an Invoice Billing.
Prior to applying for monthly invoicing, please review the following minimum requirements to determine if you are eligible to apply. These requirements include, but are not limited to:
1. Being registered as a business for a minimum of one year.
2. Spending a minimum of $2,500 a month for the last 3 months.
You can also look through the Cloud Billing Help Center which should hopefully provide answers to your questions. You can also contact the billing team through the Cloud Console Billing Support Request Form in the event you need more assistance.
Thank you.
Google improving their support? I'll see it when I believe it.
We used to pay for gold level support. It's hilariously bad.
I once had a production outage that ended up being a low-level GKE bug. Support de-escalated my ticket and linked me to this blog post: https://cloud.google.com/solutions/scalable-and-resilient-ap...
I ended up getting help by going into the google slack and kicking up a fuss. Never heard back from the "official" support channels.
We used to pay for gold level support. It's hilariously bad.
I once had a production outage that ended up being a low-level GKE bug. Support de-escalated my ticket and linked me to this blog post: https://cloud.google.com/solutions/scalable-and-resilient-ap...
I ended up getting help by going into the google slack and kicking up a fuss. Never heard back from the "official" support channels.
Sounds just like SUSE Enterprise Support, quality wise. When in doubt "we don't support it" or "we don't support it like that".
This is encouraging that Google listens to it's customers' concerns and candidly addresses them.
Have you heard of marketing before?
Online customers with established payment history, operating in compliance with our TOS, AUP and local laws, will receive advance notification
Isn't this a meaningless promise since a customer that's been compromised is likely to be violating their TOS or AUP?
So, for example, a customer that's had their webserver hacked and used to host some malware can still have their entire account shut down (as opposed to just the compromised server).
Isn't this a meaningless promise since a customer that's been compromised is likely to be violating their TOS or AUP?
So, for example, a customer that's had their webserver hacked and used to host some malware can still have their entire account shut down (as opposed to just the compromised server).
It's great that Google is taking these steps. An over-reliance on automation can be hazardous.
Where money is immediately on the line, second line human review is still a requirement for reliable operation - at the world’s most commercially successful ML powered company.
What event happened to make google come out with this blog post? I missed the hub bub apparently
Can someone link the original blog post and its HN thread? I'm having trouble finding it.
jbob2000(1)
What incentives exist for Google employees to fix these problems at the lowest levels other then feeling like they work for a great company? If I'm making 300k on Monday, delete someones entire production stack to great disdain on Tuesday, and am still making 300k on Wednesday, how incentivized am I really? Google's macro level incentives based on contrived metrics have proven to not be very good i.e. "let's launch a chat app" style promotions and bonuses. Presumably there are high level metrics for "customer success" but I feel like those are too hollow to solve this.
I'm interested in how this hole was actually dug in the first place, presumably by intelligent and well paid individuals, and how it might be fixed. Maybe its already on the path to being fixed and it'll be great to recap in a couple years, maybe these attempts will also prove hollow. Time will tell. They deserve some credit as individuals for trying.
I'm interested in how this hole was actually dug in the first place, presumably by intelligent and well paid individuals, and how it might be fixed. Maybe its already on the path to being fixed and it'll be great to recap in a couple years, maybe these attempts will also prove hollow. Time will tell. They deserve some credit as individuals for trying.
There's no law barring the having ordinary good intentions while also earning $300k. Just as there isn't a law against being a jerk at any price.
I'm more curious how Amazon avoided it. Switching from low-touch/trust consumer culture to high-touch/trust enterprise culture is tough.
What they are really trying to say is “Google Cloud is now less likely to delete your entire business computing systems over some minor credit card issues.” For this reason, this press release will not get the attention that google wants it to.
Note however that google has included human reviewers in the process and googles humsn reviewers are notorious for being ruthless.
It’s googles attitude to customers that is the ingrained and deep problem. Google has built into its DNA from the beginning that it does not want to be in contact with you, that you cannot get emergency attention, and that you are probably wrong.