Student Charged $14k on Stolen Google Cloud Credentials
36 comments
Sorry this happened to you! Feel free to send me your case number (email in profile), and I'll escalate it.
The Support personnel have hopefully been helping out, as all Billing Issues are covered regardless of support tier. I obviously don't know the ins and outs of payment instrument refunds / do debit cards mean that you actually do have to contact your bank, but I'm sure people in Support do.
The Support personnel have hopefully been helping out, as all Billing Issues are covered regardless of support tier. I obviously don't know the ins and outs of payment instrument refunds / do debit cards mean that you actually do have to contact your bank, but I'm sure people in Support do.
Thanks for responding to this and offering to escalate boulos.
I certainly don't think a student just learning the ins and outs of a cloud provider's services should be able to spend 10k+ without warnings/thresholds that require configuration to exceed. It would be positive for platform adoption to make that process better.
I certainly don't think a student just learning the ins and outs of a cloud provider's services should be able to spend 10k+ without warnings/thresholds that require configuration to exceed. It would be positive for platform adoption to make that process better.
This is hilarious. Student doesn’t understand security in depth model, gets owned. Has a sour taste with said cloud provider. At what level do you accept responsibility for shoddy security practices. If the project was truly defunct then you should have closed the project or removed everyone’s access who isn’t project owner.
Hindsight is 20/20.
Nice victim shaming you got there. The fault of all of this is 100% on whoever stole the credentials and made those calls. OP could maybe have been more careful but that doesn't mean it's all his fault or that we should be shaming in oblivion. And Google can still try to help rather than just take advantage of the situation. Life is easier when we are not dicks to each other, a little empathy can go a long way.
>>>Life is easier when we are not dicks to each other, a little empathy can go a long way.
I found my new email signature.
I found my new email signature.
I am seriously wondering whether it's a good idea to put it in my business email or not :D
I think a lot of the problems we face today are largely unacknowledged by those who create them. The delivery lacked any empathy but does not make it less true. Actions have consequences and ignorance of those outcomes doesn’t mean you can escape them. The world would be a little bit better of a place if we’d stop coddling those who float haphazardly through their own existence.
> The world would be a little bit better of a place if we’d stop coddling those who float haphazardly through their own existence.
Why? Do you never ever make mistakes?
Why? Do you never ever make mistakes?
> At what level do you accept responsibility for shoddy security practices.
I agree completely-- Google's practices are terrible here. Who in their right mind would render $14,000 worth of services to a customer for which no due diligence was performed? They never stopped to make sure someone whose usage went from zero to the stratosphere was legit or has the ability to pay such a bill?
No other industry would do something so amateur. Lawyers work on retainer. Bartenders will preauthorize your card before letting you clean out the top shelf. Landlords do credit/background checks before letting you assume tenant rights under their roof. Steam will block your credit card until verification if you buy one too many hats. Know your fucking customer!
eFax and stamps.com are the only other businesses I'm aware of who do stuff like this, and it's done by design. You forget to cancel your free trial or account, they'll let the subscription bills accrue into the thousands and then send debt collectors after you to shake you down for a settlement.
I agree completely-- Google's practices are terrible here. Who in their right mind would render $14,000 worth of services to a customer for which no due diligence was performed? They never stopped to make sure someone whose usage went from zero to the stratosphere was legit or has the ability to pay such a bill?
No other industry would do something so amateur. Lawyers work on retainer. Bartenders will preauthorize your card before letting you clean out the top shelf. Landlords do credit/background checks before letting you assume tenant rights under their roof. Steam will block your credit card until verification if you buy one too many hats. Know your fucking customer!
eFax and stamps.com are the only other businesses I'm aware of who do stuff like this, and it's done by design. You forget to cancel your free trial or account, they'll let the subscription bills accrue into the thousands and then send debt collectors after you to shake you down for a settlement.
To close the loop here, Support fixed this within a few hours of opening a billing support case.
Not to blame the victim, but at the time the original charges happened, OP didn't respond to the Support person for 3-days (our usual timeout) on "Hey, is this resolved on your side?". Tell them no! :). Glad this got resolved quickly though.
Not to blame the victim, but at the time the original charges happened, OP didn't respond to the Support person for 3-days (our usual timeout) on "Hey, is this resolved on your side?". Tell them no! :). Glad this got resolved quickly though.
I’ve heard so many stories of something similar happening on AWS and after an email to support, all of the charges were dropped.
This isn’t exactly helping Google to fight the narrative that it isn’t good with customer support and they can’t be trusted as a platform for business.
So if you were a person deciding who to choose as your cloud provider, who are you going to choose?
AWS - “No one ever got fired for choosing AWS”
Microsoft - well known for their enterprise support and there are plenty of MS Shops out there.
Or
Google?
This isn’t exactly helping Google to fight the narrative that it isn’t good with customer support and they can’t be trusted as a platform for business.
So if you were a person deciding who to choose as your cloud provider, who are you going to choose?
AWS - “No one ever got fired for choosing AWS”
Microsoft - well known for their enterprise support and there are plenty of MS Shops out there.
Or
Google?
I agree with you 100%. This has certainly left a sour taste in my mouth. Unfortunately Google Maps API is much better than the competition. AWS has none here.
The google customer experience is just horrific.
The google customer experience is just horrific.
While you're figuring this out, backup all your data on Google. Google is crazy and could possibly delete all your accounts and data.
Backing up now, thanks. Even worse I'm an Android user. I don't have a good solution for the people that mail me every few months. Oh hey, Google might lock my account because I didn't fork over 14k in fraudulent charges, can you start emailing me from my Outlook email instead?
The idea email is your own domain name. I’m yet to take my own advice though!
Before disputing the charge, be sure to back up all data and contact info from your Google accounts. Fighting charges has been known to trigger account lockouts with no appeal.
This is even more terrible as an Android user. So much for "don't be evil". Thankful for the HN forum.
Hope it works out with a minimum of hassle!
Did you check your Github repos and associated commit history for accidental push of secret files? There's an article on the HN front page describing secret leakage in Github repos (the most common is Google API keys, go figure)[1]. I imagine somebody out there has a bot to monitor pushes in realtime to extract secrets. You or a team member might have leaked keys in a similar manner.
[1]: https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-chara...
[1]: https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-chara...
Google will typically waive charges in cases like this.
The only time they won't is if (by looking at the logs) they decide you were probably scraping and storing all their data.
The only time they won't is if (by looking at the logs) they decide you were probably scraping and storing all their data.
[deleted]
Make them prove you used it to generate the charges. Make them provide IPs etc.
You need to say it was used fraudulently and you don’t agree to the charges.
You need to say it was used fraudulently and you don’t agree to the charges.
Will do, thanks for the advice. The only thing they did was validate the API count, but I was not disputing this, only that I made the calls.
The call volume itself looks suspicious. No calls for well over a year and then suddenly an incredible amount? You would think Google would have some algorithm to detect this?
The call volume itself looks suspicious. No calls for well over a year and then suddenly an incredible amount? You would think Google would have some algorithm to detect this?
Yeah exactly. But the key is not to say “lower my bill”, but rather “these charges aren’t mine”.
If they send it to collections you can dispute it. A lot of laws and things around that.
It sounds like it hasn’t been charged off yet though, since they want you to pay Google, not them.
I wish you luck. :)
If they send it to collections you can dispute it. A lot of laws and things around that.
It sounds like it hasn’t been charged off yet though, since they want you to pay Google, not them.
I wish you luck. :)
Can you get any information about the party who stole your credentials? What were the domains and urls? What is the domain whois (probably private), their IP's? Where are they hosting? This information might help you make a case of fraudent abuse of your GCP account.
Will try to ask, seems unlikely they have tools out of the box for billing support to do this. I bet they have to have an SWE grep logs. Thanks for the advice!
Google doesn't have any burden of proof here. You're responsible for your api keys and any use that occurs with them.
Next use a credit card. Basically thanks to credit card laws the bank will go tell google to f off and give you your money back. Debit cards don't have the same protection, but just call your bank or OCS (https://www.occ.treas.gov/). They have a little more bite.
His debit card ended up declining it anyway, but that's not the point. The point is Google thinks he owes them thousands of dollars, and a collection agency is hounding him for the money.
Disputing a charge with your credit card company doesn't change that. The credit card company might side in your favor and reverse the charges, but the original company can still pursue payment with you directly. Their claim can even show up on your credit report. I remember working with a payment processor in Europe that even automated sending a bill to collections if it was charged back by the consumer's credit card company.
Disputing a charge with your credit card company doesn't change that. The credit card company might side in your favor and reverse the charges, but the original company can still pursue payment with you directly. Their claim can even show up on your credit report. I remember working with a payment processor in Europe that even automated sending a bill to collections if it was charged back by the consumer's credit card company.
Us credit card law is pretty strict. Don't know about Europe. I don't see Google trying it.
playing devils advocate, but taking peoples word prima facie is dangerous. Especially when it deals with financial issues. Pretty sure you can any contempt person could game the system if people found out “students” automatically get refunds for bs charges. I think AWS is different as they have their educate program and distinguish between students and free trial customers. AFAIK gcp doesn’t have an education system for students, so it’s harder to differentiate a legitimate request
What will these system gamers do with computer power? Mining crypto will make them pennies, and running a business on stolen accounts isn’t sustainable.
there are entire dark market businesses built out of finding secrets and deploying a crypto miner as fast as possible and mining as much as possible.
that's just one use case, DDOSing an API/website through seemingly legit requests from the G3's cloud infrastructure is another.
Lots of things you can do with massive amounts of computing power. It doesn't have to be sustainable in the short term, you just keep repeating the process and the value adds up fast
that's just one use case, DDOSing an API/website through seemingly legit requests from the G3's cloud infrastructure is another.
Lots of things you can do with massive amounts of computing power. It doesn't have to be sustainable in the short term, you just keep repeating the process and the value adds up fast
He was a student so he may not have had access to a credit card when he started the account. But yes, a credit card will have more protection.
Keep trying to talk to them and explain -- so far every instance like this I've heard about was refunded. Good luck.
I dunno if google lets you do this but amazon/azure will pretty reliably let you create new free tier accounts with fake emails and access them from the same IP. i just create a new debit/credit card every 6 months(it's pretty hassle free in india).
i do pay for production instances, i just don't want to mess around on those production instances
i do pay for production instances, i just don't want to mess around on those production instances
In 2017, I made a Google Cloud Account to use Google Maps API for a Computer Science student group project and put my debit card in. I naively put a $5 account notification in, thinking it was a cap. This project was defunct after 2017 and I should have just closed the Cloud account.
All was fine up until January 2019 when the Google Cloud Credentials were somehow stolen and over the course of two days on Google Maps API, racked up enough API calls to generate over $14k invoice. I disabled the Google Cloud Account a day after I noticed an email from Google Cloud. Google Cloud did try to use debit card to deduct from checking account, but I don't leave thousands sitting around in it, so charge was declined.
I talked to Google Cloud Billing and they have not been helpful, telling me to contact my bank. Today, I got a scary email from a collections agency demanding I login to my Google Cloud account and pay the bill! Worst part is, this API used to be free, until Google started charging exorbitant amounts for it.
I know I did not make these API calls -- if you looked at the call volume history, there was nothing for well over a year, until those two days in 2019, it started going crazy (and the project is not running on any server or being used in any way). I suspect a group member might have accidentally leaked the credentials.
I know AWS has waived costs[1] like this in the past, but Google is not known for customer support. I should have been more proactive in setting up a cap.
Appreciate any advice or Google contacts to talk to an actual human. Should I see if Google is willing to actually verify this was unauthorized usage or just lower the bill? I'll eat a few thousand just to make this go away.
To say GCP has left a sour taste in my mouth is an understatement!
Thanks for reading.
[1] https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn