Google’s painful Gmail OAuth verification process(aura.app)
aura.app
Google’s painful Gmail OAuth verification process
https://aura.app/blog/google-oauth-verification/
51 comments
Stop using it and show google you don't care about their "services" as market dictates.
To the contrary it will be very good for innovation because it means people will build extensions for other email providers such as Fastmail.
No. If you want, build an email app with features or with it's own plugins. Not extensions around closed, single point of failure services.
Does Fastmail even provide an API like this?
Sure, IMAP, POP, CALDav, same as all the other mail providers...
And the optimized JMAP protocol too.
The fact that I am getting voted down proves how absolute the groupthink is... If you work in Google's ecosystem, however, the only possible exit is being bought by Google... Get bought by a competitor and they'll just turn you off.
And the optimized JMAP protocol too.
The fact that I am getting voted down proves how absolute the groupthink is... If you work in Google's ecosystem, however, the only possible exit is being bought by Google... Get bought by a competitor and they'll just turn you off.
So the answer is no then, because Gmail still provides imap and pop, which are unaffected by this.
Both disabled by default (ridiculous for an email provider), and if you enable 2FA get ready to do some yak shaving.
But how does the credential management work?
I don't recall since the last time I set it up was a while ago, but I think just your password + an app specific code if you use 2fa normally. But yeah I'm pretty confident this doesn't apply to outlook.
So why don't we see more startups use open protocols for access to email rather than make things that are GMail specific?
Why do you get voted down for just suggesting that they do so?
Why do you get voted down for just suggesting that they do so?
I'm not sure, my expectation would be because oauth is more secure and feature rich than either imap or pop. Jmap I'm less familiar with.
IMAP can be awkward with sync, pagination and has never played great with labels vs folders. JMAP looks promising.
As a gmail user - good to hear this. In the long run trust is going to be a much more important commodity that letting a spam app into your gmail.
If you look at the service that want access to all your gmail data - many promise something "free" but then mine that data (in the fine print) to send you offers, alert you to "savings" etc.
I automatically turn down apps that say they need access to my entire google drive and all email. Why not just ask for permissions for a single app specific folder? Ie, fax apps -> they should just store inbound faxes into one folder rather than asking for full drive access.
If you look at the service that want access to all your gmail data - many promise something "free" but then mine that data (in the fine print) to send you offers, alert you to "savings" etc.
I automatically turn down apps that say they need access to my entire google drive and all email. Why not just ask for permissions for a single app specific folder? Ie, fax apps -> they should just store inbound faxes into one folder rather than asking for full drive access.
> Why not just ask for permissions for a single app specific folder?
Basically because Gmail A) doesn't have folders B) doesn't have permissions based on tags. Otherwise most Gmail API apps would have this option.
The same thing is a problem when you want to delegate access to an email account, where there should be a way to delegate access based on tags, but there just isn't.
What you can now do is create Gmail Add-Ons, which only have access to this specific thread that's open when you click to activate the add-on. E.g. this is how we created https://www.prettyfwd.com
Basically because Gmail A) doesn't have folders B) doesn't have permissions based on tags. Otherwise most Gmail API apps would have this option.
The same thing is a problem when you want to delegate access to an email account, where there should be a way to delegate access based on tags, but there just isn't.
What you can now do is create Gmail Add-Ons, which only have access to this specific thread that's open when you click to activate the add-on. E.g. this is how we created https://www.prettyfwd.com
Looks interesting but it’s unclear to me, do you copy the threads content?
One other comment, there seems to be a spammer on the review page...
One other comment, there seems to be a spammer on the review page...
> do you copy the threads content?
Basically we take each message of the thread, normalize the typography so that it can be consistently styled, and then strip off the signatures and quoted reply text while leaving any quoted inline replies. There are some open source libraries that do parts of this, but none of them worked well enough for this particular use case so we ended up just developing some new techniques ourselves.
The other thing the tech does is it lets people run much more accurate NLP on threads. This isn't fully productized yet, so right now there are just a handful of companies we're working with to clean and normalize their email data.
In terms of the spammer, yeah I see that but I'm not sure what I can really do. As the app creator I don't seem to have any special ability to delete comments or flag things as spam.
Basically we take each message of the thread, normalize the typography so that it can be consistently styled, and then strip off the signatures and quoted reply text while leaving any quoted inline replies. There are some open source libraries that do parts of this, but none of them worked well enough for this particular use case so we ended up just developing some new techniques ourselves.
The other thing the tech does is it lets people run much more accurate NLP on threads. This isn't fully productized yet, so right now there are just a handful of companies we're working with to clean and normalize their email data.
In terms of the spammer, yeah I see that but I'm not sure what I can really do. As the app creator I don't seem to have any special ability to delete comments or flag things as spam.
>> Why not just ask for permissions for a single app specific folder
That's not possible. Many people and apps have been asking for more fine-grained permissions from Google APIs for years. It hasn't been done other than a few changes on Android.
This is common with most large providers that have big 3rd party ecosystems but very poor permissions that only offer all-or-nothing access.
That's not possible. Many people and apps have been asking for more fine-grained permissions from Google APIs for years. It hasn't been done other than a few changes on Android.
This is common with most large providers that have big 3rd party ecosystems but very poor permissions that only offer all-or-nothing access.
>> Why not just ask for permissions for a single app specific folder
Dropbox lets apps work on either a single app specific folder, or on your entire dropbox.
But the app has to be one type or the other, the user cannot choose what access they want to give out.
Disclosure: I work at dropbox, previously on the api-platform team.
Dropbox lets apps work on either a single app specific folder, or on your entire dropbox.
But the app has to be one type or the other, the user cannot choose what access they want to give out.
Disclosure: I work at dropbox, previously on the api-platform team.
This is very helpful to see, along with the gmass blog mentioned within. We've been going through this process for months and it's definitely a moving target with no clear path to resolution. The whole process feels a bit Kafkaesque.
As both a gmail user and developer interested in applications to help me manage my personal information, this is incredibly depressing to hear.
The idea of a verification process itself is great, and I applaud that effort. But some of these barriers seems put in place solely to kill competition and prevent startups from filling the personal data needs before Google comes up with its own plan.
These exorbitant fees of $15,000 and $75,000 are completely unjustifiable.
The idea of a verification process itself is great, and I applaud that effort. But some of these barriers seems put in place solely to kill competition and prevent startups from filling the personal data needs before Google comes up with its own plan.
These exorbitant fees of $15,000 and $75,000 are completely unjustifiable.
It's the direction that everyone seems to be moving. Wall up the gardens, remove your own access to your information, and remove the ability to share and integrate across platforms.
The weird thing is how quickly the sentiment turned from my perspective. I felt like one day most technical people applauded the ability to have total real-time access to your data, to be able to write code or use open source code to plug into these systems and augment them for better. To be able to have a startup or small company write software that can work with your google/facebook/apple/whatever account and use the information there in new ways.
Then all of a sudden (I feel like it was between 1-2 years ago, but I can also barely remember what I had for lunch yesterday so don't quote me on that!), technical people started slamming companies for "allowing someone to access their data", I saw lots of headlines about how it was unethical for Facebook to allow users to share their information with other companies (don't get me wrong, facebook does plenty wrong, but to call out the ability to share info specifically seemed so wrong). Then APIs started shutting down, access is now only allowed for other big players, and it's getting harder and harder to integrate outside of a single player's walled garden.
I get why the companies are doing it (someone told them the only ethical thing was to lock users in!?), but I don't get why HN and other technical circles are applauding it. Maybe i'm on the wrong side of history here, but I just feel like it's never a bad thing to allow me to share my information if I want. I think it should be clearly defined what i'm sharing, I think it should be obvious that i'm sharing it, and I think that some auditing and controls are obviously a good thing, but not this almost absolute shutdown of any ability for me to export or use information from these services on my own.
But maybe I'm really in a bubble, and maybe users really shouldn't be given the choice to share their personal information, but it just feels so wrong and so "holier than thou" to make that choice for them.
The weird thing is how quickly the sentiment turned from my perspective. I felt like one day most technical people applauded the ability to have total real-time access to your data, to be able to write code or use open source code to plug into these systems and augment them for better. To be able to have a startup or small company write software that can work with your google/facebook/apple/whatever account and use the information there in new ways.
Then all of a sudden (I feel like it was between 1-2 years ago, but I can also barely remember what I had for lunch yesterday so don't quote me on that!), technical people started slamming companies for "allowing someone to access their data", I saw lots of headlines about how it was unethical for Facebook to allow users to share their information with other companies (don't get me wrong, facebook does plenty wrong, but to call out the ability to share info specifically seemed so wrong). Then APIs started shutting down, access is now only allowed for other big players, and it's getting harder and harder to integrate outside of a single player's walled garden.
I get why the companies are doing it (someone told them the only ethical thing was to lock users in!?), but I don't get why HN and other technical circles are applauding it. Maybe i'm on the wrong side of history here, but I just feel like it's never a bad thing to allow me to share my information if I want. I think it should be clearly defined what i'm sharing, I think it should be obvious that i'm sharing it, and I think that some auditing and controls are obviously a good thing, but not this almost absolute shutdown of any ability for me to export or use information from these services on my own.
But maybe I'm really in a bubble, and maybe users really shouldn't be given the choice to share their personal information, but it just feels so wrong and so "holier than thou" to make that choice for them.
You're confusing 2 things.
Facebook shared data on millions of users to Cambridge Analytica. I never gave my consent, you never did.
It is a different matter than having open APIs that allow you to get your own data out of Facebook.
I want to be able to get my own data on Facebook through open APIs. What I don't want is for Facebook to give away my data to other people without my consent. You can advocate for those 2 things at the same time.
Facebook shared data on millions of users to Cambridge Analytica. I never gave my consent, you never did.
It is a different matter than having open APIs that allow you to get your own data out of Facebook.
I want to be able to get my own data on Facebook through open APIs. What I don't want is for Facebook to give away my data to other people without my consent. You can advocate for those 2 things at the same time.
Both cases are the same.
Alice sends Bob an email and Bob then shares it with an evil 3rd party spell check extension... Alice's privacy has now been breached without her consent.
Alice posts her contact info to her wall, and her friend Bob (who has read access) shares it with a third party (Cambridge Analytica). Alice's privacy has now been breached without her consent.
Cambridge analytica collected the bulk of their private data with the consent of a friend of the victim.
Alice sends Bob an email and Bob then shares it with an evil 3rd party spell check extension... Alice's privacy has now been breached without her consent.
Alice posts her contact info to her wall, and her friend Bob (who has read access) shares it with a third party (Cambridge Analytica). Alice's privacy has now been breached without her consent.
Cambridge analytica collected the bulk of their private data with the consent of a friend of the victim.
You're in part right. In Europe GDPR requires companies to get consent to share data, so it's still possible. Furthermore it requires them to let users (the owners of the data) extract their own data in some common format. That can be used to migrate to other services. However GDPR doesn't require real time webhooks and this is where you're right: no APIs and wallet gardens are a pain. There is an EU directive [1] that is forcing banks to provide open APIs, but that's only one vertical.
[1] https://ec.europa.eu/info/law/payment-services-psd-2-directi...
[1] https://ec.europa.eu/info/law/payment-services-psd-2-directi...
The difference is that the internet happened and it became way too easy for anyone in the world to exfiltrate all your data with a simple free program. This is why we can't have nice things.
Even for non-Gmail apps, this process is incredibly painful. I have an app that has been stuck in the process for weeks. Once you have read through the incredibly confusing and out of date documentation and submit what you think is the correct set of setting to comply with their policy, you then have to deal with the reviewer who will email you once every week if you are lucky. Usually to understand what they are asking you to fix you have to email them back and forth a few times. I love the platform, but they need to fix this aspect of it.
They emailed you once a week? You are indeed lucky.
I guess I need to bug them more, I haven't heard anything in weeks (busy with implementation).
One warning: choose the email address for your Google developer account carefully, there doesn't seem to be a way to change it later. It is forever tied to your permissions and approvals, afaict.
I guess I need to bug them more, I haven't heard anything in weeks (busy with implementation).
One warning: choose the email address for your Google developer account carefully, there doesn't seem to be a way to change it later. It is forever tied to your permissions and approvals, afaict.
We've been in the Oauth review process for almost a month, and getting maybe 1 response per week as well. Plenty of times it's just the reviewer not reading the instructions we sent them and, well, time to wait another week. Then there's them saying our app doesn't need to be verified (apparently reviewer was looking at a different API permission instead of the ones we requested for OAuth), so there's been a bunch of unnecessary back and forth.
We've gone through app review processes at other companies like Facebook, and it's all the same - plenty of time wasted with mostly ineptitude on the reviewer's side. Sometimes it feels like there's just one person working in Google/Facebook's basement doing these app reviews for minimum wage.
I understand the need to be thorough on these app reviews especially if the app touches sensitive user data, but when the reviewer doesn't even read the instructions provided to them properly, would you trust them to be thorough when it comes to ensuring the apps aren't malicious?
We've gone through app review processes at other companies like Facebook, and it's all the same - plenty of time wasted with mostly ineptitude on the reviewer's side. Sometimes it feels like there's just one person working in Google/Facebook's basement doing these app reviews for minimum wage.
I understand the need to be thorough on these app reviews especially if the app touches sensitive user data, but when the reviewer doesn't even read the instructions provided to them properly, would you trust them to be thorough when it comes to ensuring the apps aren't malicious?
Worse still, in six months times the requirements will change and your previously approved scopes will no longer be approved. I've also had to deal with broken OAuth verification forms on Google's site (400 errors from their backend, with no UI feedback), and the complete inability to get a response from a human.
[deleted]
This terrible, but building a business based on third party api's is always a tremendous risk. This isn't the first time a bunch of small apps have been killed off by some company making their api's inaccessible.
Also, for people who are pushing for more government regulation of service providers - this is the lite version of what you are asking for.
Also, for people who are pushing for more government regulation of service providers - this is the lite version of what you are asking for.
> this is the lite version of what you are asking for
I'd say this is the heavy version. A new startup can put together and run a GDPR compliant web app for over a year for far less than $15,000.
I'd say this is the heavy version. A new startup can put together and run a GDPR compliant web app for over a year for far less than $15,000.
As long as no requests that you delete their data or provide them with a copy. As soon as that starts happening $15k doesn't go very far in man hours to comply.
Just because you’re terrible at documenting where you put users’ information doesn’t mean everyone is.
From personal experience, yes, it really does. Why do you think otherwise?
It seems easy because you don't need to prove it. You can do your best and then launch it.
But you can't prove it for under $15k and if you are successful eventually you'll need to prove it.
Anyway, I'm not talking about the GDPR. That's just bureaucrats testing the water. The US congress is starting to look for ways to get their cut as well.
It will get to the point where you need to prove compliance with multiple conflicting regulations first, before launching. Now you need to spend $100,000 on auditors before your first push to heroku.
Most people in software, especially startups, have never had any contact with a regulated industry and don't know what they are in for.
But you can't prove it for under $15k and if you are successful eventually you'll need to prove it.
Anyway, I'm not talking about the GDPR. That's just bureaucrats testing the water. The US congress is starting to look for ways to get their cut as well.
It will get to the point where you need to prove compliance with multiple conflicting regulations first, before launching. Now you need to spend $100,000 on auditors before your first push to heroku.
Most people in software, especially startups, have never had any contact with a regulated industry and don't know what they are in for.
As someone who tried to develop a tool to help pause their box to improve focus throughout the day, I got bit by this process as well.
Basically google just went dark on me altogether. Has been months since their last reply and I kept trying to follow up. The feature I needed elevated permissions on was the ability to add filters, which unfortunately is buried with a bunch of other more dangerous permissions.
Looks like I’ll never get to launch the product :(
On the plus side, it works fine for just me! So, I just built a tool only I can use.
Basically google just went dark on me altogether. Has been months since their last reply and I kept trying to follow up. The feature I needed elevated permissions on was the ability to add filters, which unfortunately is buried with a bunch of other more dangerous permissions.
Looks like I’ll never get to launch the product :(
On the plus side, it works fine for just me! So, I just built a tool only I can use.
This all makes sense to me. If you're not providing enough value to users to cover the >$15k fee, you're just an attack vector for user data.
Consistency of the process aside, I'm really not sure what people would expect.
(I work at Google, yadda yadda, but have nothing to do with this.)
Consistency of the process aside, I'm really not sure what people would expect.
(I work at Google, yadda yadda, but have nothing to do with this.)
Fuck this, it's EXACTLY the problem in the valley. Small players should be empowered, not stifled.
I wouldn't want someone who won't bet $15K-$75K on their own product to have access to any of my data.
Your making a lot of assumptions. For example, who said anything about a product. Not all "players" small or large are selling something. Sometimes people just like to make stuff that works well for themselves and others.
Things like gsyn.ch will just not be written. Google already throws scary warnings about unverified apps using OAuth tokens when you use this service, but Google's use of non-standard CSV entries rather than vCards for sharing contacts has made for a terrible user experience if you want to use your phonebook outside an Android phone/Gmail.
At $10 a month, that’s only 125 users. How many apps with less than 125 users have we seen become a target for hackers? Has there ever been a case like that?
Startups and open source projects get screwed because of Google's lack of nuance on this issue. Google could have created a tiered fee structure based on number of users, threat vector, etc. but didn't for whatever reason.
Goal #1: Don't let the users "get screwed."
Goal #2: Don't have a solution that requires an army of people to manage.
If no one's willing to help fund your idea, you're out of luck. That seems... really understandable, given that every major government is literally investing millions trying to hack into Google's user data.
Goal #2: Don't have a solution that requires an army of people to manage.
If no one's willing to help fund your idea, you're out of luck. That seems... really understandable, given that every major government is literally investing millions trying to hack into Google's user data.
It is not too hard to understand them trying to leverage the potential market of identity providers. Facebooks Libra basically tries to do the same.
It would be a waste to use any services attached to it in my opinion. Otherwise oauth is a great technology, but interests may make it not worthwhile.
It would be a waste to use any services attached to it in my opinion. Otherwise oauth is a great technology, but interests may make it not worthwhile.
A friend and I have been working on a side project that depends on Google Auth to send emails on a user's behalf. It's one of the app's two critical features. We're not necessarily deterred by this story, but we'll start rethinking our dependence on Google.
A $15-75k fee is something that's hard to stomach at our stage. We have about 10 Gmail users excited to try our product and they might not have an issue accepting the "Unverified App" screen because we have earned some of their trust through phone calls and meetings. However, converting people that come across the app organically will be difficult.
We aren't sure when the right time will be to pay the fee and become verified. Anyone have ideas on strategy here? It could help us and other developers in the same position. We'd like to avoid raising money but this might be a good reason to - investors may see Google verification as a competitive advantage.
A $15-75k fee is something that's hard to stomach at our stage. We have about 10 Gmail users excited to try our product and they might not have an issue accepting the "Unverified App" screen because we have earned some of their trust through phone calls and meetings. However, converting people that come across the app organically will be difficult.
We aren't sure when the right time will be to pay the fee and become verified. Anyone have ideas on strategy here? It could help us and other developers in the same position. We'd like to avoid raising money but this might be a good reason to - investors may see Google verification as a competitive advantage.
I'm the author of the Mailspring email client and I've been dealing with this Oauth verification process for the last three months. Mailspring has "pro" features that leverage a small backend API, but it syncs your mail on your computer and your mail data, passwords, tokens, etc. never leave your machine. I care very much about data privacy and I wouldn't use the app myself if it was sending mail data to the cloud.
I'm a big fan of Google watching out for their users. I know of at least one very sketchy company that has shut down because of this new policy, which is great.
But after three months, they basically told me: "Your desktop app makes a network request to a third party server, you must pay $15,000 for a security audit." Their process has been vague and I wish they'd make an effort to understand whether an audit is really necessary. Their security contractors are going to be laughing all the way to the bank as they review my web service that never sees Gmail data in the first place.
Thankfully, Mailspring makes a bit of money and I can afford to do this to keep it alive. But fast-forward a few years and this is going to devastate innovation and development of third party mail clients. (And I think Google prefers it this way.) If the app didn't already have critical mass, or if I was just starting a mail app now, I'd probably throw my hands up and give up rather than emailing them dozens of times and coughing up $15k.
I'm a big fan of Google watching out for their users. I know of at least one very sketchy company that has shut down because of this new policy, which is great.
But after three months, they basically told me: "Your desktop app makes a network request to a third party server, you must pay $15,000 for a security audit." Their process has been vague and I wish they'd make an effort to understand whether an audit is really necessary. Their security contractors are going to be laughing all the way to the bank as they review my web service that never sees Gmail data in the first place.
Thankfully, Mailspring makes a bit of money and I can afford to do this to keep it alive. But fast-forward a few years and this is going to devastate innovation and development of third party mail clients. (And I think Google prefers it this way.) If the app didn't already have critical mass, or if I was just starting a mail app now, I'd probably throw my hands up and give up rather than emailing them dozens of times and coughing up $15k.