Ask HN: Anyone else getting a 500 on their Google Calendars?
Why do we even have status pages anymore.
87 comments
I thought I and all my colleagues were getting 500s, but then I checked the status page and saw a green check. Silly me!
A status page properly done is just a link to HN.
That very much is what https://downdetector.com/ and https://downforeveryoneorjustme.com/ are, crowd-sourced downtime reports. HN as a whole doesn't care if some niche industry site is up/down, so would never surface past the 'new' page. It's also helpful to see if a problem is localized to a particular region.
Hah, I just did the same thing - before clicking through to the status page, I had a bet with myself that everything would be green. Imagine my surprise when indeed everything was green.
If companies like Amazon and Google are going to outright lie on their status pages, why bother having them at all?
If companies like Amazon and Google are going to outright lie on their status pages, why bother having them at all?
Because somebody (customer) is monitoring the status page and measuring how long it was not green and hold you accountable for your service being down. So it‘s „better“ to update your status page afterwards when your legal team is all on board.
That doesn't answer the question of why bothering to have them at all.
Partially. Don‘t have one: customers complain. Have one, but with lies: customers complain. Have one that shows the truth: legal department complains.
Because it's very profitable to be able to say that you have 5 nines (99.999%) uptime, over your competition with 4 nines or less.
For the afterwards part, I assume
>So it‘s „better“ to update your status page afterwards when your legal team is all on board
>So it‘s „better“ to update your status page afterwards when your legal team is all on board
Because somebody (customer) wants to monitor a status page
But that purpose was explicitly undermined by the status page not providing the actual status!
>„better“
Is the initial double quote at the bottom something that is language specific?
Is the initial double quote at the bottom something that is language specific?
It seems to be German: https://en.m.wikipedia.org/wiki/Quotation_mark
I did not explicitly create them. HN-app on iOS did them by itself.
And it‘s probably specifiy to German.
And it‘s probably specifiy to German.
Technically, German should have them, but barely anyone bothers since they're not on a normal keyboard layout. Some word processors replace them automatically though.
Also, the upper one is “, not ".
Also, the upper one is “, not ".
Status pages of big companies like Google, Facebook etc. seems to never work. They are all green even when no one can access the service
Sounds like they don't have a status page, but just a placebo.
It probably causes no end of headaches fir IT departments and ISPs:
*"The site is up, I just can't get to it. Fix my network!"
*"The site is up, I just can't get to it. Fix my network!"
I don't think status pages are so easy in practice. Me and everyone I know have not had any problems with google calendar today, so who is right? Together we're probably <0.001% of all google calendar users so I don't think our sample is very representative.
Presumably there is some internal threshold for when to flick the switch on the service dashboard, but without knowing anything about the scale of the outage or what the threshold is we're kinda shooting in the dark.
Presumably there is some internal threshold for when to flick the switch on the service dashboard, but without knowing anything about the scale of the outage or what the threshold is we're kinda shooting in the dark.
That's why you see the orange color instead of red. To indicate that 1. The survive is not fully functional, or 2. The service is not functional for some regions. And I've seen some companies add comments to each status to explain what's going on.
I'm sure the statuses are being very liberal at their reporting thresholds towards the end user. /s
Perhaps it would help if the dashboards gave a tiny inkling of transparency, like what the thresholds are so you could gauge the relative significance of your personal service outage. If my page is out and vendor shows green, then maybe those relying on my service ontop of vendor won't believe me as much when I pass blame to them and/or maybe the vendor isn't even aware of the issue, so I should contact them because I'm an edge case.
If my service is out and the status page shows red with a nifty "were working on it" I can very clearly show anyone I report to or who is dependent on my service the parent service outage causing the issue, maybe even link them to it. In addition, I know that I don't need to contact the vendors support because they're aware of the issue and actively working on resolving it. I'm sure their support staff will appreciate it that millions of people can avoid contacting their support service reporting the same outage and having to repeat the same information through more costly reporting mechanisms than a page everyone can simply observe.
Every power company I've been with for at least the past 10 years basically do this. If I'm out at work and come home to no power, I first load my power company's outage report site (service status dashboard). I check my area and magically I know if the power company is aware and likely working on it, I can even see their updated time estimates when service will be restored. Sometimes they even tell me what caused the issue (oh boy, information). I never even have to make a call.
Meanwhile if I'm sitting here sipping coffee and hear a loud noise down the street, power is out in the middle of the day and I look across the street and see the neighbor's interior lights on, check the power company's outage map and don't see it: I give them a call. Tada, I know my isolated outage will be addressed and the service provider is aware of the issue. I also know if it looks limited, I'm probably going to be a lower priority if there are multiple outages and limited maintenance staff, so I can sort of expect it may be several hours. I can head out and not waste my time and effort sitting around.
Let's treat critical internet service infrastructure (that's what 'cloud' wanted to be, so let them have all that entails) like we treat other critical service infrastructure. It works quite well.
The underlying issue I see is that my utilities are well regulated and watched over so there often is a lot of transparency about what's going on. Heck, they even have public hearings if they want to increase their prices. Meanwhile, private company offering some service isn't very regulated and has every incentive not to let you know to manage the perception of their image and reliability vs providing empirical actual data. So will we see any sort of transparency? I doubt it, just as I highly doubt the size of represntative truthfulness of any data they report.
Perhaps it would help if the dashboards gave a tiny inkling of transparency, like what the thresholds are so you could gauge the relative significance of your personal service outage. If my page is out and vendor shows green, then maybe those relying on my service ontop of vendor won't believe me as much when I pass blame to them and/or maybe the vendor isn't even aware of the issue, so I should contact them because I'm an edge case.
If my service is out and the status page shows red with a nifty "were working on it" I can very clearly show anyone I report to or who is dependent on my service the parent service outage causing the issue, maybe even link them to it. In addition, I know that I don't need to contact the vendors support because they're aware of the issue and actively working on resolving it. I'm sure their support staff will appreciate it that millions of people can avoid contacting their support service reporting the same outage and having to repeat the same information through more costly reporting mechanisms than a page everyone can simply observe.
Every power company I've been with for at least the past 10 years basically do this. If I'm out at work and come home to no power, I first load my power company's outage report site (service status dashboard). I check my area and magically I know if the power company is aware and likely working on it, I can even see their updated time estimates when service will be restored. Sometimes they even tell me what caused the issue (oh boy, information). I never even have to make a call.
Meanwhile if I'm sitting here sipping coffee and hear a loud noise down the street, power is out in the middle of the day and I look across the street and see the neighbor's interior lights on, check the power company's outage map and don't see it: I give them a call. Tada, I know my isolated outage will be addressed and the service provider is aware of the issue. I also know if it looks limited, I'm probably going to be a lower priority if there are multiple outages and limited maintenance staff, so I can sort of expect it may be several hours. I can head out and not waste my time and effort sitting around.
Let's treat critical internet service infrastructure (that's what 'cloud' wanted to be, so let them have all that entails) like we treat other critical service infrastructure. It works quite well.
The underlying issue I see is that my utilities are well regulated and watched over so there often is a lot of transparency about what's going on. Heck, they even have public hearings if they want to increase their prices. Meanwhile, private company offering some service isn't very regulated and has every incentive not to let you know to manage the perception of their image and reliability vs providing empirical actual data. So will we see any sort of transparency? I doubt it, just as I highly doubt the size of represntative truthfulness of any data they report.
Appears so. We can expect a productivity spike as many people are missing their meetings at the moment.
https://downdetector.com/status/google-calendar/
https://downdetector.com/status/google-calendar/
Yes, both Gmail and Calendar are impacted.
You can still access your calendar via https://calendar.google.com/u/0/embed?mode=week
This brings back the old UI (Google Agenda times).
I'm amazed by how little that page downloads. I'm getting less than 1MB here. Gmail downloads 20MB to display my inbox.
You can still access your calendar via https://calendar.google.com/u/0/embed?mode=week
This brings back the old UI (Google Agenda times).
I'm amazed by how little that page downloads. I'm getting less than 1MB here. Gmail downloads 20MB to display my inbox.
I've used html-mode gmail since way back when. it's really nice, fast and simple. Unfortunately at some point several months ago, google stopped respecting having set html mode as default. the only way to get it now is to append "/h" at the end of the gmail link manually (or click the "use html mode" button while it loads, but that usually leads to it registering the click just as it finished loading, trashing a random email because the trash button is in the same location of the screen)
it's genuinely so frustrating, especially because if you use the html-link and then have to log in, it sends you back to javascript gmail again. I think this might actually be the thing that'll get me to switch from gmail, which I've been too lazy to do, but meaning to do for ages now
the worst part I think is that you can't middle click to open new tabs. I don't know if it's just me, but I constantly find myself wanting to cross reference two things that require multiple tabs, and SPAs never let you open anything in a new tab (even though they could if they bothered to implement it).
it's genuinely so frustrating, especially because if you use the html-link and then have to log in, it sends you back to javascript gmail again. I think this might actually be the thing that'll get me to switch from gmail, which I've been too lazy to do, but meaning to do for ages now
the worst part I think is that you can't middle click to open new tabs. I don't know if it's just me, but I constantly find myself wanting to cross reference two things that require multiple tabs, and SPAs never let you open anything in a new tab (even though they could if they bothered to implement it).
> SPAs never let you open anything in a new tab (even though they could if they bothered to implement it).
There's really nothing special about SPAs that prevent opening new tabs with middle click. All they need to do is use a href element instead of, e.g., a span with a click handler. For some inexplicable reason, gmail has opted for the latter even though most click targets update the url and navigate the user somewhere.
It just seems lazy to me.
There's really nothing special about SPAs that prevent opening new tabs with middle click. All they need to do is use a href element instead of, e.g., a span with a click handler. For some inexplicable reason, gmail has opted for the latter even though most click targets update the url and navigate the user somewhere.
It just seems lazy to me.
It's like everyone using these bloated JS front ends has forgotten that. Google, Reddit, Wells Fargo, Gusto, Twitter. They are actively breaking the built in browser features.
It's nowhere near as simple as that. Many SPAs don't even have URLs for referring to the items they display, so they'd also need to invent a complete URL for everything they might display and then also handle people visiting that URL. And if you change URLs in an SPA, you then need to interact with the browser history API, which, historically, has had a large number of disastrously bad bugs.
Things don't exist by default and every feature takes effort.
Things don't exist by default and every feature takes effort.
For gmail it is that simple. They've already solved providing unique URLs for all the internal app paths I bothered checking.
For SPAs in general, it's true that there is some extra work in gluing URLs and app state together, but there's no lack of libraries that make that process easy and painless.
For SPAs in general, it's true that there is some extra work in gluing URLs and app state together, but there's no lack of libraries that make that process easy and painless.
a lot of SPAs do this already to make page refreshes not kill state though, but still don't bother with links
Of course for GMail that would result in taking 30s to load the email. If it wanted to have reasonable performance for opening the message in a new tab they would need to do some magic to transfer just the state required.
You can use an IMAP client. But for me, the initial unsubscribe from everything irrelevant was painful.
Ah that's a pity :(
I got excited by the first sentence. Wish they had kept it around.
well, it still is around, it's just a hassle to get to these days, rather than loading automatically for those who opted into it. I doubt they even broke it on purpose, it's just one of those things that no one at google cares about, so it gets broken by something and no one notices or cares, and it'll never ever get fixed, because it's not like you can notify anyone at google who would care or have power to do anything.
I found a similarly great UI pattern the other day when leading a conference call. I clicked to toggle the attendee list, only to find that when I clicked a second time to hide it again, the controls had moved across so my mouse was over the hang-up button.
Joy.
Joy.
Have you recently cleared your cache? For me gmail transfers 1.5 MB before displaying inbox and loading 40MB resources from cache. I am using default gmail config and latest chrome. I have 10s of thousands messages in inbox.
That might be it. But isn't it still nuts it loads such a huge amount of resources? 40MB is even worse
After refreshing a few times, I'm now greeted by impossible to solve captchas.
https://imgur.com/a/GQhvUen
Also, this isn't normal captchas where you can get sounds or stuff. These are completely inaccessible for many people.
https://imgur.com/a/GQhvUen
Also, this isn't normal captchas where you can get sounds or stuff. These are completely inaccessible for many people.
This is a typical anti-bot Google infra CAPTCHA. Happens if you suddenly fire off many requests, well known to anyone who ever attempted to scrape Google or even did a very quick burst of searches while trying to solve a problem.
Did it not accept "contour"?
I couldn't figure out the "t" on that one. Looked like an "r". I'd fail that one too.
I see conrollr
Is contour a valid Norweigan word?
And of course, like yesterday's AWS outage, their status page is all green: https://www.google.com/appsstatus/dashboard/
These pages are so useless. For my bank (which has its outages) I use this page: [0], which immediately starts to spike when there is something going on, while the official status page [1] lags by hours at best and never reports anything at worst. There is probably some money to be made by allestoringen.nl ;)
[0]: https://allestoringen.nl/storing/rabobank/
[1]: https://www.rabobank.nl/particulieren/storing/
[0]: https://allestoringen.nl/storing/rabobank/
[1]: https://www.rabobank.nl/particulieren/storing/
The founder of Allestoringen sold it to Ookla in 2018 [1]. It seems they do have an enterprise offering these days [2].
Re: banks, there is also the iDEAL status page [3].
[1] https://tweakers.net/nieuws/142053/storingswebsite-allestori...
[2] https://downdetector.com/enterprise/
[3] https://beschikbaarheid.ideal.nl/
Re: banks, there is also the iDEAL status page [3].
[1] https://tweakers.net/nieuws/142053/storingswebsite-allestori...
[2] https://downdetector.com/enterprise/
[3] https://beschikbaarheid.ideal.nl/
When you're too big to fail, why maintain a status page?
I guess status pages are a conflict of interest
Rest and vest.
Last week Google Play was crashing on my phone over and over again (making the device an expensive paperweight over time since banking apps have mandatory updates)... I reinstalled and that fixed it for a short while and then after it started perma-crashing again I received the mail about mandatory 2FA, did that and since then no problems.
We need to move away from these big companies NOW!
Get your own HTTP, DNS and SMTP server (in that order), your own domains, host at home with fiber and lead-acid backup!
Make your ISPs enable static IP and open DNS + SMTP ports.
We need to move away from these big companies NOW!
Get your own HTTP, DNS and SMTP server (in that order), your own domains, host at home with fiber and lead-acid backup!
Make your ISPs enable static IP and open DNS + SMTP ports.
Next time that ever happens have something like `adb logcat | tee log.txt` running in the background so you can actually figure out what's crashing.
My old Android phone has gotten into a very sorry state, I badly need to factory reset it to fix it (eh, and maybe root it while I'm at it) once I get some data off of it - but the other day my bank app happened to be crashing, and I was able to determine it was because Chrome's WebView hadn't updated correctly. So (the Play Store has completely wedged itself and the app has disappeared, seemingly a catastrophically failed update) I grabbed an updated WebView APK from Ze Internetz, and yay, fixed.
Although yeah, I can't really recommend that particular strategy as a particularly secure first-class solution...
But watching logcat can be very useful. And you can at least wave the log info above your head like an annoying "I did my due dilligence" flag :P
My old Android phone has gotten into a very sorry state, I badly need to factory reset it to fix it (eh, and maybe root it while I'm at it) once I get some data off of it - but the other day my bank app happened to be crashing, and I was able to determine it was because Chrome's WebView hadn't updated correctly. So (the Play Store has completely wedged itself and the app has disappeared, seemingly a catastrophically failed update) I grabbed an updated WebView APK from Ze Internetz, and yay, fixed.
Although yeah, I can't really recommend that particular strategy as a particularly secure first-class solution...
But watching logcat can be very useful. And you can at least wave the log info above your head like an annoying "I did my due dilligence" flag :P
> mandatory 2FA
> expensive paperweight
Imagine if the expensive paperweight stops working as the mandatory 2FA.
> expensive paperweight
Imagine if the expensive paperweight stops working as the mandatory 2FA.
https://www.google.com/appsstatus/dashboard/incidents/zURR7m...
> 10:40 AM UTC The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
> Incident began at 2021-12-08 08:40 and ended at 2021-12-08 10:20 (times are in Coordinated Universal Time (UTC)).
> 10:40 AM UTC The problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
> Incident began at 2021-12-08 08:40 and ended at 2021-12-08 10:20 (times are in Coordinated Universal Time (UTC)).
Google 500 == new snowday ? :D
I can imagine it now, workers eagerly awaiting the radio announcements over breakfast: "The following regions have Google datacenter outages..."
I can imagine it now, workers eagerly awaiting the radio announcements over breakfast: "The following regions have Google datacenter outages..."
It would be more interesting if the status page also had a test function scoped to the specific user. For something Google Calendar it could OAuth, but for things like public websites/services it could do a HTTP GET (or whatever makes sense).
Obviously there is no problem: https://www.google.com/appsstatus/dashboard/#v=status
But I'm getting 500 anyway ;)
But I'm getting 500 anyway ;)
Yeah; it is 500. But it works for me with `gcalcli`[1] — a Python command-line tool that uses GCal API.
[1] https://github.com/insanum/gcalcli
[1] https://github.com/insanum/gcalcli
Maybe google calendar runs on AWS
Yes, and Google Meet is buggy and slow for me. Couldn't join my standup on desktop (managed on mobile).
Anyone else got opinion that some sites such as social media, calendar apps should have scheduled one day mandatory outage per year by law :-).
Certain customer may opt-out, but by default we should do global unplug once per year.
Certain customer may opt-out, but by default we should do global unplug once per year.
I once read a Twitter thread about the idea of an explicitly flaky AWS region.
It honestly sounds like it would be a really interesting itch to scratch - VMs with explicitly bad network, bad disk, etc. You could specify the severity and randomness of failures, or what days you want things to crash. You could even go nuts and corrupt memory, or randomly trigger GPFs or fault random instructions.
The best part is that the corruption could be cooperative, such that the initial handshakes of TLS connections always succeed, and things like database connections always establish correctly, and PID 1 never faults. So you always end up crashing somewhere obscure, infuriatingly annoying, and very interesting.
I understand that both NFS and rsync-without-SSH don't carry any form of authenticity checksum (IIUC NFS encryption does provide this if you set up Kerberos). Would be fun to corrupt random packets so they get the same TCP+IP checksum, eh...? ;)
And you could even go absolutely ballistic and borrow techniques from fuzzers like AFL to observe and trace long-running executions, and compute ways to trip a program up to get maximally divergent results. I wonder if you could build an inference model for that...
Yeah, this sounds unreasonably fun.
It honestly sounds like it would be a really interesting itch to scratch - VMs with explicitly bad network, bad disk, etc. You could specify the severity and randomness of failures, or what days you want things to crash. You could even go nuts and corrupt memory, or randomly trigger GPFs or fault random instructions.
The best part is that the corruption could be cooperative, such that the initial handshakes of TLS connections always succeed, and things like database connections always establish correctly, and PID 1 never faults. So you always end up crashing somewhere obscure, infuriatingly annoying, and very interesting.
I understand that both NFS and rsync-without-SSH don't carry any form of authenticity checksum (IIUC NFS encryption does provide this if you set up Kerberos). Would be fun to corrupt random packets so they get the same TCP+IP checksum, eh...? ;)
And you could even go absolutely ballistic and borrow techniques from fuzzers like AFL to observe and trace long-running executions, and compute ways to trip a program up to get maximally divergent results. I wonder if you could build an inference model for that...
Yeah, this sounds unreasonably fun.
I vote for once per quarter, with certain exceptions which have to keep running such as utilities etc.
Outage should take a minimum of 8 hours
Outage should take a minimum of 8 hours
As a (virtual) pager carrier, you've got my vote to mandate it for every service!
I had a problem, too. Intermittent 500, then Offline. Seems to work right now.
The calendar sidebar in mailbox does not load, but opening the full calendar seems to work.
Interestingly, it asked me to fill a captca test. Could it be an attack?
Interestingly, it asked me to fill a captca test. Could it be an attack?
>getting a 500 on their calendars
If you have 500 meetings in Gcal on anything but a multi-year view, you may want to consider a new job.
If you have 500 meetings in Gcal on anything but a multi-year view, you may want to consider a new job.
i've been getting a 'server problem [400]' in my music.youtube app on android for weeks when trying to play my own uploaded albums. reported through every channel i could find. i think nobody at G. is even trying to pretend they care.
companies die once their legal dept. ends up in charge of everything. at G this happened around 2010.
companies die once their legal dept. ends up in charge of everything. at G this happened around 2010.
Seems like an intermittent issue... Got one or two 500 errors in the past 5 minutes, sometimes just loads very slowly.
So now Google even copies AWS outages...
And their status page. All green in both cases.
[deleted]
Nope - all good here. Personal and corporate Gmail/Gcal working currently.
Working for me -- Seattle.
Yes. We did. But it seems to be back now.
Slower than usual to load but have access
Gotta love that SaaS reliability
Yep, same here (in the EU).
I did