Gmail is having an outage(outage.report)

316 pointsby gdeglin7 tahun yang lalu138 comments

https://outage.report/gmail#2019-03-13

148 comments

Someone12347 tahun yang lalu

It isn't just gmail, Google Music, Google Play, Google Drive, and others are also impacted. Seems to be a large scale Google outage of some kind.

cazum7 tahun yang lalu

Google photos is absolutely struggling to load my jpegs right now, though I'm not sure how separate that service is from Drive, if at all.

awill7 tahun yang lalu

I would bet they're both using the same storage layer (Google Cloud Storage?)

r_deckard7 tahun yang lalu

I also have problems with firebase virtual device testing service and google play console api sice yesterday. I keep getting 500 as response, dont know if related though.

jjeaff7 tahun yang lalu

Maybe unrelated but a large portion of the Google Cloud web Console was down yesterday for hours.

noitsnot7 tahun yang lalu

Add YouTube to the list.

jsaxton867 tahun yang lalu

YouTube seems to be in this amazing degraded state for me where videos play just fine, but it isn't showing me ads.

ashelmire7 tahun yang lalu

That state is called “using ublock”.

eeeeeeeeeeeee7 tahun yang lalu

Hah I had the opposite. All my ads were playing fine but not the actual videos.

palerdot7 tahun yang lalu

I was in the complete messed up scenario. Youtube loads ads fine, but after playing them, was not able to load the actual content.

[deleted]7 tahun yang lalu

sitkack7 tahun yang lalu

Youtube was giving me 500s in ways that have never occurred before.

mholt7 tahun yang lalu

Of course this would happen while I'm trying to submit my thesis at the last minute. https://twitter.com/mholt6/status/1105703745143205889

diafygi7 tahun yang lalu

Good thing https://send.firefox.com/ was launched today :)

mholt7 tahun yang lalu

Except I'd still have to send the link in our email thread!

askvictor7 tahun yang lalu

I found that emails seem to work, but attachments don't. So an email with a send link might be exactly the thing. Good luck!

diafygi7 tahun yang lalu

Maybe text the link?

mholt7 tahun yang lalu

I don't have their numbers :(

nurettin7 tahun yang lalu

Time to register for a competitor email service that doesn't use google infrastructure. Live.com, maybe?

ghthor7 tahun yang lalu

Mercury, god of communication, retrograde is kicking it up a notch this year.

9080877 tahun yang lalu

Google forgot to charge their energy crystals?

xbmcuser7 tahun yang lalu

Seems to me a location specific breakdown as all services I tried like gmail, photos and youtube all are working for me

thrusong7 tahun yang lalu

I thought outages weren't supposed to be posted here? Every time I've posted one about Facebook, it's been flagged and removed.

cedricium7 tahun yang lalu

Google Maps (specifically Street View) is showing up as a black screen for me as well.

ssambros7 tahun yang lalu

If anyone is interested: Memorystore for Redis is experiencing intermittent failures during instance creation. Existing instances are not affected.

aj77 tahun yang lalu

Is having to load two tabs of gmail to get my mail to actually appear also a type of outage?

bluedino7 tahun yang lalu

So this is why we couldn't send emails last night (9pm est) Watched Youtube instead :)

config_yml7 tahun yang lalu

Probably caused by a typo in the configuration for the new region they just launched ;)

QuamStiver7 tahun yang lalu

Happy that I was sleeping at that time.

Tsubasachan7 tahun yang lalu

Ah gmail. Where spam goes to die.

byoung27 tahun yang lalu

I was trying to send an email and kept getting "Oops there was a problem" errors, but it appears to be working now.

[deleted]7 tahun yang lalu

sureaboutthis7 tahun yang lalu

No issues here in the Midwest.

cableshaft7 tahun yang lalu

Same, at least not yet. Chicago area here. It does seem slower than usual, but might just be my imagination.

[deleted]7 tahun yang lalu

quickthrower27 tahun yang lalu

All Ok in Sydney

rorykoehler7 tahun yang lalu

Half of my colleagues had issues in Sydney. OK in Singapore apart from images attaching to docs in drive

PudgePacket7 tahun yang lalu

Having issues in Sydney :)

laurencei7 tahun yang lalu

same - I'm in Sydney - no issues (so far).

Perhaps it's user based, rather than location based?

codemati7 tahun yang lalu

Seems to be. I'm on the same network as others not having any issues.

AceyMan7 tahun yang lalu

Can confirm wrt Drive. I was just at work trying to upload files (and one-deep folders) and barely anything completed. I finally gave up.

Our edge/gateway is in metro SEA, fwiw.

ardenpm7 tahun yang lalu

Definitely having problems in Melbourne, Australia

prawn7 tahun yang lalu

In Adelaide also. Things I'm experiencing:

  can't reliably download attachments in Gmail
  messages not getting through in Hangouts
  constant "Oops" auto-save modals when typing Gmail messages

Gustomaximus7 tahun yang lalu

Brisbane too - Google Photos would sync about the time you wrote this comment.

samplatt7 tahun yang lalu

GMail's working fine for me (8min later) in Melbourne, Australia.

fourier_mode7 tahun yang lalu

Works fine in Illinois.

mnyary7 tahun yang lalu

Having problems in Budapest, Hungary.

mav3rick7 tahun yang lalu

What's the tech scene in Budapest like ?

samstave7 tahun yang lalu

He told you already, Hungry.

warp_factor7 tahun yang lalu

it's funny, not sure why you are downvoted

gabeh7 tahun yang lalu

Google seems to acknowledge the reports now. https://www.google.com/appsstatus#hl=en&v=status

gowld7 tahun yang lalu

https://status.cloud.google.com/

https://status.cloud.google.com/incident/appengine/19007

" Elevated error rate with Google App Engine Blobstore API and App Engine Version Deployment

Incident began at 2019-03-12 19:49 (all times are US/Pacific).

Mitigation work is currently underway by our Engineering Team. We will provide another status update by Tuesday, 2019-03-12 20:45 US/Pacific with current details. "

etaioinshrdlu7 tahun yang lalu

Status page is down for me.

kerng7 tahun yang lalu

These are the not that well designed scenarios, when the status pages are hosted on the very same infrastructure that suffers an outage. This is a common pattern across cloud providers I have seen happen.

cheerlessbog7 tahun yang lalu

That page is not very color blind friendly, incidentally. The information is distinguished only by color.

R4V37 tahun yang lalu

Hit that feedback button in the bottom corner.

skrebbel7 tahun yang lalu

You're kidding, right? Sending feedback to Google? I mean isn't that a bit like talking to a tortoise?

rattlesnakedave7 tahun yang lalu

In general, maybe, but they take accessibility concerns very seriously in my experience.

tomcam7 tahun yang lalu

Aren’t the icons all different?

manigandham7 tahun yang lalu

No, they're all circles but in different colors.

EDIT: you might be looking at the Google Cloud status, not the Google services status in the first post.

culturestate7 tahun yang lalu

Not for me - 'available' is a checkmark, 'service disruption' is an exclamation point, and 'unavailable' is an x.

(I notice though that they're images, not text icons, which isn't ideal.)

manigandham7 tahun yang lalu

You're talking about the Google Cloud status page which is in the 2nd post: https://status.cloud.google.com/

This thread is about Google Services linked in the top post which only shows circles: https://www.google.com/appsstatus#hl=en&v=status

crazygringo7 tahun yang lalu

Weird. Clearly Google is serving up different versions to different people.

To me they're 3 different colors of circles, no symbols whatsoever.

culturestate7 tahun yang lalu

It's more likely that your browser just isn't loading the CSS background images for some reason. If you inspect the circles, do you still see the background-image declaration?

bschwindHN7 tahun yang lalu

Here's what I see:

https://imgur.com/IrL5JBo.png

prawn7 tahun yang lalu

You might be talking about the page in the first reply? Not in the post that someone was responding to?

I see all mid-toned circles with no icons (Chrome, Australia).

Pristina7 tahun yang lalu

(3)

crazygringo7 tahun yang lalu

The orange dots are much darker and the green ones are much lighter.

I'd assume anyone who is colorblind can still distinguish between the two based on brightness and not color?

Is there any reason to think that's not the case?

tzs7 tahun yang lalu

Heck, I'm not colorblind and I had trouble seeing the difference between "service disruption" and "service outage" at first in the legend at the bottom of the page.

It wasn't until I zoomed in on them that I could see that one was orange and the other red. Once I saw them zoomed, I could then identify which was which at normal size on the status part of the page.

BTW, the orange circle is actually a span whose class is "aad-yellow-circle", and whose CSS loads the colored circle from the file yellow_circle.png.

This suggests that at one time they intended it to be a yellow circle, not an orange circle [1]. I wonder why they switched from yellow to orange?

[1] Actually, RGB to name sites suggest that it is neon carrot.

culturestate7 tahun yang lalu

If the user had e.g. red/green colorblindness, that wouldn't help. Google's made a nice tradeoff for this application, though, and used differently-shaped icons (checkmark vs. exclamation point) as well.

Edit: looks like the icons are served as images. Google should probably consider making them text icons instead to mitigate loading problems.

FYI, Toptal makes a helpful tool for quickly checking live pages with colorblindness filters: https://www.toptal.com/designers/colorfilter/

manigandham7 tahun yang lalu

You're looking at the wrong page. This thread is about https://www.google.com/appsstatus#hl=en&v=status

culturestate7 tahun yang lalu

Ahhh, got it now, thanks!

0x8BADF00D7 tahun yang lalu

Noticed this a few minutes ago. It’s pretty rare for Google to have an outage, don’t they have massive replication and failover?

baroffoos7 tahun yang lalu

Youtube was down for a few hours late last year.

rubicon337 tahun yang lalu

Fine here in CA.

stevenwoo7 tahun yang lalu

I'm in Mountain View, receiving/sending email works but downloading full size attachments does not for me personally.

taeric7 tahun yang lalu

I just emailed some folks images. Got a few warnings of something going wrong, so I then checked in my outbox. It showed that the images went, but there was a message that the "virus scanners were down, download at your own risk." Or some such. Seems to be back to working now, so I just assumed it was a time delay.

Edit: Just saw it again. Full message is: Gmail virus scanners are temporarily unavailable – The attached files haven't been scanned for viruses. Download these files at your own risk.

First time I've seen something like this, actually kind of glad they had this failure case coded for. :)

muststopmyths7 tahun yang lalu

I've been seeing that for the last couple of weeks, off and on.

taeric7 tahun yang lalu

My guess is there is always a time delay of some sort on files. The transient "something went wrong" errors were surprising.

rconti7 tahun yang lalu

That error happens all the time on files it can't handle; i think .tgz type files come to mind.

phantom7847 tahun yang lalu

Also Mountain View, but had issues sending/receiving as wells. I'm on Comcast, you?

stevenwoo7 tahun yang lalu

I am using both a T-Mobile hotspot and Sonic.net DSL.

jszymborski7 tahun yang lalu

And in Montreal, Canada too

AndrewWarner7 tahun yang lalu

YouTube seems down too:

https://outage.report/youtube

[deleted]7 tahun yang lalu

the4ner7 tahun yang lalu

drive, maps, photos all having issues as well

simonhorlick7 tahun yang lalu

I'm getting a ton of 503 Service Unavailable errors from the Google Drive API. I can't wait to see what caused such a large outage.

carmate3837 tahun yang lalu

Can't attach files to drafts nor send emails from Philippines

rum37 tahun yang lalu

(1)

bluenose7 tahun yang lalu

Elizabeth Warren is winning :(

nickchuck7 tahun yang lalu

SHE'S HACKED THE SYSTEM

jhgg7 tahun yang lalu

The root of this is definitely GCS. We noticed some keys in a GCS bucket go inaccessible since like 19:15-19:20ish PDT, and then we noticed increased timeouts/503's ramping up at 19:30 and plateauing at 19:45. But error rate for us seems to have gone back to nominal levels since 20:10... but the keys that originally went unavailable are still returning 503's.

Fun times.

jedmeyers7 tahun yang lalu

GCS might be relying on the same system as Gmail/YouTube does, so it’s not necessarily GCS itself. Spanner?

ohaideredevs7 tahun yang lalu

If Google's GCP marketing material is to be believed, you are truly on their infrastructure on GCP, so if GCP is having issues so are Google's main services.

mehrdadn7 tahun yang lalu

It will be very interesting to see what timing issues might be involved if this is indeed due to Spanner.

parent54467 tahun yang lalu

Blobstore

jchallis7 tahun yang lalu

Down in Singapore

dickeytk7 tahun yang lalu

I had trouble replying to an email but it worked once I removed the images inside the quoted signature. Makes me think this is file-related.

minikomi7 tahun yang lalu

Street view is iffy as well

graposaymaname7 tahun yang lalu

Fine here in Kerala, India.

EpicEng7 tahun yang lalu

Working fine in Austin TX

giarc7 tahun yang lalu

Down in Calgary still (10:24pmMST). Was going to do work but not confident my email drafts will save.

awal7 tahun yang lalu

My bad guys. I'm running out storage space on my google account and I think I might have pushed it over.

mehrdadn7 tahun yang lalu

On Gmail or on Drive? :-)

awal7 tahun yang lalu

Would you believe I have over 10GB of old emails sitting around?

mehrdadn7 tahun yang lalu

Of course :-)

jedmeyers7 tahun yang lalu

Isn't it shared between services?

mehrdadn7 tahun yang lalu

Well that's why I asked which one, otherwise it would've obviously been Gmail...

sn_master7 tahun yang lalu

Google Photos too. Unable to download images :(

nukemandan7 tahun yang lalu

First time this has happened to me. Historic moment. Might be time to get less completely reliant on google?

nukemandan7 tahun yang lalu

AND we are back

nukemandan7 tahun yang lalu

May I ask - why the downvote? I know it was a blip, but still, a reminder for a moment that most of life relies on them. And some of my data is not replicated else ware... that is changing.

diminoten7 tahun yang lalu

I personally downvoted because of how a single incident is being extrapolated into a new trend by you, and that feels unhelpful/fear mongering.

9080877 tahun yang lalu

zerop7 tahun yang lalu

Is Gmail onto kubernetes?

hknd7 tahun yang lalu

Google is not using kubernetes internally but it's predecessor "borg". Kubernetes was build from scratch with the same idea, but not used internally.

https://www.quora.com/How-are-Borg-and-Kubernetes-different

londons_explore7 tahun yang lalu

The general sentiment seems to be that Kubernetes is better than Borg in almost all ways.

The only thing keeping Google on Borg is the massive amount of work to migrate.

ttul7 tahun yang lalu

I think it may be a touch beyond Kubernetes.

BucketSort7 tahun yang lalu

Living for the postmortem on this one.

bsilvereagle7 tahun yang lalu

As far as I know, there wasn't a public postmortem for the Oct 16, 2018 YouTube outage. Based on that, I'm not expecting one for this outage.

afniljl7 tahun yang lalu

YouTube ain't a Gsuite service with SLA. So this is much more likely to have a public report. Whether it's detailed or not is another thing. Also it's probably due to the underlying GAE or GCS issue is my guess.

jpatokal7 tahun yang lalu

"Red" incidents on the GCP status dashboard, like this one, will typically have public incident reports.

Disclaimer: I used to work in GCP Support, but no longer do.

jimmy_ruska7 tahun yang lalu

Sorry, Bill tripped on the server cable

sachin185907 tahun yang lalu

lol. email outage is the new dog ate my homework

[deleted]7 tahun yang lalu

nopacience7 tahun yang lalu

vinayan37 tahun yang lalu

In CA.

Colaboratory Research is experiencing issues loading and saving notebooks in Google Drive.

Sad :(.

wybiral7 tahun yang lalu

Happy 30th birthday, Internet. Enjoy your consolidated cloud infrastructure.

kerng7 tahun yang lalu

Fitting that something like this happens when we celebrate 30 years of the decentralized world wide web!

wereHamster7 tahun yang lalu

Only (some) Google services are affected. My email still works, as it's hosted elsewhere.

ian07 tahun yang lalu

Email is working for me, but attachments are failing. Opening up the console and it has a message about a database connection loss.

skilled7 tahun yang lalu

Attachments are definitely down for me. Can't view received images.