Facebook Platform was down
39 comments
This should be an interesting postmortem, if they have one. I think Netflix never released a postmortem of their latest downtime.
Unrelated to OP, but did Google ever release a postmortem for the YouTube outage a few weeks back? Was interested to hear the cause of that one.
[deleted]
I love reading the postmortems. I've read some from a few different tech sites and it's fascinating to hear just how people are doing things and how they fixed it.
My semi-dormant website: https://www.outagereports.net/
I started compiling a bunch of outage reports from around the web. I have another bunch stored in Pocket that I need to add.
I started compiling a bunch of outage reports from around the web. I have another bunch stored in Pocket that I need to add.
Back up for me.
Facebook makes ~$105k in revenue per minute [1]. Since the outage was seemingly only US-based, and it lasted ~30 minutes, this outage probably cost them just over a million bucks.
Pretty staggering to think about.
[1]: https://ycharts.com/companies/FB/revenues
Facebook makes ~$105k in revenue per minute [1]. Since the outage was seemingly only US-based, and it lasted ~30 minutes, this outage probably cost them just over a million bucks.
Pretty staggering to think about.
[1]: https://ycharts.com/companies/FB/revenues
> this outage probably cost them just over a million bucks
I have always wondered whether it is accurate to think about it like this. It sure is interesting to know how much companies make in revenue per minute, but it doesn't necessarily 100% translate to lost revenue in an outage.
For example, if I wanted to purchase something on Amazon, but the retail site went down I would not visit another site to purchase it. I would just wait until the site came back up. In that sense, the revenue isn't lost just delayed.
I have always wondered whether it is accurate to think about it like this. It sure is interesting to know how much companies make in revenue per minute, but it doesn't necessarily 100% translate to lost revenue in an outage.
For example, if I wanted to purchase something on Amazon, but the retail site went down I would not visit another site to purchase it. I would just wait until the site came back up. In that sense, the revenue isn't lost just delayed.
That works with e-commerce, but not necessarily with advertising. If you were going to browse Facebook during lunch in the office (and be shown ads), but Facebook is down, then you won't necessarily just "browse Facebook later in the day instead." You weren't on Facebook to catch up; you were on Facebook because you had nothing better to do.
This is definitely true, I expect that users will spend a bit less time on Facebook today.
However, Facebook doesn't have a 100% fill rate of high value ads, so during the rest of the day they'll be showing somewhat more expensive ads to users, earning a higher average revenue. That will make up for some of the loss, despite lower usage.
However, Facebook doesn't have a 100% fill rate of high value ads, so during the rest of the day they'll be showing somewhat more expensive ads to users, earning a higher average revenue. That will make up for some of the loss, despite lower usage.
For e-commerce that argument makes sense, but not for advertising.
They lost 30 minutes of primetime eyeballs from tens or hundreds of millions of people that would have seen ads but didn't.
They lost 30 minutes of primetime eyeballs from tens or hundreds of millions of people that would have seen ads but didn't.
Moving fast and breaking things!
Interestingly I VPN'd into a node in Dublin and it is still working there.
Iowa (direct) and SF (VPN) were both erroring out sitewide. Anecdotally it seemed to start in Analytics and spread to Facebook.com and finally to the platform over the course of ~15 minutes. But not sure if that's representative.
Iowa (direct) and SF (VPN) were both erroring out sitewide. Anecdotally it seemed to start in Analytics and spread to Facebook.com and finally to the platform over the course of ~15 minutes. But not sure if that's representative.
Using a completely oversimplified estimation method and 2017 revenue numbers, they're losing about $1300 every second their website is down.
Not Facebook.com, the platform that Facebook apps are built on I think.
It's down for a lot. Seems to primarily be NE US: https://downdetector.com/status/facebook/map/
Facebook.com was definitely down for me for several minutes at least.
So much for the pivot to "move fast with stable infrastructure."
They've been down for, what, 20 minutes in the past year?
Most of their outages appear to be regional, and only part of their site. This one also appears to have been regional. So it's hard for us outsiders to know the statistics.
These kinds of outages are pretty typical for "move fast and break things" development -- instead of spending huge effort making sure that no outage ever happens, spend effort testing and limiting outages to a small part of production. The intended benefit is much faster development.
These kinds of outages are pretty typical for "move fast and break things" development -- instead of spending huge effort making sure that no outage ever happens, spend effort testing and limiting outages to a small part of production. The intended benefit is much faster development.
broken image from static.facebook.com for the logo, too.
Looks like they had a little blip [1]
[1] - https://downdetector.com/
[1] - https://downdetector.com/
same. downdetector reporting 1200 plus reports of facebook down https://downdetector.com/status/facebook
Strange, I just logged into the website fine from Portland, OR via Comcast
Down on latin america too
Ideally it would never come back up.
[deleted]
Regular login down too.
It's working now.
Is down here
same
as well as facebook.com
same
me too.
EDIT: Commenters are reporting that this appears to only impact users that are hitting the US data center[1], and that it may only impact users through specific ISPs
1.https://news.ycombinator.com/item?id=18434457
EDIT: Appears to be back up