Facebook Down?(facebook.com)
facebook.com
Facebook Down?
https://www.facebook.com/
207 comments
Finally a status page that updates the moment an issue starts
for posterity:
<?xml version="1.0"?>
<!DOCTYPE html>
<html lang="en" id="facebook">
<head>
<title>Error</title>
<meta charset="utf-8"/>
<meta http-equiv="Cache-Control" content="no-cache"/>
<meta name="robots" content="noindex,nofollow"/>
<style>
html, body { color: #333; font-family: 'Lucida Grande', 'Tahoma', 'Verdana', 'Arial', sans-serif; margin: 0; padding: 0; text-align: center;}
#header { height: 30px; padding-bottom: 10px; padding-top: 10px; text-align: center;}
#icon { width: 30px;}
.core { margin: auto; padding: 1em 0; text-align: left; width: 904px;}
h1 { font-size: 18px;}
p { font-size: 13px;}
.footer { border-top: 1px solid #ddd; color: #777; float: left; font-size: 11px; padding: 5px 8px 6px 0; width: 904px;}
</style>
</head>
<body>
<div id="header">
<a href="//www.facebook.com/">
<img id="icon" src="//static.facebook.com/images/logos/facebook_2x.png"/>
</a>
</div>
<div class="core">
<h1>Sorry, something went wrong.</h1>
<p>We're working on getting this fixed as soon as we can.</p>
<p>
<a id="back" href="//www.facebook.com/">Go Back</a>
</p>
<div class="footer"> Facebook © 2021 · <a href="//www.facebook.com/help/">Help</a></div>
</div>
<script>
document.getElementById("back").onclick = function() {
if (history.length > 1) {
history.back();
return false;
}
};
</script>
</body>
</html>
And here are the headers:
# curl -i https://www.facebook.com/
HTTP/1.1 500 Internal Server Error
X-Frame-Options: DENY
X-XSS-Protection: 0
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=15552000; preload
Set-Cookie: ...
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Cache-Control: private, no-cache, no-store, must-revalidate
Vary: Accept-Encoding
Pragma: no-cache
x-fb-rlafr: 0
Content-Type: text/html; charset="utf-8"
X-FB-Debug: ...
Date: Thu, 08 Apr 2021 21:41:49 GMT
Priority: u=3,i
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600
Connection: keep-alive
<!DOCTYPE html><html lang="en" id="facebook"><head><title>Error</title>...>>Expires: Sat, 01 Jan 2000 00:00:00 GMT
The year 2000 doesn’t sound right.
The year 2000 doesn’t sound right.
This is common, it essentially means “don’t cache this”.
Ok, I thought this was related to SSL certs and maybe wrong dates might have caused FB to go haywire.
Seriously, they host the status page on their own infrastructure? That's ... not smart.
Every company's actual status page (and news feed) is on Twitter, no matter what else they say. I don't know where Twitter hosts their real status page. Probably Facebook.
Notably, Facebook hasn't updated theirs, though.
Notably, Facebook hasn't updated theirs, though.
> every company's actual status page is on Twitter
Clearly you have not experienced the absolute joy of a mission-critical locked-in-vendor B2B SaaS with no status page, no active Twitter presence, and the effective status page being a chat widget that routes to a person overseas who says "oh yeah this is a known outage on our main product please stay tuned" but there's no attempt to proactively make it visible to clients that they know there is an outage.
EDIT: fun fact, said vendor lets us use a subdomain we own, so we just route them through a Cloudflare Worker that injects Sentry into their HTML, so that we can monitor errors ourselves and raise tickets with them pretending that we know less than we actually know, because somehow we have a better observability culture than a SaaS vendor that's been around for 20 years. Don't underestimate the difference between vendor.mybrand.com and mybrand.vendor.com on a feature matrix, it may save your sanity.
Clearly you have not experienced the absolute joy of a mission-critical locked-in-vendor B2B SaaS with no status page, no active Twitter presence, and the effective status page being a chat widget that routes to a person overseas who says "oh yeah this is a known outage on our main product please stay tuned" but there's no attempt to proactively make it visible to clients that they know there is an outage.
EDIT: fun fact, said vendor lets us use a subdomain we own, so we just route them through a Cloudflare Worker that injects Sentry into their HTML, so that we can monitor errors ourselves and raise tickets with them pretending that we know less than we actually know, because somehow we have a better observability culture than a SaaS vendor that's been around for 20 years. Don't underestimate the difference between vendor.mybrand.com and mybrand.vendor.com on a feature matrix, it may save your sanity.
During the recent Microsoft Azure AD outage, I attempted to use the chat widget. Selecting "Technical support" or "Billing Support" both seemed to return 500 status codes while the "Sales Support" routed somewhere else and connected me to a person, perfectly fine. Presumably the sales team don't use Azure for anything.
Circa 2008-2009 when I was at FB I wrote a dashboard widget for the ops team that scraped Twitter for mentions of phrases like "Facebook down" in the past 5m. It was in use for a while.
> I don't know where Twitter hosts their real status page
What are the odds it's some IRC bot running under someone's desk?
What are the odds it's some IRC bot running under someone's desk?
Seems like Giphy is up at least, also facebook owned.
Anecdotally, whatsapp worked for me (message was sent and received successfully) though Facebook gave an error. I was already logged into both services.
whatsapp.com as in the web-based client, which is down.
It loads fine for me and shows no downtime or incidents. It even says "Facebook Platform is Healthy"
Try posting
Not here. Status page still does not load at all.
same, and can't visit the site at all
Do they share infrastructure?
Given the fact that it's also down, I'd say yes
It could be some rouge admin who actually deleted Facebook.
At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.
I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue.
For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined...
I guess they have enough pending stock to deter them...
For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined...
I guess they have enough pending stock to deter them...
This wouldn't "delete" Facebook or many much smaller companies. It would result in maybe a small outage and get restored immediately in most cases. It's also an infra change you'd need across many systems - this isn't possible as a single change "across entire fleet".
This is not how non-trivial services work.
This is not how non-trivial services work.
That’s...now how any of this works. You can’t just change integrity-bearing things without FIM systems kicking in. And you’d need collusion to get something mainlined that would bypass that.
You don't need collusion - just a code reviewer not paying proper attention.
How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.
It would be hidden amongst a big refactor, and it would have a bunch of unit and integration tests that all work correctly (because they do set the environment variables correctly).
Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line.
Besides, the vast majority of code reviews are 10 minutes or less.
Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line.
Besides, the vast majority of code reviews are 10 minutes or less.
I think you oversimplified the code merge process at Facebook. Their internal code review system (Phabricator) would automatically add tens of reviewers to your PR because you changed an important script authored by those people. Also, there are linters that would call out the use of potentially hazardous commands. Actually deploying the thing is another process that requires jumping through a series of other review hoops.
p.s. I had a short internship there.
p.s. I had a short internship there.
Facebook servers do not have cron installed.
Example using widely understood tooling... But there are plenty of other one line changes with similar impact
one could hope
Time for them to hit the gym and call a lawyer!
Yeah, that's what I was wondering. Strange that there is a SPOF for Facebook/Insta/Whatsapp.
I had the impression that they are still run like mostly independent companies who only share some data on the backend.
Might I ask what’s SPOF?
Even the status page is down
I like that it doesn't know that anything went wrong.
We're obviously going to take this opportunity to dunk of facebook because it's funny, but let's not forget that there are some valuable cultural assets that facebook hosts, with groups such as "Foods with threatening auras" and "Animals in predicaments."
I long ago tamed the facebook engagement algorithm by simply unfollowing everything and everyone except these legitimate, useful types of entertainment venues.
I long ago tamed the facebook engagement algorithm by simply unfollowing everything and everyone except these legitimate, useful types of entertainment venues.
Don't forget "The Same Photo of a Gravel Pile Every Day." Almost makes the decline of Western civilization worth it.
Is there a group that saves and reuploads the same photo every day, compressing it into oblivion?
If you start one, please share a link.
There is one in VK (not mine), been going on since 2012: https://vk.com/putineveryday
"The Same Photo of Jeff Goldblum Every Day"
I love these appreciation posts so much more than the typical Facebook-hating comments you get on Reddit in contrast. Yes, we know, FB is evil, but it doesn't mean there isn't actually some good stuff on there.
I hope they get FB back up and running within the end of the year, otherwise I may actually start missing it.
I walked away from fb a year and half ago, and I don't miss any of it.
Hope it stays down, it would be an overall net gain for humanity.
Hope it stays down, it would be an overall net gain for humanity.
How on earth does EVERYTHING go down like this? It just seems unfathomable to me. Where is the single point of failure?
The global outage drinking game:
bad DNS config push - 1 shot
routing loop - 2 shots
third party advertising your routes - 3 shots
power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots
Wolves ate through fiber - 5 shots
And it was a full moon - 6 shots
Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots
And service developers left the company to found a startup - 8 shots
Expired internal SSL certificate - 9 shots
Daylight savings changeover-induced database corruption - 10 shots
Windows Update - 11 shots
bad DNS config push - 1 shot
routing loop - 2 shots
third party advertising your routes - 3 shots
power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots
Wolves ate through fiber - 5 shots
And it was a full moon - 6 shots
Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots
And service developers left the company to found a startup - 8 shots
Expired internal SSL certificate - 9 shots
Daylight savings changeover-induced database corruption - 10 shots
Windows Update - 11 shots
5 is often broken down to “rain fade” and “backhoe fade.”
My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”
My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”
Coincidentally, I'm in the same predicament with Facebook, itself.
I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.
Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.
I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.
Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.
isn't that like throwin the baby out with the bath water? i've never had problems with duo mobile, but then i don't use FB so i've never seen how it behaves with FB. so dump FB, but why dump duo mobile?
Because the only thing I used it for was to login to Facebook. All of my other 2FA is through other methods.
In a similar vein, the alternative option for 7 was a cyclical dependency introduced since last restart
I heard a story that Google SRE had to up a sacrificial Bigtable in any new zone before they could up enough services to finally up a real Bigtable for the zone.
I was going to link a related comment but you're actually the author of it!
Here it is anyway: https://news.ycombinator.com/item?id=22316230
Here it is anyway: https://news.ycombinator.com/item?id=22316230
Please, please tell me this is a true story. It's amazing.
I can only verify being unable to get the wi-fi password. Otherwise it’s all secondhand, I wasn’t on call.
> Wolves ate through fiber - 5 shots
I hope it's wolves, that would create a new category in https://cybersquirrel1.com/
I hope it's wolves, that would create a new category in https://cybersquirrel1.com/
You forgot:
Hunter shot your fiber relay with a hunting rifle - 6 shots
Hunter shot your fiber relay with a hunting rifle - 6 shots
DST can be such a hassle.
It affects our reports and change requests.
An adtech company once freaked out over a sudden 4% revenue drop, until someone pointed out that yesterday was 4% shorter than the day before.
Don't forget cosmic rays!
Apple decided to shut down Facebook.
Large scale network partitioning can happen. Data center ingress/egress can go down. If you shard to multiple data centers, it's still possible for an entire autonomous system to have an interconnect with a different AS break. That is rare as hell with the actual Internet because of all the route redundancy even between ASes, but I've seen it happen on more specialized networks. For a few days last year, the entire western half of the US was cutoff from the eastern half on Raytheon's internal TS network because someone digging a hole somewhere accidentally cut the wrong fiber line.
DNS (not necessarily this case, but man it can bring things down fast)
How does it stay up as much as it does :)
"And if these people stop, the world burns. Most people don't even know what sysadmins do, but trust me, if they all took a lunch break at the same time they wouldn't make it to the deli before you ran out of bullets protecting your canned goods from roving bands of mutants."[1]
1. https://www.stilldrinking.org/programming-sucks
1. https://www.stilldrinking.org/programming-sucks
Blood, sweat, tears and caffeine.
also, some arcane shell script written by a retired engineer somewhere in the early start of the company.
I'll be curious about it coming back up. It's the sort of site where there's an army of people continually trying to reload the page if something isn't working.
Probably an intern doing something dumb.
If an intern can bring down the whole presence of your company online it's not the intern who is dumb.
While that's perfectly true, I'm sure a smart and determined intern could cause an astonishing amount of damage in many many orgs. Hopefully not at Facebook though.
The difference is whether or not it was intentional. If someone can bring everything down accidentally, the system is broken.
Ditto if they can bring it down intentionally and cover their tracks so no one finds out who did it.
Ditto if they can bring it down intentionally and cover their tracks so no one finds out who did it.
some startups would like to disagree
bingo
The dumb thing would be giving the intern permission to bring down the whole site.
An intern shouldn't have those permissions.
Likely not related, but you can imagine that they are quite busy these days with two unplanned projects:
1. Retooling the underpinnings of their ad analytics to survive iOS14. Big advertisers are about to lose a lot of their insights into attribution, partially directly related to the new limitations of iOS14, and partially due to assumptions that Facebook had made about tracking and which are no longer accurate and require a big rewrite.
2. Building an internal Shopify to allow customers to click on ads and complete transactions without ever leaving their app. This already exists but is practically unusable and nowhere close to the polish of Shopify.
1. Retooling the underpinnings of their ad analytics to survive iOS14. Big advertisers are about to lose a lot of their insights into attribution, partially directly related to the new limitations of iOS14, and partially due to assumptions that Facebook had made about tracking and which are no longer accurate and require a big rewrite.
2. Building an internal Shopify to allow customers to click on ads and complete transactions without ever leaving their app. This already exists but is practically unusable and nowhere close to the polish of Shopify.
3) Frantically planning Zuck's next apology tour after he leaked personal data on half a billion people.
This is an Orville reference isn't it. (or at least, it could be and would be extremely related)
No, I've just picked up the pattern that we only hear from Mark Zuckerberg when he's apologizing for a grievous "oversight" involving user data/abuse* of the Facebook platform.
* Also, underhanded use as intended.
* Also, underhanded use as intended.
Perhaps some security patches after the data leak? https://news.ycombinator.com/item?id=26691868
I'm getting two different error pages with the same "Sorry, something went wrong. We're working on getting this fixed as soon as we can" message.
One has the older site look, copyright 2020, the other has the newer FB5 look, copyright 2021.
The logo is not loading on either page.
Part of me wonders if they discovered a big/new hole that led to the data leak, although Instagram being down too suggests maybe there's something happening behind the load balancers?
One has the older site look, copyright 2020, the other has the newer FB5 look, copyright 2021.
The logo is not loading on either page.
Part of me wonders if they discovered a big/new hole that led to the data leak, although Instagram being down too suggests maybe there's something happening behind the load balancers?
Someone moved fast.
I thought they banned me. Retried in an incognito window.
Time to enjoy a brief social media pause, thanks to the deity who might have been behind this!
Time to enjoy a brief social media pause, thanks to the deity who might have been behind this!
Required clip:
https://youtu.be/wZ6ZlRnS40s?t=12
https://youtu.be/wZ6ZlRnS40s?t=12
I was thinking of this:
https://youtu.be/uRGljemfwUE
Gold!
Love to see it. I hate this company.
Weird that people almost celebrating in this thread? Facebook is a tool like any other and is only 'bad' if you allow that to happen.
That's like saying that any Turing complete language is equivalent. Some languages encourage you to do things well, others do not. It was totally possible to write code with a lot of "GOTO" statements in it that was well structured, but the actual real-world results with it were bad. It may be possible to use FB in a constructive way, but that doesn't mean we are better off having it.
Social media is a square wheel. Sure, it sort of goes and if you know no alternative, it might be better than no wheel at all. But there's no good way to use it because it's fundamentally flawed.
My hammer - a tool - doesn't want me to look at more ads. No, FB is not a tool.
Downdetector is a bit more useful than facebook.com for this: https://downdetector.com/status/facebook/
Of course the first time in months I've used any of Facebook API services, and they go down mid-implementation.
Do it again. You know, for science. It doesn't count if it isn't repeatable.
And of course Instagram tells me, "No Internet Connection"
Like you couldn't connect to your server dude, don't pass the blame on!
Like you couldn't connect to your server dude, don't pass the blame on!
Perhaps they had to patch some critical vulnerability like the one that exposed 533M personal records and did the right thing by pressing the red button™
This has to be one of those rare HN instances where a popular site has gone down and people are actually happy..
I can’t think of another such instance..
I can’t think of another such instance..
popular? you must be new here...
I had put popular in italics. ;)
It’s not popular here on HN, but in the rest of the world it surely is.
It’s not popular here on HN, but in the rest of the world it surely is.
Popular in the same sense that tobacco is popular.
In Denmark right now, everything down. Was on the phone with someone (in the US) through Messenger without issues. Hung up and tried to send them a message and nothing. Instagram non-functional as well.
Oh dear. Mark Zuckerbergs website, photo gallery and his SMS messenger are down.
Time to use something that doesn't scan your photo book, address book, and ultimately; everything about YOU.
Time to use something that doesn't scan your photo book, address book, and ultimately; everything about YOU.
I'm guessing this has do something with the recent graph api scraping?
Facebook is kinda rude now, they probably adding api bot detection and more phone number verification.
Facebook is kinda rude now, they probably adding api bot detection and more phone number verification.
Quickly, let your FB engineering friends know. If their status page is down, their monitoring may be down as well and they're none the wiser.
> If their status page is down, their monitoring may be down as well
That is not how monitoring should work. But this is also not how status pages should work.
That is not how monitoring should work. But this is also not how status pages should work.
I bet FB Workplace went down too heh [1]
[1] https://workplace.com
[1] https://workplace.com
I'm wondering if they're still using IRC internally.
Nah, the longer it is down, the better.
[deleted]
[deleted]
mbasic.facebook.com was also down in New Zealand. Thought it was a company firewall issue at first, but checked https://downforeveryoneorjustme.com/ and confirmed that it's happening to others.
Anyone knows what might have caused the outage?
Someone working in data centers might have a clue.
Black hawk down... hmm not this one. Blue bird down.. not again.. ohh yea facebook is down :)
I hope they publish a postmortem.
Do they usually?
They don’t but they should.
Postmortems are exciting reads and they should have a section for postmortems in their engineering blog.
Postmortems are exciting reads and they should have a section for postmortems in their engineering blog.
[deleted]
It was down for sure for me. I am in Los Angeles area.
Error occurred message.
I can't tell if it's down or not because I block it at the router.
Everything up and working again (in Thailand at least), that was fast
Productivity around the world likely increased while this lasted.
The website is still down for me, I'm in San Francisco.
Facebook, Messenger, Instagram and Whatsapp sites all down
Looks like facebook auth is still working?
I once accidentally knocked out 65K websites during a "fix update". Back then I was mortified. Now, I just feel like that was amateur hour...
Any bets on what happened? Fat Fingers?
Snapd autoupdate.
Containment breach in the AI department, they are currently bombing the site from space, the outages are collateral damage.
There's a page saying something is wrong, which could just be a very efficient summary feature for the site, that is working as intended.
Now stay down.
Quickly, let’s get shit done!
It's working for me now.
germany/berlin also outage of all fb services using http protocol
Berlin. Facebook is down.
Down in Austria as well.
Back up on Tenerife
Could this have anything to do with the recent data breach?
How long before the riots and total societal breakdown?
Yay!
Oh noes
Oh noes
Back up
Yea
And nothing of value was lost
down it Italy as well
Hope it's down forever!
real talk can any facebook employees remind me how to get to the IRC chats
Down in Australia as well
and its back
[deleted]
Yup, getting oauth errors.
Also instagram, but not whatsapp
Whatsapp (the website) now, too.
Whatsapp (the messenger) is still transmitting messages.
Whoa. Wonder what on earth is going on. Can confirm here in Canada, FB and Messenger are both down.
I don't understand how can you allow this to happen? I get one platform being down, but for all of your platforms to be down is not a good sign.
I only realized now that I only used Facebook platforms to communicate with my friends, and guess what I cannot reach them by Messenger, Instagram or WhatsApp.
So I am downloading Signal... I mean come on, you are literally loosing users because of this...
I only realized now that I only used Facebook platforms to communicate with my friends, and guess what I cannot reach them by Messenger, Instagram or WhatsApp.
So I am downloading Signal... I mean come on, you are literally loosing users because of this...
PS. Instagram app/site, whatsapp.com, messenger.com, oculus.com also down, anything else?
UPDATE: Everything back up! (Even the status page)