Slack System Status(status.slack.com)
status.slack.com
Slack System Status
https://status.slack.com/
110 comments
Entertaining to see them simultaneously claim 100% uptime in February and list outages both yesterday and today. If 'users can't send or receive messages' on a messaging platform, your service is down.
Can't fail those SLAs if you make up the numbers
What's also interesting is their carefully crafted language. They have "Slack is not loading for some users" listed everywhere instead of "most users" or "all users" and also "Something's not quite right" instead of "Something's not working" or "Something's wrong".
If you look today (one day after the incident) you'll see that the status page now shows only 3 9s of uptime for the quarter.
For most large companies, their health page is updated manually by project managers or directors. The reason is simple: accurate reports make them look bad. And make them liable for refunds due to broken SLAs.
Amazon's famous for this; it's no surprise that Slack does it too.
Amazon's famous for this; it's no surprise that Slack does it too.
> They don't want to lose another decimal on their uptime percentage.
I was curious and found it:
> September 2021 99.999% uptime
Loonies.
I was curious and found it:
> September 2021 99.999% uptime
Loonies.
Their incident was reported a full 33 minutes after downdetector first started showing spikes in reports. That's pretty bad.
A sampling of Down Detector comments on https://downdetector.com/status/aws-amazon-web-services/ (which is also showing a spike right now, with no indication of an actual AWS outage I'm aware of):
> Was the internet and #AWS recently hacked? Seems like a lot of softwares are not working properly
> That’s so odd. Wi-Fi is out for me too. AWS outage?
> Another problem which definitely doesn’t make sense why server itself gone wild and not responsive to ssh. We will keep investigating this but it can be aws virtualization issues but interesting.
Slack has to do a little more due dilligence than "wifi's down, must be AWS" for their status page.
> Was the internet and #AWS recently hacked? Seems like a lot of softwares are not working properly
> That’s so odd. Wi-Fi is out for me too. AWS outage?
> Another problem which definitely doesn’t make sense why server itself gone wild and not responsive to ssh. We will keep investigating this but it can be aws virtualization issues but interesting.
Slack has to do a little more due dilligence than "wifi's down, must be AWS" for their status page.
They need to do more due diligence on the cause before announcing the cause, but they don't need to do more than confirm users are having problems to update the status page that problems exist.
Even assuming they can't instantly see the problems in a dashboard and need to take time to understand user feedback to see which aspects specifically are affected to update the page in more detail, nothing is stopping them from very quickly putting a "Connectivity issues reported by some users, investigating the extent and cause currently" banner at the top of the page which should be ready to go at a moment's notice anyway.
Even assuming they can't instantly see the problems in a dashboard and need to take time to understand user feedback to see which aspects specifically are affected to update the page in more detail, nothing is stopping them from very quickly putting a "Connectivity issues reported by some users, investigating the extent and cause currently" banner at the top of the page which should be ready to go at a moment's notice anyway.
I started having failed messages in channels and threads around 0605 PST... so about 20 minutes. Not too terrible considering the usual due diligence to escalate, determine scope of impact, and then push the buttons to update the status page with the approved messaging.
I completely disagree. It should go yellow the second they see that there is a spike in any abnormality. This type of issue would have been obvious seconds after it occurred. They don't need to report why but they should report that something is wrong. It's not fair to say "it's not me it's you" when they definitely know it's them.
time to start up a SaaS service status SaaS to serve SaaS status
I've definitely already seen that before
edit: Yeah it was Atlassian's Statuspage. I am pretty sure I saw them before they were acquired though and never realized
edit: Yeah it was Atlassian's Statuspage. I am pretty sure I saw them before they were acquired though and never realized
doesn't atlassian do that? and Pingdom?
https://documentation.solarwinds.com/en/success_center/pingd...
https://www.atlassian.com/software/statuspage
https://documentation.solarwinds.com/en/success_center/pingd...
https://www.atlassian.com/software/statuspage
oh true! I guess there really is nothing new under the sun anymore.
downdetector is my go to status page
2022 Checklist for Confirming a Service is Down:
[ ] Do a Twitter search [ ] Scan through HN [ ] Check with peers [ ] Consult downdetector.com
…in this order. Do not bother with official status pages.
[ ] Do a Twitter search [ ] Scan through HN [ ] Check with peers [ ] Consult downdetector.com
…in this order. Do not bother with official status pages.
> Do not bother with official status pages.
You should. If the status page is also down, you know it's going to take a while ;-)
You should. If the status page is also down, you know it's going to take a while ;-)
> You should. If the status page is also down, you know it's going to take a while ;-)
Maybe. I've been shocked by some big companies co-hosting their status pages.
Maybe. I've been shocked by some big companies co-hosting their status pages.
In this case - having happened to have noticed this at the exact time it occurred, I observed:
downdetector had it first
twitter had it second
peers were unfortunately unreachable (slack)
HN lagged by a significant margin
downdetector had it first
twitter had it second
peers were unfortunately unreachable (slack)
HN lagged by a significant margin
that has also been my recent experience. works best for large services like discord and twitter which are recent examples for me.
HN pointed me to the official status page though
...and cue the thread about how IRC is a better medium and SAAS has ruined communication.
I feel like that's a pretty reactionary painting of the issue, but wanting large platforms to provide access through open protocols that have been in use for decades and have many accessibility and usability benefits really seems pretty much a no brainer for anyone reading Hackernews.
should be
But how to you get that $1b valuation by making yet another client based on open standards? You have to corner the market, and 90% of HN readership want to create or work for a new unicorn, not just improve technology for the sake of it.
But how to you get that $1b valuation by making yet another client based on open standards? You have to corner the market, and 90% of HN readership want to create or work for a new unicorn, not just improve technology for the sake of it.
> But how to you get that $1b valuation by making yet another client based on open standards? You have to corner the market, and 90% of HN readership want to create or work for a new unicorn, not just improve technology for the sake of it.
Feel free to make a better Slack, that's more usable, more reliable, with more integrations, and run it on IRC. The implementation details mean little to users. Thinking Slack won because it was a closed standard and not that it's better than IRC for the majority of its users is quite the reach.
Your colleague from marketing:
- ... how do I get on it?
- How do I upload files to a channel?
- How do I search for something I wrote a month ago?
Feel free to make a better Slack, that's more usable, more reliable, with more integrations, and run it on IRC. The implementation details mean little to users. Thinking Slack won because it was a closed standard and not that it's better than IRC for the majority of its users is quite the reach.
Your colleague from marketing:
- ... how do I get on it?
- How do I upload files to a channel?
- How do I search for something I wrote a month ago?
I definitely prefer Slack over IRC for internal communication.
But I've also kind of come to hate live chat for internal communication entirely...but email isn't a great alternative either. I'm beginning to think there are just irreducible complexities to human communication that make every platform non-ideal, and the best thing to do is to just set and manage expectations properly as a team and as individuals.
But I've also kind of come to hate live chat for internal communication entirely...but email isn't a great alternative either. I'm beginning to think there are just irreducible complexities to human communication that make every platform non-ideal, and the best thing to do is to just set and manage expectations properly as a team and as individuals.
Zulip is a great alternative (hi Austin!).
IRC has existed for over 30 years and was widely popular for the big part of that timespan. Yet no one ever made it into or extended it into a realistic competitor to the modern wall garden proprietary chat platforms.
Is there a fundamental issue with IRC that precludes that? Did no one just bother? Is it the specific mentality among its core users that "absence of certain features in IRC considered a feature" that lead to its demise?
Is there a fundamental issue with IRC that precludes that? Did no one just bother? Is it the specific mentality among its core users that "absence of certain features in IRC considered a feature" that lead to its demise?
Money.
You can't make as much money with open protocols.
Worst of all, it's very hard to pitch such a business to people who would fund you, so your business is outcompeted very early on.
What users want and what entrepreneurs and investors want are two, largely uncorrelated, things.
You can't make as much money with open protocols.
Worst of all, it's very hard to pitch such a business to people who would fund you, so your business is outcompeted very early on.
What users want and what entrepreneurs and investors want are two, largely uncorrelated, things.
IRC was never widely popular outside of a gamer/geek niche. At its peak, QuakeNet never broke 250,000 "users", many of which were bots.
[deleted]
You are so incredibly wrong about this, it saddens me.
IRC usage stats are freely available. It was never popular.
https://netsplit.de/networks/top10.php?year=2006
https://netsplit.de/networks/top10.php?year=2006
If you're gonna tell someone they're wrong in such a dramatic way you need to explain why.
Even if his numbers are wrong, I'll never believe you that IRC was more used than say, AIM or ICQ, let alone any of the popular modern instant messenger and chat apps.
Ok, fine. But can we at least reverse the gamer/geek label. It was always a geek tool - gaming was far secondary. Look at bash.org and tell me the majority of IRC was used for gaming vs. geeking.
There's no business model in IRC, and thus no marketing money.
looks like the guy who took over freenode is doing his best to create a walled garden. Also plenty of people still on IRC, including me :)
FWIW, I like irssi much better than the slack front end, but as font ends go, slack's is not terrible.
FWIW, I like irssi much better than the slack front end, but as font ends go, slack's is not terrible.
There's Pest [0], but I don't think it tries to be more usable, maybe even less so. I think it's hard to take old standards and put modern features on them; at some point it's more beneficial to just invent your own thing.
[0] http://www.loper-os.org/?p=4003#11-what-is-pest
[0] http://www.loper-os.org/?p=4003#11-what-is-pest
I just don't see why everyone doesn't just run their own always-online IRC bouncer and tweak their irssi config for hours to their liking to get an experience that's worse than Teams.
So while I often use irssi, I have become recently fixated on how to compartmentalize all my different chats and I keep coming back to weechat on a vps that bridges slack, discord etc, so weechat bridges are another alternative. If you have a good idempotent spinup method with your vps it can also mean less data hanging around to be grabbed because you can just make a fresh one anytime, in any datacenter.
SAAS / outsourcing has ruined organisation self-responsibility, but the MBA said it was a good idea, and the ill-informed CEO just followed.
SAAS companies cannot be responsible for their own incident/status pages. This has been proven time and again.
Sounds like a SaaS opportunity.
I assume a joke here haha -- no way are any of these companies pay another to hold them ACTUALLY responsible for their outage %. They would MUCH rather just lie.
Who'd pay?
https://stop.lying.cloud/ exists for AWS.
https://stop.lying.cloud/ exists for AWS.
I would consider paying a 3p a reasonable price for accurate/reliable visibility into status of cloud services, if my business depended on it.
I’ve thought about setting something like this up for the major clouds & SaaS.
The biggest problem is that I’d have to run some dummy loads on all their services in all regions - even at the lowest tiers I’d probably be looking at a few thousand a month just to keep the service up.
The biggest problem is that I’d have to run some dummy loads on all their services in all regions - even at the lowest tiers I’d probably be looking at a few thousand a month just to keep the service up.
I've considered doing this but I think it would only be feasible as an enterprise offering. The costs of continuously making API calls to start machines, test network performance etc would be quite high.
Couldn't Catchpoint or 1000eyes do it with their existing infrastructure?
Wow, this might be more disruptive than the AWS outages. Remote companies should have some sort of backup plan for when this happens.
GitLab has this publicly documented as for what to do in such a situation https://about.gitlab.com/handbook/communication/#emergency-c...
Funny, my first impulse was to share this with my coworkers on the #engineering channel on Slack.
Its time for Teams to shine
The duration of constant freezes Teams has on every run outlast the total outage time Slack has per month.
Zoom?
:lolsob:
:lolsob:
Discord? Also supports VC so better than Slack.
It's probably just a coincidence, but about 1 hour ago I was using Slack normally and I was sending some regular expression tips [1] for some teammates. I was explaining how \d and \D works, and how they must be escaped when inside a Ruby double quoted string, like "\\d" and "\\D".
First I tried to send the message and the Slack app showed a tiny error message in that specific message I've just sent. Then I've pressed the retry button. Immediately the Slack app stopped working... And here I am, browsing on HN.
It immediately reminded me of this joke [2]
[1] https://www.regular-expressions.info/shorthand.html [2] https://twitter.com/brenankeller/status/1068615953989087232
First I tried to send the message and the Slack app showed a tiny error message in that specific message I've just sent. Then I've pressed the retry button. Immediately the Slack app stopped working... And here I am, browsing on HN.
It immediately reminded me of this joke [2]
[1] https://www.regular-expressions.info/shorthand.html [2] https://twitter.com/brenankeller/status/1068615953989087232
As usual - status page is green, you can't use slack app or web.
Your message is showing as "6 minutes ago" for me, and when I load the status page, it's full of yellow warning icons now.
[deleted]
Slack being down is so detrimental… it’s crazy.
I’ve almost forgotten how to send my coworkers email. It blows my mind how much we are so reliant on a few tools to carry out so much work.
I’ve almost forgotten how to send my coworkers email. It blows my mind how much we are so reliant on a few tools to carry out so much work.
> Uptime for the current quarter: 100%
Hahaha Slack and I have very different definitions of uptime, as I'm currently unable to load the app. Company status pages are a joke these days. Workplace politics (stakeholders making an impossible SLA) have ruined yet another great idea.
Hahaha Slack and I have very different definitions of uptime, as I'm currently unable to load the app. Company status pages are a joke these days. Workplace politics (stakeholders making an impossible SLA) have ruined yet another great idea.
Why did this fall from #1 on HN to not on the front page in minutes? Is Slack a YC company?
[deleted, it's quite obvious HN is not interested in actual discussion]
> wretchedly low-quality conversation
Some of the best conversations I've seen on HN spawn from these threads. They make us aware that we cannot depend on these service for mission critical things. They help us explore alternative options and allow us to share how we discovered the issue and what we did to weather the storm.
> yes, status pages are slow to change. There's good reason for that
What on earth could be the reason for that? They are the best source to tell us that people are having issues. Why not report it the second it's detected?
> how did people, especially those on "Hacker News" get so depressingly helpless?
Helpless? No - we depend on this news source to report relevant information the second it becomes relevant.
Some of the best conversations I've seen on HN spawn from these threads. They make us aware that we cannot depend on these service for mission critical things. They help us explore alternative options and allow us to share how we discovered the issue and what we did to weather the storm.
> yes, status pages are slow to change. There's good reason for that
What on earth could be the reason for that? They are the best source to tell us that people are having issues. Why not report it the second it's detected?
> how did people, especially those on "Hacker News" get so depressingly helpless?
Helpless? No - we depend on this news source to report relevant information the second it becomes relevant.
[deleted](1)
From https://status.slack.com/
> Uptime for the current quarter: 100%
This is another one of those system status pages that appears to be 100% for SLA reasons. An outage and partial outage are the same in my mind. I understand that money is involved, but the current trend into no visibility into company outages is frustrating to say the least.
> Uptime for the current quarter: 100%
This is another one of those system status pages that appears to be 100% for SLA reasons. An outage and partial outage are the same in my mind. I understand that money is involved, but the current trend into no visibility into company outages is frustrating to say the least.
Just to preserve the full message :)
"Something’s gone awry and we’re having trouble loading your workspace. Sorry we can’t be more specific - this is one of those cases where we don’t know what’s gone wrong either. A restart of Slack might help, and you can always contact us."
"Something’s gone awry and we’re having trouble loading your workspace. Sorry we can’t be more specific - this is one of those cases where we don’t know what’s gone wrong either. A restart of Slack might help, and you can always contact us."
Free at last! Woohoo.
6 minutes ago a collegue asked "issues with slack"
4 minutes ago I posted a screenshot of this being top story on HN so "yes" is a good bet
Now it's nowhere on the front page?
4 minutes ago I posted a screenshot of this being top story on HN so "yes" is a good bet
Now it's nowhere on the front page?
Looks like github is down, and aws is having issues, too.
https://downdetector.com/
I should've stayed in bed
https://downdetector.com/
I should've stayed in bed
I am seeing a lot of errors with the following message: 'Message undefined (message missing text)'
Why this message lost first HN page?
And google chat is working very slow and weird... They can't figure out IM still in 2022.
How come there are two active threads on this? It's pretty confusing.
https://news.ycombinator.com/item?id=30428788
> Something’s gone awry, and we’re having trouble loading your workspace.
Same here - UE
Same here - UE
It's been hours already, and it's pretty harsh and punishing for its users to wait for its uptime. I hope that they can take good care of this issue as soon as possible.
What’s odd is that we have back-end services that send messages to Slack and those seem to be working absolutely fine.
Guess the failure is in a different part of their platform…
Guess the failure is in a different part of their platform…
another data point to encourage groups of a certain size to host their own instance of something like https://zulip.com/ / https://zulip.readthedocs.io/en/stable/production/install.ht...
This option is dead for most of us because of Slack Connect (which allows cross-tenant communication, such as from x.slack.com to y.slack.com).
I now have 5-10 other organizations connected to any one of my Slack tenants, and now we're just stuck in Slack's ecosystem.
I now have 5-10 other organizations connected to any one of my Slack tenants, and now we're just stuck in Slack's ecosystem.
cross-tenant communication as a method of a vendor locking in their customers, hah!
Myself & others at the office can only use Slack on mobile. The Electron app and website both fail to load. Messages feel sluggish and delayed.
Slack has been struggling for my workspace the last 10 minutes. Can still get messages through (sometimes) but threads won't load at all.
Github is also unreachable for quite a few people. But works through a VPN to another geography. Wonder if there is any connection.
Just as I was searching my slack workspace and found info to help fix a problem, what are the chances
Maddening. “Can’t send message. Tap to try again” - when Slack’s own status page shows green.
(For me at least...) The client.boot file is returning a 429 (too many requests).
Finally, a chance for Zoom Chat to shine! /s
Something’s gone awry alright
hmm didnt this happen once before after a long weekend?
You can use http://chat.google.com/ as a backup chat. I started using it with my team. It does not come with desktop app, but usable
There's a "progressive web app" - if you're in Chrome on chat.google.com, there's a new icon on the right hand side of the URL bar which permits you to download it. Not discoverable at all - this was mentioned in the help pages [https://support.google.com/chat/answer/9455386]
Try down detector. https://downdetector.com/status/slack/