Incident at Slack(status.slack.com)
status.slack.com
Incident at Slack
https://status.slack.com/2022-03/8a9e98bfaa32ef5f
95 comments
I find slack so bloated and slow it would probably take me about 15 minutes to realise it was actually down.
Slack should develop native apps now as they are no longer a lean startup and can afford to build native apps.
It’s also important to remember that writing native apps doesn’t inherently inherently give you a big performance boost. With the type of app Slack has, I would expect the default implementation in a native app to be pretty slow as well. You’d have to do a significant amount of further optimization to make it competitive, imo. And to be honest, I imagine if someone applied that same amount of optimization and effort to the electron app, they could make the performance much better there as well.
Slack should at least bridge to an open protocol, if not use one outright, so I can use any client I want.
Like they used to.
Like they used to.
Ripcord [0] supports slack, but probably lacks a few features. It's very fast.
[0] https://cancel.fm/ripcord/
[0] https://cancel.fm/ripcord/
Friendly reminder that using third party clients is against Discord ToS and under certain circumstances can get your account banned[0].
[0] https://twitter.com/anno0770/status/1349187069793460225
[0] https://twitter.com/anno0770/status/1349187069793460225
The gloss of Slack was lost to me when they took down the irc and xmpp bridges. I really liked the freedom to do it my way.
Then it became just another crappy messaging app.
Then it became just another crappy messaging app.
If all you want is messaging, use irc, matrix or an xmpp system. The real value of slack is all the integrations and ecosystem built on top of it. That is much harder to integrate with an arbitrary client. It could be done, they could publish "enriched messaging" standards or similar and breakout capabilities that clients could support, but just supporting vanilla messaging at this point would be a severe degradation of slack's capabilities.
I can't work out why you wouldn't develop a native app anyway - all this scaffolding in a Web UI to imitate native controls (like menus) seems completely backwards to me. Seems just lazy, but in an odd way because writing native apps isn't rocket science or difficult.
It's not like they're implementing thousands of native controls or user interface components - a listview with an edit box at the bottom...
It's not like they're implementing thousands of native controls or user interface components - a listview with an edit box at the bottom...
> It's not like they're implementing thousands of native controls or user interface components
This is not true in any way. Slack also have workspaces, channels, notifications, video calls, file sharing, admin tools and probably even more features. It's not just IRC.
Maybe writing a web-based app is lazy... except when you have to write native apps for macOS, Windows, Linux, Android, and iOS, and some people still need access via a browser. And each one has different ways of displaying images and videos and doing calls and handling file access. Now you have 5+ apps that you're maintaining, each with their own dedicated team since each one is unique. Want to roll out a new feature? Cool, now you have to write it 5 times across 5 teams. Good look coordinating that.
I know there are technologies that let you share code across all of the different native clients. But a webview wrapped in i.e. Electron handles most of it for you! Why would you do it any other way except to appease people who complain about bloat?
This is not true in any way. Slack also have workspaces, channels, notifications, video calls, file sharing, admin tools and probably even more features. It's not just IRC.
Maybe writing a web-based app is lazy... except when you have to write native apps for macOS, Windows, Linux, Android, and iOS, and some people still need access via a browser. And each one has different ways of displaying images and videos and doing calls and handling file access. Now you have 5+ apps that you're maintaining, each with their own dedicated team since each one is unique. Want to roll out a new feature? Cool, now you have to write it 5 times across 5 teams. Good look coordinating that.
I know there are technologies that let you share code across all of the different native clients. But a webview wrapped in i.e. Electron handles most of it for you! Why would you do it any other way except to appease people who complain about bloat?
This is a bad argument that is factually incorrect in some ways.
Slack already has three different clients - a web one (which includes the Electron desktop wrappers), one for Android, and one for iOS.
They've already demonstrated that they have the engineering resources and coordination to develop and maintain three completely separate clients (albeit poorly) - so adding a fourth one (native C++ codebase using Qt that would work well on Windows, Mac, and Linux) would not be the Herculean engineering effort required to go from one to five, and your implicit assertions that (1) they currently only have a single client and (2) that a different client would be required for each of Windows, macOS, and Linux are simply false.
> But a webview wrapped in i.e. Electron handles most of it for you! Why would you do it any other way except to appease people who complain about bloat?
Because (1) performance (2) accessibility and (3) stability - the things that users actually need. Slack's performance is extremely bad and you get people constantly complaining about it; their accessibility is janky and suboptimal because they're reinventing desktop UI controls on the web; and their applications have repeatedly broken (such as the disastrous input box rewrite[1]) because they're trying to do things with the web platform that it wasn't meant to do. (in case you're wondering how using a desktop client would help - you expose a "lite" version of your functionality in the web application and put the full functionality in the desktop tool)
You're using "appease" as marketing-speak for "fix the very real problems with their web application that people constantly complain about". Slack should fix their application because it isn't serving the users well. Performance problems are functionality problems.
As it stands, Slack is showing us that they aren't even pretending to care about their users - they're just trying to make a buck selling to large organizations.
[1] https://news.ycombinator.com/item?id=21589647
Slack already has three different clients - a web one (which includes the Electron desktop wrappers), one for Android, and one for iOS.
They've already demonstrated that they have the engineering resources and coordination to develop and maintain three completely separate clients (albeit poorly) - so adding a fourth one (native C++ codebase using Qt that would work well on Windows, Mac, and Linux) would not be the Herculean engineering effort required to go from one to five, and your implicit assertions that (1) they currently only have a single client and (2) that a different client would be required for each of Windows, macOS, and Linux are simply false.
> But a webview wrapped in i.e. Electron handles most of it for you! Why would you do it any other way except to appease people who complain about bloat?
Because (1) performance (2) accessibility and (3) stability - the things that users actually need. Slack's performance is extremely bad and you get people constantly complaining about it; their accessibility is janky and suboptimal because they're reinventing desktop UI controls on the web; and their applications have repeatedly broken (such as the disastrous input box rewrite[1]) because they're trying to do things with the web platform that it wasn't meant to do. (in case you're wondering how using a desktop client would help - you expose a "lite" version of your functionality in the web application and put the full functionality in the desktop tool)
You're using "appease" as marketing-speak for "fix the very real problems with their web application that people constantly complain about". Slack should fix their application because it isn't serving the users well. Performance problems are functionality problems.
As it stands, Slack is showing us that they aren't even pretending to care about their users - they're just trying to make a buck selling to large organizations.
[1] https://news.ycombinator.com/item?id=21589647
I am a fan of native apps but I do have to admit it is nice that slack is -exactly- the same on any platform you access it from and that will continue to be the case without any effort from their developers. Different ports of native apps invariably start drifting unless lots of care is taken to make sure they don’t, and that does not happen to always be a priority for companies.
This is true and a valid use case I think. Slack has abstracted enough of their UI to be re-usable cross platform that any re-write would likely be a regression.
Look at 1Password and the amount of shit they get on HN with the beta version. Rewrites are hard and they take a tooooooon of time to do correctly and require a lot of engineering work to keep the product consistent across platforms.
Do I wish that electron apps were better optimized? Yes, however, I also can accept the tradeoff for apps that need to work "everywhere".
For the most part, Slack works just fine. Their problems are less the UI/UX and more to do with infrastructure. Slack rewriting to a native app would have a tiny (if any) effect on these outages.
Look at 1Password and the amount of shit they get on HN with the beta version. Rewrites are hard and they take a tooooooon of time to do correctly and require a lot of engineering work to keep the product consistent across platforms.
Do I wish that electron apps were better optimized? Yes, however, I also can accept the tradeoff for apps that need to work "everywhere".
For the most part, Slack works just fine. Their problems are less the UI/UX and more to do with infrastructure. Slack rewriting to a native app would have a tiny (if any) effect on these outages.
> I can't work out why you wouldn't develop a native app anyway
Really? One code base, one language, write features once. I think it's more about eng resourcing (headcount, onboarding, roadmap parity / feature drift, etc) than whether making a native app is hard.
Anecdotal, but I feel like the React Natives and Electrons have made it so you see less "Sorry we're only on Mac", and "We're only on Android, iOS coming soon!" from apps
Really? One code base, one language, write features once. I think it's more about eng resourcing (headcount, onboarding, roadmap parity / feature drift, etc) than whether making a native app is hard.
Anecdotal, but I feel like the React Natives and Electrons have made it so you see less "Sorry we're only on Mac", and "We're only on Android, iOS coming soon!" from apps
> so you see less "Sorry we're only on Mac"
There's the problem, nobody's sorry for offering only these static 100MB chat clients taking 2GB+ of RAM once launched. These apps are technically ridiculous, especially since all they do is embed a website. I mean, if all your company's willing to deliver is website, why even bother with the apps... But that's all fine right? We'll just upgrade to 64GB and 128GB machines soon!
There's the problem, nobody's sorry for offering only these static 100MB chat clients taking 2GB+ of RAM once launched. These apps are technically ridiculous, especially since all they do is embed a website. I mean, if all your company's willing to deliver is website, why even bother with the apps... But that's all fine right? We'll just upgrade to 64GB and 128GB machines soon!
Don't underestimate just how much devs can dig their heels in because they want to work in JS.
Is there a problem with this? Other than "JS Bad" chest-thumping? If so many devs want to work in it, there's probably a reason.
If you're a JS developer and don't know (or enjoy) other languages, the most lucrative choice is to keep using JS and advocating for it whenever possible.
If they've already got many JS developers onboard, it's normal that their opinion will be biased and they wouldn't want to switch to another language and essentially make their own job obsolete (except for the minority that also happens to be equally-proficient in whatever desktop programming language is chosen).
If they've already got many JS developers onboard, it's normal that their opinion will be biased and they wouldn't want to switch to another language and essentially make their own job obsolete (except for the minority that also happens to be equally-proficient in whatever desktop programming language is chosen).
I've been working at web-based startups for over 10 years now, and I know VERY few devs who only know JS. Most are also proficient (or at least have working knowledge of) whatever other languages we're using. Usually JVM-based like Java, Kotlin, Scala or something else like Ruby or Elixir.
Everywhere I've been, we always "choose the right tool for the job". Recently, making apps that can run across every OS and in the browser, the right tool for the job has been JS (well, TypeScript these days, but same thing).
Everywhere I've been, we always "choose the right tool for the job". Recently, making apps that can run across every OS and in the browser, the right tool for the job has been JS (well, TypeScript these days, but same thing).
We are seeing a bit of a backlash nowadays to "SPA everything // webdevs will save the world" naive techno-utopianism. Turns out that nail actually has some funny curves and the hammer isn't enough!
Salesforce bought Slack last year. They can afford to build native apps, but that might go against the whole cloud computing thing they've got going.
https://www.salesforce.com/news/press-releases/2021/07/21/sa...
https://www.salesforce.com/news/press-releases/2021/07/21/sa...
Slack: the best tool for the job! (when the job is firing off gifs and rocket ship emojis to your coworkers instead of whatever it is that keeps the lights on at your company).
It's fun sometimes, but other times it really takes away from the conversation.
Yesterday Discord & Spotify, today Slack. Is it normal to have these big guys fail back to back?
Yes, that is completely normal. SAAS service disruptions happen all the time.
So why do they pride themselves on a 99.9% uptime if downtime is common and normal?
Uptime has nothing to do with anything in the real world. It's a number you put in an SLA of how much downtime is acceptable before they start paying you back money. And 99.9% still means 43 minutes of downtime every month. And even after that it's a calculation of risk. What does full redundancy cost vs paying back the fees for a say 24 hours of downtime? A lot of the time paying back money is way way cheaper then actually investing in technical redundancy.
99.9% uptime is still more than 40 minutes downtime per month. According to their status page they are at 99.93% uptime for the current quarter even with the current incident.
99.9% = 8.77 hours per year
Multiply that out by 1000s of providers.
Multiply that out by 1000s of providers.
Downtime is not the same as service disruption
Ah, you must work for Amazon ;)
huh?
You can announce downtime for a certain capacity of a service ahead of time. Which is essentially planned maintenance with some capacity hit, but when it's unplanned it's called disruption. Also, generally disruptions come with a bigger capacity loss, even if you don't lose the service outright.
These terms are not limited to cloud infrastructures and not new either. Computation clusters use these terms for at least 12-13 years (from my experience) or for even longer.
These terms are not limited to cloud infrastructures and not new either. Computation clusters use these terms for at least 12-13 years (from my experience) or for even longer.
Slack has a major outage almost weekly
We deal with issues because something is screwed up with their APIs badly enough that service is degraded all the time.
We deal with issues because something is screwed up with their APIs badly enough that service is degraded all the time.
During a war with a major cyber battle space?
I think it’s going to be the new normal
I think it’s going to be the new normal
Another day, another SAAS outage.
My mind went completely diff direction after reading "Incident AT".
Mine as well. I understand that Slack is widely used and I appreciate seeing their post-mortems discussed on HN but I don't consider image uploads issues with no technical explanations news-worthy and valuable enough for HN.
> but I don't consider image uploads issues with no technical explanations news-worthy and valuable enough for HN
Have you read the status entry? Among other things, calls is also affected.
Have you read the status entry? Among other things, calls is also affected.
[deleted]
Yeah really bad choice of word. “Availability” or “Degraded” is what they’re looking for.
Same, this read like workplace violence. "Outage At" would be more accurate.
This is more a service degradation / outage than an incident.
How do you differentiate outage from incident?
The title here might as well be called — "Something happened at Slack".
Could you be more specific? — "An incident at Slack"
Could you be more specific? — "An outage at Slack"
Could you be more specific? — "..."
Would rather see something more specific:
"File Upload & Emoji Service Degradation at Slack" ( — insert emoji memes ).
Arguably, title is a bit misleading whether it warrants front page news for this community.
Could you be more specific? — "An incident at Slack"
Could you be more specific? — "An outage at Slack"
Could you be more specific? — "..."
Would rather see something more specific:
"File Upload & Emoji Service Degradation at Slack" ( — insert emoji memes ).
Arguably, title is a bit misleading whether it warrants front page news for this community.
I know a bunch of SaaS and other services have been hit, but at this is like the 3rd major slack outage this month. And typically my colleagues and I need to search third party sites to get any notification of what’s happening.
I'm a bit baffled that a company with a communication tool like Slack does not communicate about these incidents _through_ slack itself. Why can't there be a little service notification somewhere in the screen? I know it wouldn't help much when the entire service is down, but when parts fail won't it be helpful to at least indicate this to the user?
Or might that be due to potential negative brand marketing that springs into existence when users see such a notice inside their application?
Or might that be due to potential negative brand marketing that springs into existence when users see such a notice inside their application?
That feels like spam to me. Tell me image uploads aren't working exactly when I try to upload an image and no sooner. Or possibly keep some global in-app feature status page I can check on demand, but I don't want notifications about services and/or features I'm not currently using.
Wouldn't that be an anti-feature?
When Slack is down, Slack can't tell you when it is down. It's why status pages are often on other domains/systems etc, so the status page remains up while everything else is on fire.
When Slack is down, Slack can't tell you when it is down. It's why status pages are often on other domains/systems etc, so the status page remains up while everything else is on fire.
While it of course won't work in every case, in this case Slack wasn't down, you just couldn't attach files in some situations. This is something Slack easily could have communicated as soon as you tried to attach a file. Now it look like it worked and then times out after a couple of minutes and deleted your message
Ever since this started, been seeing 100% failure on webhooks for Event subscriptions.
But in this case Slack was not down. Parts of the service were frustratingly lagging, like uploading a picture or using reactions. Circle CI does have this sort of warning on their sidebar. There's so many parts to a big software product that it would definitely useful to know if some part of it was having issues.
If it's on a different domain and servers, I don't think it will be an issue.
If they have it on an API call to (making it up) api.slack.com/status it might very well go down when things go wrong, making it useless.
However if they physically host it elsewhere ideally on a different cloud provider, with a different domain, say, slackstatus.com/api-status that is completely separate from the actual services, which can both query the actual service with a call from outside and can be updated manually with physical persons at Slack, and the client checks that domain, I think it can work.
If they have it on an API call to (making it up) api.slack.com/status it might very well go down when things go wrong, making it useless.
However if they physically host it elsewhere ideally on a different cloud provider, with a different domain, say, slackstatus.com/api-status that is completely separate from the actual services, which can both query the actual service with a call from outside and can be updated manually with physical persons at Slack, and the client checks that domain, I think it can work.
this sounds very complicated lol
No, not really. If your calls time out, ping the status server. Done.
It sounds more like laziness than complication to me. But that typifies Slack dev culture—“just use duct tape”
It sounds more like laziness than complication to me. But that typifies Slack dev culture—“just use duct tape”
And that's exactly how you DDOS your status server.
Put a CDN in front of the status server. If that overwhelms your CDN then add some exponential back off to the mix.
Why can't they put a status.json file on S3/CloudFront/R2/wherever?
And in an absolute worst-case infra collapse, someone there could manually edit/upload a new one...
And in an absolute worst-case infra collapse, someone there could manually edit/upload a new one...
Initial setup is not much different than creating a new website, rest is trivial in practice. Easier than said.
It makes perfect sense from a user perspective but I suspect the company isn't interested in making mere users aware of ongoing issues via the service itself.
Documenting on a webpage allows a historical record of the incident long after the incident is over. I agree that an in-app notification would also be a nice feature.
Add their status to one of your channels (https://slack.com/apps/A0F81R7U7-rss)
/feed subscribe https://status.slack.com/feed/rss
/feed subscribe https://status.slack.com/feed/rss
Another week, another slack outage. Somehow it feels like they've had very frequent outages the past few weeks.
Time for self-hosted https://rocket.chat :)
Slack is not a good enough replacement for IRC.
Centrally controlled, heavy and bloated - nope, nope, it's just not it.
Centrally controlled, heavy and bloated - nope, nope, it's just not it.
In my dreams I manage a locally-hosted Matrix homeserver in our company's VPN. (We're not big.)
They just added voice calls via web UI. Not E2EE yet but if you're on an encrypted VPN then that's kinda moot anyway.
They just added voice calls via web UI. Not E2EE yet but if you're on an encrypted VPN then that's kinda moot anyway.
I like the idea of Matrix, but - can you get simple, thin, GUI clients for it like you have for IRC? IIRC, most GUI clients for it feel like something running inside my web browser...
I haven't had much occasion to use it (I dipped into a gaming group that used Matrix but quickly fell out of it) but https://github.com/poljar/weechat-matrix seems mostly adequate if you are a weechat user and just need the text chat features. Feels just like IRC.
Many options here: https://matrix.org/clients/
Seems to be more on the "glossy" and "web client" end of the spectrum in general but probably there is some reasonable compromise to be reached from the proliferation of different clients.
Seems to be more on the "glossy" and "web client" end of the spectrum in general but probably there is some reasonable compromise to be reached from the proliferation of different clients.
Element is a pretty nice client for matrix. Only been using it for a week but it's not too shabby on Windows/Android.
https://element.io/
https://element.io/
ffs -- hoping my company just moves their stuff to self hosted mattermost and calls it a day
Ahhh... so this explain why suddenly I was having problems to share a ODT file.
It will replace email.
I wonder if there is a market for making a clone of slack, which syncs all data from slack, simply for use when slack is down.
Big companies could pay monthly for "slack redundancy" from a third party.
Big companies could pay monthly for "slack redundancy" from a third party.
What I really wish is some glue layer kind of service that will sync slack with email or teams or something like that so I can do without any one service.
Well, I wouldn't use such a clone. When Slack is down, I actually feel relieved. I know I can turn off Slack any time I want, but the reality is that companies expect one to be online all the time (except during breaks of course) in regular working hours. I try to keep Slack as an async communication tool, but stakeholders just don't care about that. Also, I have a bunch of channels I cannot mute, so there is always something going on on such channels (which brings anxiety and discomfort).
So, when Slack is down, it's actually a nice "everybody is offline: either take some time off or work focused on whatever you need to work on"... which is a nice thing to experience from time to time during regular business hours.
So, when Slack is down, it's actually a nice "everybody is offline: either take some time off or work focused on whatever you need to work on"... which is a nice thing to experience from time to time during regular business hours.
Reading people's experiences in their companies makes me feel a) sad that there's so many people that have to endure this kind of work environment, b) so fortunate that I can't identify myself with any of this.
If I text someone and they reply within a day, that's fine to me, and it's fine to any of my colleagues. If it takes more than a day, I'll ping them again, and everyone can still chill.
If I text someone and they reply within a day, that's fine to me, and it's fine to any of my colleagues. If it takes more than a day, I'll ping them again, and everyone can still chill.
what industry are you in?
You could break that expectation. I personally turn off notifications except for @-ing my name or @channel, and check DMs hourly. It has not been a problem, well, people don't DM me as much because I don't reply. Half the time they figure out the answer by themselves anyway.
Ideally you could discuss that the notifications are making you anxious and less productive and your employee could accommodate that. If your employer is not interested in helping you become more productive then maybe look somewhere else? You don't have to settle. Your mental health is paramount.
Ideally you could discuss that the notifications are making you anxious and less productive and your employee could accommodate that. If your employer is not interested in helping you become more productive then maybe look somewhere else? You don't have to settle. Your mental health is paramount.
Sounds like slack is the new "spend all day in your cubicle".
This could be achieved by using a Matrix bridge to Slack: https://matrix.org/bridges/
Actually a clone wouldn't help much, because Slack itself is not really that great, even when it is up. There are better alternatives, which do not cause high CPU load every time one switches a channel, render Markdown properly (seriously, what is it with all the "enterprise service providers" being unable to use a decent markdown parser!?) and have have better integration with other open source tools.