Slack using React Dev Build in Production(twitter.com)
twitter.com
Slack using React Dev Build in Production
https://twitter.com/pankowecki/status/892294002040594434
108 comments
React dev builds are terribly slow at rendering large lists. Open Slack and scroll up for about a minute. It becomes unresponsive very quickly.
I don't understand how deploying React's dev build would help anyway... Do they regularly run into bugs where they need to step into React code?
I don't understand how deploying React's dev build would help anyway... Do they regularly run into bugs where they need to step into React code?
scrolling up also makes CPU usage spike on chrome (and by extension, on the electron app). This doesn't happen with Firefox
[deleted]
Atom and Slack often get critiqued for their use of an HTML5 stack. I'd also like better performance when it comes to startup time, memory usage and responsiveness regarding interactions (I'm using a 12" Macbook) but the stack is so useful and universal that it is here to stay and worth figuring out how to get to great performance (similar to how Emacs figured out how to work around slow startup).
Are there examples for HTML5 apps that manage to have great performance? Are there inherent limits?
Are there examples for HTML5 apps that manage to have great performance? Are there inherent limits?
If you look outside first world countries, yes. Check out the progressive web apps talks in Google IO 2017 - most of the examples on stage were indian companies flipkart.com, housing.com and a few others. I work on www.justickets.in myself.
We have a huge Android market here, but they're all phones with very low storage and slow 3G connections, so the default way to get an 'app' out is a HTML5 progressive web app. And the companies here work hard to make things run fast and smooth on slow devices.
We have a huge Android market here, but they're all phones with very low storage and slow 3G connections, so the default way to get an 'app' out is a HTML5 progressive web app. And the companies here work hard to make things run fast and smooth on slow devices.
I'm not sure how effective it is but I always liked the idea of forcing engineers to use those 3g speeds 1 day a week so they are more inclined to support those customers. Iirc FB does this to some degree, but I think they have a different lightweight version of FB intended for those audiences
Sounds good, but in my experience 3g also varies by an order of magnitude (5 KB/s - 50 KB/s).
When I have 20-50 KB/s stable 3g, browsing the web is a bit slow, but most sites work pretty good. Also messenger aren't a problem.
But working below 20 KB/s is a pain.
When I have 20-50 KB/s stable 3g, browsing the web is a bit slow, but most sites work pretty good. Also messenger aren't a problem.
But working below 20 KB/s is a pain.
Chrome dev tools has a reasonable connection bandwidth simulator. Is there a good one that also simulates variability / packet loss in addition to bandwidth caps?
Yup – I love this idea. I actually occasionally test my single page apps under 3g connections to see how the loading experience works.
Anecdotal, I remember a few months back someone showed here on HN "skytorrents.in", which I assume was made by an Indian, it was blazzingly fast.
Yeah I remember back in India page speed was a real concern not because google could penalize you but the network speed was so slow. Not sure now but back then Bsnl used to suck big time.
After coming to Australia, everyone here treats +4mb websites as normal which was a real mind twister for me. Thankfully pagespeed has always been my concern regardless of network speeds due to my 'conditioning' years ago.
After coming to Australia, everyone here treats +4mb websites as normal which was a real mind twister for me. Thankfully pagespeed has always been my concern regardless of network speeds due to my 'conditioning' years ago.
> similar to how Emacs figured out how to work around slow startup
By dumping binaries, creating emacsclient, but mostly by waiting 25+ years for fast computers w/SSDs to come along?
I love emacs, but slow startup was a problem for multiple decades, IMO.
By dumping binaries, creating emacsclient, but mostly by waiting 25+ years for fast computers w/SSDs to come along?
I love emacs, but slow startup was a problem for multiple decades, IMO.
The emacs analogy is a terrible one. I start emacs once a month (or longer). I start slack once a day, sometimes more.
Discord is pretty snappy, even with tons of users..
It's pretty buggy, though. On Windows, at least once a day it resizes down to a window only big enough to hold the X button to close the window. And it drains the hell out of my battery on Android.
VS Code performs well.
I'll second this.
There's been articles on HN specifically about how slow VS Code is, particularly the one about rendering cursor blinking, and of course anything running on Electron will never compete work the likes of vim, &c.
However, as Electron apps go, I've found it noticeably faster and less resource intensive than any others. Slack in particular is comparatively terrible, as was Atom in the past (people have said it's improved but I've not felt the need to try it again recently)
There's been articles on HN specifically about how slow VS Code is, particularly the one about rendering cursor blinking, and of course anything running on Electron will never compete work the likes of vim, &c.
However, as Electron apps go, I've found it noticeably faster and less resource intensive than any others. Slack in particular is comparatively terrible, as was Atom in the past (people have said it's improved but I've not felt the need to try it again recently)
I've turned my vim into a monstrosity with plugins in the past - believe me, it's possible to make your vim just as laggy :)
Currently running Slack App, Atom App, and Spotify App on my MBP. According to my system stats, I'm consuming 3GB of RAM across the entire system (OS and all of the above).
So what I would like to know is wtf everyone is doing in these apps to create this CRAZY amount of resource hogging they keep bitching about? Or are they rockin chomebooks? Seriously? I want to know where all of this nonsense is coming from?
So what I would like to know is wtf everyone is doing in these apps to create this CRAZY amount of resource hogging they keep bitching about? Or are they rockin chomebooks? Seriously? I want to know where all of this nonsense is coming from?
That is a crazy amount of resource hogging at 3GB.
You're running a music player, a text editor (or IDE), and a chat client. I would expect reasonably that would use less memory. One point you could make is that even native application alternatives, ones that don't use Electron, would be bloated and I would have to agree. Visual Studio .NET is pretty darn slow/cumbersome itself.
You're running a music player, a text editor (or IDE), and a chat client. I would expect reasonably that would use less memory. One point you could make is that even native application alternatives, ones that don't use Electron, would be bloated and I would have to agree. Visual Studio .NET is pretty darn slow/cumbersome itself.
Back in 1999 I was running Emacs, xchat and freeamp on my Pentium 66 with 16Mb of RAM. (an weak machine, even for 1999 but with Linux Mandrake and WindowMaker it was usable)
The UI was less sophisticated and it doesn't involve network connection but sometimes I'm wondering how did we get there...
A lot of this comes back to higher expectations: we now have much larger displays, color depths, instead of simple bitmap fonts we have much higher-quality rendered vectors with advanced layout systems which can complex scripts, etc. Instead of rendering into a shared buffer, each window has at least one (on OS X, two) full buffers and the whole thing is composited, which is great for responsiveness and visual quality but definitely uses In the 90s, 640x480 was a common display resolution — now Apple recommends that developers ship 512x512 icons.
That's not to say that there aren't decisions to reconsider about code size, resource formats, etc. but I think it's easy to forget how much more behaviour has moved into our default baseline assumptions.
That's not to say that there aren't decisions to reconsider about code size, resource formats, etc. but I think it's easy to forget how much more behaviour has moved into our default baseline assumptions.
> doesn't involve network connection
In my very limited experience, this makes programming anything several times more complicated. Writing a document editor? Difficult. Writing a networked, multi-user document editor? As difficult as the last thing but with asynchrony and lots of new failure modes.
In my very limited experience, this makes programming anything several times more complicated. Writing a document editor? Difficult. Writing a networked, multi-user document editor? As difficult as the last thing but with asynchrony and lots of new failure modes.
Slack on my computer, with five teams, consumes 1.8gb of RAM.
Microsoft Word, with a multi-page complicated document, is using 200mb.
Just because we _have_ the resources to allow applications to bloat doesn't mean we should be alright with it. Its a similar argument to electric versus ICE cars; just because gas is cheap doesn't mean we shouldn't support EVs. EVs are fundamentally better for the future.
If we focus on performance optimizing the apps we use, it opens up a wide range of new computing hardware. You make fun of chromebooks, but imagine how much more interesting your work would be if you could use a machine with that performance. Battery life would be substantially higher. The cost of the machine would be substantially lower. The only reason most people need massive performance beasts is because we've pushed abstraction hell and bad engineering practices, so all of our tools suck.
Docker is another offender in recent times. Its great if you're on Linux, but few people are. So on OSX, we need to run a VM, so there goes another gig of memory.
Microsoft Word, with a multi-page complicated document, is using 200mb.
Just because we _have_ the resources to allow applications to bloat doesn't mean we should be alright with it. Its a similar argument to electric versus ICE cars; just because gas is cheap doesn't mean we shouldn't support EVs. EVs are fundamentally better for the future.
If we focus on performance optimizing the apps we use, it opens up a wide range of new computing hardware. You make fun of chromebooks, but imagine how much more interesting your work would be if you could use a machine with that performance. Battery life would be substantially higher. The cost of the machine would be substantially lower. The only reason most people need massive performance beasts is because we've pushed abstraction hell and bad engineering practices, so all of our tools suck.
Docker is another offender in recent times. Its great if you're on Linux, but few people are. So on OSX, we need to run a VM, so there goes another gig of memory.
3GB of RAM is precious (and I say that as someone who has 128GB)!
Your bar should not be, "Well, I don't see a slowdown." Many people still develop software on computers with only 8GB total RAM. Throw in a browser and one or two more applications and you're getting close to exhausting that.
Eventually you are going to exhaust your memory resources; using software with poor performance constraints is (in my opinion) inherently antagonistic to a development machine. That's not to say I'm against desktop JavaScript/HTML5 apps categorically, I just think they need much better optimization.
Your bar should not be, "Well, I don't see a slowdown." Many people still develop software on computers with only 8GB total RAM. Throw in a browser and one or two more applications and you're getting close to exhausting that.
Eventually you are going to exhaust your memory resources; using software with poor performance constraints is (in my opinion) inherently antagonistic to a development machine. That's not to say I'm against desktop JavaScript/HTML5 apps categorically, I just think they need much better optimization.
At least on Windows Slack is very memory-hungry, here's mine right now on my work laptop: http://i.imgur.com/76RQeHv.png
No idea why, it's only joining one team with under 20 people and ~5 channels.
No idea why, it's only joining one team with under 20 people and ~5 channels.
Fun Fact! The Spotify desktop client is written in C++! https://www.quora.com/What-is-the-technology-behind-the-Spot....
But I agree, Atom and Slack are really useful to if they consume another 250mb of ram because "It's not optimal", let them use it.
But I agree, Atom and Slack are really useful to if they consume another 250mb of ram because "It's not optimal", let them use it.
The view layer is powered by Chromium Embedded Framework, so it's not entirely C++.
Very true! I wonder how much more efficient the c++ makes the app in contrast to the node based electron infrastructure
Most of the time Slack is behaving quite well but I've experience it running into 80+% CPU for long periods of time while eating 2-3GiB of RAM, I always forced quit and didn't bother. But yes, sometimes it turns into a resource hog, same with VSCode or Atom, something sometimes makes them feel unresponsive and "laggy".
[deleted]
how's that even possible? I mean just the kernel + mds_store and mdworker of mac take like 2gb of memory.
I also run two java processes with 1gb predefined (well probably I could fine tune them to use 512mb). and one angular/cli which accounts for 800mb. plus my database which got 500mb aswell.
spotify on mac used way more than 500mb memory, the last time I used it and also had an extreme amount of i/o pressure. atom actually did use at least 500 mb memory as well, with just the c# plugin.
I also run two java processes with 1gb predefined (well probably I could fine tune them to use 512mb). and one angular/cli which accounts for 800mb. plus my database which got 500mb aswell.
spotify on mac used way more than 500mb memory, the last time I used it and also had an extreme amount of i/o pressure. atom actually did use at least 500 mb memory as well, with just the c# plugin.
Plenty. Discord, VSCode, and MongoBooster are all apps that I feel hit 90% of native performance while still being based on web tech. Hyper terminal is another one which has very specific reproducible performance issues, but overall is pretty great and fast.
Its incompetence on behalf of Slack and Github, not the tech itself.
Its incompetence on behalf of Slack and Github, not the tech itself.
I've always found Fastmail's mobile app to be amazingly native-like even though it uses HTML5.
I honestly don't understand why you need a desktop app to run chat.
[deleted](3)
Once you tried Discord you wonder why people are still using Slack.
Btw, Discord also surpassed Slack in traffic on SimilarWeb.
Btw, Discord also surpassed Slack in traffic on SimilarWeb.
It is even more surprising that by just adding a few privacy features, they can start targeting the corporate market.
I would gladly pay them.
I would gladly pay them.
Privacy is a huge concern. It's difficult to phrase this properly without sounding like an asshole, but Discord's development team has a lot of very young people (no proof, but I'd swear some of them are 13-16 years old). If you've ever listened in on their Discord voice channels, the level of immaturity is astounding. Power to them - it's great to see talented youth create a great product. I just wouldn't trust them with a business's confidential information.
Discord's specifics aside, it blows my mind that people use any off-site SAAS for internal communications, including Slack. The database set contains so much critical information, including infrastructure passwords, private SSL and SSH certs, etc. Employees will send anything and everything over the company's chat client. Even if you trust the company's employees and general security, the fact remains that your uploaded attachments typically have a public URL not requiring authentication to download.
Discord's specifics aside, it blows my mind that people use any off-site SAAS for internal communications, including Slack. The database set contains so much critical information, including infrastructure passwords, private SSL and SSH certs, etc. Employees will send anything and everything over the company's chat client. Even if you trust the company's employees and general security, the fact remains that your uploaded attachments typically have a public URL not requiring authentication to download.
> Discord's development team has a lot of very young people (no proof, but I'd swear some of them are 13-16 years old)
Someone spreading FUD?
Everything with Discord feels like a super mature well develoved product.
Someone spreading FUD?
Everything with Discord feels like a super mature well develoved product.
The technology is there to solve this. End to end encryption solves this problem and lets the devs wash their hands off the privacy concerns.
End-to-end encryption is essentially useless/unsolved when it comes to large, dynamic groups like Slack channels. Any new member who joined wouldn't be able to see any history in any channel.
But this is not a problem of Discord but of all group chats. The initial discussion point was that Discord might be insecure because of young hackers building the product.
Do you find Discord to perform better than Slack? Any thoughts as to why?
I've used Discord for gaming related purposes on another computer, but it didn't feel... "better" to me. Sluggish at times, but in understandably large rooms with lots of traffic.
I'm mainly curious in your statement as to how Discord performs compared to Slack, and if better, how they achieve "better".
I've used Discord for gaming related purposes on another computer, but it didn't feel... "better" to me. Sluggish at times, but in understandably large rooms with lots of traffic.
I'm mainly curious in your statement as to how Discord performs compared to Slack, and if better, how they achieve "better".
Onboarding is a millions times easier.
With Slack you need for every chat a new login/password and 100 clicks for sign-up, skip tutorial and email-verification, WTH why? Getting users into Discord is one single click. With Discord you have one login and you can have different names per server.
And Slacks admin pages are just a confusing pile of mess, my .vimrc is light-weight compared. Slack is so overrated and people just use it because everybody does.
Discord has never been sluggish for me.
However, both are timewasters.
With Slack you need for every chat a new login/password and 100 clicks for sign-up, skip tutorial and email-verification, WTH why? Getting users into Discord is one single click. With Discord you have one login and you can have different names per server.
And Slacks admin pages are just a confusing pile of mess, my .vimrc is light-weight compared. Slack is so overrated and people just use it because everybody does.
Discord has never been sluggish for me.
However, both are timewasters.
> Slack is so overrated and people just use it because everybody does.
Sidenote, I use it because the integrations take a burden off of me. Ie, my CI might not have integrations for Discord. My CI has plugins, and do those have integrations for Discord too? Etcetc. Being the most popular is not always just due to "we use it because everyone does".
Personally, I dislike how expensive Slack is. I'm not sure if I prefer Slack over Discord or not, but even if I preferred Discord, I'd still choose Slack for the integrations that I don't have to write myself.
Sidenote, I use it because the integrations take a burden off of me. Ie, my CI might not have integrations for Discord. My CI has plugins, and do those have integrations for Discord too? Etcetc. Being the most popular is not always just due to "we use it because everyone does".
Personally, I dislike how expensive Slack is. I'm not sure if I prefer Slack over Discord or not, but even if I preferred Discord, I'd still choose Slack for the integrations that I don't have to write myself.
> My CI has plugins, and do those have integrations for Discord too?
Yes and if not there is a Slack-compatible API, check Discord's API doc.
Just try Discord, takes one minute and will know yourself if you like it. It's not a behemoth like Slack.
Yes and if not there is a Slack-compatible API, check Discord's API doc.
Just try Discord, takes one minute and will know yourself if you like it. It's not a behemoth like Slack.
I've already used it, quite a bit, as I mentioned. Not for work specifically, though. And never the API of course. Heck, I probably have it open at home right now haha.
Discord is too branded as a gaming tool. No way I could convince any professionals to use it.
FWIW, I tried Discord recently and it felt more sluggish and heavier than Slack on macOS.
I made the opposite experiece and just made a test with some medium and similar sized chat servers with many users amd history (bitcoin related chat servers) on Slack and Discord in Chrome which should represent their native Electron apps:
Data from Chrome Inspector's network tab:
Data from Chrome Inspector's network tab:
Slack Discord
Req. 93 103
Transf.22.4MB 5.2MB
Finish 39.19s 6.48s
DomCL. 6.53s 4.01s
Load 21.70s 4.16s
Do you use discord at work?
I do for outside-asking-for-help. At my current client internal communications have to be self-hosted so neither Slack nor Discord is an option; if I were still at the startup where I worked previously (where teams chose their own tools) I'd certainly be pushing Discord.
Discord feels very responsive and lightweight. Anyone knows what their stack looks like?
They have some fantastic dev blogs[1]. Their backend seems to mostly be elixir.
1: https://blog.discordapp.com/tagged/engineering
1: https://blog.discordapp.com/tagged/engineering
Elixir for the backend as others have said, and the frontend is React on Electron.
Elixir for the backend, they have around 5 million concurrent users.
Electron
It's a wonder that any of these services enjoy success when IRC exists.
As an IRC, Slack, and Matrix user and netadmin, I will say that Slack and other nextgen chat services have a few advantages over IRC:
- automatic backlog resumption
- server stored pageable backlog
- continuous presence (you don't disappear from the channel when you leave, you're just marked as away)
- authentication built into the spec rather than implemented using a "services" layer
- general upgrades to what is allowed in the chat, such as long pastes, inline images, and file uploads
If you haven't tried Matrix yet (the best client is probably Riot at https://riot.im/ ), I encourage you to give it a try. It's essentially next generation IRC.
- automatic backlog resumption
- server stored pageable backlog
- continuous presence (you don't disappear from the channel when you leave, you're just marked as away)
- authentication built into the spec rather than implemented using a "services" layer
- general upgrades to what is allowed in the chat, such as long pastes, inline images, and file uploads
If you haven't tried Matrix yet (the best client is probably Riot at https://riot.im/ ), I encourage you to give it a try. It's essentially next generation IRC.
That sounds like having an irc client connected to say ZNC. It has all of the same features and has been around for years..
Sure, but a bouncer is another layer on top of your connection to the server, when architecturally it's much better to have the server manage that for you.
Why should end users care about the architecture? It's architecturally a hell of a lot better than a massive electron app that eats all of your RAM!
There are a million more embedded features in Slack than there are in IRC. Sure someone could spend a year trying to implement all of those plugins in their private IRC server and it be less functional and user friendly, but why would someone want to waste their time and money doing that?
IRC as a service. Allows users to pay a static amount each month and have a solid chat solution.
"We use the dev build of React. It makes it easier for us to debug issues our customers might be experiencing. "
Is it that hard to debug the production build?
Is it that hard to debug the production build?
I wonder if they were for some reason unable to emit a sourcemap of their js bundle. Unless they are doing something special here, that should be sufficient for debugging.
Surprising given that Slack's performance is something people regularly complain about
Or.... it's not surprising at all.
Surprising because it's probably the lowest hanging fruit possible
That really does explain why the RAM usage is so damn bad. Comparing it to Discord for example is night and day.
I've pretty much resorted to using slack in my browser. It all runs so much better.
Actually that is exactly what I'd been doing. I recently upgraded to 32 GB of RAM and now the desktop app is usable again, but obviously this is still not even reasonable.
Really, I just wish there was a terminal slack client. It'd probably run so much better.
WTF. I remember running a network of 10+ servers and worker-machines on a 4GB hosts not many years ago. And now you can't run a "modern" chat-app without having 32GBs of ram?
You know what? I'm happy with IRC. weechat uses maybe 28MBs and runs stellar.
Maybe you should just consider that the tools you are using simply are flawed and move to something proven instead?
You know what? I'm happy with IRC. weechat uses maybe 28MBs and runs stellar.
Maybe you should just consider that the tools you are using simply are flawed and move to something proven instead?
Not true. Slack does not take 32GB of RAM to be usable, more likely the rest of his tools take 30GB of RAM. My laptop only has 8GB of RAM and I have Chrome and Firefox open and active while using Slack. Slack has been open for a couple of weeks and is using less than 500mb of RAM on my machine.
This is correct- my daily environment usually involves multiple VMs and a Docker Compose setup, either a JetBrains IDE or Visual Studio Code, Chrome with some reasonable number of tabs open, Slack, and a few utilities. To further illustrate the reason why this sounds insane, before I was at 32GB, I was at 8GB, and conserved RAM by running Linux with an extremely lean setup (i3 + st + Chrome.) Now I have plenty to spare, and I'm using a much more rich environment.
So yeah, it is a little more complicated than just "Slack needed 32GB" - in fact, and thankfully, even my whole stack fits comfortably. It's what I upgraded from that it didn't work well on.
So yeah, it is a little more complicated than just "Slack needed 32GB" - in fact, and thankfully, even my whole stack fits comfortably. It's what I upgraded from that it didn't work well on.
Someone needs to explain Slack what sourcemaps are and what they are used for ;P
I gave up on the Slack desktop client after regularly finding it using 3GB of RAM for no obvious reason. Switched to the browser client but found it buggier and needed to be refreshed periodically due to becoming so slow that there was a noticeable delay on keypresses. Now I'm using Slack via XMPP using Pidgin and things are much, much better.
You're all looking at this from a dev perspective ... look at Slack's customer base.
Slack Enterprise is expensive for a messaging service, and one of their main selling points is support and reliability. When a huge customer reports a breaking bug they need to resolve it ASAP, turnaround time is probably the number-one priority.
Slack Enterprise is expensive for a messaging service, and one of their main selling points is support and reliability. When a huge customer reports a breaking bug they need to resolve it ASAP, turnaround time is probably the number-one priority.
Would this be a factor in Slack's consistent 20% CPU usage when it's idle? (electron app/linux)
[deleted]
This makes absolutely no sense. In no way would the debug output help them remotely. Hopefully the visibility will now cause this to get fixed.
[deleted]
Spotify was also doing this until very recently
The Wall Street Journal [1] also does this.
[1] https://www.wsj.com
[1] https://www.wsj.com
More evidence of my theory that the worst solution for the majority always seems to win right now...
That's short-sighted. Electron sucks, but the fact people use it means there's a lack of proper, easy solutions for multiplatform development.
Companies will always take the easy path because capitalism. It's up to us (native toolkit devs) to make the easier path better.
Companies will always take the easy path because capitalism. It's up to us (native toolkit devs) to make the easier path better.
Bundling an X11 server and writing your app in Motif/Lesstif would be easier and less resource intensive honestly.
Is there a platform out there that has no X server?
Is there a platform out there that has no X server?
>Is there a platform out there that has no X server?
XQuartz on Mac OS X sucks (maybe it's just Inkscape though) and I'm not sure if there's decent Windows support.
Qt might work, though.
XQuartz on Mac OS X sucks (maybe it's just Inkscape though) and I'm not sure if there's decent Windows support.
Qt might work, though.
>XQuartz on Mac OS X sucks
I've used it, it wasn't bad at all but it definitely clashed visually (not that electron apps don't.)
>I'm not sure if there's decent Windows support.
Cygwin with X11 is one of the first things I install on my work machine whenever I get a new job. It can't do fancy GTK stuff super well but most of the apps I use work great.
I've used it, it wasn't bad at all but it definitely clashed visually (not that electron apps don't.)
>I'm not sure if there's decent Windows support.
Cygwin with X11 is one of the first things I install on my work machine whenever I get a new job. It can't do fancy GTK stuff super well but most of the apps I use work great.
The tweet discussed is about their web version
But they don't have a desktop version. Their so called native apps are html in a browser.
I believe this is a response saying that electron made cross platform easy but at the cost of performance. Native clients still aren't as easy to build as html+css+js and the barrier to entry or cost to maintain is too high vs wrapping the web app you already built.
cool, but I still get very high CPU when a gif from giphy is running in background macOS Slack app!
I can't really imagine what type of benefit they believe they're getting. Your error tracking system does re-assemble the stack from a out-of-band delivered sourcemap, even in prod, even with multiple versions deployed simultaneously... right? Like any decent error tracking system such as Rollbar or Sentry will do out of the box. Tell me you didn't build your own and can't even unwind a production error trace.