Mozilla finds CPU bug (bad store forwarding) in Samsung Galaxy S20(bugzilla.mozilla.org)
bugzilla.mozilla.org
Mozilla finds CPU bug (bad store forwarding) in Samsung Galaxy S20
https://bugzilla.mozilla.org/show_bug.cgi?id=1833315
77 comments
How do you trace what the JIT is doing in an Android app?
If I recall correctly, there were some TRACE or DEBUG logs one could turn on that would - amid a ton of unhelpful output - also emit logs when the JIT was doing its thing. Then, while running a small reproducer app and combing through the output, we noticed values were always correct before the JIT ran, then when the JIT started to kick in, we had a transition period where sometimes wrong values started to crop up (I assume until all threads had switched to the optimized code), and then always produce the wrong result.
This gave me a hunch that it could be related to JIT, and then I tried completely turning it off, which caused the issue to disappear.
This gave me a hunch that it could be related to JIT, and then I tried completely turning it off, which caused the issue to disappear.
The only thing better than being able to blame the compiler for something, is being able to blame the CPU
Saving that quote.
But how are you going to fix it then?
Change your code to not trigger the bug, and leave that workaround there for the next decade or so until the buggy CPU generation isn't relevant anymore.
Or hope for microcode updates to fix the bug at the CPU level.
Or hope for microcode updates to fix the bug at the CPU level.
That's my point ... it's a lot harder to do this than to just fix it in your own code.
You change the compiler to carefully step around the stuff that doesn't work in the hardware.
Compilers can be patched to automatically do some masking in the code generation step after every LDRB instruction in this case, when compiling for that specific CPU. For example, the Blackfin DSPs have many similar bugs which are fixed this way. Some of which the fixes done automatically, while others could be disabled for performance critical code which has been implemented while keeping the various issues in mind, like -mcsync-anomaly and -mspecld-anomaly.
Certain MIPS CPUs also have incorrectly implemented instructions and the -mfix-* compiler parameters could be used to enable workarounds.
JIT compilers can do the same.
Certain MIPS CPUs also have incorrectly implemented instructions and the -mfix-* compiler parameters could be used to enable workarounds.
JIT compilers can do the same.
Great tshirt
Summary: Samsung's LDRB doesn't zero-extend when reading a byte, so it's not following ARM specs[1]. Fix was to workaround in the JIT[2].
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1833315#c13
[2] https://hg.mozilla.org/mozilla-central/rev/c6a272593c07#l7.1...
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1833315#c13
[2] https://hg.mozilla.org/mozilla-central/rev/c6a272593c07#l7.1...
When I was a kid, I always suspected a compiler or CPU bug when my program did not behave as expected. I heard similar stories on more than several occasions from other people, so being naïve must be a common thing among us software engineers when we make our first steps. Kudos to Mozilla, their finding is a real CPU bug.
It's not a real CPU bug unless you can write a 20 line program in C/assembly and run it to demonstrate the problem.
Then, when you've done that, send it to samsung so they patch it in their CPU design. They may even be able to patch it in their devices in the field (it's common for CPU designs to have special hardware whose purpose is runtime detecting certain trigger conditions and running replacement code to avoid hardware bugs discovered after the CPU was produced).
They can also issue an errata which might let compiler authors modify their code generators to avoid any pattern that triggers the bug, therefore avoiding other projects being impacted.
Then, when you've done that, send it to samsung so they patch it in their CPU design. They may even be able to patch it in their devices in the field (it's common for CPU designs to have special hardware whose purpose is runtime detecting certain trigger conditions and running replacement code to avoid hardware bugs discovered after the CPU was produced).
They can also issue an errata which might let compiler authors modify their code generators to avoid any pattern that triggers the bug, therefore avoiding other projects being impacted.
There are a surprising (or unsurprising depending on your perspective) number of hardware bugs out there. There are published errata that I had hit before they were documented. It’s not always very much fun - and leads to a lot of head scratching.
yeah same, from almost 30 years ago. Interesting shift. I distinctly remember having the instinct that it must be the computer that must be wrong, and what I wrote was obviously correct. Thanks for reminding me of that. With some experience, you figure out that you are wrong %99.99 of the time and after that loop, the thought never crosses your mind. I've never had to deal with hardware bugs myself, but had my share of compiler and platform bugs over the years.
For ages in *nixland the phrase "select is not broken" has been an aphorism for such situations. I'm not sure of its origin, but it's referenced in The Pragmatic Programmer:
> Tip #33, pg. 95:
>
> “select” Isn't Broken
>
> It is rare to find a bug in the OS or the compiler, or even a third-party
> product or library. The bug is most likely in the application.I really don't know when this became "common knowledge". Most OSes and compilers were utter crap not more than 10-20 years ago. We would spend afternoons ranting about MSVC++ bugs. gcc was a joke, and even if quickly improved, does nobody really remember that by the 2010's gcc was so buggy -O3 was barely usable? No Gentoo users around?
I recall doing some Gentoo installs using -Os at least for some packages, just to see how much that would shrink binary sizes.
Documentation / wikis etc. advised against that not just for performance-of-generated-code reasons, but also because of compiler bugs using -Os was considered unreliable.
Note this was with ancient GCC (v2.9.x, early 3.x releases iirc)
Documentation / wikis etc. advised against that not just for performance-of-generated-code reasons, but also because of compiler bugs using -Os was considered unreliable.
Note this was with ancient GCC (v2.9.x, early 3.x releases iirc)
`-O3` was known/documented to be unsafe.
Yeah, I don't know where that comes from. Everything is broken.
If you think select is broken, read the docs and make sure you're using it correctly. If you still think select isn't doing what the docs say, try to reduce your actual use into a minimal test case and see. Go look at the source for select on your OS. Search the internet and see if anyone else ran into the same thing (but be careful, because there's lots of bad leads out there).
If you think select is broken, read the docs and make sure you're using it correctly. If you still think select isn't doing what the docs say, try to reduce your actual use into a minimal test case and see. Go look at the source for select on your OS. Search the internet and see if anyone else ran into the same thing (but be careful, because there's lots of bad leads out there).
Ironically we later found that `select` is broken by design. `poll` was created as an improvement, and syscalls like `epoll` later refined it further.
FD_SETSIZE ought to be enough for anybody
If you step into microcontroller land you run into hardware bugs all the time, at least if you have to write any low-level code. My very first major project I ran into at least 3 silicon bugs which wasted weeks of my life. Rookie mistake: check the errata. Though one of the bugs wasn't in the errata.
It's a good guideline when you're early on in your programming career, but I've been surprised by how many compiler bugs I have independently re-discovered in my career. One of the curses of my job is that I work with ancient compilers. About half a dozen times I have stumbled on a bug, spent the better part of a day debugging my code, only to find that I'm being bitten by a documented GCC bug that was fixed a decade ago.
A few years ago I was doing Android native development. Back then Samsung as well as Huawei (and other cheap Chinese vendors) have been known to be doing weird things to the operating system and in Google Play console there'd often be wild device-specific crashes coming from those vendored Androids.
I never loved Samsung devices but since then I avoid them even more.
I never loved Samsung devices but since then I avoid them even more.
Here’s the first story I read (~11 years ago) with an occurrence like this that cracked my concept of hardware/software divide…
https://www.theregister.com/2012/03/07/amd_opteron_bug_drago...
https://www.theregister.com/2012/03/07/amd_opteron_bug_drago...
It's never a compiler or CPU bug, until it is.
This site has a list of phones with the same CPU as the Samsung Galaxy S20 they mentioned. https://en.wikichip.org/wiki/qualcomm/snapdragon_800/865
I wonder if it happens to them too?
I wonder if it happens to them too?
Depending on the region where they were sold, Samsung Galaxy S20 could use either Snapdragon 865 or Exynos 990 CPUs.
The Mozilla messages fail to mention this information that is essential for any CPU bug, which CPU model was used in the phones with crashes.
It is more likely that the bug is present in the Samsung Exynos 990 CPUs and not in Snapdragon, because Cortex-A77 was used much more widely, so it is improbable for such a bug to not also be noticed elsewhere.
The Mozilla messages fail to mention this information that is essential for any CPU bug, which CPU model was used in the phones with crashes.
It is more likely that the bug is present in the Samsung Exynos 990 CPUs and not in Snapdragon, because Cortex-A77 was used much more widely, so it is improbable for such a bug to not also be noticed elsewhere.
The bug report shows a Mali GPU, which rules out Snapdragon (Adreno GPU). Also, the phone model when googled shows the S20 FE, which has Exynos.
There are two variants.
The standard S20 used Snapdragon in North America, and Exynos in Europe. For the S20FE the Snapdragon version was sold in Europe - which was a selling point by itself.
I did not know that the S20FE had Exynos at all until I just looked it up... Apparently version 1 does, while version 2 uses Snapdragon.
https://www.gsmarena.com/samsung_galaxy_s20_fe-10428.php
The standard S20 used Snapdragon in North America, and Exynos in Europe. For the S20FE the Snapdragon version was sold in Europe - which was a selling point by itself.
I did not know that the S20FE had Exynos at all until I just looked it up... Apparently version 1 does, while version 2 uses Snapdragon.
https://www.gsmarena.com/samsung_galaxy_s20_fe-10428.php
I have it and when I bought it you could get both in Europe. I planned to send it back if it would be exynos...
Given that it's only effecting S20's I'd be surprised if it was the snapdragon 865 as that's in a ton of phones as you point out. I'd expect it to be the Exynos 990, which is Samsung's custom chip for every region but North America and was only used in the S20 series.
Wouldn't be surprised if it was an Exynos bug either. Samsung LSI haven't produced a decent flagship Exynos SOC for multiple generations now. Always behind Snapdragon on overall balance of performance (CPU/GPU/radios), heat and battery life.
I refuse to buy any phone with an Exynos SOC ever since I got screwed with a poorly tuned Exynos 9810 in my S9. Andrei Frumusanu did a great two part article on turning the 9810 when he used to write for Anandtech:
https://www.anandtech.com/show/12615/improving-exynos-9810-g...
https://www.anandtech.com/show/12620/improving-the-exynos-98...
I refuse to buy any phone with an Exynos SOC ever since I got screwed with a poorly tuned Exynos 9810 in my S9. Andrei Frumusanu did a great two part article on turning the 9810 when he used to write for Anandtech:
https://www.anandtech.com/show/12615/improving-exynos-9810-g...
https://www.anandtech.com/show/12620/improving-the-exynos-98...
Meh, decisions decisions. I bought a Pixel 6a which uses Samsung technology for the Google SoC. I bought it anyway because I wanted to install GrapheneOS but since I assumed it was evolved-but-still-Exynos I did hesitate.
There used to be a reason: Exynos variants had unlockable bootloader. Not sure how feasible unlocking is nowadays on the Qualcomm variants.
IIRC locked bootloaders on Qualcomm variants was found on US market phones, with non-US (e.g. Hong Kong) markets having a unlockable bootloader.
One of the crashes listed is https://crash-stats.mozilla.org/report/index/f585f1a9-0ed8-4... . That lists a "Mali-G77" GPU, which isn't used by Qualcomm. So the affected chip is probably the Exynos 990, which uses
* Octa-core (2x2.73 GHz Mongoose M5 & 2x2.50 GHz Cortex-A76 & 4x2.0 GHz Cortex-A55) - Global
* Octa-core (1x2.84 GHz Cortex-A77 & 3x2.42 GHz Cortex-A77 & 4x1.80 GHz Cortex-A55) - USA
(from https://www.gsmarena.com/samsung_galaxy_s20-10081.php )
* Octa-core (2x2.73 GHz Mongoose M5 & 2x2.50 GHz Cortex-A76 & 4x2.0 GHz Cortex-A55) - Global
* Octa-core (1x2.84 GHz Cortex-A77 & 3x2.42 GHz Cortex-A77 & 4x1.80 GHz Cortex-A55) - USA
(from https://www.gsmarena.com/samsung_galaxy_s20-10081.php )
Kudos to these guys for figuring this out from the dump, not a trivial task! These kind of issues can take months to triage if one is unlucky. I remember our team going crazy trying to figure out the reason for random crashes until finally discovering a firmware bug in proprietary hardware.
This is where the vocal minority of HN posters who are in a snit about telemetry are wrong. This is the kind of issue you can discover at global scale, but only if you collect the data.
Even if telemetry is useful (this is a rare example - the vast majority of telemetry is either useless or actively works against the users), it's still a breach of privacy and confidentiality (crash dumps - which are necessary to investigate/reproduce such bugs - can contain highly confidential data including passwords or session cookies).
> but only if you collect the data.
If the problem is impacting the user and they want it fixed they'll happily tell you. This bug (which causes crashes) would've been investigated just as well by asking users "Firefox has crashed - send bug report?", while giving the user the choice to decline if they had sensitive data in the browser's state.
Obviously nobody in their right mind should opt-in to generic telemetry because all the "product improvements" excuses never panned out - in the last decade software has only become worse, more annoying, and less feature-full, so people are making the rational decision and are not opting into something that's not actually benefiting them. But opt-in telemetry in specific cases where the user actually has a problem like this occurrence can work just fine.
> but only if you collect the data.
If the problem is impacting the user and they want it fixed they'll happily tell you. This bug (which causes crashes) would've been investigated just as well by asking users "Firefox has crashed - send bug report?", while giving the user the choice to decline if they had sensitive data in the browser's state.
Obviously nobody in their right mind should opt-in to generic telemetry because all the "product improvements" excuses never panned out - in the last decade software has only become worse, more annoying, and less feature-full, so people are making the rational decision and are not opting into something that's not actually benefiting them. But opt-in telemetry in specific cases where the user actually has a problem like this occurrence can work just fine.
We've had crash reports for about as long as we've had computer programs. The problem is not in the mechanism itself, but in its abuse for (overwhelmingly, non-consensual) behavioral profiling or spying.
Even when not abused to violate our privacy, the end result is often highly questionable. In place of good design, common sense, and user studies, companies try to extrapolate trends/patterns from data where there are none, resulting in mediocre decisions at best, and self-feeding cycles of devolution. ("Users are clicking this button a lot, let's make it larger" - users are actually clicking the button by accident.)
Even when not abused to violate our privacy, the end result is often highly questionable. In place of good design, common sense, and user studies, companies try to extrapolate trends/patterns from data where there are none, resulting in mediocre decisions at best, and self-feeding cycles of devolution. ("Users are clicking this button a lot, let's make it larger" - users are actually clicking the button by accident.)
It looks like this could have been discovered by simply having (anonymised) crash report telemetry? I don't know about others, but that sort of telemetry doesn't sound so bad.
It's very hard to have useful anonymized telemetry. In this case, you actually need at least a partial memory or register dump to understand what's going on, but those can contain sensitive data that the failing code was processing.
Telemetry doesn't have to be 100% anonymized - it can (and in some cases needs) to contain PII, just needs to be collected respectfully and transparently. Ask the user before sending something, and let the user review it. The user can then make their own decision.
Telemetry doesn't have to be 100% anonymized - it can (and in some cases needs) to contain PII, just needs to be collected respectfully and transparently. Ask the user before sending something, and let the user review it. The user can then make their own decision.
> Ask the user before sending something, and let the user review it. The user can then make their own decision.
I under The sentiment, but say you were sending a memory dump as part of your crash reporting. How is a user going to review that?
I under The sentiment, but say you were sending a memory dump as part of your crash reporting. How is a user going to review that?
Personally I don't have a problem with automatically sending the type of telemetry that makes detecting this possible, but Mozilla could also discover this issue by in-house testing on a wide variety of devices, and/or prompting users to submit telemetry after a crash.
It's not feasible for a smaller developer, but it wouldn't be unreasonable for Mozilla to have a big lab bench with like 100 different devices on it that are just constantly running a script to start Firefox and load a suite of test sites.
It's not feasible for a smaller developer, but it wouldn't be unreasonable for Mozilla to have a big lab bench with like 100 different devices on it that are just constantly running a script to start Firefox and load a suite of test sites.
I don't think you could really approach the kind of coverage automatic crash telemetry can give with a bunch of test devices. You will absolutely find a non-zero number of issues with the test devices, but it's not really the same.
Without substantially increasing the cost and work to maintain the test pool, you'll be testing exactly one combination of OS version/patch level/settings, one internet connection medium/carrier, one set of browser config options, etc for each device in the pool.
The investigation into any fault isolated to a single device in the test pool has to start with suspicion of the health of the specific device. The test S20 crashing doesn't tell you that every S20 will crash in the same way, it could be an isolated local fault. Yes you can throw more devices but that's further increasing the cost and effort of maintaining the test pool.
Without substantially increasing the cost and work to maintain the test pool, you'll be testing exactly one combination of OS version/patch level/settings, one internet connection medium/carrier, one set of browser config options, etc for each device in the pool.
The investigation into any fault isolated to a single device in the test pool has to start with suspicion of the health of the specific device. The test S20 crashing doesn't tell you that every S20 will crash in the same way, it could be an isolated local fault. Yes you can throw more devices but that's further increasing the cost and effort of maintaining the test pool.
It's not remotely feasible to test more than a tiny fraction of the variety of devices, configurations, and sites that a product like Firefox is going to encounter in the real world.
I agree you'd never catch everything, but if it's crashing on the latest Samsung flagship phone you could probably catch that.
User volunteered crash telemetry is better than nothing, but automatic crash telemetry is going to have a more representative sample. A compromise would be to grab just a stack trace and phone model / firmware version etc automatically, and a full dump with user affirmative consent; but if that's not compatible with your privacy stance, asking everybody for everything is reasonable but limiting.
Android has an immense variety of devices. Without a representative sample, you wouldn't know which devices to buy. Regional variants sometimes have a huge difference and sometimes not; this bug seems to be tied to non-US Samsung flagships, which you might not buy if you're US based and don't know what people are using.
Android has an immense variety of devices. Without a representative sample, you wouldn't know which devices to buy. Regional variants sometimes have a huge difference and sometimes not; this bug seems to be tied to non-US Samsung flagships, which you might not buy if you're US based and don't know what people are using.
Indeed. Telemetry is often used for good purposes, like improving features that cause trouble to end users and to fix crashes.
I'm not saying bad actors don't exist, just that telemetry is often used for good, with no nefarious goals.
I'm not saying bad actors don't exist, just that telemetry is often used for good, with no nefarious goals.
Your point would be correct if that was the only telemetry being collected. The pile of stuff most things are sending back home contains much that would not be useful here but is potentially privacy breaking (or just purely for marketing purposes with no intention of being useful to the user). Similarly, in either case telemetry is fine if it is opt-in, and if extra is collected the opt-in is repeated (i.e. no stalking via the back door by changing what is collected and sent home after consent is given).
Bullshit, CPU bugs have been found before wide usage of telemetry and will continue to be found in various context, number of them not doing it even if they could.
Telemetry is probably just one more excuse to ship crap in the first place.
Telemetry is probably just one more excuse to ship crap in the first place.
I allow telemetry to Mozilla but no one else
> but only if you collect the data
You spelled 'send crash dumps with the user's permission' wrong.
You spelled 'send crash dumps with the user's permission' wrong.
You're wrong about them being wrong. Nobody is arguing that telemetry could not possibly have any upsides (primarily for the developer), only that the downsides (primarily for users) make the tradeoff unacceptable.
Normal software testing (remember that?) can catch bugs without invasions of privacy.
Have you ever worked on a large-scale software product used by millions of people across thousands of kinds of devices? Telemetry is always going to be useful.
> Telemetry is always going to be useful.
Useful for what? We've had an explosion of telemetry and "analytics" the past decade and software quality and functionality has only gone downhill. In some fields we've outright regressed compared to early 2000s tooling.
A simple example is that despite all the telemetry, manpower and tech innovation, every single mainstream communications product out there is inferior to pre-Microsoft Skype (despite the latter being built with much less manpower and running on much more primitive hardware & bandwidth).
Of course, not all of this devolution is to blame on telemetry alone, but I disagree that telemetry is some sort of requirement or game-changer when it comes to software quality. We've successfully built good quality software before it and within reasonable budgets.
Useful for what? We've had an explosion of telemetry and "analytics" the past decade and software quality and functionality has only gone downhill. In some fields we've outright regressed compared to early 2000s tooling.
A simple example is that despite all the telemetry, manpower and tech innovation, every single mainstream communications product out there is inferior to pre-Microsoft Skype (despite the latter being built with much less manpower and running on much more primitive hardware & bandwidth).
Of course, not all of this devolution is to blame on telemetry alone, but I disagree that telemetry is some sort of requirement or game-changer when it comes to software quality. We've successfully built good quality software before it and within reasonable budgets.
I find Discord more usable than pre-Microsoft Skype, though I still miss the peer-to-peer capability and built in file transfer. Skype admittedly had a better UI in some ways and didn't demand nearly as many resources.
I do agree that it seems that despite increased capabilities across the board and piles and piles of newer and better ways to do things, the quality of consumer software has dropped somewhat.
I do agree that it seems that despite increased capabilities across the board and piles and piles of newer and better ways to do things, the quality of consumer software has dropped somewhat.
always2slow(2)
I had taken over maintenance of a library that allowed predicting satellite orbits from TLE data, which was used by the European Space Agency (and others) for the occasional mobile app.
Sporadically, we were getting reports of strange situations where the altitude of a satellite was way off, by hundreds or thousands of kilometers. The bug was really difficult to track down and reproduce, and by chance one of our Samsung tablets finally started showing the behaviour.
It turned out that Dalvik (the now obsolete Android VM) was figuring out that the TLE calculation was in a hot path and would JIT it to run more efficiently, and the optimized code used some native arithmetic (or was it trigonometic? can't recall) call that had a bug. We saw that the incorrect values started showing up right around the time when the JIT started kicking in. Fortunately, we could turn off JIT for specific devices and we rolled out an update that fixed the issue.
Fun times!