Fans of a 2013 Japanese soccer game tracked down a bizarre floating point bug(dolphin-emu.org)
dolphin-emu.org
Fans of a 2013 Japanese soccer game tracked down a bizarre floating point bug
https://dolphin-emu.org/blog/2021/09/07/dolphin-progress-report-august-2021/
50 comments
In the spec, the pseudocode defines them as
IEEE 754-2008 only defines that you have to provide a fusedMultiplyAdd() operation, so it's up to the CPU architecture whether folding in the free negations is something they want to provide, and if so, where exactly the negation is done.
FMADD: addend + ( op1 * op2)
FMSUB: addend + (-op1 * op2)
FNMADD: -addend + (-op1 * op2)
FNMSUB: -addend + ( op1 * op2)
IEEE 754-2008 only defines that you have to provide a fusedMultiplyAdd() operation, so it's up to the CPU architecture whether folding in the free negations is something they want to provide, and if so, where exactly the negation is done.
The negated variant make sense compared to the original, it's really the definition of msub which is strange, I would expect msub to subtract the last operand from the multiplication, not the other way around. The AMD64 mnemonics actually encode this order e.g. VFMADD213PDy is 2 * 1 + 3 (ba + c) while VFMADD132PDy is 1 * 3 + 2 (ac+b).
Though apparently the Dolphin folks disagree either way and interpret nmadd as (nm)add, rather than n(madd).
Though apparently the Dolphin folks disagree either way and interpret nmadd as (nm)add, rather than n(madd).
The Arm ARM says FNMADD is "Floating-point Negated fused Multiply-Add", so the position of the hyphens points strongly to "n(madd)" rather than "(nm)add".
> ARM says FNMADD is "Floating-point Negated fused Multiply-Add", so the position of the hyphens points
Never mind the hyphens, what do they think a "Negated fused Multiply" is?
Never mind the hyphens, what do they think a "Negated fused Multiply" is?
[deleted]
The order of negation in those instructions may have been due to a very low-level hardware optimisation: it could save some time and circuit area to have the negation happen "inside" instead of "after", because the former is part of existing circuitry to do FP multiplication (ermember that FP is sign-magnitude) whereas the latter would require an additional stage on the output.
{paragraphs of subtle floating point math differences between architectures}
“We've learned not to question it.”
Truly a veteran developer crew! Congrats on the amazing fix.
“We've learned not to question it.”
Truly a veteran developer crew! Congrats on the amazing fix.
Its always fun to read Dolphin Progress Reports
The original title: Dolphin Progress Report: August 2021
I've emailed the mods to suggest a change.
I've emailed the mods to suggest a change.
Honest question:
Hows the experience with dolphin and the remotes ??
I assume you have to buy a wii-remote to have a decent experience ?
The few times i tried to play games with keyboard or gamepad(logitech) it kinda was frustrating !
*Otherwise - good job on a great project !
The few times i tried to play games with keyboard or gamepad(logitech) it kinda was frustrating !
*Otherwise - good job on a great project !
Depends on what you want to play (a regular controller is a fine substitute for the Wii Classic Control Pad) but the Wiimote works well over Bluetooth.
Funny that they are finally starting to bypass a few of the protections of the closed-source version of the Homebrew channel. This highlights how hard it can be to accurately emulate some hardware (mis)features.
To be fair, we deliberately used very obscure hardware "features" which we knew were not implemented by any emulators and probably not used by any games to build these protections :-)
I'd have to dig up the old code but I'm fairly sure some of them rely on an operating system (Nintendo IOS, unrelated to both Cisco's IOS and Apple's iOS) running on the co-processor (nicknamed "Starlet"). Dolphin doesn't emulate that part at all because IOS exposes a high-level interface that can just be emulated instead. Works amazingly well for games, but will probably trip our protections.
I'd have to dig up the old code but I'm fairly sure some of them rely on an operating system (Nintendo IOS, unrelated to both Cisco's IOS and Apple's iOS) running on the co-processor (nicknamed "Starlet"). Dolphin doesn't emulate that part at all because IOS exposes a high-level interface that can just be emulated instead. Works amazingly well for games, but will probably trip our protections.
What I find interesting in addition to that is how much slack most software allows. I remember lurking on the alt groups when emus were being discussed for things like an amiga or pc. They were talking sub cycle accuracy would be totally necessary for anything to work at all. Yet most software seems pretty chill with 'sort of close' results. It is oddly counter intuitive. Intuitively they were right but ended up being wrong (mostly). Some bits though you need that accuracy. Mostly you dont.
I think a lot of software companies would do like what you did. Where they would check for some sort of thing that should be there, or not, or too much of something. But the emulator would or would not have it. I think in many cases it would be things like checking to see if 512k of memory was available, when the real box would have 128k. That was usually for things like copiers I think?
I think a lot of software companies would do like what you did. Where they would check for some sort of thing that should be there, or not, or too much of something. But the emulator would or would not have it. I think in many cases it would be things like checking to see if 512k of memory was available, when the real box would have 128k. That was usually for things like copiers I think?
Yup, I'm pretty sure many software companies have done similar things. It's a bit easier on video game consoles where your software runs on the very same hardware everywhere and you can pull off more subtle tricks. There were a few commercial games that also used a similar approach [1].
[1] https://dolphin-emu.org/blog/2017/02/01/dolphin-progress-rep...
[1] https://dolphin-emu.org/blog/2017/02/01/dolphin-progress-rep...
Is it weird that you sound proud about this? Is there something noble about doing work that makes lives harder for hobbyists trying to preserve video games for the future, and has nearly zero impact on the actual sales of the game at release?
> Is it weird that you sound proud about this? Is there something noble about doing work that makes lives harder for hobbyists trying to preserve video games for the future, and has nearly zero impact on the actual sales of the game at release?
What makes you think we did it for any of those reasons? And how does us abusing hardware bugs make video game preservation any harder?
This was not for a commercial game, this was for an entry point to load your own software on a locked down video game console.
And back then people were selling our (free!) software and we added those protections to make sure we could show a "you have been scammed if you paid for this" screen that couldn't be removed. Unfortunately that also meant breaking our loader from running in emulators, but that didn't matter at all: Those could just directly launch actual .elf files anyway and didn't need the detour through the homebrew channel.
The code (minus those protections) for the Homebrew Channel is also available as open source these days.
What makes you think we did it for any of those reasons? And how does us abusing hardware bugs make video game preservation any harder?
This was not for a commercial game, this was for an entry point to load your own software on a locked down video game console.
And back then people were selling our (free!) software and we added those protections to make sure we could show a "you have been scammed if you paid for this" screen that couldn't be removed. Unfortunately that also meant breaking our loader from running in emulators, but that didn't matter at all: Those could just directly launch actual .elf files anyway and didn't need the detour through the homebrew channel.
The code (minus those protections) for the Homebrew Channel is also available as open source these days.
I'm curious how well can my PC handle emulated console games (esp. newer gens). Where can I get some ROMs?
For research purpose, of course.
For research purpose, of course.
This "magic" URL showing the most-favorited items on the Internet Archive is very interesting.
https://archive.org/advancedsearch.php?q=*&fl%5B%5D=identifi...
https://archive.org/advancedsearch.php?q=*&fl%5B%5D=identifi...
[deleted]
I think changing game to video game would make the title less confusing.
Ya, I initially imagined it was some passionate sports fans with a sharp eye for sportscast overlay graphics/animations noticed some subtle glitches that made it back to some rendering engine.
I agree. I thought it's a bug with a software used by the Japanese football association.
I think floating point would give the assumption that it has nothing to do with meatspace, but it would allow room for a fascinating surprise if it did!
I vote for keeping it as is.
I vote for keeping it as is.
If we're following HN guidelines correctly, the title should be 'Dolphin Progress Report: August 2021' -- what has been submitted has been editorialised.
I overlooked the URL. Seeing that or the original title I would have understood what this was.
But now, my first guess was actually that this was about some much debated football match and that the floating point bug was present in some software used by the referees when reviewing replays.
But now, my first guess was actually that this was about some much debated football match and that the floating point bug was present in some software used by the referees when reviewing replays.
Football referees didn't review replays in 2013.
I can personally attest to creating and fixing floating point bugs to do with real life football games.
Counterpoint: I assumed real life scoreboard.
I was immediately trying to figure out if it somehow had to do with a stop watch being used in meatspace.
tokai(1)
That's a great initiative by the fans right here
"This makes sense, so of course nothing else does it this way." :)
Is the "broken" approach something from the Intel-designed x86 or the AMD-designed AMD64? And is aarch64 just following what the PC world did? Seems like it.
Too bad PowerPC fell by the wayside.
Too bad PowerPC fell by the wayside.
Nothing was really broken, the problem was just that the 3 CPU architectures, POWER, Intel/AMD and 64-bit ARM include in their instruction sets slightly different FMA instructions, so porting a program between them requires care.
The fused multiply-add instructions (FMA) have been used for the first time in 1990, in the first IBM POWER CPUs.
Some years later FMA instructions have been included in many other CPUs from various vendors, the main exceptions being the Intel/AMD CPUs (where the first CPU with FMA was AMD Bulldozer in 2011, 21 years later than POWER) and the ARM CPUs, which also added FMA quite late, but before the introduction of the 64-bit variant.
The problem is that both Intel and ARM have chosen to define FMA instructions that differ between themselves and also from POWER, in the position of the negation signs.
With real numbers of unlimited precision, that would not have mattered, because the negation is commutative with the multiplication and distributive over addition.
However with floating-point numbers these mathematical properties do not hold exactly, which is what the developers of the Dolphin emulator have discovered now.
This is a general rule for any floating-point computation, that changing a formula using associativity/distributivity/commutativity is not guaranteed to produce the same results.
Sometimes the differences are too small to matter, but in this case they made a game unplayable in the emulator, so they had to emulate the exact formula used in the original game, without trying to reduce the execution time by reducing the number of emulated instructions (emulating the original formula requires 2 instructions, while they initially optimized the code to use a single instruction but that computed a formula with the negation sign moved in another place).
The fused multiply-add instructions (FMA) have been used for the first time in 1990, in the first IBM POWER CPUs.
Some years later FMA instructions have been included in many other CPUs from various vendors, the main exceptions being the Intel/AMD CPUs (where the first CPU with FMA was AMD Bulldozer in 2011, 21 years later than POWER) and the ARM CPUs, which also added FMA quite late, but before the introduction of the 64-bit variant.
The problem is that both Intel and ARM have chosen to define FMA instructions that differ between themselves and also from POWER, in the position of the negation signs.
With real numbers of unlimited precision, that would not have mattered, because the negation is commutative with the multiplication and distributive over addition.
However with floating-point numbers these mathematical properties do not hold exactly, which is what the developers of the Dolphin emulator have discovered now.
This is a general rule for any floating-point computation, that changing a formula using associativity/distributivity/commutativity is not guaranteed to produce the same results.
Sometimes the differences are too small to matter, but in this case they made a game unplayable in the emulator, so they had to emulate the exact formula used in the original game, without trying to reduce the execution time by reducing the number of emulated instructions (emulating the original formula requires 2 instructions, while they initially optimized the code to use a single instruction but that computed a formula with the negation sign moved in another place).
They mention precision bit then actually gave the example that if all inputs were exactly 0 that it gave differing results of +0 vs -0. Presumably the game checked the sign of the result and hit an error recovery codepath if it was negative.
That's not really something that unlimited precision would avoid, but also not something that "real" arithmetic suffers from since real arithmetic doesn't have -0 and +0 as distinguishable concepts.
That's not really something that unlimited precision would avoid, but also not something that "real" arithmetic suffers from since real arithmetic doesn't have -0 and +0 as distinguishable concepts.
You are right that for that place in the game this probably exposed a bug in the original game.
Normally testing the sign of a floating-point number that is the result of a multiplication or division should always be preceded by a test for zero, unless you intentionally want to take different paths for zeros with different signs.
Normally testing the sign of a floating-point number that is the result of a multiplication or division should always be preceded by a test for zero, unless you intentionally want to take different paths for zeros with different signs.
Based on the article it sounds like the main difference comes down to calculations resulting in +0 or -0. As long as the architecture is self-consistent, it wouldn’t be a problem. Once that floating point value is sent to another architecture, things could go bad. I suppose sending FP values over the wire and used for dsync detection doesn’t happen often?
> AArch64 nmadd equation is -(A * C) - B [...] Having an add instruction subtract is a curious decision from ARM [...] AArch64's nmsub equation is (A * C) - B, which isn't negated at all. Yet AArch64's msub is negated, for whatever reason, so we used that instead. We've learned not to question it.
Re-arrange those equations a bit, and you get:
And suddenly, it all makes sense :-)