Hardware Security Exploit Research – Xbox 360(github.com)
github.com
Hardware Security Exploit Research – Xbox 360
https://github.com/kooscode/srgh-matrix-trinity/blob/main/README.md
101 comments
Have you seen Tony Chen's (development lead for Xbox One security) description of the Xbox 360 reset glitch hack at [1] and the effect this (and other console exploits of that time) had on Xbox One development?
This is one of my go-to case study videos for the development effort required to architect a computer to resist attackers who have physical access.
[1] https://youtu.be/U7VwtOrwceo?t=536
This is one of my go-to case study videos for the development effort required to architect a computer to resist attackers who have physical access.
[1] https://youtu.be/U7VwtOrwceo?t=536
[deleted]
Congratulations, haven't had a reason to mess with it myself, but I've heard it described online as the most secure piece of consumer hardware before or since
I think you might be mixing up the Xbox 360 with the Xbox One, the former was ultimately compromised in several ways, but the latter's security has held up extremely well for 11 years and counting. The Xbox One and its successor are easily the most secure consoles ever made.
Obligatory: https://www.youtube.com/watch?v=U7VwtOrwceo
Obligatory: https://www.youtube.com/watch?v=U7VwtOrwceo
> The Xbox One and its successor are easily the most secure consoles ever made.
Microsoft also allowed any console to switch to developer mode and run homebrew, massively reducing the need for people to try find exploits.
Microsoft also allowed any console to switch to developer mode and run homebrew, massively reducing the need for people to try find exploits.
Found it! I misremembered, it was making a slightly different claim
"The Xbox 360 hypervisor is probably the most secure piece of code Microsoft has ever written." from the excellent article Tony Hawk's Pro Strcpy
https://icode4.coffee/?p=954
"The Xbox 360 hypervisor is probably the most secure piece of code Microsoft has ever written." from the excellent article Tony Hawk's Pro Strcpy
https://icode4.coffee/?p=954
The Xbox 360 was overall a very, very secure device. While we don't know exactly how the folks who discovered the hypervisor syscall handler bug were able to get plaintext, it's theorized that it came from development kit and SDK leaks. With an SDK and dev kit someone could dump boot loaders and the HV.
Otherwise on a retail console you can't do much. The hard drives are not encrypted but all content that can possibly contain code / save data is signed. Save data cannot contain code but introduces scripting engine / save parsing attack surface, but you can't modify it without first dumping keys from a retail console.
To dump keys from a retail console you have to get code exec in the hypervisor. To attack the hypervisor you have be able to dump the hypervisor to audit it.
To dump the hypervisor you have to be able to read its contents or dump it from flash. The flash is encrypted with a per-console key (and I don't think you can sniff the bus?) and RAM is encrypted.
Realistically if it weren't for the original syscall handler bug and dev kits getting into researcher's hands, the Xbox 360 may have never been hacked.
Otherwise on a retail console you can't do much. The hard drives are not encrypted but all content that can possibly contain code / save data is signed. Save data cannot contain code but introduces scripting engine / save parsing attack surface, but you can't modify it without first dumping keys from a retail console.
To dump keys from a retail console you have to get code exec in the hypervisor. To attack the hypervisor you have be able to dump the hypervisor to audit it.
To dump the hypervisor you have to be able to read its contents or dump it from flash. The flash is encrypted with a per-console key (and I don't think you can sniff the bus?) and RAM is encrypted.
Realistically if it weren't for the original syscall handler bug and dev kits getting into researcher's hands, the Xbox 360 may have never been hacked.
Stupid question, is the reason that people cannot simply dump the ROM as they do with say routers is that the rom is encrypted? But if they have the SDK they can decrypt it?
The flash chip is encrypted with the console's CPU key, and the CPU key is unique per-console and encoded in efuses. So even if one person manages to dump keys they're mostly useless for hacking other consoles. The exception to this is the "keyvault" which is the console's own private key used for signing save games. You can take save games from console A and load them on console B, so console B is able to verify console A's signature based off the public key certificate embedded in the save. Microsoft had a revocation process for revoking keyvaults if they ever leaked but they just gave up once too many were in the wild.
Dev kits are keyed differently and most of the console's keys for signing / encryption are in various SDK DLLs that if you reverse engineer you can find.
Dev kits are keyed differently and most of the console's keys for signing / encryption are in various SDK DLLs that if you reverse engineer you can find.
Thanks, MS did take a lot of efforts on the security. Searching for X360 keyvault gives a lot of webpages. I'll read a bit.
https://free60.org/ probably has all the information you need.
Thank you! Now I need to buy a used XBox 360...
I have a hard time believing the ‘since’ part of that description. Intuition suggests the latest iPhone would take that crown each year.
> the latest iPhone would take that crown each year
Apple continuously patches zero-day kernel exploits against the latest iOS and hardware, https://support.apple.com/en-us/100100
Apple continuously patches zero-day kernel exploits against the latest iOS and hardware, https://support.apple.com/en-us/100100
Sure, but there are many more people looking at iPhone security than at Xboxes. The incentives, both monetary and otherwise, are much greater with iPhones than they ever were with consoles.
That is far from the only thing that would be relevant for such a metric. (For one thing, you also have to ask which kernel.)
iPhone and iPad devices have been breached by zero-days for years, unlike Xbox One.
Still simplistic. "Breached" can mean a number of things, and the incentives here are very different.
Breached = remote control of device, bypass all vendor and owner security policy, and exfiltration of non-public data.
And that was achieved?
Multiple times via iOS zero days. One example, https://www.theguardian.com/technology/2024/dec/20/whatsapp-...
> NSO Group Technologies, was accused in a lawsuit by Meta’s messaging app of infecting and surveilling the phones of 1,400 people over a two-week period in May 2019 via its notorious Pegasus software. The judge.. found the company had violated state and federal US hacking laws.. was used to infiltrate not only WhatsApp but also iPhones to extract pictures, emails and texts.. victims of the hack identified by Meta were senior government officials, journalists, human rights activists, political dissidents and diplomats.
> NSO Group Technologies, was accused in a lawsuit by Meta’s messaging app of infecting and surveilling the phones of 1,400 people over a two-week period in May 2019 via its notorious Pegasus software. The judge.. found the company had violated state and federal US hacking laws.. was used to infiltrate not only WhatsApp but also iPhones to extract pictures, emails and texts.. victims of the hack identified by Meta were senior government officials, journalists, human rights activists, political dissidents and diplomats.
5 years ago, by state level actors. How many state level actors are trying to breach the Xbox? Your original statement was “there are kernel vulnerabilities being fixed, therefore iPhones are less secure than the Xbox one”. It’s just not that simple. I can’t make any assessment about which of those are more secure without a serious look, but this isn’t it.
“How many state level actors are trying to breach the Xbox?”
The U.S. Guvmint for one: https://filesharingtalk.com/content/1654-US-government-hires...
The U.S. Guvmint for one: https://filesharingtalk.com/content/1654-US-government-hires...
Misremembered, it was saying it was the most secure code Microsoft has written, not anyone
From the excellent: https://icode4.coffee/?p=954
From the excellent: https://icode4.coffee/?p=954
What are the reasons for why Microsoft wanted to lock down consoles to only run signed code? As a games console manufacturer, what are the business reasons for doing so? Thanks
Limiting piracy is the ongoing reason, but there is also the historical reason of the Video game crash of 1983 which led to Nintendo's Seal of Quality.
Essentially as the platform owner, you want to ensure games sold for the platform "just work", and if you have a bunch of third parties running bad software, consumers would lose faith in the platform altogether.
Essentially as the platform owner, you want to ensure games sold for the platform "just work", and if you have a bunch of third parties running bad software, consumers would lose faith in the platform altogether.
> ensure games sold for the platform "just work"
To add a little more color to this, it wasn't solely to ensure games worked. The lesson of the video game crash was that third party publishers would make knock-off games similar to popular titles and flood the market with them at much lower cost - sometimes as low as $5 vs for a $40 for a top title. These games were generally low budget and rushed to market to capitalize on looking like a top-selling title - while being just different enough to (hopefully) avoid trademark infringement.
These games usually "worked" (as in booting up and playing), the issue was more that that they were just bad versions of the title they were ripping off due to having little development time and minimal play testing along with poorer artwork and fewer levels (thus saving ROM memory). The flood of cheap, bad versions of more popular games is credited as the main factor that killed the Atari VCS.
Another big factor was that later console manufacturers charged game publishers a license fee for the proprietary library code required for a console to run a game. This fee could allow manufacturers to sell game consoles at cost or even below cost and recoup the lost profit over time in the per game license fee.
This wasn't always the case in the early days of hardware cartridge systems. Initially, some early console manufacturers didn't charge much more than a game publisher could buy blank cartridges for from a third party. Some other manufacturers chose to generate revenue simply by building more margin into the wholesale price they charged game publishers for blank cartridges. Of course, when console manufacturers started increasing their cartridge profit margin, game publishers were motivated to use third party cartridges - which led to console makers deploying "genuine hardware" checks or, later, disc checks and encryption. Nintendo popularized enforcing their business model both technically and legally (by requiring an IP license). Today, console manufacturer business models rely on 1) Collecting per game license fees, 2) Blocking piracy, 3) Limiting game supply.
There is a lot of interesting history around how game console business models and the legal landscape evolved over time. (https://en.wikipedia.org/wiki/Atari_Games_Corp._v._Nintendo_....)
To add a little more color to this, it wasn't solely to ensure games worked. The lesson of the video game crash was that third party publishers would make knock-off games similar to popular titles and flood the market with them at much lower cost - sometimes as low as $5 vs for a $40 for a top title. These games were generally low budget and rushed to market to capitalize on looking like a top-selling title - while being just different enough to (hopefully) avoid trademark infringement.
These games usually "worked" (as in booting up and playing), the issue was more that that they were just bad versions of the title they were ripping off due to having little development time and minimal play testing along with poorer artwork and fewer levels (thus saving ROM memory). The flood of cheap, bad versions of more popular games is credited as the main factor that killed the Atari VCS.
Another big factor was that later console manufacturers charged game publishers a license fee for the proprietary library code required for a console to run a game. This fee could allow manufacturers to sell game consoles at cost or even below cost and recoup the lost profit over time in the per game license fee.
This wasn't always the case in the early days of hardware cartridge systems. Initially, some early console manufacturers didn't charge much more than a game publisher could buy blank cartridges for from a third party. Some other manufacturers chose to generate revenue simply by building more margin into the wholesale price they charged game publishers for blank cartridges. Of course, when console manufacturers started increasing their cartridge profit margin, game publishers were motivated to use third party cartridges - which led to console makers deploying "genuine hardware" checks or, later, disc checks and encryption. Nintendo popularized enforcing their business model both technically and legally (by requiring an IP license). Today, console manufacturer business models rely on 1) Collecting per game license fees, 2) Blocking piracy, 3) Limiting game supply.
There is a lot of interesting history around how game console business models and the legal landscape evolved over time. (https://en.wikipedia.org/wiki/Atari_Games_Corp._v._Nintendo_....)
I think it's also worth pointing out that the console makers (and developers) pour a lot more resources into ensuring that the products released for their platform are of a suitable quality than, say, phone app store gatekeepers.
A big draw as well is that people can't (within the economic viability timeframe of the games/console) hack the games on a console, meaning you get a much more predictable online experience than you might on PC.
A big draw as well is that people can't (within the economic viability timeframe of the games/console) hack the games on a console, meaning you get a much more predictable online experience than you might on PC.
There was a period of time when this was true but at least with nintendo, the eshop is full of shovelware now
Microsoft actually reversed course on this. You can make a one-time purchase to access "developer mode" and then run whatever you want. It's been suggested that this is the reason there's been less interest in hacking the Xbox. Ironically it also means you have more computational freedom on the Xbox than on the iPhone/iPad.
> then run whatever you want
Not quite. You (were? I don't know if this is changed) limited in how much of the hardware you could access: it wasn't 100% access. Enough for most homebrew, emulators and so on, but it wasn't carte blanche "replacement for a dev-kit" access.
Not quite. You (were? I don't know if this is changed) limited in how much of the hardware you could access: it wasn't 100% access. Enough for most homebrew, emulators and so on, but it wasn't carte blanche "replacement for a dev-kit" access.
I don't think it much different from the official games, which also don't have full access to the console.
The "not-behind-the-paywall-and-NDA" GDK version is severely more limited than the invite-only GDKX.
IIRC the homebrew you can run is mostly UWP stuff? But if you want to launch a _game_, built for an Xbox, it requires to be in the program.
IIRC the homebrew you can run is mostly UWP stuff? But if you want to launch a _game_, built for an Xbox, it requires to be in the program.
No it was quite different (again, unless they changed it? They might've, I'm long out of this scene). UWP apps had a lot more restrictions on them, the SDK was different, and you didn't have the full consoles hardware to use, compared to the "behind the NDA" SDK: it'd run slower, basically.
Still great, and good enough for most use cases
Still great, and good enough for most use cases
[deleted]
> and if you have a bunch of third parties running bad software, consumers would lose faith in the platform altogether.
Famously the reason no one ever used Microsoft Windows.
Famously the reason no one ever used Microsoft Windows.
> Famously the reason no one ever used Microsoft Windows.
Microsoft in its early days invested a shit ton of money and effort into backwards compatibility testing and fix development. Up until Windows 7 you could be reasonably sure that any piece of software from the Windows 95 32-bit days would still work without major issues - even 16-bit software would run under a 32-bit W7 host, only W7 x64 finally dropped support for that.
Microsoft in its early days invested a shit ton of money and effort into backwards compatibility testing and fix development. Up until Windows 7 you could be reasonably sure that any piece of software from the Windows 95 32-bit days would still work without major issues - even 16-bit software would run under a 32-bit W7 host, only W7 x64 finally dropped support for that.
I think they even run windows 3.1 games back then. It is very impressive. Actually I think some still do. But the ones using WinG do not, like Fury3.
Yet Microsoft bought Bethesda
They sell the consoles at a loss, so if you could port your own games to the consoles instead of buying the games that they could take a royalty from then they lose money. It doesn't have to be an effective circumvention to trigger the DMCA making it illegal.
They sell the console at a loss but make 30% on every game sold. The business model is variant of "dumping" but antitrust isn't enforced anymore.
If it were possible to go the wrong way on the expanding brain meme, this would be it.
Nobody is buying consoles for the hardware. 99% of the product is software. The accounting that you are using, a popular one, is so opppositely-informative that people who make consoles - and smartphones for that matter - clearly do not make decisions with it. It is 200% wrong to characterize it as dumping. Nothing is being dumped.
Here’s a simple idea for you: show me the vibrant market for Xbox 360 2005 era computer hardware. There isn’t any right? What about Xbox One 2013 era? And yet we still play games that were developed earlier than 2013, like League of Legends. The product is software. Nobody loses anything by being unable to run Linux on 2013 hardware today, and nobody loses anything by being unable to run Linux on the Xbox One in 2013 because, if they wanted cheap computers then, they had plenty to choose from!
Nobody is buying consoles for the hardware. 99% of the product is software. The accounting that you are using, a popular one, is so opppositely-informative that people who make consoles - and smartphones for that matter - clearly do not make decisions with it. It is 200% wrong to characterize it as dumping. Nothing is being dumped.
Here’s a simple idea for you: show me the vibrant market for Xbox 360 2005 era computer hardware. There isn’t any right? What about Xbox One 2013 era? And yet we still play games that were developed earlier than 2013, like League of Legends. The product is software. Nobody loses anything by being unable to run Linux on 2013 hardware today, and nobody loses anything by being unable to run Linux on the Xbox One in 2013 because, if they wanted cheap computers then, they had plenty to choose from!
Both the hardware and software are products. No one selling open platform PCs can compete with negative margin console sales, which is anticompetive. If antitrust prevented dumping consoles would be more expensive and consumers would buy PCs instead. In both software and hardware the open PC platform is far more competitive which drives value for consumers.
> No one selling open platform PCs can compete with negative margin console sales
This makes zero sense. Both have existed simultaneously forever, and hundreds of millions of people eagerly buy both for the same households. I cannot understand your point of view here, other than invoking a word "dumping" and "anticompetive" that you are using 200% wrong. Consoles and open platform PCs do not compete with each other.
> In both software and hardware the open PC platform is far more competitive which drives value for consumers.
The market for high budget single player games exists solely because of DRM protected consoles. So this category of product, that people eagerly have paid for for decades, to the tune of hundreds of billions of dollars, would cease to exist if you required console makers to allow people to bypass DRM. Ask 20 people in the game industry and 19 would agree. I understand the core and spirit of what you are saying, but it is reflecting your aspirations for a world that doesn't exist. Markets aren't art exhibits!
This makes zero sense. Both have existed simultaneously forever, and hundreds of millions of people eagerly buy both for the same households. I cannot understand your point of view here, other than invoking a word "dumping" and "anticompetive" that you are using 200% wrong. Consoles and open platform PCs do not compete with each other.
> In both software and hardware the open PC platform is far more competitive which drives value for consumers.
The market for high budget single player games exists solely because of DRM protected consoles. So this category of product, that people eagerly have paid for for decades, to the tune of hundreds of billions of dollars, would cease to exist if you required console makers to allow people to bypass DRM. Ask 20 people in the game industry and 19 would agree. I understand the core and spirit of what you are saying, but it is reflecting your aspirations for a world that doesn't exist. Markets aren't art exhibits!
That both exist does not make negative margin sales not an anticompetive practice. Dumping is the correct term for such anticompetive behavior. Consoles are just x86 PCs. They run the same application software.
The Witcher 4 is a high budget single player game and sold 30 million copies despite having no DRM. I don't think you're familiar with the games industry.
The Witcher 4 is a high budget single player game and sold 30 million copies despite having no DRM. I don't think you're familiar with the games industry.
Just curious do we know how much loss for one hardware piece? What's the margin on extra controllers and peripherals?
There are firms that specialize in competitive analysis and cost estimates. The margin on controllers and peripherals is fairly high.
https://www.gamespot.com/articles/ps3-manufacturing-costs-do...
https://www.gamespot.com/articles/ps3-manufacturing-costs-do...
Thank you!
They are rent seekers.
I’m not sure that’s a fair perspective, they built a proprietary product that’s intended to be a loss leader, the opposite of that is a desktop which costs substantially more.
It would only cost more upfront, definitely not over the life of the console.
Rent seekers don't do anything for your benefit. The definition of false economy.
The only difference between their "Proprietary product" and a PC is DRM.
Rent seekers don't do anything for your benefit. The definition of false economy.
The only difference between their "Proprietary product" and a PC is DRM.
business reasons, the console is sold as a loss leader and the real money is made in licensing the games. also serves as brand protection i.e. preventing poor quality third party games from tarnishing the reputation of the console.
A games console provided a platform where they could more effectively argue that “their” works “““needed””” to be protected so they could farm us (people who want to run their own code on hardware they purchased) for digital-jail technologies which would never otherwise have reason to exist. Then those technologies can metastasize fully-formed over to general-purpose computing in a way that's harder to argue against. They learned with Clipper and Palladium that trying to develop jail tech on PC would be vehemently opposed.
The opposition was pointless though, like everything has TPM/IME/etc. nowadays so we lost that war awhile ago. I don't see how consoles helped them win that war though.
When everything is locked down for good I'll go back to physical books. Vaporware games and media are not worth preserving anyway. And the games and media I do have free from prison can entertain me for a very long time.
Another possible (even worse) future could be cloudification of everything. Enjoy your thin client.
Of course software is bigger than entertainment which might represent a problem. We're increasingly societally locked into this digital shit.
Another possible (even worse) future could be cloudification of everything. Enjoy your thin client.
Of course software is bigger than entertainment which might represent a problem. We're increasingly societally locked into this digital shit.
Can you speak to some of the harder or more interesting challenges you faced during that time?
One challenge was that while I started working on the Xbox 360 about three years before it would ship, we knew that the custom CPU would not be available until early 2005 (first chips arrived in early February). And there was only supposed to be one hardware spin before final release.
So I had no real hardware to test any of the software I was writing, and no other chips (like the Apple G5 we used as alpha kits) had the custom security hardware or boot sequence like the custom chip would have. But I still needed to provide the first stage boot loader which is stored in ROM inside the CPU weeks before first manufacture.
I ended up writing a simulator of the CPU (instruction level), to make progress on writing the boot code. Obviously my boot code and hypervisor would run perfectly on my simulator since I wrote both!
But IBM had also had a hardware accelerated cycle-accurate simulator that I got to use. I was required to boot the entire Xbox 360 kernel in their simulator before I could release the boot ROM. What takes a few seconds on hardware to boot took over 3 hours in simulation. The POST codes would be displayed every so often to let me know that progress was still being made.
The first CPU arrived on a Friday, by Saturday the electrical engineers flew to Austin to help get the chip on the motherboard and make sure FSB and other busses were all working. I arrived on Monday evening with a laptop containing the source code to the kernel, on Tuesday I compiled and flashed various versions, working through the typical bring-up issues. By Wednesday afternoon the kernel was running Quake, including sound output and controller input.
Three years of preparation to make my contribution to hardware bring-up as short as possible, since I would bottleneck everyone else in the development team until the CPU booted the kernel.
So I had no real hardware to test any of the software I was writing, and no other chips (like the Apple G5 we used as alpha kits) had the custom security hardware or boot sequence like the custom chip would have. But I still needed to provide the first stage boot loader which is stored in ROM inside the CPU weeks before first manufacture.
I ended up writing a simulator of the CPU (instruction level), to make progress on writing the boot code. Obviously my boot code and hypervisor would run perfectly on my simulator since I wrote both!
But IBM had also had a hardware accelerated cycle-accurate simulator that I got to use. I was required to boot the entire Xbox 360 kernel in their simulator before I could release the boot ROM. What takes a few seconds on hardware to boot took over 3 hours in simulation. The POST codes would be displayed every so often to let me know that progress was still being made.
The first CPU arrived on a Friday, by Saturday the electrical engineers flew to Austin to help get the chip on the motherboard and make sure FSB and other busses were all working. I arrived on Monday evening with a laptop containing the source code to the kernel, on Tuesday I compiled and flashed various versions, working through the typical bring-up issues. By Wednesday afternoon the kernel was running Quake, including sound output and controller input.
Three years of preparation to make my contribution to hardware bring-up as short as possible, since I would bottleneck everyone else in the development team until the CPU booted the kernel.
You have an awesome memory, Dinarte!
Eric Mejdric from IBM called on Friday and said we have the chips, when are you guys getting here?
I took a red eye that night and got to Austin on Saturday morning.
We brought up the board, the IBM debugger, and then got stuck.
I remember calling you on Sunday morning. You had just got a big screen TV for the Super bowl and had people over and in-between hosting them you dropped us new bits to make progress.
I think Tracy came on Sunday or Monday and with you got the Kernel booted.
Good times!
This is Harjit by the way.
Edit: added super bowl.
Eric Mejdric from IBM called on Friday and said we have the chips, when are you guys getting here?
I took a red eye that night and got to Austin on Saturday morning.
We brought up the board, the IBM debugger, and then got stuck.
I remember calling you on Sunday morning. You had just got a big screen TV for the Super bowl and had people over and in-between hosting them you dropped us new bits to make progress.
I think Tracy came on Sunday or Monday and with you got the Kernel booted.
Good times!
This is Harjit by the way.
Edit: added super bowl.
OMG Harjit! I saw you in the documentary! You and the entire team are total rockstars! I just cannot fathom ever being in a position to design something that provided so much joy and happy core memories to countless people around the world...you guys did it!
Just the thought of how many people you touched with your work....just amazing! :)
Had a question if you don't mind: Can you talk about the thought process behind the power supply design? Its very large even in the super slim models. Were you following a specific design driven by the hardware architecture or were there other reasons? I always wondered about that.
Just the thought of how many people you touched with your work....just amazing! :)
Had a question if you don't mind: Can you talk about the thought process behind the power supply design? Its very large even in the super slim models. Were you following a specific design driven by the hardware architecture or were there other reasons? I always wondered about that.
>I saw you in the documentary!
I presume you're referring to this one: https://www.xbox.com/en-US/power-on#watch
I presume you're referring to this one: https://www.xbox.com/en-US/power-on#watch
Yes! It is worth watching every second. What an amazing production.
Actually Tracy never made it to Austin. He was going to fly in later in the week to continue bring-up, but since we were done by Wednesday, we just sent the chips to Redmond and he continued there. He was of course always available on the phone to answer my kernel questions I had.
This is really some blast from the past. Can you please shed more light on the simulator? Is it interpretation or JIT? But then I realize XBOX uses Pentium III, so maybe virtualization? Edit Sorry it was XBOX 360 so it's not Pentium.
As someone who recently got interested in emulation and wrote two lc-3 emulators, would really love to learn from the masters.
As someone who recently got interested in emulation and wrote two lc-3 emulators, would really love to learn from the masters.
Kick ass. This kind of post is why HN is the best.
damn, that's bad ass. did that simulator run on a Windows system or was it something more esoteric?
I called the simulator Sbox and it was just a simple console app. I didn't implement the GPU, so no graphics just the hypervisor and kernel and some simple non-graphics apps. I made it so that you could build the Xbox 360 kernel on your windows machine, then just run sbox.exe and it would automatically find the just built kernel image targeting the PPC64 and boot it. Then if you typed control-C it would drop into the kernel debugger as a sub process, and you could poke around at the machine state as if it were the real Xbox hardware, showing all the PPC instructions and registers. It was a lot of fun writing it, and quite useful.
You should also talk about the lwarx/stecx bug. IIRC - in the first version of the chip there was a bug in one or both of these instructions. Your code booted on SBox but didn't on the hardware. You compared the two and then figured out it was these instructions.
You filed a bug report and then dug into them and used SBox to figure out what must have been going wrong.
The chip supplier came back with a workaround and within five minutes you simulated it on SBox and said it wouldn't work, why, and then said how it should be fixed.
The supplier didn't believe you as yet. And you worked out a workaround so we could be unblocked. Two weeks later they agreed with your fix...
You filed a bug report and then dug into them and used SBox to figure out what must have been going wrong.
The chip supplier came back with a workaround and within five minutes you simulated it on SBox and said it wouldn't work, why, and then said how it should be fixed.
The supplier didn't believe you as yet. And you worked out a workaround so we could be unblocked. Two weeks later they agreed with your fix...
I recall an issue when trying to use lwarx/stwcx on Xbox 360 directly that the compiler (or maybe even the kernel, on program load? it's been a while) raised an error and said to use the Interlocked intrinsics instead -- is that related?
So the PPC instruction set uses lwarx (load word and reserve indexed), and stwcx (store word conditional indexed), along with variations for word size, to implement atomic operations such as interlocked-increment and test-and-set.
So on PPC interlocked-increment is implemented as:
loop: lwarx r4,0,r3 # Load and reserve r4 <- (r3) addi r4,r4,1 # Increment the value stwcx. r4,0,r3 # Store the incremented value if still reserved bne- loop # Loop and try again if lost reservation
The idea is that the lwarx places a reservation on an address that it wants to update at some later time. It doesn't prevent any other thread or processor from reading or writing to that address, or cause any sort of stall, but if an address being reserved is written to, conditional or otherwise, then the reservation is lost. The stwcx instruction will perform the store to memory if the reservation still exists clears the NE flag, otherwise it doesn't do the write and sets the NE flag and software should just try again until it succeeds.
On the Xbox 360 we provided the compiler which would emit sequences like these for all atomic intrinsics, but developers could also write assembler code directly if they wanted to. We'll get back to this point in a moment.
As the V1 version of the Xbox 360 CPU was being tested by IBM, they discovered that an error with the hardware implementation of these two instructions and issued an errata for software to work around it, which we implemented. Unfortunately, after further testing IBM discovered that the errata was insufficient, so issued a second errata, which we also implemented and assumed all was well.
Then the V2 version of the CPU comes out and months go by. But early one morning I get a phone call from IBM letting me know that the latest errata was still insufficient and that the bug is in the final hardware. Further, Microsoft has already started final production of CPU parts, even before full testing was fully complete (risk buy), so that they could have sufficient supply for the upcoming November release. I was told that they are stopping manufacturing of additional CPUs, and that I had 48 hours to figure out if there is anything software can do that could work around the hardware issue. They also casually mentioned that millions of dollars of parts would need to be discarded, a hardware fixed implemented which would take weeks, then the production could resume from scratch.
Bottom line is that, yes, there was a set of software changes that would work around the bug, but it required very specific sequences of instructions, the disabling of interrupts around these sequences, a change to the hypervisor, and updating the compiler to emit the new sequences. To make sure that developers didn't introduce code sequences that uses lwarx/stwcx in a way that would expose the bug (via inline assembly, for example), the loader would scan the code and refuse to load code that didn't obey the new rules.
Interesting fact: the hardware bug existed in every version of the Xbox 360 ever shipped, because software needed to run on any console ever shipped, there was no advantage to ever fixing the bug since software always needed to work around it anyway.
So on PPC interlocked-increment is implemented as:
loop: lwarx r4,0,r3 # Load and reserve r4 <- (r3) addi r4,r4,1 # Increment the value stwcx. r4,0,r3 # Store the incremented value if still reserved bne- loop # Loop and try again if lost reservation
The idea is that the lwarx places a reservation on an address that it wants to update at some later time. It doesn't prevent any other thread or processor from reading or writing to that address, or cause any sort of stall, but if an address being reserved is written to, conditional or otherwise, then the reservation is lost. The stwcx instruction will perform the store to memory if the reservation still exists clears the NE flag, otherwise it doesn't do the write and sets the NE flag and software should just try again until it succeeds.
On the Xbox 360 we provided the compiler which would emit sequences like these for all atomic intrinsics, but developers could also write assembler code directly if they wanted to. We'll get back to this point in a moment.
As the V1 version of the Xbox 360 CPU was being tested by IBM, they discovered that an error with the hardware implementation of these two instructions and issued an errata for software to work around it, which we implemented. Unfortunately, after further testing IBM discovered that the errata was insufficient, so issued a second errata, which we also implemented and assumed all was well.
Then the V2 version of the CPU comes out and months go by. But early one morning I get a phone call from IBM letting me know that the latest errata was still insufficient and that the bug is in the final hardware. Further, Microsoft has already started final production of CPU parts, even before full testing was fully complete (risk buy), so that they could have sufficient supply for the upcoming November release. I was told that they are stopping manufacturing of additional CPUs, and that I had 48 hours to figure out if there is anything software can do that could work around the hardware issue. They also casually mentioned that millions of dollars of parts would need to be discarded, a hardware fixed implemented which would take weeks, then the production could resume from scratch.
Bottom line is that, yes, there was a set of software changes that would work around the bug, but it required very specific sequences of instructions, the disabling of interrupts around these sequences, a change to the hypervisor, and updating the compiler to emit the new sequences. To make sure that developers didn't introduce code sequences that uses lwarx/stwcx in a way that would expose the bug (via inline assembly, for example), the loader would scan the code and refuse to load code that didn't obey the new rules.
Interesting fact: the hardware bug existed in every version of the Xbox 360 ever shipped, because software needed to run on any console ever shipped, there was no advantage to ever fixing the bug since software always needed to work around it anyway.
Thank you so much. This is so awesome to know and learn.
I'm just curious, what are the instructions that replace the lwarx/stwcx "atomic" pair? From my understanding, basic you need to generate a pair of load reserved/save instructions, and you have to replace the pair with a series of instructions. But I don't understand why do you have to disable interrupts -- is it because actually multiple instructions were used to facilitate the load, and an interrupt may disturb a value stored in a register?
Sorry I know little about assembly and arch.
I'm just curious, what are the instructions that replace the lwarx/stwcx "atomic" pair? From my understanding, basic you need to generate a pair of load reserved/save instructions, and you have to replace the pair with a series of instructions. But I don't understand why do you have to disable interrupts -- is it because actually multiple instructions were used to facilitate the load, and an interrupt may disturb a value stored in a register?
Sorry I know little about assembly and arch.
We still used the lwarx/stwcx pair to implement atomic operations, but to avoid the hardware bug a strict rule needed to be followed.
Rule: On a given hardware thread (there are two hardware threads per processor on the Xbox 360), every lwarx reservation of an address must be paired with a stwcx conditional store to that same address before a reservation is made to a different address. So a sequence like lwarx A / lwarx B / stwcx B / stwcx A is forbidden. But lwarx A / stwcx A / lwarx B / stwcx B is fine.
So I changed the compiler to emit atomic intrinsics that obeyed this rule.
But there was still the issue of logical thread scheduling. Imagine there are two logical threads running, one has a sequence of lwarx A / stwcx A and the other has lwarx B / stwcx B. The first thread is running on a hardware thread and just after executing lwarx A, the timer interrupt fires and the kernel decides to switch to the second logical thread, which executes lwarx B, and thus violates the rule.
To make sure that never happens, the compiler also emits disable-interrupts / lwarx A / stwcx A / enable-interrupts. That prevents the scheduler from switching threads in the middle of the atomic sequence.
But there was still one more problem. It is possible for a page-fault to occur in the middle of the sequence should it span the end of one page and the beginning of another, and the second page is not in the TLB. So the thread is running along and executes disable-interrupts / lwarx A, then when trying to fetch the next instruction it faults to the hypervisor because it isn't yet mapped by the TLB. The hypervisor executes a bunch of code to add the mapping of the new page to the TLB and then returns to the faulting thread to complete the stwcx A / enable-interrupts sequence.
The problem is that the TLB is a shared resource between the two hardware threads of a processor, so the two hardware threads need a way to atomically update the TLB, and the obvious way to do that is to use a spin-lock that is naturally implemented by a lwarx B / stwcx B pair of instructions. But the hypervisor TLB handler can't use those instructions because the code causing the TLB fault might be in the middle of using them and thus would cause the hardware bug to manifest.
The solution was to use non-reservation load/store instructions to implement a simple spin-lock. If the two hardware threads were both trying to update the TLB then hardware thread 2 would simply wait for hardware thread 1 to clear its lock before proceeding.
Rule: On a given hardware thread (there are two hardware threads per processor on the Xbox 360), every lwarx reservation of an address must be paired with a stwcx conditional store to that same address before a reservation is made to a different address. So a sequence like lwarx A / lwarx B / stwcx B / stwcx A is forbidden. But lwarx A / stwcx A / lwarx B / stwcx B is fine.
So I changed the compiler to emit atomic intrinsics that obeyed this rule.
But there was still the issue of logical thread scheduling. Imagine there are two logical threads running, one has a sequence of lwarx A / stwcx A and the other has lwarx B / stwcx B. The first thread is running on a hardware thread and just after executing lwarx A, the timer interrupt fires and the kernel decides to switch to the second logical thread, which executes lwarx B, and thus violates the rule.
To make sure that never happens, the compiler also emits disable-interrupts / lwarx A / stwcx A / enable-interrupts. That prevents the scheduler from switching threads in the middle of the atomic sequence.
But there was still one more problem. It is possible for a page-fault to occur in the middle of the sequence should it span the end of one page and the beginning of another, and the second page is not in the TLB. So the thread is running along and executes disable-interrupts / lwarx A, then when trying to fetch the next instruction it faults to the hypervisor because it isn't yet mapped by the TLB. The hypervisor executes a bunch of code to add the mapping of the new page to the TLB and then returns to the faulting thread to complete the stwcx A / enable-interrupts sequence.
The problem is that the TLB is a shared resource between the two hardware threads of a processor, so the two hardware threads need a way to atomically update the TLB, and the obvious way to do that is to use a spin-lock that is naturally implemented by a lwarx B / stwcx B pair of instructions. But the hypervisor TLB handler can't use those instructions because the code causing the TLB fault might be in the middle of using them and thus would cause the hardware bug to manifest.
The solution was to use non-reservation load/store instructions to implement a simple spin-lock. If the two hardware threads were both trying to update the TLB then hardware thread 2 would simply wait for hardware thread 1 to clear its lock before proceeding.
Thanks so much for the input! I vaguely know a little bit about everything you talked about--the threads, TLB and such, but I have never worked with them in practice. This is so interesting.
That sounds like a lot of fun! What was your career like before that, that led you to be in the place to do such fun things?
This is the coolest HN post I’ve read in months.
Cheers.
Cheers.
Small world! I worked on Yellow Door / Golden Gate automation for releasing 360 titles and patches to prod, and the beta group / KDC service code.
You sir are an inspiration! I am but a mediocre Angular developer and reading this has me in complete awe! The kind of drive you must have had to get this done well I dont know how people manage to do it but it is so cool to see! :)
> we knew that the custom CPU would not be available until early 2005
Sounds a little bit like the situation with Xbox Series? The SDKs were released late because Microsoft was waiting for certain features in AMD APU
Sounds a little bit like the situation with Xbox Series? The SDKs were released late because Microsoft was waiting for certain features in AMD APU
Amazing! Thanks for sharing! What sort of things are you working on now?
How did you feel about teenagers and college students exploiting holes in your work? Were you impressed, disappointed, amused, etc.?
Oh and I'd just like to say thank you for your contribution to my childhood/adolescence.
Oh and I'd just like to say thank you for your contribution to my childhood/adolescence.
What was the culture like working on this project and back in those days? I’ve always been fascinated by the development of consoles, especially the story of the 360. Any sources you recommend to learn more? I thought the Microsoft documentary on Xbox was the best I’ve found so far.
The Winchester revision is still considered unhackable afaik. It's crazy how many Xbox 360 revisions there were compared to other consoles.
Was it ever explained why? This was an unanswered question I always wondered about from time to time. They must have done something to remove RGH capability?
Yes that revision is patched to specifically counter RGH. Microsoft disabled the ability to get the precise timing needed from the CPU and also added more filtering/robustness so the system will reset properly instead of getting into the inconsistent state of the old revisions.
Wait until you read Grimdoomer’s (yet to be published) write-up on his recent hypervisor exploit that works on the latest OS: https://x.com/grimdoomer/status/1847497275289063676
It’s going to be a doozy
It’s going to be a doozy
I feel like random delays would make the glitch attack harder but it would still be possible given enough attempts. Seems like the bigger issue is that you can glitch the CPU reset line which corrupts the processing rather than having no effect or resetting the CPU.
I assume those are probably very hard to fix since (again, I assume, I'm just a hobbyist in the hardware space) that sort of glitch relies on propagation delays (e.g. a short burst triggering some latches but not others, or triggering the latches in some specific synchrony).
Can anyone confirm if I'm on the right track with my guess?
Can anyone confirm if I'm on the right track with my guess?
I’m surprised this was a task that only took a single developer!
Having a single developer allows fewer offices with their windows completely covered with newspaper. Plus, there's one person doing everything, which can be a lot better than two with people who have different ideas of how to make the system work together.
> So - here is a hopefully informative write up of my Journey to figuring out how these guys were running unsigned code in 2011 on a XBOX 360
> XBOX 360 Security defeated - 2011
I realize this post is more about hardware security than software security, but if the benchmark is unsigned code execution then the author should at least mention the 2007 (King Kong shader exploit) and 2009 (SMC hack — same root flaw but executed automatically at boot) methods of achieving the same:
- https://github.com/Free60Project/wiki/blob/master/docs/Hacks...
- https://github.com/Free60Project/wiki/blob/master/docs/Hacks...
> XBOX 360 Security defeated - 2011
I realize this post is more about hardware security than software security, but if the benchmark is unsigned code execution then the author should at least mention the 2007 (King Kong shader exploit) and 2009 (SMC hack — same root flaw but executed automatically at boot) methods of achieving the same:
- https://github.com/Free60Project/wiki/blob/master/docs/Hacks...
- https://github.com/Free60Project/wiki/blob/master/docs/Hacks...
Back in the day I really liked my 360 whereas my ONE seems like a mistake.
Yeah, I still have an Xbox One hooked up (along with a SOTA gaming PC) but I now have more fun playing retro games via emulation than modern games.
Lately I've been having a blast exploring the X360 library via Xenia (since I had a PS3 during that era and never got to see some of these titles).
Lately I've been having a blast exploring the X360 library via Xenia (since I had a PS3 during that era and never got to see some of these titles).
Fun thinking back to the days of cat and mouse between MS and hackers so many cool things went down. My favorite being hackers making custom firmware for certain dvd burners so that you could burn right to the very edge of 7.5gb discs and make 8gb games that MS started using. Truncated games were being detected and then hackers came up with this idea it was so genius.
Then came along the reset glitch hack and I moved away from discs to an external hard drive and never looked back. I did a few for me and a couple friends. The soldering involved was pretty precise in the fact that it was a very very small pad you needed to solder to and if you screwed up it was very easy to lift the pad and putting yourself into a big heap of trouble fixing it. I was also using a crappy $15 soldering iron with a bad tip because I was poor. But never did I have an issue. Depending on your install you could get the glitch to happen sometimes on the first reset or for some it took multiple resets. I was happy because all mine seemed to work first if not second reset which a lot of people struggled to get. I still have my rgh 360 my kids have it with a hdd full of games I backed up from my own games you know.
Then came along the reset glitch hack and I moved away from discs to an external hard drive and never looked back. I did a few for me and a couple friends. The soldering involved was pretty precise in the fact that it was a very very small pad you needed to solder to and if you screwed up it was very easy to lift the pad and putting yourself into a big heap of trouble fixing it. I was also using a crappy $15 soldering iron with a bad tip because I was poor. But never did I have an issue. Depending on your install you could get the glitch to happen sometimes on the first reset or for some it took multiple resets. I was happy because all mine seemed to work first if not second reset which a lot of people struggled to get. I still have my rgh 360 my kids have it with a hdd full of games I backed up from my own games you know.
> Generating VERY precise timing and pulses, you need FPGA's or CPLD's
Back in the day, I managed to create a working RGH modchip based on Atmega8 (8 bit micro) running at 20 MHz with hand-crafted assembly code. I named it RWH (Reset Witch Hack) and it was able to boot Xbox 360 Jasper in 1-2 minutes. Old motherboards had a physical pad on the motherboard allowing for slowing down the CPU, so no i2c was required. I also have to connect the whole 8 bit POST bus to read the current value in one instruction.
PCB was made at home, and since AVR is 5V system, I used NPNs for voltage conversion (all values were inverted in the software).
Why? I didn't have money to buy the "real" CPLD modchip.
Rush and happiness when it first booted - priceless.
I still should have the source code for it somewhere on backup.
photos: https://gist.githubusercontent.com/JaCzekanski/c02ed11c30fac... https://gist.githubusercontent.com/JaCzekanski/c02ed11c30fac...
Back in the day, I managed to create a working RGH modchip based on Atmega8 (8 bit micro) running at 20 MHz with hand-crafted assembly code. I named it RWH (Reset Witch Hack) and it was able to boot Xbox 360 Jasper in 1-2 minutes. Old motherboards had a physical pad on the motherboard allowing for slowing down the CPU, so no i2c was required. I also have to connect the whole 8 bit POST bus to read the current value in one instruction.
PCB was made at home, and since AVR is 5V system, I used NPNs for voltage conversion (all values were inverted in the software).
Why? I didn't have money to buy the "real" CPLD modchip.
Rush and happiness when it first booted - priceless.
I still should have the source code for it somewhere on backup.
photos: https://gist.githubusercontent.com/JaCzekanski/c02ed11c30fac... https://gist.githubusercontent.com/JaCzekanski/c02ed11c30fac...
Fun article. One note:
> I have a Saleae 8 channel 100Mhz, which turned out not to be fast enough > I found a not too expensive 200Mhz Kingst LA2016 Logic Analyzer on Amazon
The author is confusing MHz with MS/s (mega samples per second). Salaea has a logic analyzer that works on 100MHz signals (with 500 MS/s), but I suspect the author had the unit with 100 MS/s that only works up to 25MHz signals.
The cheap Kingst unit has 200 MS/s but only works with signals up to 40MHz.
> I have a Saleae 8 channel 100Mhz, which turned out not to be fast enough > I found a not too expensive 200Mhz Kingst LA2016 Logic Analyzer on Amazon
The author is confusing MHz with MS/s (mega samples per second). Salaea has a logic analyzer that works on 100MHz signals (with 500 MS/s), but I suspect the author had the unit with 100 MS/s that only works up to 25MHz signals.
The cheap Kingst unit has 200 MS/s but only works with signals up to 40MHz.
> Note - Newer revisions of XBOX 360 has no access to CLK and you must use Matrix oscillator
If there's no CLK line on the mobo, does this mean newer X360s have everything that might be clocked (I assume at least CPU, GPU and V/RAM?) in a single chip, SoC-like?
If there's no CLK line on the mobo, does this mean newer X360s have everything that might be clocked (I assume at least CPU, GPU and V/RAM?) in a single chip, SoC-like?
I'm not aware of any reason nor modern computer system that would have these things clocked by a common clock line of any kind.
Today, GPUs connected via PCIe or the like use 8b/10b coding over differentially-signaled pairs. The signal itself has clock recovery.
(V)RAM is generally clocked at a different frequency than the CPU as well, and all DDR utilizes strobes to determine when data is valid because access time is variable.
In some SOCs/FPGA-based devices, a central clock generator will sometimes provide LVCMOS/HCSL/LVDS/etc. clock lines to each device, but these aren't often re-used. This allows for flexibility and later programmability. There's generally no assumed phase or frequency relationship between these derived clocks and the original source - especially after the signal has traveled 20cm across a board.
In the case of a CPU/GPU, though, a 20 cent crystal oscillator at each device feeding into internal PLLs is typically the go-to.
Today, GPUs connected via PCIe or the like use 8b/10b coding over differentially-signaled pairs. The signal itself has clock recovery.
(V)RAM is generally clocked at a different frequency than the CPU as well, and all DDR utilizes strobes to determine when data is valid because access time is variable.
In some SOCs/FPGA-based devices, a central clock generator will sometimes provide LVCMOS/HCSL/LVDS/etc. clock lines to each device, but these aren't often re-used. This allows for flexibility and later programmability. There's generally no assumed phase or frequency relationship between these derived clocks and the original source - especially after the signal has traveled 20cm across a board.
In the case of a CPU/GPU, though, a 20 cent crystal oscillator at each device feeding into internal PLLs is typically the go-to.
If you are not familiar with the fascinating story behind it, I recommend this podcast episode [1], it's one of my absolute favorites (along with its sequel)! I found out about it on this very forum a few years ago, hope to propagate the favor to somebody else out there.
[1] https://darknetdiaries.com/episode/45/
[1] https://darknetdiaries.com/episode/45/
[deleted]
Note to self: you should have added random delays before and after making the POST code visible on the external pins.