x86 prefixes and escape opcodes flowchart(soc.me)
soc.me
x86 prefixes and escape opcodes flowchart
https://soc.me/interfaces/x86-prefixes-and-escape-opcodes-flowchart.html
52 comments
This site redirects to HN when it notices HN in the referrer.
The fact sites do evil things with these headers is why I configure Firefox with
network.http.referer.XOriginTrimmingPolicy
Referrer links are a dumb idea. Why the hell do you want to know where I'm coming from other than to track me
For targeting advertising expendatures at the site level. If most of my traffic, as revealed by referrer links, comes from social-media-platform-foo and only a little from social-media-platform-bar, then I am likely to spend more on ads from foo than from bar. I'll grant that it is a noisy measure, but doesn't need to be about tracking a particular individual.
Businesses survived just fine before this. Do personalised ads earn more money? Maybe. But they're invasive and if the governments bowed down to the people instead of corporations they'd be just as illegal as stalking a potential customer to harass them when they're most likely to see you
Got same issue with Edge, but works with Firefox :)
Wow, I didn't even notice because I have extensions that strip the referrer header. Excellent.
If you have JavaScript enabled, that is. JWZ at least does the redirect on the server side.
The following is pulled in from `https://soc.me/assets/js/turnBack.js`:
The following is pulled in from `https://soc.me/assets/js/turnBack.js`:
const undesirables = [
"news.ycombinator.com/",
// "reddit.com/", // disable temporaily
"lobste.rs/"
] ;
if (undesirables.find(site => document.referrer.includes(site))) {
window.location.replace(document.referrer);
}
Git history doesn't explain it unfortunately
https://github.com/soc/soc.me/blame/main/assets/js/turnBack....
Although, when we inspect author's profile on lobste.rs, we'll see that he's banned:
https://lobste.rs/~soc [Banned 4 years ago by pushcx: Troll.]
Maybe he's banned from HN as well. And this 'undesirables' is a method of taking some kind of revenge.
https://github.com/soc/soc.me/blame/main/assets/js/turnBack....
Although, when we inspect author's profile on lobste.rs, we'll see that he's banned:
https://lobste.rs/~soc [Banned 4 years ago by pushcx: Troll.]
Maybe he's banned from HN as well. And this 'undesirables' is a method of taking some kind of revenge.
Last comment was just over 5 years ago.
https://news.ycombinator.com/user?id=soc
https://news.ycombinator.com/user?id=soc
Author has said that development is moved to codeberg, but github version was good enough for the "turnBack.js" analysis.
Uh, before I wrote my sibling comment I read 'comment' as 'commit' and because of it I somehow assumed it was a github link... not sure how that happened.
Anyway, the user named 'soc' on HN has "listbite" in the description. I saw this profile, but I think it's not the same guy. But also I wasn't sure so I didn't paste the link to HN at all.
Anyway, the user named 'soc' on HN has "listbite" in the description. I saw this profile, but I think it's not the same guy. But also I wasn't sure so I didn't paste the link to HN at all.
Why are they undesirable though
This is an interesting way to prevent the hug of death. I wonder what the author's reasoning is, also would it really be effective?
I doubt it, the redirect is client-side, I got a flash of the page before the redirect.
If anything, it's going to at least double its traffic this way when people click again assuming they hit back somehow.
I thought I was losing my mind for a second. What a strange script to run...
Copy the URL and manually paste it into a new tab, no referrer then.
open in new tab
That doesn't seem to clear the referrer, at least on Firefox. Gotta go a step further and outright copy/paste the URL into an already-created tab.
Open in private works
This flowchart hides the most awful parts (IMO) of x86 prefixes: some combinations of prefixes are invalid but still parsed and executed, like combining two segment overrides, or placing a legacy prefix after a REX prefix.
The CPU also doesn't care if you use prefixes that aren't valid for a specific instruction, for example a REP on a non-repeatable instruction. The LOCK prefix is the only prefix that makes the sane choice to reject invalid combinations, rather than silently accept them.
Also, the (E)VEX prefix doesn't behave like the other prefixes: it must be placed last, and can therefore only appear once. All other prefixes can be repeated.
The CPU also doesn't care if you use prefixes that aren't valid for a specific instruction, for example a REP on a non-repeatable instruction. The LOCK prefix is the only prefix that makes the sane choice to reject invalid combinations, rather than silently accept them.
Also, the (E)VEX prefix doesn't behave like the other prefixes: it must be placed last, and can therefore only appear once. All other prefixes can be repeated.
> The CPU also doesn't care if you use prefixes that aren't valid for a specific instruction, for example a REP on a non-repeatable instruction.
This is one of the reasons why the x86 could be extended so much. PAUSE is just REP NOP, for example. Segment prefixes in front of conditional branches were used as static branch prediction hints (which I believe have returned in some newer Intel CPUs). Useful if you want to make a hint on newer CPUs that is harmless on older CPUs.
Some prefixes have become part of the encoding for certain SIMD instructions, but that is a different case because those prefixes aren't hints.
This is one of the reasons why the x86 could be extended so much. PAUSE is just REP NOP, for example. Segment prefixes in front of conditional branches were used as static branch prediction hints (which I believe have returned in some newer Intel CPUs). Useful if you want to make a hint on newer CPUs that is harmless on older CPUs.
Some prefixes have become part of the encoding for certain SIMD instructions, but that is a different case because those prefixes aren't hints.
Not at all.
The correct behavior for allowing future extensions has already been introduced by Intel with 80186, in 1982, which has introduced an invalid instruction exception, to be used for all undefined instruction opcodes.
This behavior was unlike 8086/8088, which happily executed any undefined instructions, most of them being aliases to defined instructions.
For any opcode where current CPUs generate invalid instruction exceptions, it is very easy to define them in future CPUs to encode useful instructions. Had REP NOP generated exceptions in old CPUs, it would have been still fine for it to become PAUSE in current CPUs. Unfortunately, the designers of Intel CPUs have not always followed their own documentation, so not all invalid opcodes generate the exception, as they should. The non-enforcing of this condition has led to the existence of even commercial programs that are invalid or of compilers that generate officially invalid instructions.
It is true that there are a few cases when Intel has exploited the fact that some encodings were equivalent with a NOP on old CPUs, by reusing them for some instruction on new CPUs, where this allowed the execution of a program compiled for new CPUs on old CPUs. However this has been possible only for very few instructions, e.g. for branch direction hints, when not executing them on old CPUs does not change the result of a program.
In general the reuse of an opcode for a new instruction, when that opcode does not generate exceptions on old CPUs, is very dangerous, because the execution on old CPUs of a program compiled for new CPUs will have unpredictable consequences, like destroying some property of the user.
Your example with PAUSE is also one of the very few examples, besides branch hints, where the execution of a new program on old computers is not dangerous, despite the reassignment of the opcode.
Some time ago there was a discussion about a bug in some CPU, but I do not remember in which one, where the bug was triggered when the order of the REP prefix and of the 64-bit REX prefix was invalid, but the invalid order was ignored by the older CPUs instead of generating the appropriate exception, which allowed the execution of invalid programs, which did not have any bad effects on old CPUs, but they triggered the bug on that specific new CPU.
The new CPU should have been bug-free, but also the programs that triggered the bug should not have existed, as they should have crashed immediately on any older CPU.
The correct behavior for allowing future extensions has already been introduced by Intel with 80186, in 1982, which has introduced an invalid instruction exception, to be used for all undefined instruction opcodes.
This behavior was unlike 8086/8088, which happily executed any undefined instructions, most of them being aliases to defined instructions.
For any opcode where current CPUs generate invalid instruction exceptions, it is very easy to define them in future CPUs to encode useful instructions. Had REP NOP generated exceptions in old CPUs, it would have been still fine for it to become PAUSE in current CPUs. Unfortunately, the designers of Intel CPUs have not always followed their own documentation, so not all invalid opcodes generate the exception, as they should. The non-enforcing of this condition has led to the existence of even commercial programs that are invalid or of compilers that generate officially invalid instructions.
It is true that there are a few cases when Intel has exploited the fact that some encodings were equivalent with a NOP on old CPUs, by reusing them for some instruction on new CPUs, where this allowed the execution of a program compiled for new CPUs on old CPUs. However this has been possible only for very few instructions, e.g. for branch direction hints, when not executing them on old CPUs does not change the result of a program.
In general the reuse of an opcode for a new instruction, when that opcode does not generate exceptions on old CPUs, is very dangerous, because the execution on old CPUs of a program compiled for new CPUs will have unpredictable consequences, like destroying some property of the user.
Your example with PAUSE is also one of the very few examples, besides branch hints, where the execution of a new program on old computers is not dangerous, despite the reassignment of the opcode.
Some time ago there was a discussion about a bug in some CPU, but I do not remember in which one, where the bug was triggered when the order of the REP prefix and of the 64-bit REX prefix was invalid, but the invalid order was ignored by the older CPUs instead of generating the appropriate exception, which allowed the execution of invalid programs, which did not have any bad effects on old CPUs, but they triggered the bug on that specific new CPU.
The new CPU should have been bug-free, but also the programs that triggered the bug should not have existed, as they should have crashed immediately on any older CPU.
There is utility for having a reserved set of opcode space for "NOP if you don't know what the semantics are, but later ISAs may attach semantics for it," because this allows you to add various instructions that merely do nothing on processors that don't support them. The ENDBR32/ENDBR64 instructions for CET, XACQUIRE/XRELEASE hints for LOCK, the MPX instructions, the PREFETCH instructions all use reserved NOP space (0F0D and 0F18-0F1F opcode space).
This is true, but the encoding space reserved for future extensions that is interpreted as NOP should be many times smaller than the space for encodings that generate the invalid instruction exception.
The reason is that the number of useful instructions that are only performance hints or checks for some exceptional conditions, so that if they are ignored that does not have bad consequences, is very limited.
For the vast majority of instruction set extensions, not executing the new instructions completely changes the behavior of the program, which is not acceptable, so the execution of such programs must be prevented on older CPUs.
Regarding the order of prefixes, Intel has made mistakes in not specifying it initially in 8086 and in allowing redundant prefixes. The latter has been partially corrected in later CPUs by imposing a limit for the instruction length.
Because of this lack of specification, the various compilers and assemblers have generated any instruction formats that were accepted by an 8088, so it became impossible to tighten the specification.
However, what is really weird is why Intel and AMD have continued to accept incrorrect instruction encodings even after later extensions of the ISA that clearly specified only a certain encoding to be valid, but in reality the CPUs also accept other encodings and now there are programs that use those alternative encodings that were supposed to be invalid.
The reason is that the number of useful instructions that are only performance hints or checks for some exceptional conditions, so that if they are ignored that does not have bad consequences, is very limited.
For the vast majority of instruction set extensions, not executing the new instructions completely changes the behavior of the program, which is not acceptable, so the execution of such programs must be prevented on older CPUs.
Regarding the order of prefixes, Intel has made mistakes in not specifying it initially in 8086 and in allowing redundant prefixes. The latter has been partially corrected in later CPUs by imposing a limit for the instruction length.
Because of this lack of specification, the various compilers and assemblers have generated any instruction formats that were accepted by an 8088, so it became impossible to tighten the specification.
However, what is really weird is why Intel and AMD have continued to accept incrorrect instruction encodings even after later extensions of the ISA that clearly specified only a certain encoding to be valid, but in reality the CPUs also accept other encodings and now there are programs that use those alternative encodings that were supposed to be invalid.
The prefix structure has been enforced starting with the VEX prefixes (which is a lot later than it should have; AMD made a mistake not enforcing more rules around the REX prefix). The legacy prefixes are of course an unfixable mess because of legacy.
Yes, I wish this was this simple. :) There are many other complications:
* Some instructions require VEX.L or VEX.W to be 0 or 1, and some encodings result in completely different instructions if you change VEX.L.
* Different bits of the EVEX prefix are valid depending on the opcode byte.
* Some encodings (called groups) produce different instructions depending on bits 3-5 of the modrm byte (the second byte after all prefixes). Some encodings further produce different groups depending on whether bits 6-7 (mod) of the modrm byte identifies a register or not.
* Some instructions read a whole vector register but only a scalar if the same instruction has a memory operand. Sometimes this is clear in the manual, sometimes it is not, sometimes the manual is downright wrong.
* Some instructions do not allow using the legacy high-8-bits registers even though they don't do anything with bits 8 and above of the operand: they only want a 32- or 64-bit register as their operand.
* APX (EVEX map 4) looks a lot like legacy map 0, but actually a few instructions were moved there from other maps for good reasons, a few more were moved there for no apparent reason (SHLD/SHRD iirc), and a few more are new.
* REX2 does not extend SSE and AVX instructions to 32 registers even though REX does extend them to 16.
* Intel defines a thing called VEX instruction classes, which makes sense except for a dozen or two instructions where it doesn't. For these, sometimes AMD uses a different class, sometimes doesn't; sometimes AMD's choice makes sense, sometimes it doesn't.
And many more that I found out while writing QEMU's current x86 decoder (which tries to be table based but sometimes that's just impossible).
* Some instructions require VEX.L or VEX.W to be 0 or 1, and some encodings result in completely different instructions if you change VEX.L.
* Different bits of the EVEX prefix are valid depending on the opcode byte.
* Some encodings (called groups) produce different instructions depending on bits 3-5 of the modrm byte (the second byte after all prefixes). Some encodings further produce different groups depending on whether bits 6-7 (mod) of the modrm byte identifies a register or not.
* Some instructions read a whole vector register but only a scalar if the same instruction has a memory operand. Sometimes this is clear in the manual, sometimes it is not, sometimes the manual is downright wrong.
* Some instructions do not allow using the legacy high-8-bits registers even though they don't do anything with bits 8 and above of the operand: they only want a 32- or 64-bit register as their operand.
* APX (EVEX map 4) looks a lot like legacy map 0, but actually a few instructions were moved there from other maps for good reasons, a few more were moved there for no apparent reason (SHLD/SHRD iirc), and a few more are new.
* REX2 does not extend SSE and AVX instructions to 32 registers even though REX does extend them to 16.
* Intel defines a thing called VEX instruction classes, which makes sense except for a dozen or two instructions where it doesn't. For these, sometimes AMD uses a different class, sometimes doesn't; sometimes AMD's choice makes sense, sometimes it doesn't.
And many more that I found out while writing QEMU's current x86 decoder (which tries to be table based but sometimes that's just impossible).
> Some instructions require VEX.L or VEX.W to be 0 or 1, and some encodings result in completely different instructions if you change VEX.L.
There is even an instruction where AMD got this wrong! VPERMQ requires VEX.W=1, but some AMD CPUs also happily execute it when VEX.W=0 even though that is supposed to raise an exception.
There is even an instruction where AMD got this wrong! VPERMQ requires VEX.W=1, but some AMD CPUs also happily execute it when VEX.W=0 even though that is supposed to raise an exception.
I wonder whether there are some prefixes that cause (some) CPUs to execute the instruction a lot slower.
This is indeed a thing. I believe in general instructions are executed slower when there are more than 4 legacy prefixes. And there are plenty of other timing differences between different microarchitectures
Fun little tidbit: The 0x40-0x4f range used for the REX prefix actually clashes with the single-byte encodings for increment/decrement.
When AMD designed the 64 bit extension, they had run out of available single-byte opcodes to use as a prefix and decided to re-use those. The INC/DEC instructions are still available in 64 bit mode, but not in their single-byte encodings.
When AMD designed the 64 bit extension, they had run out of available single-byte opcodes to use as a prefix and decided to re-use those. The INC/DEC instructions are still available in 64 bit mode, but not in their single-byte encodings.
Which clever code can utilize to determine which mode its running in and branch appropriately depending if the inc/dec were executed or not.
It is still crazy to me that intel though having 2 1 byte instructions for something that could just be coded as a single add with immediate instruction (which they had anyway).
captcha farm?
No captcha for me, though I do often have captcha problems with the site.
On that page, there is a link to another interesting page on the same site:
https://soc.me/interfaces/intels-original-64bit-extensions-f...
where there are links to a couple of patents filed by Intel in 2000, about a 64-bit extension of the x86 ISA, which had been implemented in Pentium 4, but which had been nonetheless disabled and hidden from the users, in order to not compete with Itanium.
The page explains the content of the patents.
As already mentioned by another poster, at least on Firefox you have to open a tab and then copy this link there, to avoid being identified as an "undesirable" :-)
https://soc.me/interfaces/intels-original-64bit-extensions-f...
where there are links to a couple of patents filed by Intel in 2000, about a 64-bit extension of the x86 ISA, which had been implemented in Pentium 4, but which had been nonetheless disabled and hidden from the users, in order to not compete with Itanium.
The page explains the content of the patents.
As already mentioned by another poster, at least on Firefox you have to open a tab and then copy this link there, to avoid being identified as an "undesirable" :-)
This is in no small part why x86 code density is awful despite variable size encoding.
Awful compared to what?
I've seen benchmarks that go both ways in terms of a "winner" but in terms of overall variance there seems to be very little. There are some cases where ARM64 or RISCV do better and there are some cases where x86_64 does better. I can't see code density being a relevant factor when picking one ISA over another.
We've got good compilers now anyways.. outside of power consumption.. the ISA wars are dead.
I've seen benchmarks that go both ways in terms of a "winner" but in terms of overall variance there seems to be very little. There are some cases where ARM64 or RISCV do better and there are some cases where x86_64 does better. I can't see code density being a relevant factor when picking one ISA over another.
We've got good compilers now anyways.. outside of power consumption.. the ISA wars are dead.
Technically, code density still matters - because both L1 cache memory and L1 instruction fetch misses are very expensive.
But as you point out, code density gets far less attention in tech circles these days. And higher-level decision makers rightfully focus on higher-level system performance metrics.
But as you point out, code density gets far less attention in tech circles these days. And higher-level decision makers rightfully focus on higher-level system performance metrics.
> outside of power consumption.
This is a pretty huge caveat. >90% of cpus are <1W (usb cables, wifi cards, storage controllers etc), and 99% are <10W (phones, lots of laptops)
This is a pretty huge caveat. >90% of cpus are <1W (usb cables, wifi cards, storage controllers etc), and 99% are <10W (phones, lots of laptops)
You are right about the order of magnitude of the percentages, but the CPUs <10 W are in phones, tablets, small SBCs (single-board computers) and only in a small fraction of laptops and mini-PCs.
The vast majority of laptops and mini-PCs uses more powerful CPUs. While 15 W CPUs were normal many years ago, when Intel launched the "Ultrabook" campaign, in recent years, due to better cooling, most laptops and mini-PCs use 25 W to 30 W CPUs, and a significant fraction of them (bigger than the fraction of the laptops/mini-PCs that uses <=10 W CPUs) uses CPUs with a TDP between 35 W and 65 W.
The vast majority of laptops and mini-PCs uses more powerful CPUs. While 15 W CPUs were normal many years ago, when Intel launched the "Ultrabook" campaign, in recent years, due to better cooling, most laptops and mini-PCs use 25 W to 30 W CPUs, and a significant fraction of them (bigger than the fraction of the laptops/mini-PCs that uses <=10 W CPUs) uses CPUs with a TDP between 35 W and 65 W.
Lots of laptops have ~30W cpus, but that's peak usage. A typical laptop has a ~50WH battery, and a ~5W screen. If it's getting 5hr battery life, that means the CPU is averaging <5W.
True, but if you are classifying the CPUs by their idle power consumption, then that is completely unrelated to the ISA implemented by the CPU and even the correlation with the power consumed at maximum load is very weak.
[deleted]
Compared to VAX, of course…
I respect the disobedience.
There's EVEX2 now. It's hard to keep up...
hiding this one, i hope others downvote it as well. Dick move by the person hosting this redirecting back to HN. Such incessant levels of pettiness are so irritating.