Why Ice Lake is Important – a bit-basher’s perspective(branchfree.org)
branchfree.org
Why Ice Lake is Important – a bit-basher’s perspective
https://branchfree.org/2019/05/29/why-ice-lake-is-important-a-bit-bashers-perspective/
41 comments
Author here in case anyone wants detail, to throw rocks, mock my limited understanding of Galois Field arithmetic, etc.
I really enjoy this summary of the AVX512 instruction set evolution. It makes me curious to know more about the AVX512 microarchitecture evolution too. How practical is AVX512 likely to be over the coming years? How much do we have to worry about side-effects like throttling the CPU clock speed down when the wide datapaths are exercised, etc?
Can someone wake me up when it is a no-brainer to use AVX512 instructions on CPUs that support them? :-)
Can someone wake me up when it is a no-brainer to use AVX512 instructions on CPUs that support them? :-)
I hope you have a comfortable bed, then. It will be a while before there are machines deployed across the whole range (client laptop, client desktop, server, network edge) that are SNC. Atom isn't even supporting AVX2 in the short term either.
Daniel Lemire, who I work with on simdjson, wrote this on throttling: https://lemire.me/blog/2018/08/13/the-dangers-of-avx-512-thr...
I don't know specifics (I'm ex-Intel a good bit before SNC, and wasn't a hardware guy) but my impression is the clock throttling impact gen-on-gen goes down. If you're actually getting good value from AVX-512 instructions rather than chucking in 1 in 1000 instructions for the cool factor, you're probably fine.
Daniel Lemire, who I work with on simdjson, wrote this on throttling: https://lemire.me/blog/2018/08/13/the-dangers-of-avx-512-thr...
I don't know specifics (I'm ex-Intel a good bit before SNC, and wasn't a hardware guy) but my impression is the clock throttling impact gen-on-gen goes down. If you're actually getting good value from AVX-512 instructions rather than chucking in 1 in 1000 instructions for the cool factor, you're probably fine.
Thanks!
While you are here... :-)
What do you think about the tendency of glibc to use the latest available instruction set extensions for routines like memcpy? Should we be worried that when we call memcpy() in a program this will effectively mix some wide-datapath instructions into the CPU's instruction stream with potentially hilarious consequences like clock throttling? What's all that about?
While you are here... :-)
What do you think about the tendency of glibc to use the latest available instruction set extensions for routines like memcpy? Should we be worried that when we call memcpy() in a program this will effectively mix some wide-datapath instructions into the CPU's instruction stream with potentially hilarious consequences like clock throttling? What's all that about?
As a library author, I am very, very nervous about doing anything that might cause clock throttling. After all, you will not see a 2x gain from twice the SIMD width (there are many other limiting factors, and of course, Skylake issues 2 x 512b operations max vs 3 x 256b ops).
Even if you did get 2x, you would have to balance that against clock throttling in circumstances where you have no idea of the _relative_ importance of your library. Are you 95% of the cycles, or 10%?
Even if you did get 2x, you would have to balance that against clock throttling in circumstances where you have no idea of the _relative_ importance of your library. Are you 95% of the cycles, or 10%?
I'm always nervous about companies like Intel doing both hardware and software. If you do only hardware then you need to keep it simple or else software people won't use your stuff. If you are playing both sides then you can throw engineers at the problem and make it as complicated as you want. In this case they can just push complex AVX512 code into projects like glibc and bypass the whole "is this really worth the complexity?" discussion.
One related example is Intel CPUs having good cache-DMA integration but this not working across NUMA nodes. Ideally Intel would improve this feature to be NUMA-aware and everybody would be happy. However, it seems like they decided it would be cheaper to send forth an army of software engineers to put NUMA-scheduling kludges into a few key projects like Kubernetes and then call it a day.
/rant!
One related example is Intel CPUs having good cache-DMA integration but this not working across NUMA nodes. Ideally Intel would improve this feature to be NUMA-aware and everybody would be happy. However, it seems like they decided it would be cheaper to send forth an army of software engineers to put NUMA-scheduling kludges into a few key projects like Kubernetes and then call it a day.
/rant!
> good cache-DMA integration but this not working across NUMA nodes. Ideally Intel would improve this feature to be NUMA-aware
How do you propose they do this? You can't magically move the physical devices to the other socket, and transferring the data between sockets is what those "NUMA-scheduling kludges" are trying to avoid. The only solution is to have software put the data close to the device.
How do you propose they do this? You can't magically move the physical devices to the other socket, and transferring the data between sockets is what those "NUMA-scheduling kludges" are trying to avoid. The only solution is to have software put the data close to the device.
My take is that they should use the memory subsystem to route the DMA to the appropriate L3 cache.
I don't think the main issue is conserving QPI bandwidth: it's minimizing latency when a core loads the data into L2. That latency often causes ~30% performance drag on e.g. packet processing applications.
I don't think the main issue is conserving QPI bandwidth: it's minimizing latency when a core loads the data into L2. That latency often causes ~30% performance drag on e.g. packet processing applications.
sometimes you get more than a 2x gain when going to a wider simd instruction set, because more instructions are available. This is especially true with AVX-512 and its masks.
IIRC glibc reverted those avx-512 string functions, precisely because it turned out that in most real programs outside microbenchmarks, the clock speed hit negated the benefit from wider instructions.
But IIRC, back when AVX was introduced, there was some issue with that as well, and similar with AVX2. But now those seem to have been mostly addressed. I guess in some years the same will happen with AVX-512 too?
But IIRC, back when AVX was introduced, there was some issue with that as well, and similar with AVX2. But now those seem to have been mostly addressed. I guess in some years the same will happen with AVX-512 too?
Is there definitive information on these issues?
The pattern for me is that I'm excited about a new instruction set when it is announced, then disappointed when it is shipped in silicon, then I have a hard time knowing when it is truly resolved and worth taking a second look at.
My recollection with AVX2 is that glibc shipped those routines since before silicon was generally available, and then kept them enabled even when the clock-throttling issues became apparent. (The water was muddied a bit in one glibc release that shipped a bug in selecting which code to use for the running uarch, but that's a separate issue.)
The pattern for me is that I'm excited about a new instruction set when it is announced, then disappointed when it is shipped in silicon, then I have a hard time knowing when it is truly resolved and worth taking a second look at.
My recollection with AVX2 is that glibc shipped those routines since before silicon was generally available, and then kept them enabled even when the clock-throttling issues became apparent. (The water was muddied a bit in one glibc release that shipped a bug in selecting which code to use for the running uarch, but that's a separate issue.)
Not specifically glibc but some data on the frequency scaling impact: https://blog.cloudflare.com/on-the-dangers-of-intels-frequen...
Re AVX-512 and throttling, may I suggest this post instead https://lemire.me/blog/2018/09/07/avx-512-when-and-how-to-us..., again by Daniel.
I would point out that there _are_ consumer-grade chips with AVX-512 beyond the Xeons and Cannonlake: the Skylake-X chips such as i7-7800X, which even happen to have 2 512-bit FMA units, unlike some of the cheaper Xeons.
I will point out three things that make the Sunny Cove exciting to me, beyond the obvious new instructions (based on the available slides):
- There are now two ports for vector shuffles. For shuffle-heavy applications, which is often the case with bit-manipulation kernels, this is great news. This seems improved from Cannonlake.
- This was already present on the Cannonlake, but integer division is drastically improved, and goes from ~30ish uops to 4. Divisions that could take up to 90 cycles will now take <=18.
- There are now 4 LEA ports, up from 2, which for address calculation and small integer multiplications are quite useful.
I will point out three things that make the Sunny Cove exciting to me, beyond the obvious new instructions (based on the available slides):
- There are now two ports for vector shuffles. For shuffle-heavy applications, which is often the case with bit-manipulation kernels, this is great news. This seems improved from Cannonlake.
- This was already present on the Cannonlake, but integer division is drastically improved, and goes from ~30ish uops to 4. Divisions that could take up to 90 cycles will now take <=18.
- There are now 4 LEA ports, up from 2, which for address calculation and small integer multiplications are quite useful.
"Execution port Twitter" seems divided on what the extra shuffle port will do. I'm cautiously optimistic.
Integer division is never really on my radar, but yes, that's exciting.
LEA is very useful, but not something I spend a huge amount of time thinking about. It would be interesting to see whether the penalty for a "slow LEA" is relaxed.
You're right that the Skylake-X processors have AVX-512, but they kinda push the 'consumer' envelope. There's also no consistent branding for 'having AVX-512 or not' (they have happily adopted i7/i9 for products that don't have it). These are really obviously rebadged Xeons. Nothing against them, but it's a tiny niche.
Integer division is never really on my radar, but yes, that's exciting.
LEA is very useful, but not something I spend a huge amount of time thinking about. It would be interesting to see whether the penalty for a "slow LEA" is relaxed.
You're right that the Skylake-X processors have AVX-512, but they kinda push the 'consumer' envelope. There's also no consistent branding for 'having AVX-512 or not' (they have happily adopted i7/i9 for products that don't have it). These are really obviously rebadged Xeons. Nothing against them, but it's a tiny niche.
I am really enjoying this thread!
Faster integer division is exciting to me, and so is the prospect of faster "slow LEAs."
The other year I was able to speed up a Lua MD5 benchmark by 15% by eliminating slow-LEAs from the JIT backend. I only tracked down half of them though, so if the problem has eased that would save me tracking down the other half :). https://github.com/raptorjit/raptorjit/pull/55
Faster integer division is exciting to me, and so is the prospect of faster "slow LEAs."
The other year I was able to speed up a Lua MD5 benchmark by 15% by eliminating slow-LEAs from the JIT backend. I only tracked down half of them though, so if the problem has eased that would save me tracking down the other half :). https://github.com/raptorjit/raptorjit/pull/55
What is the state of compiler support for these very advanced instruction sets ? Can the average developer benefit by basically adding a few compiler flags ?
Also, each new Intel platform seems to bring additional instructions but most software isn't made available in a wide range of microarchitecture-specific builds. Is there typically capability detection going on behind the scenes ?
Some of the operations described here seem so specific, that I have a hard time imagining compilers being able to spot the relevant patterns in source code that can make use of them (then again, I'm not a specialist). I guess these are explicitly coded for in Assembly ?
Also, each new Intel platform seems to bring additional instructions but most software isn't made available in a wide range of microarchitecture-specific builds. Is there typically capability detection going on behind the scenes ?
Some of the operations described here seem so specific, that I have a hard time imagining compilers being able to spot the relevant patterns in source code that can make use of them (then again, I'm not a specialist). I guess these are explicitly coded for in Assembly ?
Using these instruction sets usually requires explicitly writing code using C language intrinsic functions, which is a reasonable compromise between transparent automatic usage in C++ and needing to write assembly. On the positive side, most compilers have full support for these intrinsics and the compiler can even do compile-time evaluation for a few of them e.g. some intrinsics are treated as 'constexpr' in C++ by the compiler, but the compiler won't automagically use them for much and is generally poor at identifying ordinary code that can be reduced to these intrinsics. So for designing algorithms that take advantage of the advanced instruction sets, you are mostly on your own. Writing C/C++ code with heavy use of intrinsics isn't hard to learn or particularly opaque, just arcane.
These instruction sets can offer very substantial speedups for code paths when applied judiciously. You do need to check the CPU capabilities at start up time for practical software, you can't assume the environment supports it even when these instructions have been around a long time. Some had very uneven implementation timelines across CPU product lines. Even for explicitly high-performance software, I invariably would get bug reports because someone tried to run it on a 10+ year old machine that did not support the instructions I was using. There are standard open source libraries for doing this feature detection using CPUID.
These instruction sets can offer very substantial speedups for code paths when applied judiciously. You do need to check the CPU capabilities at start up time for practical software, you can't assume the environment supports it even when these instructions have been around a long time. Some had very uneven implementation timelines across CPU product lines. Even for explicitly high-performance software, I invariably would get bug reports because someone tried to run it on a 10+ year old machine that did not support the instructions I was using. There are standard open source libraries for doing this feature detection using CPUID.
Toolchain support, at the minimum, often consists of having a compiler flag to allow the use of intrinsics for the new architecture - pretty much the opposite of what you're hoping (automagical support).
There's a fair bit of work where people try to autovectorize code. I'm happy that folks are pushing this work, but, well, I drive a manual (in this regard).
It does happen that code does auto-detection and skips to a particularly compelling platform-specific optimization path, but it's a huge pain. It's unlikely that someone will optimize for each and every variant, but they might draw a line here and there through the myriad of architecture choices and offer something.
Sorry that the story isn't better. I generally hand-code stuff - not in asm but with intrinsics (which is getting a lot of the benefit of asm, but without doing all your own codegen for the boring bits, scheduling and register allocation).
There's a fair bit of work where people try to autovectorize code. I'm happy that folks are pushing this work, but, well, I drive a manual (in this regard).
It does happen that code does auto-detection and skips to a particularly compelling platform-specific optimization path, but it's a huge pain. It's unlikely that someone will optimize for each and every variant, but they might draw a line here and there through the myriad of architecture choices and offer something.
Sorry that the story isn't better. I generally hand-code stuff - not in asm but with intrinsics (which is getting a lot of the benefit of asm, but without doing all your own codegen for the boring bits, scheduling and register allocation).
Thanks for writeup. Do you see AVX512 or it's successors being able to rival GPUs as the linear algebra champs in the near future?
GPU HW dude, here: I’m not sure of AVX-512’s current fma16 story, but it’s predecessor (LRBni) was terrifyingly good at fma32–much better than contemporary GPUs. The problem with those parts is total power displacement (TDP). The market is really geared to mobile; so, so that became less of an issue. INTC-for-Graphics is hampered by the lack of deep, dedicated latency hiding fixed-function HW. Otherwise, INTC is probably a strict win over any discrete GPU.
(Disclaimer: I wrote software rasterizers, drivers, and compilers for Larrabee.)
(Disclaimer: I wrote software rasterizers, drivers, and compilers for Larrabee.)
That's exactly the kind of thing I don't really know about. I'm not hostile to that sort of work, I just can't really add any information to it. There's clearly a huge market to building the Biggest Baddest Linear Algebra Engine out there, but I hope that other uses of SIMD don't get neglected.
At this moment, as a consumer, I'm more concerned about Spectre mitigation impact, power draw and cost - among other things.
And currently, AMD is definitely winning there, and it may be my processor of choice for the next few years, until Intel fixes shortcomings in all those areas. 10nm is a step in the right directions, but the price/performance ratio is nowhere close to the mark.
But that's just my take on that.
And currently, AMD is definitely winning there, and it may be my processor of choice for the next few years, until Intel fixes shortcomings in all those areas. 10nm is a step in the right directions, but the price/performance ratio is nowhere close to the mark.
But that's just my take on that.
Ice Lake's Wikipedia page mentions that Intel is adding hardware mitigations to Spectre and Meltdown.
Something which I found odd in the article:
In the beginning you write that "VBMI [...] is the only extension that we’ve seen before – it’s in Cannonlake." but later you write that "VPOPCNTDQ is older (from the MIC product line)"
So which is it? Or am I misunderstanding something?
In the beginning you write that "VBMI [...] is the only extension that we’ve seen before – it’s in Cannonlake." but later you write that "VPOPCNTDQ is older (from the MIC product line)"
So which is it? Or am I misunderstanding something?
Ach, you're right. Just a mistake. I tend to forget the existence of any operations that are (a) in MIC and (b) not available for bytes, so this was a double-whammy and even after writing that sentence I forgot that I was contradicting myself.
I wonder if this will lead to significant speed ups in databases like ElasticSearch, where operations on large bitmaps (each pixel representing a term’s presence in a document) are commonplace. Do you have any insight into this?
It partially depends on the design of the database engine. Some classic designs are not that amenable to it but newer designs do tend to be. Many database operations (like operations on large bitmaps) are effectively memory-bandwidth bound, unless they are running completely out of close CPU cache. Some scenarios where vector intrinsics are commonly used:
- Using these instructions to encode/decode compressed data structures where evaluation would otherwise be memory-bound, trading the "free" CPU time to reduce memory bandwidth consumption. Large bitmaps are amenable to this.
- Increasing throughput of operations that are CPU bound so that they become (closer to) memory bound, recognizing that the memory bandwidth available to CPU cache-friendly structures may effectively be much higher than RAM bandwidth. Format parsing and cache replacement is a common use case for vector intrinsics in databases for this purpose.
- Reducing branch prediction penalties for data structures that don't play well with branch predictors. Commonly used hot path radix-trie structures often have this issue. Some of the intrinsics don't have a direct scalar equivalent and redesigning around them can significantly reduce branching pressure (both number and predictability).
- Scan operators for vectors of high-level data types, ironically because they allow you to efficiently use a more complex representation than a simple vector. Pages in databases are large enough these days that this is worth doing and you'll spend much of your CPU time here anyway.
I have not (yet) built an operational database kernel that uses AVX-512 intrinsics, though the designs are setup for it, mostly because support is still insufficiently common at the moment. I think the CPU throttling aspects of vector instructions are less of a concern for database engines because good designs tend not to have clock rate as a major throughput limitation in practice, though I've never actually measured it.
- Using these instructions to encode/decode compressed data structures where evaluation would otherwise be memory-bound, trading the "free" CPU time to reduce memory bandwidth consumption. Large bitmaps are amenable to this.
- Increasing throughput of operations that are CPU bound so that they become (closer to) memory bound, recognizing that the memory bandwidth available to CPU cache-friendly structures may effectively be much higher than RAM bandwidth. Format parsing and cache replacement is a common use case for vector intrinsics in databases for this purpose.
- Reducing branch prediction penalties for data structures that don't play well with branch predictors. Commonly used hot path radix-trie structures often have this issue. Some of the intrinsics don't have a direct scalar equivalent and redesigning around them can significantly reduce branching pressure (both number and predictability).
- Scan operators for vectors of high-level data types, ironically because they allow you to efficiently use a more complex representation than a simple vector. Pages in databases are large enough these days that this is worth doing and you'll spend much of your CPU time here anyway.
I have not (yet) built an operational database kernel that uses AVX-512 intrinsics, though the designs are setup for it, mostly because support is still insufficiently common at the moment. I think the CPU throttling aspects of vector instructions are less of a concern for database engines because good designs tend not to have clock rate as a major throughput limitation in practice, though I've never actually measured it.
I think it can help. Traditional DBs generally worked on the assumption that if you were scanning through records one-by-one you'd already lost, so naive ideas that you can just write Really Good SIMD won't help. But there are plenty of smart people who understand this well and can write SIMD code for what remains.
Scanning non-DBMS stuff can benefit a lot from good SIMD (although writing code that's tremendously clever that allows you to run faster than memory bandwidth permits is also fairly self-limiting; the faster you go, the less likely anything can keep you fed).
Scanning non-DBMS stuff can benefit a lot from good SIMD (although writing code that's tremendously clever that allows you to run faster than memory bandwidth permits is also fairly self-limiting; the faster you go, the less likely anything can keep you fed).
You mentioned there are NUCs that have AVX-512, which surprised me. Those looking for such a NUC: search for NUCs with the Core i3-8121U CPU. This is a cheap AVX-512 entry point if you want to experiment. (~$400 for the NUC).
Fwiw bit bashing is great for fields of characteristic 2 and 3
> I will also admit that I’m not very interested in deep learning, floating point or crypto
I feel like I met that one other guy in the world.
I feel like I met that one other guy in the world.
There are dozens of us ... dozens!
>>floating point
One of these is not like the others
One of these is not like the others
For anyone else confused by the title, this is about the upcoming Intel CPU generation.
GF2P8AFFINEQB sounds a lot like MXOR from Donald Knuth's MMIX (19 years prior). Now x86 just need MOR as well. EDIT: This link may be helpful: https://math.stackexchange.com/questions/1107839/how-is-d-kn...
Thank you - this led to some interesting background. I could definitely have used MOR during bitwise regex (Glushkov NFA) implementation.
I hope they release more information soon, like latency and throughput for all these avx512 instructions on ice lake.
No 2nd FMA really sucks, hope they add it when they do desktop-
I can't tell from the micro architecture slide if the yellow box labeled "ALU" that is found on port 0 and 5 refers to only integer ops, or if that includes float(add/mul).
No 2nd FMA really sucks, hope they add it when they do desktop-
I can't tell from the micro architecture slide if the yellow box labeled "ALU" that is found on port 0 and 5 refers to only integer ops, or if that includes float(add/mul).
Food for the brain.
I never thought about it before tbh