Avoid speculative indirect calls in kernel(lkml.org)
lkml.org
Avoid speculative indirect calls in kernel
https://lkml.org/lkml/2018/1/3/797
96 comments
I'm frankly shocked Linus was anywhere near as restrained as he was upon seeing this monstrosity. Nothing of this sort should ever make it into the kernel without explicit and loudly announced opt-out switches. If anything close to this does make it in, even as an opt-in, this will easily be the biggest performance regression in recent (perhaps all-time?) kernel history.
His restraint and the fact that this stuff is even being considered tells me this is a really really bad bug. I doubt he would be so cordial in any other situation.
Nothing like this should have ever made it into hardware. It's lazy implementation (or performance at the expense of security).
Performance at the expense of security is a compromise made daily in every aspect of IT. The only way to make systems perfectly secure is to make them unusable.
Now, you can certainly argue that this particular tradeoff wasn't worth it, and in hindsight it's certainly a questionable one. However, do note that side channel attacks have been known for decades (since out-of-order CPUs became widespread in the 90s) and until this point have rarely presented a significant threat to most systems.
Now, you can certainly argue that this particular tradeoff wasn't worth it, and in hindsight it's certainly a questionable one. However, do note that side channel attacks have been known for decades (since out-of-order CPUs became widespread in the 90s) and until this point have rarely presented a significant threat to most systems.
I'd be more inclined to think this was someone who didn't understand the implications implementing a performance improvement. This bug is clearly deep into unk-unk (i.e. well past the known areas of concern) territory and took a great deal of insight to the fundamentals of speculative execution to discover.
I missed that, actually. But you are right, it's not just the crucial bits of inline assembly they are patching here, they expect your kernel to be built with their special version of GCC (yet unreleased) that mutilates every single freaking indirect jmp/call into what could cost more than a virtual function call..
We use a new compiler option -mindirect-branch=thunk-extern
We use a new compiler option -mindirect-branch=thunk-extern
I missed that it was indirect jumps too, which means switch statements also get affected. That's even worse.
Here is an example with the very common programming patterns that result in indirect jumps/calls:
https://godbolt.org/g/LPycpY
Every single call rax and jump rax that you see on the right will be replaced with the following:
https://godbolt.org/g/LPycpY
Every single call rax and jump rax that you see on the right will be replaced with the following:
push rax
jmp .indirect_thunk
That has to be crazy slow. They probably haven’t published benchmarks yet because they’re still waiting for it to finish running!
Interacting terribly with CPU branch prediction is the whole point -- Spectre attacks work by messing with the branch predictor, to misdirect speculative execution onto paths that reveal information useful to the attacker. So, the point of this whole crazy dance is to effectively disable the branch predictor.
Of course. But it needs spelling out, because branch prediction is essential to good performance and why todays CPUs can be much faster despite not reaching higher clocks.
It's much more than just the cost of a few more instructions.
Apparently the perf impact isn't so bad outside of microbenchmarks actually. 10% for real world programs ... as long as you use profile guided optimisation to devirt as many calls as possible!
https://support.google.com/faqs/answer/7625886
http://webcache.googleusercontent.com/search?q=cache%3Ahttps...
(some websites with info on this are DoSd and down right now, including the Arm website and LLVM code review site so switched to a cache link)
Pretty painful for C++ but fixable with build changes. JVM users and other JIT-compiled languages will probably be fixable without any serious impact, as HotSpot and other advanced runtimes devirtualise calls (convert indirect to direct) as much as they can anyway, meaning most calls won't be indirect when finished and once the code has warmed up.
https://support.google.com/faqs/answer/7625886
http://webcache.googleusercontent.com/search?q=cache%3Ahttps...
(some websites with info on this are DoSd and down right now, including the Arm website and LLVM code review site so switched to a cache link)
Pretty painful for C++ but fixable with build changes. JVM users and other JIT-compiled languages will probably be fixable without any serious impact, as HotSpot and other advanced runtimes devirtualise calls (convert indirect to direct) as much as they can anyway, meaning most calls won't be indirect when finished and once the code has warmed up.
The above example compiled with optimizations and with `indirect_jump` altered so the compiler actually emits a jump table instead of an array lookup:
https://godbolt.org/g/MPs43x
https://godbolt.org/g/MPs43x
Clang has a new code generation option, too.
* https://reviews.llvm.org/D41723
* https://news.ycombinator.com/item?id=16070050
* https://news.ycombinator.com/item?id=16070356
* https://reviews.llvm.org/D41723
* https://news.ycombinator.com/item?id=16070050
* https://news.ycombinator.com/item?id=16070356
This is my understanding, please correct me if wrong:
There's a basic class of bugs (let's call it Spectre-class) that's about exfiltration of memory contents via side channels introduced by speculative execution. Basically every processor that can speculatively execute code (i.e., every modern one in the past two decades or so) is potentially vulnerable.
Of known Spectre-class bugs, one (Spectre) relies on branch predictor tables. All tested CPUs (ARM, AMD, Intel) were vulnerable to Spectre. The other bug publicly announced (Meltdown) relies on when the #GP(0) exception actually gets thrown and is specific to Intel CPUs. The fix for Spectre is to basically banish branch prediction, and the fix for Meltdown to is to unmap kernel pages when in userspace code.
Beyond these two bugs, there may exist other Spectre-class bugs specific to different architectures/vendors that are not yet thoroughly investigated.
What did I get right and wrong?
There's a basic class of bugs (let's call it Spectre-class) that's about exfiltration of memory contents via side channels introduced by speculative execution. Basically every processor that can speculatively execute code (i.e., every modern one in the past two decades or so) is potentially vulnerable.
Of known Spectre-class bugs, one (Spectre) relies on branch predictor tables. All tested CPUs (ARM, AMD, Intel) were vulnerable to Spectre. The other bug publicly announced (Meltdown) relies on when the #GP(0) exception actually gets thrown and is specific to Intel CPUs. The fix for Spectre is to basically banish branch prediction, and the fix for Meltdown to is to unmap kernel pages when in userspace code.
Beyond these two bugs, there may exist other Spectre-class bugs specific to different architectures/vendors that are not yet thoroughly investigated.
What did I get right and wrong?
AMD alleges that the fix for Spectrr can be ameliorated with minimal performance overhead.
Considering Intel is working on this for the last 6 months, it is indeed a little telling that they did not have the patch behind a feature flag. It also explains why AMD was so passive aggressive while inserting the exception for their CPUs. Seems like Intel was trying to make this some sort of act of god type of deal instead of writing out the if(intel){horrible mitigation}.
Can I see an article/discussion about AMD being passive-agressive about it? I want my popcorn.
It is dry - https://lkml.org/lkml/2017/12/27/2
Hardly passive-aggressive. What would the alternative have been? Leave PTI on on AMD just out of sympathy for Intel?!
+1 :D
So is there no way to disable the performance crippling fix? If you ran an air gapped or highly controlled environment wouldn’t you want th option of disabling this to keep performance? How many Giga Terra or Exo FLops will be lost to this fix in super computers? Or is the way to disable it just not merge this patch into your custom kernel code?
Correct me if I'm wrong, but I don't think super computers are updating there kernels at all.
New software is written with newer kernels in mind. To continue running newer software, you will need to update your kernels.
Additionally: you might update infrequently when it comes to feature releases, but you'll definitely need security patches.
Additionally: you might update infrequently when it comes to feature releases, but you'll definitely need security patches.
They do, but not often. Maybe once a year or so.
It can be controlled by a boot option in linux.
Not this one, which basically penalises every indirect call by preventing them from being speculatively executed.
I don't see why the text could not be fixed at boot time. It has been done before.
Because this actually changes how every indirect call is compiled.
So what? IIRC we used to have (and maybe still have) a feature that patched out spinlocks in binary code when it detects it is run on monoprocessor systems. You "just" have to identify all the call sites, and patch the code there during the boot process.
This isn't the fix with the 5% quoted performance loss (that one clears all of TLB on every syscall), I don't think this is as costly. But it's certainly not free and it's certainly not clean.
I'm disappointed Intel didn't bother to test the performance impact of this and report it in the patchset. Benchmarks are ordinarily expected with something like this (very hot paths they are changing), and they did have 6 months..
I'm disappointed Intel didn't bother to test the performance impact of this and report it in the patchset. Benchmarks are ordinarily expected with something like this (very hot paths they are changing), and they did have 6 months..
I'm disappointed Intel didn't bother to test the performance impact of this and report it in the patchset.
You should be disappointed that they didn't report it. But I'd be willing to bet every last dollar of my net worth that Intel has done very extensive performance testing.
I'd venture that the results are far too embarrassing to report.
You should be disappointed that they didn't report it. But I'd be willing to bet every last dollar of my net worth that Intel has done very extensive performance testing.
I'd venture that the results are far too embarrassing to report.
Yup this for sure. They probably ran the performance test bench thousands of times, and only then released it with the PR department fully briefed about how bad it was going to be.
This is worse, far worse --- it basically disables prediction for indirect jumps and calls, plus adds another few extra clock cycles to each one of those for the "retoline" instruction sequence.
Love it, an LFENCE on every indirect jump in core 32/64bit entry assembler code, not to mention the cost of the trampoline itself. Linus is right to say that this hack should be reserved for configured as broken CPUs..
If I'm reading it correctly, the lfence is never reached for real (only speculatively..)
code invoking the thunk pushes the target address on the stack, then branches to or calls the thunk.
The first instruction of the thunk is a "call" to a nearby address, skipping over an infinite loop (2: lfence; jmp 2b)
the lea n(%xsp),%xsp discards the return address pushed by "call", and the subsequent "ret" "returns" to the branch target rather than to label 2.
perhaps the speculative execution engine doesn't take the stack pointer adjustment into account and so gets stuck in the lfence loop, only being snapped back to its senses when the lea completes..
code invoking the thunk pushes the target address on the stack, then branches to or calls the thunk.
The first instruction of the thunk is a "call" to a nearby address, skipping over an infinite loop (2: lfence; jmp 2b)
the lea n(%xsp),%xsp discards the return address pushed by "call", and the subsequent "ret" "returns" to the branch target rather than to label 2.
perhaps the speculative execution engine doesn't take the stack pointer adjustment into account and so gets stuck in the lfence loop, only being snapped back to its senses when the lea completes..
Linus missed the bus. The Spectre attack doesn't require speculation across protection domains.
Why does he mention ARM64 instead of AMD?
Because Intel writes almost all of the x86 arch code for Linux, so AMD gets stuck footing the bill for hardware bugs like this, even if AMD isn’t affected. From Linus’s perspective it’s the same arch, because, well, it is.
Well he does say patches should written that don't treat all CPUs as crap. AMD had to basically say "Wait a moment! This does not affect our CPUs what are you doing Intel?"
As far as I understand, one of the two attacks does affect AMD, and it's hard to solve as it relies on normal branch prediction.
Amd claims that two of the three spectre attacks don't work on their hardware and the last one (variant one) is fixable with negligible performance impact.
https://www.amd.com/en/corporate/speculative-execution
https://www.amd.com/en/corporate/speculative-execution
I still don‘t get how variant 1 is fixable with negligible impact. Do we have any details on this as of now?
A bit of PR spin in Variant 2:
Differences in AMD architecture mean there is a near zero risk of exploitation of this variant. Vulnerability to Variant 2 has not been demonstrated on AMD processors to date.
Near-zero is not zero.
Differences in AMD architecture mean there is a near zero risk of exploitation of this variant. Vulnerability to Variant 2 has not been demonstrated on AMD processors to date.
Near-zero is not zero.
That's fantastic. I know whom my next CPU will be made by.
Well there is spectre, and meltdown. Spectre is much more limited than meltdown. The patch AMD was referring to was for Meltdown.
Spectre at least does not leak ring-0 memory. Sure it means that some application might be able to read/infer memory values of other user-mode process.
Spectre at least does not leak ring-0 memory. Sure it means that some application might be able to read/infer memory values of other user-mode process.
Spectre is much scarier than Meltdown, from my understanding, because it represents a whole class of strategies that lack a single fix, even at the hardware level.
My understanding is that it's not a bug of unintended behaviour, but rather a problem with the specification, all behaviour is as intended, but we need to have different behaviour. Which means that basically we need to get a new ISA -- or a new version of an existing one -- and transition everything to it.
[deleted]
Linus has a long history with, a deep understanding of, and I presume a special place in his heart for x86 microarchitectures.
My favorite is espfix. espfix64 is worse than espfix32 too. I have suggested limiting modify_ldt to root instead.
Does this disable the branch predictor for all function calls?
Somebody needs to tell him that the plural of CPU is "CPUs", not "CPU's" :-). I am afraid that even more people will start overusing apostrophes.
Let's try avoiding being pedants. Human language is quite squishy and imprecise anyways.
Agreed. Whether or not you think grammar matters, bringing it up is destructive to the conversation this is meant to start.
As long as the style is consistent, it doesn't really matter.
The general newspaper style is acronyms with periods use apostrophe to separate period from s. Therefore M.D.'s and Ph.D.'s are plural. Where period is not used in the acronym, the apostrophe is not used, so CPU not C.P.U., therefore it's CPUs. Anyway, it's just a convention, the key is consistency.
The general newspaper style is acronyms with periods use apostrophe to separate period from s. Therefore M.D.'s and Ph.D.'s are plural. Where period is not used in the acronym, the apostrophe is not used, so CPU not C.P.U., therefore it's CPUs. Anyway, it's just a convention, the key is consistency.
Is this mentioned in any particular newspaper style guide that a general member of the public can get their hands on? Like the AP style guide for instance?
Using an apostrophe is the technically correct way to pluralize acronyms.
> Using an apostrophe is the technically correct way to pluralize acronyms.
No, it's not the “technically correct way”. It's true that some style guides recommend something like this (but always, AFAIK, more limited); e.g., the New York Times style (and Chicago style) uses apostrophes for plurals of abbreviatiins formed with capital letters and periods, and plurals of single letters. [0] AP, on the other hand, only uses apostrophes for plurals of single letters, but not for abbreviations/initialisms/acronyms. Most British styles I've seen don't use apostrophes for plurals at all, IIRC.
[0] https://afterdeadline.blogs.nytimes.com/2010/04/13/faqs-on-s...
No, it's not the “technically correct way”. It's true that some style guides recommend something like this (but always, AFAIK, more limited); e.g., the New York Times style (and Chicago style) uses apostrophes for plurals of abbreviatiins formed with capital letters and periods, and plurals of single letters. [0] AP, on the other hand, only uses apostrophes for plurals of single letters, but not for abbreviations/initialisms/acronyms. Most British styles I've seen don't use apostrophes for plurals at all, IIRC.
[0] https://afterdeadline.blogs.nytimes.com/2010/04/13/faqs-on-s...
I’d love a reference for this convention.
Says whom?
Pretty common in our industry to use apostrophes after an acronym to make it plural. An example is VM's and VMS are two different things. When you get lazy and do vm's and vms you can not always be sure, I have even written an email where I stated "we need multiple VMS VM's".
This seems unnecessary hostile. It's quite obvious that these patches had to be completed in a hurry, and in addition to any number of similar patches for other systems. Configurability is currently useless for Intel CPUs, as all of them seem to be affected.
Depending on how their CPU development pipeline works, it's quite likely that even the next generation will still be affected, giving everyone plenty of time later for such niceties.
It's also slightly too harsh to call all of Intel's work "crap" when this bug has apparently existed for the better part of two decades without being noticed.
Depending on how their CPU development pipeline works, it's quite likely that even the next generation will still be affected, giving everyone plenty of time later for such niceties.
It's also slightly too harsh to call all of Intel's work "crap" when this bug has apparently existed for the better part of two decades without being noticed.
I didn't take the comment as "all Intel CPUs are crap," but that (1) the fix is needed for crap CPUs and (2) the fix is turned on unconditionally on Intel CPUs, even future CPUs. That is, the code itself assumes that all CPUs that could be crap actually are crap. Torvalds wants a flag that could be set appropriately when Intel starts shipping CPUs that don't have this problem.
> had to be completed in a hurry
"We reported this issue to Intel, AMD and ARM on 2017-06-01"
https://googleprojectzero.blogspot.com/2018/01/reading-privi...
"We reported this issue to Intel, AMD and ARM on 2017-06-01"
https://googleprojectzero.blogspot.com/2018/01/reading-privi...
I agree with Linus though. Not that I've ever worked on a big project like the kernel but things like these will take a backburner.
The kernel maintainers don't have a way to bring these things up in Intel's boardroom like Apple can and it is possible that Intel will never produce a proper fix.
The kernel maintainers don't have a way to bring these things up in Intel's boardroom like Apple can and it is possible that Intel will never produce a proper fix.
That's Linus. But he's not wrong on this one. If anything he's restrained.
>unnecessarily hostile
Not that I disagree with you, but are you familiar with Linus’ other uhhh...critiques?
http://lkml.iu.edu/hypermail/linux/kernel/1510.3/02866.html
Not that I disagree with you, but are you familiar with Linus’ other uhhh...critiques?
http://lkml.iu.edu/hypermail/linux/kernel/1510.3/02866.html
This would be more convincing if the x86 CPU manufacturer he used to work for were still in business.
I mean, I could sit here and criticize how Linus runs his kernel development project and how so much insecure code gets in, and talk about what a "competent" kernel developer would do, but since I don't run an equally successful, more secure kernel, it's just armchair quarterbacking.
I mean, I could sit here and criticize how Linus runs his kernel development project and how so much insecure code gets in, and talk about what a "competent" kernel developer would do, but since I don't run an equally successful, more secure kernel, it's just armchair quarterbacking.
Interesting concept - we should discount all criticism except from equally or more successful companies than Intel?
if geofft.karma >= foxylad.karma:
print('No comment.')
else:
print('This was a really badly thought out comment.')
I wonder why it is that when I say "I don't think that X should be doing Y in this situation because Z," I so often get responses of the form "So you think that nobody should be doing generic-version-of-Y ever?" That's probably a flaw in my own ability to convey my points. Let me see if I can convey this one a bit better.
First, I think that there's a distinction between constructive, humble criticism, of the form "Why can't the CPU be designed in such a way that ___", and statements like "Well, a competent CPU designer would have done ___". That distinction is not just politeness (although politeness is of intrinsic worth); that decision leads to better technical outcomes. There might be a good answer, like "It can't be designed like that because of the following constraints" or "We tried it and couldn't get it to work"; there might just be an answer like "That's totally the right thing to do and we just cut corners" or "We never thought of it but our competitor does that and it seems to work for them, yeah." Unlike arguments about how competent people actually are, these discussions contain actual technical content, and help different people understand each other's points of view (which is the whole point of having an online, public, decentralized development community, is it not?).
Second, yes, I think that we should discount, though not by much, criticism from people who say "I could totally do that in a better way" and have not actually tried to do that (or have tried and failed to launch a product). There's a lot of things about building complex systems (like modern CPUs) that aren't apparent on paper, and only become apparent once you try to reach some challenging level of scale or performance. This is not at all to say that there's no place for outside criticism; often outside criticism can offer fresh perspectives, or suggest a way to simplify the system and avoid the emergent problems of complexity. And often there are people who are genuinely qualified and really could build a better system, but happen to be working on something else. But it's still fundamentally outside criticism. The only people who can really say "Yes, you totally could have done this instead" are those who actually did.
Third, in this particular scenario, it happens to be the case that ex-Intel employees with insider knowledge have called out Intel for cutting corners (see the "Update" section of https://danluu.com/cpu-bugs/), which seems like a much more productive basis to have a technical conversation.
And if those stories are to be believed and if they're relevant to this problem, the problem is not the competence of engineers - the problem is the competence (or the confused incentives, really) of management who are not telling their engineers to invest time into doing things the engineers know perfectly well how to do, or are telling their engineers to invest time into building other features that make those things harder. (For instance, quite possibly some part of management got worried about benchmarks of syscall performance and told their engineers to prioritize that over making sure that there aren't side-channel leaks between rings.)
First, I think that there's a distinction between constructive, humble criticism, of the form "Why can't the CPU be designed in such a way that ___", and statements like "Well, a competent CPU designer would have done ___". That distinction is not just politeness (although politeness is of intrinsic worth); that decision leads to better technical outcomes. There might be a good answer, like "It can't be designed like that because of the following constraints" or "We tried it and couldn't get it to work"; there might just be an answer like "That's totally the right thing to do and we just cut corners" or "We never thought of it but our competitor does that and it seems to work for them, yeah." Unlike arguments about how competent people actually are, these discussions contain actual technical content, and help different people understand each other's points of view (which is the whole point of having an online, public, decentralized development community, is it not?).
Second, yes, I think that we should discount, though not by much, criticism from people who say "I could totally do that in a better way" and have not actually tried to do that (or have tried and failed to launch a product). There's a lot of things about building complex systems (like modern CPUs) that aren't apparent on paper, and only become apparent once you try to reach some challenging level of scale or performance. This is not at all to say that there's no place for outside criticism; often outside criticism can offer fresh perspectives, or suggest a way to simplify the system and avoid the emergent problems of complexity. And often there are people who are genuinely qualified and really could build a better system, but happen to be working on something else. But it's still fundamentally outside criticism. The only people who can really say "Yes, you totally could have done this instead" are those who actually did.
Third, in this particular scenario, it happens to be the case that ex-Intel employees with insider knowledge have called out Intel for cutting corners (see the "Update" section of https://danluu.com/cpu-bugs/), which seems like a much more productive basis to have a technical conversation.
And if those stories are to be believed and if they're relevant to this problem, the problem is not the competence of engineers - the problem is the competence (or the confused incentives, really) of management who are not telling their engineers to invest time into doing things the engineers know perfectly well how to do, or are telling their engineers to invest time into building other features that make those things harder. (For instance, quite possibly some part of management got worried about benchmarks of syscall performance and told their engineers to prioritize that over making sure that there aren't side-channel leaks between rings.)
Sorry, I have a weakness for reductio ad absurdum, so fair enough to call me out on that. But in this case I'm glad I gave in to temptation, because your response was really valuable.
I think a kernel developer, let alone a core kernel developer and the guy who started the entire project by himself, is probably one of the BEST qualified to comment on the situation.
That was my point - on kernel design, yes, he's way more qualified to comment than I am. But his claim was about CPU design, not kernel design. He's about as qualified to comment about CPU design as I am on kernel design.
He worked at Transmeta, after all.
Yes, that's what I said. Transmeta is out of business; all their clever ideas have nothing to show. It is entirely possible that a "competent CPU engineer" needs to make certain things absolutely work with high performance even if the cost is leaky side channels, and someone who worked at a failed CPU company does not know that because they never got to see their product succeed in the market.
Do you seriously think that winning the market equals being right? "The market" is a horribly ill-idealized model full of wrong assumptions. Uninformed customers, unfair practices and outright illegal behaviour need to be expected. "The market" is a very bad indicator as to who was right.
If the claim being made is "well obviously they should have just done foo and it would have worked," there's reason to believe that foo gives you a significantly less competitive product, and the speaker is not giving any evidence for why it's possible to do foo and still succeed in the market, the only thing we have to go on is the speaker's experience in that market.
A rigorous argument for why foo can be done and doesn't kill your product would be fine too, but this email doesn't even attempt to do that. What can be asserted without proof can be dismissed without proof.
A rigorous argument for why foo can be done and doesn't kill your product would be fine too, but this email doesn't even attempt to do that. What can be asserted without proof can be dismissed without proof.
Transmeta's IP, and its ideas, were gobbled up and put to good use when the company folded.
It was a clever microarchitecture; they just targeted the wrong consumer segment (breaking into x86 was nigh-impossible that late; chip internals on the other hand... =D)
It was a clever microarchitecture; they just targeted the wrong consumer segment (breaking into x86 was nigh-impossible that late; chip internals on the other hand... =D)
...which is why AMD with just as good performance doesn’t have the same flaws.
I'd say having worked on CPU architecture, regardless of the success or failure of it, gives Torvalds even more credibility talking about this sort of thing.
Having said that, though, I think calling it "crap" is a bit much. But that's still a pretty mild/weak insult in Torvalds-speak.
Having said that, though, I think calling it "crap" is a bit much. But that's still a pretty mild/weak insult in Torvalds-speak.
But do you disagree with his points?
Yes, sort of - I don't think he has the evidence to claim that a competent CPU engineer would have done things in a certain way, because all the really theoretically good CPUs (and I count Transmeta among them!) seem to have not succeeded in the market nearly as well as the messy, awful ones, and a CPU that no longer exists doesn't actually count for anything. A competent CPU engineer is balancing lots of tradeoffs. It might be the case that he's right, but in the absence of evidence I weakly believe he's not, and I strongly believe that he's not in possession of strong enough evidence.
https://lkml.org/lkml/2018/1/3/770
Unlike the KPTI patches, which only affect things on each system call, this happens on every indirect call and probably bloats the code considerably too. I can see why Linus is not happy.
Edit: To give a bit more background, predicting indirect calls and speculating into them is absolutely critical to getting good performance from OOP-ish code which tends to use them a lot (virtual functions in C++, function pointers in C). The Linux kernel is (thankfully?) not very OOP-ish, but it does rely on indirect calls (function pointers) extensively.