Dave Jones used to have a series of "Why user space sucks" Linux kernel conference talks with many such examples, usually with dumb and redundant system calls.
However as someone who looks a lot at instruction traces I could probably write on e on why Linux kernel code sucks too. One of my current pet peeves is the way Linux walks bitmasks of CPU bits, which is a reasonably common operation. Due to a chain of unfortunate changes and decisions it currently needs 16+ instructions to find the next bit for something which the x86 instruction set has a single instruction. Of course that is so big that it is even outlined, adding even more overhead.
Early x86-64 Linux had a similar problem. The x86-64 ABI uses registers for the first 6 arguments. To support variable number of arguments (like printf) requires passing the number of arguments in an extra register (RAX), so that the callee can save the registers to memory for va_arg() and friends. Doing this for every call is too expensive, so it's only done when the prototype is marked as stdarg.
Now the initial gcc implemented this saving to memory with a kind of duffs device, with a computed jump into a block of register saving instructions to only save the needed registers. There was no boundary check, so if the no argument register (RAX) was not initialized correctly it would jump randomly based on the junk, and cause very confusing bug reports.
This bit quite some software which didn't use correct prototypes, calling stdarg functions without indicating that in the prototype. On 32bit code which didn't use register arguments this wasn't a problem.
Later compiler versions switched to saving all registers unconditionally.
If you read Anderson "A history of Aerodynamics" it disagrees on this point. It states that the Wright's didn't have a good way to calculate drag, and they didn't understand many of the side effects from real wings (like flow separation) which caused wrong measurements initially. Later on they apparently came back to something that was closer to Lilienthal's numbers, even though the problem simply wasn't fully understood at the time.
"When the Wright brothers compared their results with those of Lilienthal, they found some
disagreement, but not as much as they expected. As Wilbur states in his diary for October
16, 1901: "It would appear that Lilienthal is very much nearer the truth then we have
heretofore been disposed to think." [Wolko, 21]. 17 The formulas were still not producing
the lift and drag that were actually being produced. The only other possible source of error
in these equations was the Smeaton coefficient of air pressure."
This technique works with any instruction that clobbers a register, not just with CPUID. In the worst case you could just single step the other CPUs until you hit an instruction that overwrites a register too. These are common.
In my case (for a custom hypervisor for a sadly cancelled project) it wasn't a problem because the hypervisor quit itself in early guest boot, which is single CPU only.
However as someone who looks a lot at instruction traces I could probably write on e on why Linux kernel code sucks too. One of my current pet peeves is the way Linux walks bitmasks of CPU bits, which is a reasonably common operation. Due to a chain of unfortunate changes and decisions it currently needs 16+ instructions to find the next bit for something which the x86 instruction set has a single instruction. Of course that is so big that it is even outlined, adding even more overhead.