"high level" languages are easier to optimize(jyn.dev)
jyn.dev
"high level" languages are easier to optimize
https://jyn.dev/high-level-languages-are-easier-to-optimize/
10 comments
High level languages are easier to optimize because they are starting with more de-optimizations. Haskell can be optimized to perform stream operations on unboxed integers, precisely because integers are boxed to start with! Whereas the code you would have written in C to do the same thing would already be using unboxed integers, no?
The greatest value brought by compiler optimisations is removing the overhead of convenience. Sometimes that is about avoiding the boxing that is a necessity in many high level languages, but in other cases it serves to allow a more modular programming style without overhead. Stream fusion is a good example: it lets you structure your program as small and composable units, without the cost of manifesting intermediate results. That is not merely about avoiding the inherent inefficiency of e.g. Haskell, but about permitting different styles of programming, and the argument is that a low level language simply cannot allow such a style (without overhead), because the required optimisations are not practical to implement.
Sometimes. Other times you write inneficient C because it makes the code simpler.
you can imagine a version of Haskell that doesn’t have polymorphism or laziness, that only has unboxed integers, and as a result doesn’t need a GC. in such a language it’s still easy for the compiler to do stream fusion; whereas it’s still hard in C because the compiler needs to prove the loop doesn’t have side effects.
The argument isn't wrong and people have been making it for decades that everything from Java to standard ML should be theoretically faster than C. I think it's telling that despite that, the comparison the author chose is a functional language running in parallel (after compiling to C) vs single threaded C. That comparison is obviously nonsense.
The argument is wrong because the best optimizer is the stuff between your ears, not a compiler. C and to a lesser extent C++ have always won because they allow you to spend those neurons thinking about the underlying machine with a thin(ner) abstraction layer instead of wrapping it in layers of conceptually elegant abstractions that get dissected by a sufficiently smart compiler.
The argument is wrong because the best optimizer is the stuff between your ears, not a compiler. C and to a lesser extent C++ have always won because they allow you to spend those neurons thinking about the underlying machine with a thin(ner) abstraction layer instead of wrapping it in layers of conceptually elegant abstractions that get dissected by a sufficiently smart compiler.
Is there any work trying to apply transformers to compiler optimization? Seems like we have some new tools to potentially make compilers even better.
But yes, humans can still usually optimize better.
The other thing you see with high level languages is a death by a thousand cuts due to things like cache locality or instruction level parallelism. It’s very, very hard to write a VM with stuff going on like JIT and GC and a heap allocator that gives you good locality or ILP. A major problem is that yes you can optimize for that, including in real time, but that not only adds work but adds work that implies a cache flush.
The latter point — cache locality and ILP — is why some C/C++/Rust code is faster when compiled optimizing for space instead of for “speed.” Less code means it fits in cache.
All that being said, HLLs usually offer superior programmer productivity, especially for novice to mid career devs who aren’t quite up to things like comprehending the Rust borrow checker. Machine time has to be weighed against human time. The latter is usually more expensive (but not always at scale!).
But yes, humans can still usually optimize better.
The other thing you see with high level languages is a death by a thousand cuts due to things like cache locality or instruction level parallelism. It’s very, very hard to write a VM with stuff going on like JIT and GC and a heap allocator that gives you good locality or ILP. A major problem is that yes you can optimize for that, including in real time, but that not only adds work but adds work that implies a cache flush.
The latter point — cache locality and ILP — is why some C/C++/Rust code is faster when compiled optimizing for space instead of for “speed.” Less code means it fits in cache.
All that being said, HLLs usually offer superior programmer productivity, especially for novice to mid career devs who aren’t quite up to things like comprehending the Rust borrow checker. Machine time has to be weighed against human time. The latter is usually more expensive (but not always at scale!).
when’s the last time you wrote a parallel array traversal in C?
also, consider reading the linked post about how assembly instructions are no longer a good approximation of how your computer works: https://queue.acm.org/detail.cfm?id=3212479. in general, writing in a language that is not close to the hardware allows the compiler to adapt when the hardware changes; for example futhark has the ability to execute using either SIMD or GPUs precisely because it’s not over-determined by the source language. C ties processors to the model of the PDP-11, which hasn’t been manufactured for 30 years.
also, consider reading the linked post about how assembly instructions are no longer a good approximation of how your computer works: https://queue.acm.org/detail.cfm?id=3212479. in general, writing in a language that is not close to the hardware allows the compiler to adapt when the hardware changes; for example futhark has the ability to execute using either SIMD or GPUs precisely because it’s not over-determined by the source language. C ties processors to the model of the PDP-11, which hasn’t been manufactured for 30 years.
It's not especially difficult to write a parallel array sum in CUDA, which is C++ with a couple of keywords bolted on. Haven't done that in a bit, but I wrote a SIMD hsum not long ago without much difficulty either.
C was of course originally designed for the PDP-11, but neither the standard nor the implementations have assumed that anytime this century. It would be a quite a stretch to say that thread local storage, atomics, the weird restrictions on pointers to deal with segmented architectures, IEEE floats, and other "modern" additions have anything to do with PDP-11s. And obviously you can take C/C++ code and efficiently build it for a wildly different architecture, like you do every time you use a compiler (including NVCC).
I'm not even saying that C is the fastest possible language because it really shouldn't be. What I'm saying is that decades of HLL advocates saying that we just need a sufficiently smart compiler to beat C have failed to produce one. C-family languages remain the gold standard for performance, and there's not much that even reliably competes beyond Rust and Fortran. Fortran is also an interesting example of a "low level" language without many of the bad ideas of C that ends up not much faster these days.
C was of course originally designed for the PDP-11, but neither the standard nor the implementations have assumed that anytime this century. It would be a quite a stretch to say that thread local storage, atomics, the weird restrictions on pointers to deal with segmented architectures, IEEE floats, and other "modern" additions have anything to do with PDP-11s. And obviously you can take C/C++ code and efficiently build it for a wildly different architecture, like you do every time you use a compiler (including NVCC).
I'm not even saying that C is the fastest possible language because it really shouldn't be. What I'm saying is that decades of HLL advocates saying that we just need a sufficiently smart compiler to beat C have failed to produce one. C-family languages remain the gold standard for performance, and there's not much that even reliably competes beyond Rust and Fortran. Fortran is also an interesting example of a "low level" language without many of the bad ideas of C that ends up not much faster these days.
Fortran is nothing like a low-level language.
Typed languages are even more easier to optimize. No need for the box/unboxing dance and escape analysis