Performance of ES6 features relative to ES5(kpdecker.github.io)
kpdecker.github.io
Performance of ES6 features relative to ES5
https://kpdecker.github.io/six-speed/?utm_source=ESnextNews.com&utm_medium=Weekly%20Newsletter&utm_campaign=Week%2010
70 comments
I assumed the "es6" row header refers to the engine's native implementation of the feature. In which case it should grey out "destructuring" under Node 4.2.6. But they don't, it's listed as "10x slower". What am I missing?
This makes me think of Python 2 vs 3 benchmarks for some reason, although I'm sure I'm not remembering very well.
I assume there's plenty of scope for ES6 to improve and (hopefully?) surpass ES5 in performance in many, if not all, areas?
What would be causing the large slowdowns in ES6 here? Is this a case of having features that make JS nicer to code in, but giving up some performance? Or is this more down to immature compilers not yet optimising ES6 as substantially?
I assume there's plenty of scope for ES6 to improve and (hopefully?) surpass ES5 in performance in many, if not all, areas?
What would be causing the large slowdowns in ES6 here? Is this a case of having features that make JS nicer to code in, but giving up some performance? Or is this more down to immature compilers not yet optimising ES6 as substantially?
Why is arrow-declare is so much slower on Firefox (16x to 325x slower) compared to everything else (identical to 2.4x slower)?
There is also https://github.com/bevry/esnext-benchmarks
I'd like to see Closure Compiler thrown into the mix. It not only translates ES6 to ES5, but applies optimizations.
The "commentariat" charge is legitimate, which is why there is a field known as "mathematics" which is unambiguous. 1.6x by anybody's non-commentariat definition is 60% faster, as anybody who has ever even scratched the surface of any mathematical, engineering, scientific, statistical or indeed, computer science discipline knows (though this is perhaps unknown to the Creative Suite crowd). This comment is completely bogus, with apologies if you don't understand. But you really should know what you don't know before posting such garbage.
Please don't post uncivil comments to Hacker News.
We detached this subthread from https://news.ycombinator.com/item?id=11204967 and marked it off-topic.
We detached this subthread from https://news.ycombinator.com/item?id=11204967 and marked it off-topic.
Don't be so smug.
Also you're wrong. "__ faster" implies addition in many/most cases. Since you know what 1.6x is by itself, I'll let you use your math knowledge to calculate x + 1.6x.
Also you're wrong. "__ faster" implies addition in many/most cases. Since you know what 1.6x is by itself, I'll let you use your math knowledge to calculate x + 1.6x.
The thing about JIT languages is that there's no such thing as overall speed of a particular operation; you can only sample current speed on current implementations.
New features pursue correctness, then as their place in the JITs mature interesting new approaches will change their speed characteristics later. Things like contrasting the speed of various ways of iterating an array are all semantically equivalent, and as the JIT learns the various semantics they should all theoretically approach similar speeds, even if the newer ones are slower now.
New features pursue correctness, then as their place in the JITs mature interesting new approaches will change their speed characteristics later. Things like contrasting the speed of various ways of iterating an array are all semantically equivalent, and as the JIT learns the various semantics they should all theoretically approach similar speeds, even if the newer ones are slower now.
> The thing about JIT languages is that there's no such thing as overall speed of a particular operation
Except for the ASM.js subset of operations perhaps.
Except for the ASM.js subset of operations perhaps.
That applies to any language implementation.
C compilers nowadays praised for speed used to generate worse code than junior Assembly developers on home micros.
C compilers nowadays praised for speed used to generate worse code than junior Assembly developers on home micros.
> The thing about JIT languages is that there's no such thing as overall speed of a particular operation; you can only sample current speed on current implementations.
How in the world is that specific to JIT? Have you never heard of a Sufficiently Smart Compiler?
How in the world is that specific to JIT? Have you never heard of a Sufficiently Smart Compiler?
The main difference is that you the developer can test with a known compiler and know what the performance characteristics are before you release your code: when Microsoft releases VisualStudio 2018, your existing application doesn't change until you rebuild it. In contrast, for code running in a web browser or something like the JVM, your existing code may run faster or slower without you even knowing about the new version.
> The main difference is that you the developer can test with a known compiler and know what the performance characteristics are before you release your code: when Microsoft releases VisualStudio 2018, your existing application doesn't change until you rebuild it. In contrast, for code running in a web browser or something like the JVM, your existing code may run faster or slower without you even knowing about the new version.
While the release of a new AOT compiler version might not affect the performance characteristics of your application without an active choice to recompile, the release of updates to any OS components your application interacts with, may, in much the same way that a new web browser or JVM version would. The only way you get immunity from that is to bundle the whole software stack running on the bare metal as your "app", rather than relying on lower-level software that may change independently.
While the release of a new AOT compiler version might not affect the performance characteristics of your application without an active choice to recompile, the release of updates to any OS components your application interacts with, may, in much the same way that a new web browser or JVM version would. The only way you get immunity from that is to bundle the whole software stack running on the bare metal as your "app", rather than relying on lower-level software that may change independently.
Right – that's why I didn't say “known compiler and standard library”. That said, it's a difficult comparison because OS updates tend to be more conservative and are definitely less frequent than browser updates, AOT apps in lower-level languages like C/C++ are going to manage a higher percentage of their core data structures because they're pinning something like Boost whereas a JavaScript developer just gets whatever version of Object/Array/etc. is available, and there's not really a concept of a pinned release: if you install Windows apps, you'll pull in major versions of things like .NET or the JVM but there's no analogous practice of pinning a particular browser version. The closest we've come was Internet Explorer's compatibility modes which they've deprecated and which never changed the core JavaScript engine's performance – e.g. IE8-running-as-IE7 still applied the IE8 JavaScript engine's validation and optimizations.
I'm not sure this is true anymore even in this case - with multicore processors, cache dependency, and pervasive virtualization, the speed of your program can be significantly affected by what else is running on the box. Remember that the x86 itself has a JIT on the chip, converting x86 machine code to whatever microcode the processor uses. When I was at Google, they provided special dedicated machines for benchmarking, with custom run-times that disabled a lot of the containerization/virtualization features, and there was still a lot of noise in benchmark times.
That's certainly true and I hope I didn't give the impression that I thought this was a binary situation. It's just that code which runs in a JITed environment, particularly one like the browser JavaScript runtimes which aren't even versioned, has an even wider exposure to skew. Everything running on a multiprocess/user operating system is exposed to resource contention but e.g. your JavaScript code also has to worry about things like Chrome disabling optimizations anywhere you use try/catch.
It that case, what's so special about JIT?
Wouldn't that apply to any interpreted language?
Wouldn't that apply to any interpreted language?
It does but the most common JITed language in existence is JavaScript, where the uncertainty is especially pronounced. For example, Python isn't [traditionally] a JITed language but most Python developers know that their code will run on CPython, which is pretty stable and you have a rough idea of which versions it'll run on, whereas something running in a browser can see huge shifts in what is an expensive or cheap operation from user to user over the course of a single day, much less release to release.
Remember when WebKit started JITing regular expressions? That changed the benchmark situation so much that some people called it cheating and not a single line of user code changed. That's not impossible in other environments – I'm sure there's been a case where a shared library update made a big difference – but it's far less common in my experience.
Remember when WebKit started JITing regular expressions? That changed the benchmark situation so much that some people called it cheating and not a single line of user code changed. That's not impossible in other environments – I'm sure there's been a case where a shared library update made a big difference – but it's far less common in my experience.
[deleted]
I would like to see Babel ES6-to-ES5 converted performance here too, because that's how most real-world ES6 code is going to be run --- there are too many non-ES6-compatible browsers out there.
That's exactly the information the "babel" rows provide? Each row is a given "ES6 implementation" and how it performs compared to the baseline "ES5 native" implementation of a feature.
So the first row of each section is the babel-compiled (to ES5) ES6, the second is traceur-compiled, third is typescript-compiled and last is native ES6 runtime. If you look at the test files, most of them only have an es5 and an es6 versions.
So the first row of each section is the babel-compiled (to ES5) ES6, the second is traceur-compiled, third is typescript-compiled and last is native ES6 runtime. If you look at the test files, most of them only have an es5 and an es6 versions.
Oh, FFS. I managed to spend some time looking at the table without actually seeing what the entire point of it was.
Tea, you have failed me!
Sorry about that.
Tea, you have failed me!
Sorry about that.
Isn't that what the 'babel' rows show?
Note, when it says "1.6x faster" what is really means is "1.6 as fast" or "60% faster", or really "operates at 1.6x the speed of the baseline", not "1.6x increase in speed".
For example, when looking at the data for Chrome 48's "arrow" tests [1], the baseline number is 57,858,016 and the traceur number is 91,556,806, which is 158.2% of the baseline, but is reported as "1.6x faster".
I know this seems like semantics, but "1.0x faster" sounds a lot like "twice as fast". (In this case, "1.0x faster" is reported as "identical" [2]).
[1] https://github.com/kpdecker/six-speed/blob/master/data.json#...
[2] https://github.com/kpdecker/six-speed/blob/master/tasks%2Fre...
For example, when looking at the data for Chrome 48's "arrow" tests [1], the baseline number is 57,858,016 and the traceur number is 91,556,806, which is 158.2% of the baseline, but is reported as "1.6x faster".
I know this seems like semantics, but "1.0x faster" sounds a lot like "twice as fast". (In this case, "1.0x faster" is reported as "identical" [2]).
[1] https://github.com/kpdecker/six-speed/blob/master/data.json#...
[2] https://github.com/kpdecker/six-speed/blob/master/tasks%2Fre...
This is the convention when talking about speedup in software. 1x speedup means identical time.
[1] https://en.wikipedia.org/wiki/Speedup
[1] https://en.wikipedia.org/wiki/Speedup
If the phrase was "1.6x speedup" then I would agree and not have commented. But the phrase is "1.6x faster", which is a bit ambiguous.
That there is disagreement among the commentariat both ways is indicative of ambiguity. (Though, I am assuming no trolling here.)
While not exactly analogous, try replacing it with a percentage and re-evaluate: What does "60% faster" mean and what does "160% faster" mean?
That there is disagreement among the commentariat both ways is indicative of ambiguity. (Though, I am assuming no trolling here.)
While not exactly analogous, try replacing it with a percentage and re-evaluate: What does "60% faster" mean and what does "160% faster" mean?
They did say "1.6x faster", not using the term "speedup" (which is terrible nomenclature too, but that's not the point).
To say that something that's no faster is identical to something that's "1x faster" is a terrible confusion of thought.
It's reasonable to expect that, for example, "1x faster" would mean "for a baseline of 1 operation per second, it's faster by 1 operation per second, i.e. a total of 2 operations per second."
EDIT: Oh god, that's even before I saw the "1.2x slower" entries…
To say that something that's no faster is identical to something that's "1x faster" is a terrible confusion of thought.
It's reasonable to expect that, for example, "1x faster" would mean "for a baseline of 1 operation per second, it's faster by 1 operation per second, i.e. a total of 2 operations per second."
EDIT: Oh god, that's even before I saw the "1.2x slower" entries…
I agree based on my experience reading these types of comparisons. I've never thought there was any ambiguity to saying "1.6 times faster." It means performance_1 = performance_0 * 1.6.
>Note, when it says "1.6x faster" what is really means is "1.6 as fast" or "60% faster", or really "operates at 1.6x the speed of the baseline", not "1.6x increase in speed".
Isn't this always the case with such comparisons? One is meant to multiple initialValue by e.g. 1.6 (hence the "x", for multiple), not calculate initialValue + 1.6*initialValue.
Isn't this always the case with such comparisons? One is meant to multiple initialValue by e.g. 1.6 (hence the "x", for multiple), not calculate initialValue + 1.6*initialValue.
No, the word faster means an increase by that amount. 100% faster means twice as fast. 1x faster means twice as fast. 100% as fast means no change. 1x as fast means no change.
English, you subtle trickster!
English, you subtle trickster!
I'm not sure I've ever seen anything described as being '1x' faster, without any fractional part. It's such an uncommon usage, I really don't think you can categorically declare that '1x faster means twice as fast'.
If I said A was 50% faster than B which is faster? Because it means the same as A is 0.5x faster than B.
Not really, that doesn't make sense. It's "0.5x the speed of b". Saying "faster" when it's really actually slower is just confusing and ungrammatical. Likewise saying "1x faster" when it's actually the same speed is gratuitously confusing.
What doesn't make sense? Saying "A is 50% faster than B" is equivalent to "A is 150% as fast as B". Same goes for "A is 0.5x faster than B" which is equivalent to "A is 1.5x as fast as B".
There is a very distinct difference between "faster than" and "as fast as" which should be very clear now.
There is a very distinct difference between "faster than" and "as fast as" which should be very clear now.
You're 100% right (2x right?), but I think this is one of those situations like "bimonthly", where people usually intend it to mean twice a month instead of every two months.
It's completely ambiguous with potentially significant costs for misunderstanding, but people refuse to make themselves clear, just because.
It's completely ambiguous with potentially significant costs for misunderstanding, but people refuse to make themselves clear, just because.
For "A was 50% faster than B", it could very easily be B which is faster -- and A is half its speed.
What I meant to say in my parent comment is not the use of "faster" vs "as fast" in English, but that usually in those benchmarks "Nx" means N times the speed of the baseline.
What I meant to say in my parent comment is not the use of "faster" vs "as fast" in English, but that usually in those benchmarks "Nx" means N times the speed of the baseline.
In English, "50% faster" is unambiguously "1.5x as fast". No one would ever interpret it as meaning that B is faster.
Again, I wasn't talking about what's it in "proper english" -- but how it's used in benchmarks.
Kevin Decker, who did the site in TFA, is a native english speaker, and yet he writes "1.5x faster" to mean baseline1.5 (and not baseline + 1.5baseline).
So clearly it's not so "unambiguous" whether you can use faster in this way or not. And in fact, I've never seen any of those benchmarks where "Nx" doesn't mean the final value is N*baseline.
Kevin Decker, who did the site in TFA, is a native english speaker, and yet he writes "1.5x faster" to mean baseline1.5 (and not baseline + 1.5baseline).
So clearly it's not so "unambiguous" whether you can use faster in this way or not. And in fact, I've never seen any of those benchmarks where "Nx" doesn't mean the final value is N*baseline.
This confused the heck out of me as a kid. Saying something is 50% faster means the same as being 150% faster. Or 100% faster could also be 200%, depending on subtle language choices.
Very unintuitive.
Very unintuitive.
I still don't see how it could be confusing.
"faster than" != "as fast as"
A is 50% faster than B == A is 150% as fast as B
"faster than" != "as fast as"
A is 50% faster than B == A is 150% as fast as B
Yea in that example it might matter, but I'm seeing places where it's 10x or 100x slower. At that point, it hardly matters.
Correct, at the extremes, the numbers don't matter. But where it says "1.2x slower", it really means "84% of baseline" [1], which might be a bit misleading.
[1] https://github.com/kpdecker/six-speed/blob/master/data.json#...
[1] https://github.com/kpdecker/six-speed/blob/master/data.json#...
vegabook(1)
Thanks Captain Obvious
It's kind of awesome how good TypeScript looks in all this. Testament to the compiler talent MS has on staff.
These measures are JS performance are important for VM implementors, for people writing node.js applications to be used at large scale and people doing computationally-complex work in the browser (eg creating physics engines).
It's worth noting that for most of us, most of the time, this kind of JS performance is less important than user-perceptible optimizations, like batching DOM reads and writes and decreasing asset size
It's worth noting that for most of us, most of the time, this kind of JS performance is less important than user-perceptible optimizations, like batching DOM reads and writes and decreasing asset size
The results given by this benchmark are bothering me because they do not fit with what I've seen in production.
For example, the `for-of-array` is transpiled by babel to something that has 2 nested try/catch blocks, and unfortunately, V8 has a nasty deoptimisation when dealing with these, even if no exception is ever thrown.
This deopt led to a several orders of magnitude slowdown compare to a simple `for` loop in Google Chrome, forcing use to abandon the `for of` construction.
I suspect that the benchmark shown on this page does not expose this kind of behavior because it doesn't iterate enough for the JIT to kick in. If it's the case, it means that these values reflects only the behavior of the cold, interpreted code, and not the hot one. (which is quite sad for a performance benchmark, because the performance matter only for the former …)
This deopt led to a several orders of magnitude slowdown compare to a simple `for` loop in Google Chrome, forcing use to abandon the `for of` construction.
I suspect that the benchmark shown on this page does not expose this kind of behavior because it doesn't iterate enough for the JIT to kick in. If it's the case, it means that these values reflects only the behavior of the cold, interpreted code, and not the hot one. (which is quite sad for a performance benchmark, because the performance matter only for the former …)
Chrome doesn't have an interpreter, it has baseline and optimizing compiler. Also, I think try/catch preventing optimization was fixed.
Still your point could stand, or additionally there could be many optimizations that interfere with such simple micro benchmarks, turning all or parts of them into no-ops. If this is happening in ES5 and not ES6, the results could be drastic. I suspect this is what's happening with some of the larger differences in native ES6.
Still your point could stand, or additionally there could be many optimizations that interfere with such simple micro benchmarks, turning all or parts of them into no-ops. If this is happening in ES5 and not ES6, the results could be drastic. I suspect this is what's happening with some of the larger differences in native ES6.
Agreed, and issues like this make me hugely suspicious of JS performance benchmarks. I'm only familiar with V8, but my experience has been that there are more or less three "tiers" of factors that affect performance:
1. whether the optimizing compiler gets used
2. random stuff that's impossible to know about unless you grab your code's internal representation and decompile it [1]
3. the raw performance of individual statements
The effects of (1) dominate (2), and the effects of (2) dominate (3). So when I see performance micro-benchmarks that purport to measure (3) without apparently paying any attention to (1) or (2), my first inclination is to assume the results are essentially random noise.
[1] E.g. whether statements get wrapped in type checks, whether an integer variable gets handled internally as a SMI (small int) or whether it keeps getting converted to a boxed Number and back. The cost of such things can easily exceed the cost of the statement you're trying to benchmark.
1. whether the optimizing compiler gets used
2. random stuff that's impossible to know about unless you grab your code's internal representation and decompile it [1]
3. the raw performance of individual statements
The effects of (1) dominate (2), and the effects of (2) dominate (3). So when I see performance micro-benchmarks that purport to measure (3) without apparently paying any attention to (1) or (2), my first inclination is to assume the results are essentially random noise.
[1] E.g. whether statements get wrapped in type checks, whether an integer variable gets handled internally as a SMI (small int) or whether it keeps getting converted to a boxed Number and back. The cost of such things can easily exceed the cost of the statement you're trying to benchmark.
> Also, I think try/catch preventing optimization was fixed.
Do you have a source for this? Try/catch deopts still bubble up in my Chrome profiling, so it seems like they're still a problem.
Do you have a source for this? Try/catch deopts still bubble up in my Chrome profiling, so it seems like they're still a problem.
V8 uses Turbofan to optimize some functions that Crankshaft couldn't. As of July, `for-of` and `with` were optimized with Turbofan, but not `try-catch`: http://v8project.blogspot.com/2015/07/v8-45-release.html
Since then there's been work on Turbofan support for `try-catch`, but I can't tell if it's been shipped and enabled, so I could be wrong.
Since then there's been work on Turbofan support for `try-catch`, but I can't tell if it's been shipped and enabled, so I could be wrong.
Thanks for the link! That post definitely makes it sound like try/catch optimizations are on the way. I didn't know it was being worked on.
Yes, my understanding is the problem is essentially undefined behavior as exceptions can be caught at any point in the call stack which makes it basically impossible to optimize.
It's not impossible at all. try/catch in the non-throwing case is fast in both Firefox (SpiderMonkey) and IE (Chakra); I'm not sure what the state of things is in Safari (JavaScriptCore). In the throwing case you obviously have to do some work, but as long as that case is rare it's not a problem. Contrast with the V8 situation (which they are fixing), where simply having a try/catch in your function at all will deoptimize the function, even if an exception is never actually thrown.
Oh, and one optimization strategy for try catch is even pretty simple to describe in general terms: you need a cheap way to check whether an exception is thrown, and then after every operation that can throw (e.g. a call into the vm to a function that is allowed to throw) you check whether it did. If not, you just move along. If it did, you jump to an out of line path that does cleanup and callstack unwinding. The devil, as usual, is in the details.
Oh, and one optimization strategy for try catch is even pretty simple to describe in general terms: you need a cheap way to check whether an exception is thrown, and then after every operation that can throw (e.g. a call into the vm to a function that is allowed to throw) you check whether it did. If not, you just move along. If it did, you jump to an out of line path that does cleanup and callstack unwinding. The devil, as usual, is in the details.
Java optimizes try/catches.
I'm not a frontend developer but... is it still truly necessary to transpile stuff like for-of ?
It's supported by every desktop browser on the market, modulo some bugs (like inability to iterate over HTMLCollection and co. on Chrome)... as long as you only need to use it for arrays, things might work quite fine
It's supported by every desktop browser on the market, modulo some bugs (like inability to iterate over HTMLCollection and co. on Chrome)... as long as you only need to use it for arrays, things might work quite fine
Two words:
Internet Explorer.
Unless you're a startup and don't have to give a damn about what browsers your users use, IE marketshare is just too big to ignore. And depending on your audience, IE can mean anything from IE11 all the way back to IE8.
Theoretically IE is irrelevant now that Microsoft has started publicly killing it off, but you'd be surprised at the number of corporate users that are still on Windows XP (or maybe Vista/7) running IE 8 and 9.
Of course evergreen browsers are less of a problem but amazingly a lot of the same users who are stuck with IE also manage to have extremely outdated versions of Firefox.
Internet Explorer.
Unless you're a startup and don't have to give a damn about what browsers your users use, IE marketshare is just too big to ignore. And depending on your audience, IE can mean anything from IE11 all the way back to IE8.
Theoretically IE is irrelevant now that Microsoft has started publicly killing it off, but you'd be surprised at the number of corporate users that are still on Windows XP (or maybe Vista/7) running IE 8 and 9.
Of course evergreen browsers are less of a problem but amazingly a lot of the same users who are stuck with IE also manage to have extremely outdated versions of Firefox.
According to caniuse, 12% of web users in the US are on IE 11, which doesn't support for-of.
You can use loose mode which doesn't use try/catch blocks
http://babeljs.io/repl/#?experimental=false&evaluate=true&lo...
http://babeljs.io/repl/#?experimental=false&evaluate=true&lo...
More than just these, even innocuous features such as `let` and `const` over `var` cause a substantial performance decrease in V8 (but not SpiderMonkey).
I do a lot of one-off demos, that I mostly write in ES6 these days. I don't have a demonstration of pure `let` vs `var`, but if you change Babel to JavaScript in this one (and click run), you'll see a fairly substantial performance decrease in Chrome for this demo (Firefox stays the same though): https://jsfiddle.net/oa1sckzu/ . I have others, and the results are largely the same for them, but I'll spare you.
I do a lot of one-off demos, that I mostly write in ES6 these days. I don't have a demonstration of pure `let` vs `var`, but if you change Babel to JavaScript in this one (and click run), you'll see a fairly substantial performance decrease in Chrome for this demo (Firefox stays the same though): https://jsfiddle.net/oa1sckzu/ . I have others, and the results are largely the same for them, but I'll spare you.
There is a recent v8-users mailing list thread on this exact topic. Here's one possible explanation on perf differences between `let` and `var`: https://groups.google.com/d/msg/v8-users/hsUrt4I2D98/ELsfO1e...
Why is "identical" in green and "faster" in a slightly darker green? Is es6 a library or is it the browser's implementation of ES6?
ES6 is the native implementation in the browser or JS-engine you are using, indeed.
Green is probably chosen because the same performance is nice to have with this (subjectively) better syntax. The darker shade is probably chosen because more performance is nice, as it pushes you to the newer (subjectively better) syntax.
Green is probably chosen because the same performance is nice to have with this (subjectively) better syntax. The darker shade is probably chosen because more performance is nice, as it pushes you to the newer (subjectively better) syntax.