Backbone Events vs. Ember Bindings: A Benchmark(jsfiddle.net)
jsfiddle.net
Backbone Events vs. Ember Bindings: A Benchmark
http://jsfiddle.net/jashkenas/CGSd5/
35 コメント
To respond in advance: Yes, this is an entirely artificial setup. The point is to start to get a feel for the overhead that different binding strategies can incur -- a question that folks often ask about.
If you're never rendering complicated UI, or never have more than a couple models, this will never make a difference. But if you have a large list of models, or need to run in IE8, or need to run on mobile, it very well may.
Edit: For folks wondering about the cause behind the difference -- Yehuda's looked into it more, and knows more about it than I do -- but there are a number of factors that could be at play:
* bulk innerHTML updates vs. granular DOM twiddling (especially in different browsers).
* Handlebars vs. Underscore templates -- although Handlebars templates are quite fast.
* Fine-grained bindings needing to be kept in sync inside of Ember for Metamorph.js purposes.
* Other miscellaneous overhead.
Some quality time with the Chrome CPU profiler should be able to tell you more. FWIW, Yehuda seems to link it's the latter: overhead that should be possible to eliminate in time.
Edit 2: Looks like @ericf's forked and added a new YUI version: http://jsfiddle.net/ericf/NrpcQ/
If you're never rendering complicated UI, or never have more than a couple models, this will never make a difference. But if you have a large list of models, or need to run in IE8, or need to run on mobile, it very well may.
Edit: For folks wondering about the cause behind the difference -- Yehuda's looked into it more, and knows more about it than I do -- but there are a number of factors that could be at play:
* bulk innerHTML updates vs. granular DOM twiddling (especially in different browsers).
* Handlebars vs. Underscore templates -- although Handlebars templates are quite fast.
* Fine-grained bindings needing to be kept in sync inside of Ember for Metamorph.js purposes.
* Other miscellaneous overhead.
Some quality time with the Chrome CPU profiler should be able to tell you more. FWIW, Yehuda seems to link it's the latter: overhead that should be possible to eliminate in time.
Edit 2: Looks like @ericf's forked and added a new YUI version: http://jsfiddle.net/ericf/NrpcQ/
I can address some of these, as I've been staring at the profile quite a bit:
"bulk innerHTML updates vs. granular DOM twiddling (especially in different browsers)."
A part of the cost is definitely in making these small changes, but not a huge part. For what it's worth, we're in the process of implementing a feature that would allow you to specify the re-rendering granularity in the template itself, but we don't want it to be heavily used by end-users.
"Handlebars vs. Underscore templates -- although Handlebars templates are quite fast."
I don't see the cost of rendering handlebars templates at all in the top of the profile, which is not surprising, as the templates we're rendering are very small, and Handlebars templates are very fast.
"Fine-grained bindings needing to be kept in sync inside of Ember for Metamorph.js purposes."
The cost of bindings are not ongoing, but rather when a `set` happens. Internally, this code is somewhat equivalent to the Backbone event system (we even trigger `change:foo` events) with some more chrome on top to manage things like computed properties. Most of the cost here is not in the internals of the binding system, but rather in the effects of the binding system, like re-rendering the `{{content}}` and updating the style.
In short, the internals of Ember know about as much on an ongoing basis as Backbone's internals know: namely, that there is an observer on `change:content` that needs to be triggered when the underlying property changes. Again, we do more work once that event is triggered than Backbone does, but the vast majority of the cost is on the receiving end of the events, not in the event system itself.
"Other miscellaneous overhead."
From my analysis of the profile, this looks to be the primary cost. The view layer itself could stand to be more optimized. For example, we use a state machine internally to control what events are legal in different states (preRender, inBuffer, hasElement, inDOM). It was our first state machine implementation inside of Ember, and we didn't do some obvious optimizations (like cache legal transitions).
The ember-states code, which ships with ember and is used in ember-data, is far more optimized, and moving to use it should eliminate some of the cost I'm seeing. At the very least, it provides us with a central state machine implementation that we can continue to optimize over time, resulting in performance improvements all around.
There are other areas where micro-optimizations will almost certainly help (such as in the event system itself and how it's used).
"bulk innerHTML updates vs. granular DOM twiddling (especially in different browsers)."
A part of the cost is definitely in making these small changes, but not a huge part. For what it's worth, we're in the process of implementing a feature that would allow you to specify the re-rendering granularity in the template itself, but we don't want it to be heavily used by end-users.
"Handlebars vs. Underscore templates -- although Handlebars templates are quite fast."
I don't see the cost of rendering handlebars templates at all in the top of the profile, which is not surprising, as the templates we're rendering are very small, and Handlebars templates are very fast.
"Fine-grained bindings needing to be kept in sync inside of Ember for Metamorph.js purposes."
The cost of bindings are not ongoing, but rather when a `set` happens. Internally, this code is somewhat equivalent to the Backbone event system (we even trigger `change:foo` events) with some more chrome on top to manage things like computed properties. Most of the cost here is not in the internals of the binding system, but rather in the effects of the binding system, like re-rendering the `{{content}}` and updating the style.
In short, the internals of Ember know about as much on an ongoing basis as Backbone's internals know: namely, that there is an observer on `change:content` that needs to be triggered when the underlying property changes. Again, we do more work once that event is triggered than Backbone does, but the vast majority of the cost is on the receiving end of the events, not in the event system itself.
"Other miscellaneous overhead."
From my analysis of the profile, this looks to be the primary cost. The view layer itself could stand to be more optimized. For example, we use a state machine internally to control what events are legal in different states (preRender, inBuffer, hasElement, inDOM). It was our first state machine implementation inside of Ember, and we didn't do some obvious optimizations (like cache legal transitions).
The ember-states code, which ships with ember and is used in ember-data, is far more optimized, and moving to use it should eliminate some of the cost I'm seeing. At the very least, it provides us with a central state machine implementation that we can continue to optimize over time, resulting in performance improvements all around.
There are other areas where micro-optimizations will almost certainly help (such as in the event system itself and how it's used).
A bit more of an apples to apples comparison http://jsfiddle.net/krisselden/bYPPT and just to illustrate, Ember doesn't force you to micro bind and one would never suggest it for animations.
In reply to the edit -- would be cool if someone could extend this to ExtJS as well just for the heck of it.
In case anyone cares, here's a jsfiddle for KnockoutJS:
http://jsfiddle.net/2pLk3/5/
I was curious how it would stack up - the answer seems to be "pretty well, actually". Of course, all the normal warnings about synthetic benchmarks apply. :)
Edit: I'm updated the JSFiddle to attempt to properly clean up the Knockout bindings. Probably. As I said, I'm pretty new to Knockout still, and may well have made a mistake.
I was curious how it would stack up - the answer seems to be "pretty well, actually". Of course, all the normal warnings about synthetic benchmarks apply. :)
Edit: I'm updated the JSFiddle to attempt to properly clean up the Knockout bindings. Probably. As I said, I'm pretty new to Knockout still, and may well have made a mistake.
Thanks for this - I always wonder why knockout doesn't get more attention on hacker news, it's really great and seems to meet the same needs as ember. Plus, their tutorials are awesome:
http://learn.knockoutjs.com
http://learn.knockoutjs.com
Looks great -- but you need to clean up the knockout bindings when clicking a button again. After a few clicks, it gets slower and slower... Also affects which button you click first.
Feh. As multiple people have kindly pointed out to me, I had a silly typo in my code and was accidentally clobbering the ko.computed function. The updated jsfiddle is here:
http://jsfiddle.net/2pLk3/18/
Embarrassing. :)
Embarrassing. :)
You are not actually using ko.computed objects in your fiddle.
My take on it: http://jsfiddle.net/HusVw/1/
And guess what, it's _much_ slower. (And I'm a happy KO user).
My take on it: http://jsfiddle.net/HusVw/1/
And guess what, it's _much_ slower. (And I'm a happy KO user).
There's some discussion on the ko google groups about this :
https://groups.google.com/forum/?fromgroups#!topic/knockoutj...
you are right, the correct version is perhaps slightly faster than the ember, but way slower than the backbone example.
However, by leveraging an optimized ko.computed implementation
http://mbest.github.com/knockout-deferred-updates/
the results are much more impressive.
http://jsfiddle.net/2pLk3/10/
https://groups.google.com/forum/?fromgroups#!topic/knockoutj...
you are right, the correct version is perhaps slightly faster than the ember, but way slower than the backbone example.
However, by leveraging an optimized ko.computed implementation
http://mbest.github.com/knockout-deferred-updates/
the results are much more impressive.
http://jsfiddle.net/2pLk3/10/
I'm don't agree that that the version with unused intermediate computed observables is the only correct version (or that there even is only one correct version).
Avoiding the use of intermediate computeds makes the Knockout version roughly as fast as Backbone for this specific benchmark, and more closely replicates the functionality of the Backbone example. That sounds like the correct version to me.
Avoiding the use of intermediate computeds makes the Knockout version roughly as fast as Backbone for this specific benchmark, and more closely replicates the functionality of the Backbone example. That sounds like the correct version to me.
One thing that jumped out at me pretty quick in comparing the two implementations in jsFiddle is that with ember every property is set with an individual call rather than passing in an object of properties to change. Here is a version with backbone using individual set calls with a notable dip in performance: http://jsfiddle.net/6rvkC/
Is this limitation in ember because of the computed properties? Would changing this even help?
Is this limitation in ember because of the computed properties? Would changing this even help?
Interestingly, not really, because the Ember run loop implementation coalesces multiple calls to set, so using setProperties (which allows you to pass in multiple properties at once) doesn't really affect performance much.
It's one of the first things I tried, in case there was a bug in the run loop code that caused the re-render to happen multiple times.
It's one of the first things I tried, in case there was a bug in the run loop code that caused the re-render to happen multiple times.
I think it's worth noting, that Backbone (as expected, I imagine) gives you a little more rope to hang yourself with here. To me, that's a great highlight of how the frameworks differ in their approach. "Perfectly written" Backbone code is significantly faster; but, if you (perhaps unknowingly) take a slower path, Backbone isn't going to do anything to make it faster. Not advocating one approach over the other; just pointing out the juxtaposition.
Quite, and it's also worth noting that by relying on a "runloop" to defer your renders, you're giving up synchronicity in your data changes -> ui updates when you don't necessarily need to (a very common source of dev bugs in SproutCore 1).
Not having worked with SproutCore, I honestly can't comment on whether it introduced more bugs. Working with anything async is always going to introduce some complexity; but given the poor DOM performance on low-end devices, deferring rendering is often a very helpful concept.
I'd be interested to hear what type of bugs were common in this scenario; my understanding of the runloop is that after the DOM update (at the end of each loop) the UI should reflect the application state.
I'd be interested to hear what type of bugs were common in this scenario; my understanding of the runloop is that after the DOM update (at the end of each loop) the UI should reflect the application state.
For those interested in, or learning Ember, I've taken Jeremy's Ember code and rewrote it in idiomatic Ember style, with commentary - http://joubert.posterous.com/application-architecture-in-emb...
Both make my Core i7 CPU go to 100% for a simple animation of 100 small circles.
I thought HTML5 was going to solve all performance problems and make me breakfast too. Wonder if this will even run on mobile.
I thought HTML5 was going to solve all performance problems and make me breakfast too. Wonder if this will even run on mobile.
They're both pretty slow considering the ridiculous hardware of today's computers. We're really running a lot of our stuff on layers and layers of performance-draining abstractions... I remember the days when my friends were complaining about the unneccesary slowdown of MUI apps for their Amigas and I was reading Michael Abrash's Black Book to improve my line-drawing routines. Granted, I surely don't want to go back to the days of Mode X DOS programming, but the whole web stack is really one baroque mess, despite the heroic achievements of browser and Javascript VM developers.
I completely agree. So much clever engineering is being devoted right now to making HTML do things it was never intended to do. I'm not happy about the shift to proprietary native platforms but selfishly I'm so much happier coding close to the metal in iOS with a well-designed UI toolkit.
Close to the metal? Well, in your case that might be true, but I see more and more applications on mobile devices that are pretty small wrappers around Webkit views, as this makes portability easier and harnesses the existing skills of the developers.
Micro-benchmarks are useful, but please keep in mind that you'll most likely not be updating hundreds of objects multiple times a second in a real application...
You might be, it depends on the application. That's exactly what's going in in this NY Times visualization:
http://www.nytimes.com/interactive/2012/02/13/us/politics/20...
Though it uses d3 rather than Backbone or Ember.
http://www.nytimes.com/interactive/2012/02/13/us/politics/20...
Though it uses d3 rather than Backbone or Ember.
That is an amazing visualization.
That's what you think. Every application is different.
Could the difference be mostly due to handlebars vs. underscore? I'd be curious how it'd look normalizing for templating engine.
At first glance it seems that template rendering is a key point here (render-as-a-string vs. dom-manipulation). However, if you change the backbone implementation to use dom manipulation (http://jsfiddle.net/jqSK2/) it performs just as well (or better, depending on the browser). Backbone's event system is much more important in this comparison.
I've been looking at the profiler (both Chrome and Safari) a bunch and it is emphatically not an issue with handlebars template. A number of things show up, but templates don't really show up as a major factor at all.
There are a number of separate issues that bubbled up in the profile that I'm looking at. At least one seems like it may be a bug (but I may just be looking at it wrong), and the rest seem like cases where we can reduce the overall overhead of the binding system (as well as overhead in the view layer).
There are a number of separate issues that bubbled up in the profile that I'm looking at. At least one seems like it may be a bug (but I may just be looking at it wrong), and the rest seem like cases where we can reduce the overall overhead of the binding system (as well as overhead in the view layer).
Out of curiosity and apologies if this is a lazy question but why is the difference so pronounced?
[deleted]
For the past few months, we've prioritized creating an API that is friendly, conceptually simple, and correct. Our view implementation, for example, handles many edge cases that other frameworks might punt on.
We believe that nailing the API first is important for a number of reasons. We can increase performance without you having to change your app; the same is not true if we spend time making it fast, then require you to change the API once we realize it is not a good one.
The cool thing about declarative systems is that once you express your intent in a way the framework understands, we can apply heuristics and optimizations that every app developer gets for free.
Some people pooh-pooh the idea that better abstractions require a "sufficiently advanced compiler," but I think the history of the field has proved that it's the case. V8 and LLVM are examples of projects where we continue to eke out performance increases well past what I think the average person would have expected. I saw the same thing in Handlebars-- a lot of people criticize Handlebars for being separated into multiple steps, as it increases the size of the library somewhat, but being able to reason about each of the steps in the compilation pipeline has given contributors the ability to reason about possible slowdowns and turned it into one of the fastest template libraries around.
So, all of that being said, I would never recommend using bindings for animation. I'm not aware of any framework or library that recommends this. Bindings were designed for infrequently changing values that need to be displayed to the user. Since the smoothness of animation is critical to the user experience, it's hard to argue that any optimization is premature optimization. All of the animation we've done in our Ember apps is done via either CSS transitions, or using requestAnimationFrame and modifying the DOM element directly.
While Ember.js will presumably always be technically slower than Backbone, slower doesn't have to mean slow. That being said, there are obvious performance gains to be made here, and I'm glad we've got Jeremy to keep us on our toes. :P