Pushing the performance limits of Node.js(jut.io)
jut.io
Pushing the performance limits of Node.js
http://www.jut.io/blog/2015/pushing-node-js-performance-limits
21 comments
I've done CPU laden tasks many times in Node. The key is how you break up the heavy task into pieces that can be distributed more evenly on the event loop instead of blocking it. For example processing a huge data stream is typically done by breaking it into chunks where each chunk can be processed in a few ms, rather than doing one giant blocking operation on the entire stream.
Done properly this also leads to a low memory process that will grab data from an incoming socket stream, transform it, then push that data back out through an outgoing socket stream, only ever keeping a few MB of the stream in working memory at a time. Slap a cluster of these streaming processes together running on a single box to utilize all CPU cores and you can easily push many Mbps of IO through your server quickly and efficiently.
Additionally even if you do have a heavy blocking system IO operation that you can't break up very well you can increase `UV_THREADPOOL_SIZE` when needed to reduce the impact of the blocking.
Done properly this also leads to a low memory process that will grab data from an incoming socket stream, transform it, then push that data back out through an outgoing socket stream, only ever keeping a few MB of the stream in working memory at a time. Slap a cluster of these streaming processes together running on a single box to utilize all CPU cores and you can easily push many Mbps of IO through your server quickly and efficiently.
Additionally even if you do have a heavy blocking system IO operation that you can't break up very well you can increase `UV_THREADPOOL_SIZE` when needed to reduce the impact of the blocking.
But now you've just implemented thread context switching by pulling it's implementation details into your algorithm... Just use a language that supports threading if you have real work to do...
Sometimes it is more efficient to implement the context switching yourself. For extremely high performance server use cases such as realtime analytics, streaming video processing, high frequency trading, etc it isn't uncommon to have server code written in C/C++.
It really depends on what your most important metric is. If the most important metric is speed then having more control over context switching is often better.
It really depends on what your most important metric is. If the most important metric is speed then having more control over context switching is often better.
Same problem with hammers. They're great for nailing, but nearly useless when eating soup.
What I'm trying to say is: use the right tool for the job. If your code is CPU intensive either don't use Node or use it exclusively for the IO-bound tasks and spawn jobs (or use a message queue, or whatever) for your CPU-bound needs.
What I'm trying to say is: use the right tool for the job. If your code is CPU intensive either don't use Node or use it exclusively for the IO-bound tasks and spawn jobs (or use a message queue, or whatever) for your CPU-bound needs.
There are two conflated points here.
> ... things fall apart ... a wrecking ball...
This one is FUD.
I doubt there are many left on HN or anywhere else that hasn't heard this argument, and a similar proportion who don't know the solution (which is that you use clustering or worker processes for tasks the block the process more than your use case can absorb).
> Its great for rapid prototyping...
This one is true, it is great for rapid prototyping, but the implied point that it's not good for production apps is demonstrably false (as proven by the number of serious and respected companies happily using it in production).
There's an argument to be made about node.js being a form of technical debt, and that's a reasonable point, but it also needs to be made in the context of the huge value of that technical debt.
Update: I actually think Node.js is a contributor of technical debt but that it's worth the cost for the use cases I usually deal with (see point 7 on my post: https://medium.com/@richmarr/technical-debt-and-why-most-peo...)
> ... things fall apart ... a wrecking ball...
This one is FUD.
I doubt there are many left on HN or anywhere else that hasn't heard this argument, and a similar proportion who don't know the solution (which is that you use clustering or worker processes for tasks the block the process more than your use case can absorb).
> Its great for rapid prototyping...
This one is true, it is great for rapid prototyping, but the implied point that it's not good for production apps is demonstrably false (as proven by the number of serious and respected companies happily using it in production).
There's an argument to be made about node.js being a form of technical debt, and that's a reasonable point, but it also needs to be made in the context of the huge value of that technical debt.
Update: I actually think Node.js is a contributor of technical debt but that it's worth the cost for the use cases I usually deal with (see point 7 on my post: https://medium.com/@richmarr/technical-debt-and-why-most-peo...)
(Disclosure: I’m the VP Eng at Jut)
As Dave pointed out in the blog, we’ve definitely run into our fair share of challenges with the inherent limits of node.js. But on the whole, we’ve found it to strike a good balance between speed of development and performance. In addition to some of the techniques talked about in the post, we’ve also dropped down to C++ add-ons where we have really critical performance sections, and we do coarse-grained load balancing between multiple node processes to distribute computation on multiple cores.
We also have a good bit of shared code between the browser and server (including the Juttle language compiler and runtime) that has been hugely beneficial for us.
YMMV of course.
As Dave pointed out in the blog, we’ve definitely run into our fair share of challenges with the inherent limits of node.js. But on the whole, we’ve found it to strike a good balance between speed of development and performance. In addition to some of the techniques talked about in the post, we’ve also dropped down to C++ add-ons where we have really critical performance sections, and we do coarse-grained load balancing between multiple node processes to distribute computation on multiple cores.
We also have a good bit of shared code between the browser and server (including the Juttle language compiler and runtime) that has been hugely beneficial for us.
YMMV of course.
I had never read about that 1.5GB heap limit. I've come across that same error message - FATAL ERROR: JS Allocation failed - process out of memory - several times in my application but googling never gave me a good answer why this was happening, especially since I had a lot of spare memory on the server. Very good thing to know...
Seems to me like there's very little new here to anyone paying attention to the space. They didn't raise any limits - they just fixed their own badly written code.
The 1.5 gig limit is not widely known (it's a limit of V8 and I believe, correct me if I'm wrong here, that it's improved in later versions due to some changes to GC) but if you're hitting it you're doing something massively wrong: like they were here.
Splitting data up into chunks for the event loop to process is priority one in any Node app that deals with data processing. It has been for a long time. This is true anywhere you run JavaScript. It's where libraries like [Highland](http://highlandjs.org/) excel and why Node has the concept of Buffers.
Chunking data should be a no brainer and it's frankly a little strange that Jut weren't doing this in the first place. Raises questions about what else is not being done correctly under the hood.
There's no meat here (aside from learning that NPM uses them).
The 1.5 gig limit is not widely known (it's a limit of V8 and I believe, correct me if I'm wrong here, that it's improved in later versions due to some changes to GC) but if you're hitting it you're doing something massively wrong: like they were here.
Splitting data up into chunks for the event loop to process is priority one in any Node app that deals with data processing. It has been for a long time. This is true anywhere you run JavaScript. It's where libraries like [Highland](http://highlandjs.org/) excel and why Node has the concept of Buffers.
Chunking data should be a no brainer and it's frankly a little strange that Jut weren't doing this in the first place. Raises questions about what else is not being done correctly under the hood.
There's no meat here (aside from learning that NPM uses them).
> In fact, as of this writing, the JPC depends on 103 NPM packages
It's really scary that you need so many packages to build a web app using Node.js.
It's really scary that you need so many packages to build a web app using Node.js.
considering how many "packages" are implicitly provided by a traditional LAMP stack, it's not _that_ surprising.
You want cookies? package. Sessions? package. promises? package*. make http requests? package. templating? package. hashing algorithm? package. You get the picture. If I had to replace every PHP function I use with a package, I'd end up with just as many packages.
You want cookies? package. Sessions? package. promises? package*. make http requests? package. templating? package. hashing algorithm? package. You get the picture. If I had to replace every PHP function I use with a package, I'd end up with just as many packages.
That's the problem. In Python, Ruby or even PHP you usually just pick a function / module from the standard library, in Node.js you need to choose one from ~10 quite similar and quite popular third-party packages.
> In Python, Ruby or even PHP you usually just pick a function / module from the standard library
Which is usually rotten for the sake of not breaking things, and forces you to use a third party library anyways (at least it's my experience with Python and exactly the reason why PHP is a mess).
IMHO Node.js not being "batteries included" is for the best.
Which is usually rotten for the sake of not breaking things, and forces you to use a third party library anyways (at least it's my experience with Python and exactly the reason why PHP is a mess).
IMHO Node.js not being "batteries included" is for the best.
that's true. in my memory, it was that way with php 15 years ago too, thinking of phpclasses.org, hotscripts.com etc. Always sorting through the trash...
When I read 20 seconds to compile the list of the 10 most downloaded packages during the past 14 days, my first reaction was that this seems ridicules slow. How many downloads could that query have to accumulate?
A search then revealed that they have surpassed a billion downloads per month. Who on earth installs a billion packages and that every month?
A search then revealed that they have surpassed a billion downloads per month. Who on earth installs a billion packages and that every month?
Most companies using Node probably also have some form of CI. For example the company I work at probably generates tens of thousands of package downloads a day because we have CI running docker container builds for automated testing on every commit, and each container build downloads packages from scratch for an entirely fresh build.
Once a container is built and passes tests it is reused as many times as needed for deploying out on edge hosts so there are no more additional package downloads after that, but still the continual CI builds throughout the day as people commit code generate a lot of downloads.
Once a container is built and passes tests it is reused as many times as needed for deploying out on edge hosts so there are no more additional package downloads after that, but still the continual CI builds throughout the day as people commit code generate a lot of downloads.
FWIW, in that situation, a company should be using their own npm repo, not hitting the public NPM. Artifactory provides one such solution.
It blows me away that so many companies so late in the game still do things like deploy from github, or do CI/CD hitting things like PyPI or NPM constantly.
Yes, these services are generally reliable. If you have a critical release to push, should github being down be an acceptable reason to not be able to deploy it?
It blows me away that so many companies so late in the game still do things like deploy from github, or do CI/CD hitting things like PyPI or NPM constantly.
Yes, these services are generally reliable. If you have a critical release to push, should github being down be an acceptable reason to not be able to deploy it?
Why would you download dependencies on each build instead of once you integrate a new dependency into the project and then just keep it under source control?
If you have a thorough test suite then it is possible to use a package version pattern such as ~1.1.0, or even ^1.1.0 to allow the package to upgrade automatically when the maintainer releases a new version.
Obviously being able to do this is highly dependent on having really thorough test coverage to make sure that automatic package upgrades don't break stuff, but this would be one primary reason for downloading dependencies fresh for each build.
But even if you had locked down the package versions it still wouldn't be a good idea to commit the package into your source control. Part of the benefit of hitting NPM to download the package is that package maintainers will often explicitly deprecate a package version that should now be considered outdated, or perhaps which had a security vulnerability. This will show up as a very visible warning message when running npm install.
This has alerted me multiple times to issues with child modules I had added to my project, or even grandchild modules that were included by other child modules.
Obviously being able to do this is highly dependent on having really thorough test coverage to make sure that automatic package upgrades don't break stuff, but this would be one primary reason for downloading dependencies fresh for each build.
But even if you had locked down the package versions it still wouldn't be a good idea to commit the package into your source control. Part of the benefit of hitting NPM to download the package is that package maintainers will often explicitly deprecate a package version that should now be considered outdated, or perhaps which had a security vulnerability. This will show up as a very visible warning message when running npm install.
This has alerted me multiple times to issues with child modules I had added to my project, or even grandchild modules that were included by other child modules.
I think you're mixing up two different things discussed in the article. There is one remark about how many packages are used by Jut, but then the case study is about NPM using Jut to analyze traffic at the NPM registry. The billions of downloads per month is global traffic. For instance see https://twitter.com/seldo/status/631289016101441537
Nope, I totally realized that the parent comment was just talking about the number of Jut dependencies while I looked for the aggregate number of monthly downloads from NPM. Both numbers are unexpectedly large.
And therein lies the problem with node. Its great for rapid prototyping, but things fall apart pretty quickly with CPU laden tasks. That and the whole single threaded thing with node (technically JS) can be a wrecking ball when one task takes too long.