Any good Node.js parallelization libraries?
Or do I have to use something along the lines of Web Workers or NodeJS Worker Threads? I would prefer it if I had an API similar to Python’s concurrent.futures. What I really want is apply a function to an array in parallel (i.e. some parallel version of .map)
5 comments
Some ground info, Node has worker_threads for this purpose. https://nodejs.org/api/worker_threads.html
I haven't had cause to use it yet, but I very much trust/respect the team behind: https://www.npmjs.com/package/piscina . It's so popular it has well maintained forks, such as tinypool, https://www.npmjs.com/package/tinypool .
Also seeing good npm popularity metrics around: https://www.npmjs.com/package/synckit
I haven't had cause to use it yet, but I very much trust/respect the team behind: https://www.npmjs.com/package/piscina . It's so popular it has well maintained forks, such as tinypool, https://www.npmjs.com/package/tinypool .
Also seeing good npm popularity metrics around: https://www.npmjs.com/package/synckit
We used piscina at it was quite good: https://github.com/piscinajs/piscina
While it may not provide parallel map out of the box it will help to implement it a little bit easier than using nodes api.
While it may not provide parallel map out of the box it will help to implement it a little bit easier than using nodes api.
It is a single threaded runtime so to parallelize a map you will need to serialize the items and ship them to another node process. A bit like dask in Python.
Tried it, seemed to actually DECREASE my program’s performance. I think it’s because of the overhead of creating thousands of threads…