Show HN: Pipelines – Language for scripting parrallel pipelines with Python(github.com)
github.com
Show HN: Pipelines – Language for scripting parrallel pipelines with Python
https://github.com/calebwin/pipelines
17 comments
Looks interesting but I don’t understand if the “x as x” is mandatory? Seems overly verbose if you don’t want to import it with another name?
That is unnecessary, I don't understand why the Readme includes it. It does nothing.
It's a relatively simple Nim code. It takes a line that starts with "import" and searches for three components: module, function and alias. If you don't provide an alias, you'll get an index error.
[deleted]
I'm not sure I understand the wins of this over just using the standard multiprocessing module.
You can trivially define a worker pool or arbitrary size and pipe a stream of values through your function using multiple cores, avoiding GIL issues.
If you have something which scales to multiple machines just use celery or redis streams.
You can trivially define a worker pool or arbitrary size and pipe a stream of values through your function using multiple cores, avoiding GIL issues.
If you have something which scales to multiple machines just use celery or redis streams.
DASK
Nice readme! It's so common for projects to forget to include an introductory blurb, it's great to see one that does not forget.
Small bug in the example code? I think this:
Small bug in the example code? I think this:
return even % 2 == 0
return number % 2 == 0wait... `/>` and `|>` are python operators?
No. Those are Nim, or rather tokens that guides the Nim code to do the pipe thing.
The code splits the line in function names, args and those operators, to then orchestrate the pipeline flow.
The code splits the line in function names, args and those operators, to then orchestrate the pipeline flow.
Should have named it Pypelines. But seriously though I am not sure what the benefits over multiprocessing are. It looks a little cryptic at first sight.
Just curious if you looked at Apache Airflow as the integration functionality and pipeline / DAG configuration approaches are similar and it's also done in Python.
https://github.com/apache/airflow
The Nim language is new to me. What was the experience like implementing a framework in it?