Nextflow: System for creating scalable, portable, reproducible workflows(github.com)
github.com
Nextflow: System for creating scalable, portable, reproducible workflows
https://github.com/nextflow-io/nextflow
23 comments
To implement an efficient dataflow-based programming API/DSL, you better have some support for channels and lightweight threads in a scriptable language, something that you've got in Groovy with the GPars library that Nextflow uses.
We opted for implementing all of this in Go in SciPipe, where we get similar basic dataflow/flow-based functionality as Nextflow with the native concurrency primitives of Go, but the Go syntax probably/surely puts away some biologists who have written some python at most before, and Go won't let us customize the API and hide away as much of the plumbing under nice syntax, as Groovy.
In this regard, Groovy with the GPars library for the concurrency, doesn't seem as a particularly bad choice. There weren't that many options at the time either.
The downside has been tooling support though, such as editor intelligence and debugging support, although parts of that is finally improving now with a NF language server.
Today, one could probably implement something similar with Python's asyncio and queues for the channel semantics, and there is even the Crystal language that has Go-like concurrency in a much more script-like language (see a comparison between Go and Crystal concurrency syntax at [1]), but Crystal would of course be an even more fringe langauge than Groovy.
[1] https://livesys.se/posts/crystal-concurrency-easier-syntax-t...
We opted for implementing all of this in Go in SciPipe, where we get similar basic dataflow/flow-based functionality as Nextflow with the native concurrency primitives of Go, but the Go syntax probably/surely puts away some biologists who have written some python at most before, and Go won't let us customize the API and hide away as much of the plumbing under nice syntax, as Groovy.
In this regard, Groovy with the GPars library for the concurrency, doesn't seem as a particularly bad choice. There weren't that many options at the time either.
The downside has been tooling support though, such as editor intelligence and debugging support, although parts of that is finally improving now with a NF language server.
Today, one could probably implement something similar with Python's asyncio and queues for the channel semantics, and there is even the Crystal language that has Go-like concurrency in a much more script-like language (see a comparison between Go and Crystal concurrency syntax at [1]), but Crystal would of course be an even more fringe langauge than Groovy.
[1] https://livesys.se/posts/crystal-concurrency-easier-syntax-t...
I really wish Crystal had taken off a bit. I thought it had a chance in bfx with some good benchmarking and PR by lh3 in biofast.
If the jvm is acceptable, was clojure considered? It has async, data driven idioms, tooling (repl), and is not hard.
Snakemake is easy to deal with that scenario. I had a profile for each of our slightly different hpc clusters. You could throttle the array by total resources so I could say request no more than 750gb memory allocated across the array to be polite to the rest of the hpc users, and it would fit however many jobs it could within that constraint and step of the pipeline. I could have a job instead be ran on the internet connected head node vs airgapped compute node if something needed downloading. Worked great and the python syntax is pretty useful along with conda env management baked in.
Cool seeing a workflow language pop up on HN!
Nextflow and Snakemake are the two most-used options in bioinformatics these days, with WDL trailing those two.
I really wish Nextflow was based on Scala and not Groovy, but so it goes.
There is a Draft up for dsl3 that adds static types to the channels that I’m very excited about. https://github.com/nf-core/fetchngs/pull/309
Nextflow and Snakemake are the two most-used options in bioinformatics these days, with WDL trailing those two.
I really wish Nextflow was based on Scala and not Groovy, but so it goes.
There is a Draft up for dsl3 that adds static types to the channels that I’m very excited about. https://github.com/nf-core/fetchngs/pull/309
At a previous Biotech, we used Cromwell/WDL because the DSL was the most intuitive to our bioinformatics scientists. But seeing as that doesn't work as nicely on AWS (and is also supported by an organization that is imploding), we opted for Argo on our K8s cluster to process RNAseq data en masse. Getting the scientists to use YAMl has been an uphill struggle, but the same issues would apply to learning groovy I guess. We've found that the Argo engine is easier to maintain, and also we only have to support one orchestrator across our Bioinformatics and ML teams.
For industrial purposes, I've started to approach these pipelines as a special case of feature extraction and so I'm reusing our ML infrastructure as much as possible.
For industrial purposes, I've started to approach these pipelines as a special case of feature extraction and so I'm reusing our ML infrastructure as much as possible.
I would rather write Groovy than YAML any day of the week.
Why did you rule out Nextflow or Snakemake? I believe they both work with k8 clusters.
Argo doesn’t look great from my standpoint as a workflow author.
Why did you rule out Nextflow or Snakemake? I believe they both work with k8 clusters.
Argo doesn’t look great from my standpoint as a workflow author.
For both workflow languages, they are both better for building a singular reproducible workflow that can be published with an academic paper. For us, I'm looking for a workflow language that can treat the pipeline as a testable, deployable piece of software. I find that with Nextflow, scientists fall into bad patterns of mixing in the pipeline logic (eg if this sample type, then process it this way) interspersed with the bioinformatics model (eg use these bowtie2 parameters) throughout the pipeline which makes it more difficult to maintain as our platform evolves. Their K8s integration is lacking for both of them and they work much better an academic-style clusters.
YAML does leave a lot to be desired, but it also forces a degree of simplicity in architecting the pipeline because to do otherwise is too cumbersome. I really liked WDL as a language when I used to use that--seemed to have a nice balance of readability and simplicity. I believe Dyno created a python SDK for the Argo YAML syntax, and I need to look into that more.
YAML does leave a lot to be desired, but it also forces a degree of simplicity in architecting the pipeline because to do otherwise is too cumbersome. I really liked WDL as a language when I used to use that--seemed to have a nice balance of readability and simplicity. I believe Dyno created a python SDK for the Argo YAML syntax, and I need to look into that more.
I've used Snakemake my whole life, can someone experienced with both systems share whether jumping to nextflow is worth it?
I have pipelines written in both frameworks.
Nextflow (despite the questionable selection of groovy as the language of choice) is more powerful and enables greater flexibility in terms of information flow.
For example, snakemake makes it very difficult if not impossible to create pipelines that deviate from a DAG architecture. In cases where you need loops, conditionals and so on, Nextflow is a better option.
One thing that I didn't like about nextflow is that all processes can either run under apptainer or docker, you can mix and match docker/apptainer like you do in snakemake rules.
For example, snakemake makes it very difficult if not impossible to create pipelines that deviate from a DAG architecture. In cases where you need loops, conditionals and so on, Nextflow is a better option.
One thing that I didn't like about nextflow is that all processes can either run under apptainer or docker, you can mix and match docker/apptainer like you do in snakemake rules.
Can you describe a scenario that would be impossible to code for in a snakemake paradigm? For example at least with conditionals I imagine you could bake some flags into the output filename and have different jobs parse that. I’m not sure exactly what you mean by loop but if its iterating over something that can probably be handled with the expand or lambda functions.
Here is a scenario which is relatively trivial in Nextflow and difficult to write in snakemake:
1. A process that "generates" protein sequences
2. A collection of processes that perform computationally intensive downstream computations
3. A filter that decides, based on some calculation an a threshold whether the output from process (1) should move to process (2).
Furthermore, assume you'd like process (1) to continue generating new candidates continously and independently until N number of candidates pass the filter for downstream processing.
That's not something that you can do easily with snakemake since it generates the DAG before computation starts. Sure, you can create some hack or use checkpoints that forces snakemake to reevaluate the DAG and so on, and maybe --keep-going=true so that it won't end the other processes from failing, but with nextflow you just set a few channels as queues and connect them to processes, which is much easier.
1. A process that "generates" protein sequences
2. A collection of processes that perform computationally intensive downstream computations
3. A filter that decides, based on some calculation an a threshold whether the output from process (1) should move to process (2).
Furthermore, assume you'd like process (1) to continue generating new candidates continously and independently until N number of candidates pass the filter for downstream processing.
That's not something that you can do easily with snakemake since it generates the DAG before computation starts. Sure, you can create some hack or use checkpoints that forces snakemake to reevaluate the DAG and so on, and maybe --keep-going=true so that it won't end the other processes from failing, but with nextflow you just set a few channels as queues and connect them to processes, which is much easier.
Just make your N number of candidates check generate some empty file after N is reached and put that as input for the next job. For threshold example you can do the same thing or even bake the metric into a filename.
As I said, you can hack your way through snakemake to make it work probably using DAG reevaluation and tricks with filenames, but Nextflow allows it in a much more straightforward manner that's more easy to follow, understand and debug.
I’ve used both. I would say nextflow is a more production-oriented tool. Check out seqera platform to see if any of the features there seem useful. It can also be useful to get out of the wildcards/files mindset for certain workflows. Nextflow chucks the results of a step into a hashed folder, so you don’t have to worry about unique output names.
That said, I do find snakemake easier to prototype with. And it also has plenty of production features (containers, cloud, etc). For many use cases, they’re functionally equivalent
That said, I do find snakemake easier to prototype with. And it also has plenty of production features (containers, cloud, etc). For many use cases, they’re functionally equivalent
NF Tower / Seqera would be the selling points. They offer a nice UX for managing pipelines and abstract over AWS.
Technically snakemake can do it all. But in practice NF seems to scale up a bit better.
That said, if you don’t need the UI for scientists, I’d stick to snakemake.
Technically snakemake can do it all. But in practice NF seems to scale up a bit better.
That said, if you don’t need the UI for scientists, I’d stick to snakemake.
Nextflow transformed how I did bioinformatics, truly should be a top skill sought after in bioinformaticians
ayyy, they used this in one of my previous workpaces in biotech.
I have a dislike of nextflow because it submits 10s of thousands of separate jobs to our HPC scheduler which causes a number of issues, though they've now added support for array jobs which should hopefully solve that.