Amazon Genomics CLI(aws.amazon.com)
aws.amazon.com
Amazon Genomics CLI
https://aws.amazon.com/blogs/industries/announcing-amazon-genomics-cli-preview/
41 comments
c'mon, they couldn't have named it the Amazon Genomics Toolkit CLI or something to get a T in there and then made the binary be named "agtc" instead of "agc"?
Maybe they thought agc was a more peaceful, serine, choice?
You mean serene not the amino acid serine.
For the non molecular biologists out there, the 3-base codon AGC gets converted to the amino acid Serine.
https://en.m.wikipedia.org/wiki/DNA_and_RNA_codon_tables
https://en.m.wikipedia.org/wiki/DNA_and_RNA_codon_tables
You two, so good. I cannot tell which comment is more amusing. :)
AWS prefers at most three worded service names. Three letter acronyms are easier to remember and pronounce than four.
Is there data behind that claim? The Bloomberg Terminal uses up to four and they are not strictly acronyms — sure, WEI is “World Economic Indices”, but TOP is “Top News” and OMON is “Options Monitor”. Four opens up a much wider range of possibilities in English if you don’t strictly use acronyms because there’s a wealth of short words and abbreviations that are super memorable.
a few examples that come to mind: EC2, EMR, ECR, EKS, RDS, EBS, DMS, ECS, VPC, SSS (S3), SNS, SES, IAM,
although in that case agtc is the obvious choice
Amazon Quantum Ledger Database
(QLDB)
(QLDB)
> the genomics and bioinformatics communities have developed specialized workflow definition languages like WDL, Nextflow, CWL, and Snakemake.
It's funny, but after many years in the field, and many generations of workflow definitions, I'm still now happy with any of the options. Snakemake is the closest to being usable for me for both prototyping and longer term work does, but they are all still fragile and inflexible in weird ways.
It's a tough problem to solve, which I think is evidenced by the large number of solutions. I've even tried looking outside at non-bioinformatics tools and never been very happy. And for all their flaws, the bioinformatics tools are better at typical bioinformatics workflows than tools developed for other domains.
It's funny, but after many years in the field, and many generations of workflow definitions, I'm still now happy with any of the options. Snakemake is the closest to being usable for me for both prototyping and longer term work does, but they are all still fragile and inflexible in weird ways.
It's a tough problem to solve, which I think is evidenced by the large number of solutions. I've even tried looking outside at non-bioinformatics tools and never been very happy. And for all their flaws, the bioinformatics tools are better at typical bioinformatics workflows than tools developed for other domains.
Agree with all of this.
The maddening part for us has been that Snakemake, which like for the parent poster has been the best workflow management system for us, is not as widely supported across commercial platforms , and is somewhat less big PaaS enabled than e.g. nextflow.
On the one hand Snakemake does support kubernetes, but on the other hand we are a bioinformatics research group and don’t have k8s engineers :/
(Scaling) Optimal workflow management for genome informatics groups is still very much an open problem.
The maddening part for us has been that Snakemake, which like for the parent poster has been the best workflow management system for us, is not as widely supported across commercial platforms , and is somewhat less big PaaS enabled than e.g. nextflow.
On the one hand Snakemake does support kubernetes, but on the other hand we are a bioinformatics research group and don’t have k8s engineers :/
(Scaling) Optimal workflow management for genome informatics groups is still very much an open problem.
For someone from the productivity tools domain who has been thinking of launching one for bioinformatics users, where could I read more about the typical problems, workflows and existing solutions (and their shortcomings)?
It seems the domain specific aspect is important.
When I read that sentence, I was surprised not to see the classic suggestions from tech/data science:
Prefect, Airflow, Luigi, Apache NiFi, Jenkins, AWS Step Functions, or Pachyderm
When I read that sentence, I was surprised not to see the classic suggestions from tech/data science:
Prefect, Airflow, Luigi, Apache NiFi, Jenkins, AWS Step Functions, or Pachyderm
The reason they’re domain specific is that they were intended to be usable by non-programmers, namely wetlab biologists.
Of course, that didn’t end up being the case. If you can’t program at an elementary enough level to define simple workflow graphs using an API in a non-DSL (like those you mention), then the language is irrelevant—you’re just not technically sophisticated enough to be writing your own workflows. So now we have these DSLs that non-programmers still can’t use and programmers all hate. A lose-lose.
Of course, that didn’t end up being the case. If you can’t program at an elementary enough level to define simple workflow graphs using an API in a non-DSL (like those you mention), then the language is irrelevant—you’re just not technically sophisticated enough to be writing your own workflows. So now we have these DSLs that non-programmers still can’t use and programmers all hate. A lose-lose.
That's a big part of it. And I say this as someone who played a heavy role in the development of one of the named DSLs. Yes we had that exact intention and exact unintended outcome.
There's another part of it as well. Bioinformatics workflows tend to be just enough different from more standard workflows that there is friction using off the shelf tooling.
For one, the DAG nodes are often mostly/fully represented by command line tools expecting a POSIX style file system and making assumptions/asserting opinions on where files live, to where it writes outputs, etc. Bioinformatics workflow orchestrators can understand this and provide optimizations in the DSL to express how to manage the file movement. In contrast, I find many people with a more standard data/biz workflow mentality think of DAG nodes as being queries, running blocks of code, etc.
Lifecycles vary as well. The institutions who have these workflows will either be pure research or some degree of research/production hybrid. There's an advantage to being able to use the same software on both side of that hybrid. When your research workflow is ready to be blessed to production, having to translate it into a different system can be expensive in terms of both time and bugs.
Another aspect is that these workflows, until not too many years ago, were often run on HPC compute clusters using software like SGE, SLURM, PBS, etc. These DSLs can provide optimizations for tweaking parameters in a way that more mainstream tools do not.
There's another part of it as well. Bioinformatics workflows tend to be just enough different from more standard workflows that there is friction using off the shelf tooling.
For one, the DAG nodes are often mostly/fully represented by command line tools expecting a POSIX style file system and making assumptions/asserting opinions on where files live, to where it writes outputs, etc. Bioinformatics workflow orchestrators can understand this and provide optimizations in the DSL to express how to manage the file movement. In contrast, I find many people with a more standard data/biz workflow mentality think of DAG nodes as being queries, running blocks of code, etc.
Lifecycles vary as well. The institutions who have these workflows will either be pure research or some degree of research/production hybrid. There's an advantage to being able to use the same software on both side of that hybrid. When your research workflow is ready to be blessed to production, having to translate it into a different system can be expensive in terms of both time and bugs.
Another aspect is that these workflows, until not too many years ago, were often run on HPC compute clusters using software like SGE, SLURM, PBS, etc. These DSLs can provide optimizations for tweaking parameters in a way that more mainstream tools do not.
I don't think that a single one of these was meant to be used by wetlab biologists, and every single one assumes that the user will be learning an entirely new text based language to express the workflow semantics. In contrast, I see useless GUIs as fairly common on non-bioinformatics workflow systems, and when I see that I know that it is unlikely to be able to scale what I need it to do. Manually drawing every box and line assumes simple workflows that are unlikely to scale very far.
Bioinformatics and Other workflow management systems just have different focuses, more than they have different end user skills. In the 2000s I tried to shoe-horn so many CS Systems type workflow onto using clusters, and every single one had abstractions that simply did not fit what I needed. Typically they seem to require knowledge of how many tasks a job must be split up into, for example, instead of dynamically adapting the workflow to the inputs as desired. Or they hard code the method does launching processes to a certain type of compute environment. So when somebody says "workflow," I assume they mean something entirely different then what I mean until proven otherwise.
For example, if I go to the Apache NiFi docs to see if it's suitable, the Getting Started page starts with a GUI, which is a huge red flag, I don't want to do workflow entry with a GUI because a pre-defined graph will only cover a fraction of my use cases. But worse then that, when it describes where to start creating a processor, it says you need to select from a list of processors. Now this is an even BIGGER red flag because I know absolutely none of the commands I need will be available, and I'll need to add tons of metadata for whatever I want, most likely in some sort of bad GUI, or even worse, XML or some other bad data standard that software engineering adopted due to cargo-cult design. The bigger danger is that the system assumes that processors must be written in some certain language, or assumes that data itself will be introspectable, or something that will require massive amounts of work.
Every one of these systems embodies a huuuuuuge number of assumptions about what it means to run a workflow, nobody ever states what those assumptions are, or probably even realizes how different their needs are from others' needs. At least with the workflow systems from other people in bioinformatics, I know that the assumptions they make will work with the tools and data sizes in the field, and won't have some deal-breaker I only find out about after three hours of making workflows.
Bioinformatics and Other workflow management systems just have different focuses, more than they have different end user skills. In the 2000s I tried to shoe-horn so many CS Systems type workflow onto using clusters, and every single one had abstractions that simply did not fit what I needed. Typically they seem to require knowledge of how many tasks a job must be split up into, for example, instead of dynamically adapting the workflow to the inputs as desired. Or they hard code the method does launching processes to a certain type of compute environment. So when somebody says "workflow," I assume they mean something entirely different then what I mean until proven otherwise.
For example, if I go to the Apache NiFi docs to see if it's suitable, the Getting Started page starts with a GUI, which is a huge red flag, I don't want to do workflow entry with a GUI because a pre-defined graph will only cover a fraction of my use cases. But worse then that, when it describes where to start creating a processor, it says you need to select from a list of processors. Now this is an even BIGGER red flag because I know absolutely none of the commands I need will be available, and I'll need to add tons of metadata for whatever I want, most likely in some sort of bad GUI, or even worse, XML or some other bad data standard that software engineering adopted due to cargo-cult design. The bigger danger is that the system assumes that processors must be written in some certain language, or assumes that data itself will be introspectable, or something that will require massive amounts of work.
Every one of these systems embodies a huuuuuuge number of assumptions about what it means to run a workflow, nobody ever states what those assumptions are, or probably even realizes how different their needs are from others' needs. At least with the workflow systems from other people in bioinformatics, I know that the assumptions they make will work with the tools and data sizes in the field, and won't have some deal-breaker I only find out about after three hours of making workflows.
> I don't think that a single one of these was meant to be used by wetlab biologists
Speaking for myself, we didn't target wetlab biologists with WDL but we did target "non-programmers" as MontyCarloHall described. This was influenced by the groups from which the original developers originated prior to working on the DSL.
The degree to which we were successful can be debated, but hey that was the idea at least ;)
Speaking for myself, we didn't target wetlab biologists with WDL but we did target "non-programmers" as MontyCarloHall described. This was influenced by the groups from which the original developers originated prior to working on the DSL.
The degree to which we were successful can be debated, but hey that was the idea at least ;)
Wetlands biologists tend to be pretty clever, and I have no more difficulty teaching them small bits of AWK to do some quick calculations on their laptop than I do new hires that are very skilled with Python.
If you look at a small WDL tutorial, such as this:
https://www.rc.virginia.edu/userinfo/howtos/rivanna/wdl-bioi...
I think that this is more complex than, say, defining Afro data types, as it involves calling tasks like functions, string substitutions, filling in variables from a config file, etc. There’s a certain base level of complexity that can be hard to abstract away, and I don’t think this has yet got to the simplicity of, say, pivot tables in Excel.
If you look at a small WDL tutorial, such as this:
https://www.rc.virginia.edu/userinfo/howtos/rivanna/wdl-bioi...
I think that this is more complex than, say, defining Afro data types, as it involves calling tasks like functions, string substitutions, filling in variables from a config file, etc. There’s a certain base level of complexity that can be hard to abstract away, and I don’t think this has yet got to the simplicity of, say, pivot tables in Excel.
> There’s a certain base level of complexity that can be hard to abstract away
100% and a conclusion I came to far too long into my tenure in the bioinformatics workflow domain. It is also why I soured a bit on some of the GA4GH efforts in the same space (WES & TES in particular).
I still very much agree w/ the goals & ideals (respectively), but in my experience the middle ground between "I don't care, just give me the defaults" and "please do not abstract away any of the complexity, I need that" is quite small. And that zone would be the sweet spot for these efforts.
100% and a conclusion I came to far too long into my tenure in the bioinformatics workflow domain. It is also why I soured a bit on some of the GA4GH efforts in the same space (WES & TES in particular).
I still very much agree w/ the goals & ideals (respectively), but in my experience the middle ground between "I don't care, just give me the defaults" and "please do not abstract away any of the complexity, I need that" is quite small. And that zone would be the sweet spot for these efforts.
Also DVC, which has gotten really good in the last 2 years.
Snakemake works very well for me on our batch processing cluster, but for other scaling we still package the whole flow in a docker and only scale on sample. Yeah, it's not very efficient. This (AGC) would really help though.
It's nice to see cloud vendors starting to come to where the action is in standardised / open source tool sets rather than trying to capture people into their own proprietary workflow tools which has been a big turnoff for me thus far.
I'd be curious if the underpinnings of the support they have for these tools allows more toolsets to be brought over or if the integration is highly specialised.
I'd be curious if the underpinnings of the support they have for these tools allows more toolsets to be brought over or if the integration is highly specialised.
People underestimate how much compute genomics research chews up. The cloud companies (heck, even the CPU/GPU companies!) are for the most part trying to meet the researchers where they are instead of providing proprietary tools. For two reasons:
a) They can sell a lot of compute time, storage space, etc
b) It's an academic heavy field, and selling proprietary tools tend to be tough sledding
a) They can sell a lot of compute time, storage space, etc
b) It's an academic heavy field, and selling proprietary tools tend to be tough sledding
Industry Espionage as a Service?
In my old university, the people working on the genome studies had to sign a draconian NDA and they had a dedicated air-gapped compute cluster for it. If your genomics study can produce a patent for a new diagnostic method or maybe even a drug precursor molecule, that'll be way more valuable than everything you spent on security.
And already 8 years ago, hacking attempts by what was thought of as foreign government actors was a common experience in life science research. When I had to give a talk about our methods, we scrambled all the gene IDs for fear of the slides getting leaked.
So based on how people working in that field behave, I presume they would never ever agree to upload the data to servers outside of their control. Accordingly, I can only imagine this service to be used accidentally by inexperienced students (who are then on the hook for violating their NDA).
In my old university, the people working on the genome studies had to sign a draconian NDA and they had a dedicated air-gapped compute cluster for it. If your genomics study can produce a patent for a new diagnostic method or maybe even a drug precursor molecule, that'll be way more valuable than everything you spent on security.
And already 8 years ago, hacking attempts by what was thought of as foreign government actors was a common experience in life science research. When I had to give a talk about our methods, we scrambled all the gene IDs for fear of the slides getting leaked.
So based on how people working in that field behave, I presume they would never ever agree to upload the data to servers outside of their control. Accordingly, I can only imagine this service to be used accidentally by inexperienced students (who are then on the hook for violating their NDA).
I have administered an HPC cluster for a research group working on genetics and genomics. There was some data that was the “secret sauce” mostly because it was curated over a long period of time with industrial collaborators and bound by several agreements that it would be a huge task in itself to navigate the proper permissions to ever release them publicly. The default policy was to consider all of them secret and available only on a project need basis for researchers closely associated with the institute.
Now, it is technically possible to have this data in AWS with assurances of encryption and some sane security measures like not leaving it open to public. However, the big catch is - the compute part of the cluster is on-premise. We did the math of moving the compute to AWS, and it was several orders of magnitude expensive(consider that the group did not need to pay for power, network links, and maintenance of the server room by virtue of being part of the university). The only “extra cost” of being on-premise was my salary, which is substantially small portion of the increase by going to AWS. Also, guess what going to AWS doesn’t magically eliminate the need for a guy whose responsibility it is to ensure the cluster is operated, and for users to come in for help, and to manage the ecosystem in general.
The argument of moving to the cloud is more complicated than it appears on the surface. I believe the HPC shops that have a lot of compute and very little need for elasticity will be the last to head to the cloud.
Now, it is technically possible to have this data in AWS with assurances of encryption and some sane security measures like not leaving it open to public. However, the big catch is - the compute part of the cluster is on-premise. We did the math of moving the compute to AWS, and it was several orders of magnitude expensive(consider that the group did not need to pay for power, network links, and maintenance of the server room by virtue of being part of the university). The only “extra cost” of being on-premise was my salary, which is substantially small portion of the increase by going to AWS. Also, guess what going to AWS doesn’t magically eliminate the need for a guy whose responsibility it is to ensure the cluster is operated, and for users to come in for help, and to manage the ecosystem in general.
The argument of moving to the cloud is more complicated than it appears on the surface. I believe the HPC shops that have a lot of compute and very little need for elasticity will be the last to head to the cloud.
You have to take into account the structure of academic research funding as well: grant based with typical horizon of 1-3 years and then funding runs out. If you put all your data in the cloud, who is going to be paying for the "forever-costs" of storage, egress etc after you grants are gone? For researchers this typically comes "for free" with their institutional infrastructure while for cloud it introduces tremendous uncertainty because typically they are required to preserve data supporting publications etc, effectively forever while few grants actually fund that.
If you don't believe AWS's attestations about their security encryption and customer isolation and the associated certifications, then agreed, you shouldn't be sharing your data.
That restricts you to your own infrastructure then. There are tradeoffs in both directions.
That restricts you to your own infrastructure then. There are tradeoffs in both directions.
I worked at a Max Planck genetics lab and never saw an NDA, or even any reference to the notion of keeping anything secret.
The are also dozens of online databases for all sorts of data from biology.
Reading between the lines, you seem to attach certain ideas to the term „genetics“?
The are also dozens of online databases for all sorts of data from biology.
Reading between the lines, you seem to attach certain ideas to the term „genetics“?
they're talking about clinical trials with PII, and some associated genetic sequencing data.
whoever set up your system was overly paranoid. there are exabytes of PII in Amazon now. Air gapping was never a good security method.
No way in hell I’m giving my genomic data through CLI to Amazon
I am pretty sure this is not meant for an average consumer but for pharma companies and other life sciences based companies.
Also academic research, but yes.
These workflow orchestration tools are ubiquitous in the bioinformatics space. A lot mores than things like Airflow and the like.
These workflow orchestration tools are ubiquitous in the bioinformatics space. A lot mores than things like Airflow and the like.
I wonder how many of them would give that data to Amazon either. Thats some of the most valuable data a pharma company has
Amazon doesn't make websites, it gives you the tools to make one. Similarly, I don't think it would want to compete with pharma companies who have equally deep pockets and far more experience than Amazon would.
Use of Big Cloud is very common in this space, in particular AWS.
could you articulate why?
I ask, because I've shared my genome with Google, and it's online for anybody to see.
I ask, because I've shared my genome with Google, and it's online for anybody to see.