But I'd add getting rid of heatmaps on large datasets. They are information dense and pretty, but I can't see how anyone interprets them.
Better to do clustering and plot the data for each relevant cluster in a more meaningful way.
I suppose there are already many articles showing how to speed calculations by avoiding/optimizing pandas.
It does feel a little unfair a comparison. Everyone knows that for loops are slow in python.. as is much of the core library. But pushing analysis to c using pythonic APIs (numpy/numba/pytorch) is fairly trivial
So my question to the non bioinformatics - is this already a solved problem?
You have tasks which require resources based on the input parameters, these are run in docker containers to ensure the environment and you want to track the output of each step.
Often these are embarrassingly parallel operations (e.g. I have 200 samples to do the same thing on).
Something like dask perhaps,but can specify a docker image for the task?
What is the goto in DevOps for similar tasks? GitHub actions comes pretty close...
To bioinformatics what is the unique selling point of next flow over say wdl/Cromwell?
>2017/01/31 23:00-ish
YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com
>2017/01/31 23:27 YP - terminates the removal, but it’s too late. Of around 310 GB only about 4.5 GB is left