>To disagree, I'm a computational biologist and it's my firm belief 99% of the scientifically important stuff happens before the stats and plotting.
I'm a microbiologist too, but the kind that uses mostly off-the-shelf tools to do the taxonomic/functional assignment on metagenomes, and then stats/data science on the features. I kinda didn't know what you mean by "99% of the scientifically important stuff happens before the stats and the plotting".
I mean, give me a 500x2.6x10^6 sparse matrix of gene function abundances and tell me that you've done anything scientifically meaningful. Or on the other side, let me hand you a fastq file from sequencing a poorly extracted DNA sample, and you give me the best algorithm in the world, and there's nothing scientifically meaningful that's going to come out of that.
>TL;DR: wrote a benchmark article without...running it?
He benchmarks against the rust implementation, which, unless benchmarks have zero meaning, should be sufficient to get a general sense of the scale of the difference. The post is obviously not meant as the last word on this benchmark, it's meant to show that the benchmark is kinda meaningless.
>Then you conclude with "the language I use is faster!!!"
If this is your take-home from the post, it's pretty clear you didn't read it, or your reading comprehension needs some work. That sentence was obviously facetious, poking a little fun at the author of the original piece.
>However, in my opinion, this lack of validation (really, lack of parsing in any meaningful sense) means that the performance between this parser and Needletail is incomparable. So what exactly does it demonstrate? You get to claim your implementation is faster than someone else if you do the same task in less time, but not if you skip half the job.
Yeah, this makes the original post just incredibly misleading. Sure, meaningful benchmarks are hard, but before you go writing "woah, look how much faster we are!", a bit more due dilligence should be required.
Calling julia code from python or python code from julia is relatively straightforward. I haven't spent much time doing the former, but love the ergonomics of the later with PythonCall.jl and CondaPkg.jl to manage python dependencies.
It really depends on what you do - sometimes offsets are better, sometimes an index is better. If you really want 0-based indexing, you can have it! https://github.com/JuliaArrays/OffsetArrays.jl
>Julia does solve the two language issue, but not in all situations
And not instantaneously. Developer time being finite, using what works from existing libraries while waiting for a pure-julia implementation seems like an excellent solution.
Who am I to say what hills are worth dying on? But I don't want you to die at all - in case you're ever compelled to switch, I can't help you with end, but OffsetArrays.jl can help with your indexing woes, and I think there are packages to rotate matrices too (though not certain of that).
> If you take a functional approach, especially using pipes, dplyr and a split, apply, combine style, it is quite beautiful
Sure, but what if you don't? Sometimes, this is the right way to do things, other times there are other approaches that are more natural/beautiful. In many cases, a loop with conditionals is much easier to understand.
Out of curiosity, when was the last time you looked at DataFrames.jl? A huge amount has happened in the last year. Plus, if you want more tidy-like syntax, you can go with Query.jl, (or DataFramesMeta.jl, though that isn't quite finished updating to the the new DataFrames syntax), or of you just want pipes on DataFrame operations, there's Pipe.jl and Chain.jl.
I don't think your comments are harsh, you need what you need and you like what you like. I do mostly data wrangling too, but feel much less constrained with Julia than with tidyr. Sometimes having constraints and one right way to do things is good, but it's not for me.
Also worth noting it's not necessarily on the language developers to do this. Even in R, tidyverse is in packages, not in the base language.
Style guide for the base language is to generally stick to ascii, and many of the big packages do that too. It's really nice in examples, analysis scripts and the like though.
% of the repos in the org on github? That number is lower than I'd like. % of the repos that are actively maintained? Much higher.
One of the great things about julia is that it's really easy to throw together a package and register it. One of the bad things about julia is how easy it is for those one-off projects or idea dumps to pollute the space. We could definitely do a better job labeling the repos that are no longer being maintained or that aren't actually ready for prime time. There's a tradition in julia of a lot of really functional libraries to stay < v1.0, because we all take semver seriously, and if the interface is still in a bit of flux, making the switch to 1.0 is a big deal (DataFrames.jl, looking at you). But it does make it hard for new users to distinguish between a super robust package and someone's weekend hobby.
Depending on which plotting package you use. Plots.jl itself is pretty slow to get started still, though there are ideas on how to improve that.
That said, once you're up and running, plots (and everything else) are super snappy. When I'm doing plotting stuff, I'm usually doing it interactively, and when I'm scripting it, it's because I'm plotting hundreds or thousands of things (and then the startup time is vanishingly small).
I'm a microbiologist too, but the kind that uses mostly off-the-shelf tools to do the taxonomic/functional assignment on metagenomes, and then stats/data science on the features. I kinda didn't know what you mean by "99% of the scientifically important stuff happens before the stats and the plotting".
I mean, give me a 500x2.6x10^6 sparse matrix of gene function abundances and tell me that you've done anything scientifically meaningful. Or on the other side, let me hand you a fastq file from sequencing a poorly extracted DNA sample, and you give me the best algorithm in the world, and there's nothing scientifically meaningful that's going to come out of that.