>> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce.
Is this the scientific version of "rich people problems"?
> Is this the scientific version of "rich people problems"?
Author here. Yes, most certainly. In fact, it was one of the things that drew me towards the NIH for my PhD. My overall point in the post was to show that a somewhat naive Python implementation and a much faster Nim version have a small Levenshtein distance. For many people in bioinformatics who don't have a background in software engineering (that would be a significant fraction, if not a majority of them), this could be a huge boon. Combined with the fact that most bioinformatics researchers don't have the privilege of the world's largest biomedical HPC cluster at their disposal, I still think Nim would be great drop-in replacement for quick single-threaded line-oriented string processing. For numerical stuff, probably not.
However, I am mostly writing in Rust these days for longer term projects that require threading and good ecosystem support. Perhaps I'll write a follow-up retrospective on Rust versus Nim in this area.
Author here. I love the type system. By using distinct strings to represent DNA, RNA, and protein, I can avoid silly errors while still using the optimized implementations under the hood. This is what the `bioseq` library (about two hundred lines) does [1] and I find it incredibly elegant.
Author here. This is basically the approach I am using these days to get maximum multithreaded performance for when it really counts (inner loops) [1]. I draft in Python and use Copilot to convert it to Rust, then optimize from there. However, Nim is still better than Rust in my opinion for simple scripts that I don't want to spend a bunch of time writing. Its only major downside is its relative lack of support for parallelism and bioinformatics (i.e., why I used Rust for a more serious project).
Not the author of this paper but am current PhD student focused on viroid discovery. There's no TEM but there are good methods such as RNAfold [0] for predicting their structures. In the case of rod-shaped RNAs, the prediction methods are quite good since it basically comes down to looking for substrings of reverse complement sequences within the circular RNA.
This looks excellent for my use case, which is visualizing viroid and viroid-like circular genomes. Right now, I reluctantly use the R version but am eager to try this out. I spent countless hours making this figure [1] for my last paper and am quite confident that would have saved me a ton of time.
If you get a chance, please do publish this package in something like the Journal of Open Source Software so that I can cite it. Thanks for sharing this tool!
I’m a huge fan of the moonwatch both for its beauty and history with the space program. Ever since I was 10 I’ve wanted one. However, I’m a PhD student and there’s no way I can afford to get a “real” one. The Swatch release is really attractive to me since it captures the spirit of the watch while making it accessible.
For reference, there’s a very popular variant of the Speedmaster Professional that uses a sapphire crystal rather than a hesalite (basically plastic) crystal. Despite never having been used in space (since the shattering crystal is a risk) the sapphire model is still considered a “professional” edition and highly sought after. People who wear it enjoy the aesthetic of the spacefaring version with the earthly practicality of scratch-proof sapphire. The Swatch version’s desirability is just the same logic a taken a few more steps. It’s made by Omega, has the same basic design, and evokes the imagery of the space race.
Alternatively, consider the popularity of the Tesla toy car for kids. It’s not anything like the real Tesla in terms of functionality but is still a cool electric vehicle, especially if you already have a Tesla.
I followed this plan and lost over 100 lbs (45 kg) in about a year and have kept it off for the last five years. Even more impressively, I did it while eating the very same junk food that made me fat, albeit in different quantities. I tried every diet you could imagine before reading the book and failed every single time. As a programmer, the approach Walker uses just clicks with me. If you're overweight and have an engineering mindset, it's absolutely worth the read. It changed my life.
I still follow the plan to this day, albeit with more sophisticated logging. I use a combinatorial optimizer web app that tells me what to eat every day so that it completely takes the element of choice (and ability to screw up) out of the equation. I've developed it for the last five years and am hoping to release it as a product and/or open source eventually. If anyone's interested, shoot me an email (link on my website) and I'll share access.
Ah that makes sense. I agree with you; I’m not a huge fan of trying to infer where the types came from myself either when reading code on GitHub since it doesn’t have the inference that my IDE does.
There is some logic as to why that is. Here [1] is an explanation for why it makes sense but the tldr is that you don't want to be manually importing functions such as `$` and `+`. In languages like Python, those are defined as methods on the object being imported (e.g. `.__str__()`) so they come along for free. Not so in Nim. If there's a conflict (same name, same signature), the compiler will warn you but it's extremely rare.
You hit the nail on the head. This is just the lexicographically minimal string rotation with a canonicalization step. I have actually had this on my to-do list for a while. The truth is that the data set I'm applying this to is small (though I'm doing my best to change that by discovering new viroid-like agents) so the optimization has yet to become pressing. At some point, I'd like to spin this script out into a standalone tool but I'm not sure the demand is there yet.
Author here. This is spot on. The majority of the code I write is either piping data around to existing tools using shell scripting and Snakemake or writing the data processing code myself when there isn't a tool that does what I need. Usually, I'm working alone or with a few other computational biologists. Many of my scripts are one-off but they have the distinct tendency of growing in complexity and scope if they are useful. That's one of the big advantages with Nim in my mind: you can write a quick and dirty script and have it be pretty fast and then go back later and optimize it to a few percent of C without having to rewrite your code in another language. In this sense, it's quite like Julia (another really good language).
I'm sorry, I completely forgot that the file I used was from six months ago when I wrote the blog post (and then promptly forgot to publish it). In the last half year, the number of coronavirus sequences has increased dramatically. One thing that you could do to drop the file size down is to filter for only complete and unambiguous sequences, which drops the number down from 1.6 million to ~100k [1].
Alternatively, the exact file I used for the post is available for one week here with MD5 sum 3c33c3c4c2610f650c779291668450c9 [2]. Anyone who wants the file is free to reach out to me directly (email is on site).
I didn't post it because it's quite big (150M) but readily available from the NCBI Virus portal [1]. I would love to see how well other languages compete both for speed and simplicity.
I actually use TypeScript/JavaScript a lot for this reason, especially for biological algorithms that I want to run in the browser. The developer tooling is also as good as you can hope for, especially when using VS Code. I actually wrote a circular RNA sequence deduplication algorithm in it just recently [1].
With respect to the identifier resolution in Nim, it strikes me as more of a matter of preference. Especially given the universal function call syntax in Nim, at least it's consistent. For example, Nim treats "ATGCA".lowerCase() the same as lowercase("ATGCA"). I do appreciate the fact that you can use a chaining syntax instead of a nesting one when doing multiple function calls but this is also a matter of style more than substance.
This is because there's four bases and each can either be included or excluded from a given combination. So there are 4*2 = 16 combinations each of which with their own letter. In all honesty, these are pretty rarely used in practice these days except for N (any base) although they do sometimes show up when representing consensus sequences.
You make a fair point that using optimized numerical libraries instead of string methods will be ridiculously fast because they're compiled anyway. For example, scikit-bio does just this for their reverse complement operation [1]. However, they use an 8 bit representation since they need to be able to represent the extended IUPAC notation for ambiguous bases, which includes things like the character N for "aNy" nucleotide [2]. One could get creative with a 4 bit encoding and still end up saving space (assuming you don't care about the distinction between upper versus lowercase characters in your sequence [3]). Or, if you know in advance your sequence is unambiguous (unlikely in DNA sequencing-derived data) you could use the 2 bit encoding. When dealing with short nucleotide sequences, another approach is to encode the sequence as an integer. I would love to see a library—Python, Nim, or otherwise—that made using the most efficient encoding for a sequence transparent to the developer.
> Is this the scientific version of "rich people problems"?
Author here. Yes, most certainly. In fact, it was one of the things that drew me towards the NIH for my PhD. My overall point in the post was to show that a somewhat naive Python implementation and a much faster Nim version have a small Levenshtein distance. For many people in bioinformatics who don't have a background in software engineering (that would be a significant fraction, if not a majority of them), this could be a huge boon. Combined with the fact that most bioinformatics researchers don't have the privilege of the world's largest biomedical HPC cluster at their disposal, I still think Nim would be great drop-in replacement for quick single-threaded line-oriented string processing. For numerical stuff, probably not.
However, I am mostly writing in Rust these days for longer term projects that require threading and good ecosystem support. Perhaps I'll write a follow-up retrospective on Rust versus Nim in this area.