HackerTrans
TopNewTrendsCommentsPastAskShowJobs

bede

no profile record

Submissions

Removing newlines in FASTA file increases ZSTD compression ratio by 10x

log.bede.im
279 points·by bede·10 माह पहले·113 comments

comments

bede
·3 माह पहले·discuss
I too would far rather bear a token cost than have my sessions rot silently beneath my feet. I usually have ~5 running CC sessions, some of which I may leave for a week or two of inactivity at a time.
bede
·3 माह पहले·discuss
Thanks for publicising. I recently decided not to renew my Backblaze in favour of 'self hosting' encrypted backups outside the US. But I was horrified to learn that my git repos may not have been backed up, nor my Dropbox, whose subscription I also recently cancelled. Good riddance.

My experience using restic has been excellent so far, snapshots take 5 mins rather than 30 mins with backblaze's Mac client. I just hope I can trust it…
bede
·4 माह पहले·discuss
> Probably obvious for many, but I didn't realize ACs don't transport any air into the room, but just moves it around

I had the same epiphany as you days after acquiring a CO2 monitor. Most people notice poor indoor air quality from proxies such as humidity and temperature. AC (without ventilation) eliminates these and tricks our senses very effectively, giving us cool and fresh feeling indoor spaces full of CO2 and devoid of oxygen.
bede
·6 माह पहले·discuss
> There’s another distance limit at work here, and that is the speed of light. It takes milliseconds for the signal in your phone to reach the hotel above ground and be handed over to the mobile network.

It takes roughly 100us for light to travel 30km – Can you explain how the speed of light is relevant here?
bede
·9 माह पहले·discuss
For BAM this could be a good place to start: https://www.htslib.org/benchmarks/CRAM.html

Happy to discuss further
bede
·9 माह पहले·discuss
Author of [0] here. Congratulations and well done for resisting. Eager to try it!

Edit: Have you any specific advice for training a fasta compressor beyond that given in e.g. "Using OpenZL" (https://openzl.org/getting-started/using-openzl/)
bede
·10 माह पहले·discuss
A Zstd maintainer clarified this: https://news.ycombinator.com/item?id=45251544

> Ultimately, Zstd is a byte-oriented compressor that doesn't understand the semantics of the data it compresses
bede
·10 माह पहले·discuss
Fascinating, thank you.
bede
·10 माह पहले·discuss
Thanks for reminding me to benchmark this!
bede
·10 माह पहले·discuss
Yes, when doing anything intensive with lots of sequences it generally makes sense to liberate them from FASTA as early as possible and index them somehow. But as an interchange format FASTA seems quite sticky. I find the pervasiveness of fastq.gz particularly unfortunate with Gzip being as slow as it is.

> Took me a while to realize that Grace Blackwell refers to a person and not an Nvidia chip :)

I even confused myself about this while writing :-)
bede
·10 माह पहले·discuss
BAM format is widely used but assemblies still tend to be generated and exchanged in FASTA text. BAM is quite a big spec and I think it's fair to say that none of the simpler binary equivalents to FASTA and FASTQ have caught on yet (XKCD competing standards etc.)

e.g. https://github.com/ArcInstitute/binseq
bede
·10 माह पहले·discuss
Thank you for clarifying this – yes the non-semantic nature of these particular line breaks is a key detail I omitted.
bede
·10 माह पहले·discuss
Exactly. The line breaks break the runs of otherwise identical bits in identical sequences. Unless two identical subsequences are exactly in phase with respect to their line breaks, the hashes used for long range matching are different for otherwise identical subsequences.
bede
·10 माह पहले·discuss
Yes I'd expect a dict-based approach to do better here. That's probably how it should be done. But --long is compelling for me because using it requires almost no effort, it's still very fast, and yet it can dramatically improve compression ratio.
bede
·10 माह पहले·discuss
Yes, it sounds like 7-Zip/LZMA can do this using custom filters, among other more exotic (and slow) statistical compression approaches.