New, better compression for columns of numerical data(github.com)

New, better compression for columns of numerical data

https://github.com/mwlon/quantile-compression

1 comments

mwlon5 lat temu

I made this open source compression algorithm as part of a database I'm creating. It typically compresses columns of numerical data to ~25% smaller than alternatives (think .snappy.parquet or .gzip.parquet) at similar or cheaper compute cost. It decompresses 15-100 million 64-bit numbers per second on a single i5 CPU per second.

I also made a blog post that introduces the idea more from the math perspective: https://graphallthethings.com/posts/quantile-compression