Numerical data is full of rich patterns, but the general-purpose compressors we've historically used on them (e.g. snappy, gzip, zstd) are designed for unstructured, string-like data. Pcodec (or pco) is a new approach for numerical sequences that gets better compression ratio and decompression speed than alternatives. It usually improves compression ratio substantially, given the same compression time. Plus it's built to perform on all common CPU architectures, decompressing around 1-4GB/s.
You might have seen me post about Quantile Compression in previous years. Pco is its successor! Pco gets slightly better compression ratio, robustly handles more types of data, and (most importantly) decompresses much faster.
If you're interested in using it, there's a Rust API, Python (PyO3) API, and a CLI.
You might have seen me post about Quantile Compression in previous years. Pco is its successor! Pco gets slightly better compression ratio, robustly handles more types of data, and (most importantly) decompresses much faster.
If you're interested in using it, there's a Rust API, Python (PyO3) API, and a CLI.