HackerTrans
TopNewTrendsCommentsPastAskShowJobs

_willmanning

8 karmajoined 7 ปีที่แล้ว

comments

_willmanning
·7 วันที่ผ่านมา·discuss
it would have been easier to just go to FNAC and buy Rocket League like a normal person :)
_willmanning
·11 เดือนที่ผ่านมา·discuss
Vs Apache Parquet: 100x faster random access, 10-20x faster scans, 5x faster writes, similar compression ratio

And nearly as fast as duckdb's native format on Clickbench when queried from duckdb
_willmanning
·12 เดือนที่ผ่านมา·discuss
I'll chime in (as a Vortex maintainer), that we are greatly indebted to Azim's work on FastLanes & ALP. Vortex heavily utilizes his work to get state-of-the-art performance.

I would add that Vortex doesn't have standalone Clickbench results. Azim is presumably referring to the duckdb-vortex results, which were run on an older version of duckdb (1.2) than the duckdb-parquet ones (1.3). We'll get those updated shortly; we just released a new version of Vortex & the duckdb extension. Meanwhile, I believe the DataFusion-Vortex vs DataFusion-Parquet speedups show substantial improvements across the board.

The folks over at TUM (who originally authored BtrBlocks) did a reasonable amount of micro-benchmarking of Vortex vs Parquet in their recent "Anyblox" paper for VLDB 2025: https://gienieczko.com/anyblox-paper

They essentially say in the paper that Vortex is much faster than the original BtrBlocks because it uses better encodings (specifically citing FastLanes & ALP).

I'm looking forward to seeing the FastLanes Clickbench results when they're ready, and Azim, we should work together to benchmark FastLanes against Vortex!
_willmanning
·2 ปีที่แล้ว·discuss
Compression! Vortex can easily be 10x smaller than the equivalent Arrow representation (and decompresses very quickly into Arrow)
_willmanning
·2 ปีที่แล้ว·discuss
Perhaps that verbiage is just confusing. "On-disk" sort of implies "file format" but could be more explicit.

That said, the immediate next line in the README perhaps clarifies a bit?

"Vortex is designed to be to columnar file formats what Apache DataFusion is to query engines (or, analogously, what LLVM + Clang are to compilers): a highly extensible & extremely fast framework for building a modern columnar file format, with a state-of-the-art, "batteries included" reference implementation."