HackerTrans
トップ新着トレンドコメント過去質問紹介求人

pauldix

no profile record

投稿

Duck Hunt: Moving Bauplan from DuckDB to DataFusion

bauplanlabs.com
17 ポイント·投稿者 pauldix·8 か月前·1 コメント

コメント

pauldix
·8 か月前·議論
I believe you could do this effectively with COBS (COmpact Bit Sliced signature index): https://panthema.net/2019/1008-COBS-A-Compact-Bit-Sliced-Sig...

It's a pretty neat algorithm from a paper in 2019 for the application "to index k-mers of DNA samples or q-grams from text documents". You can take a collection of bloom filters built for documents and then combine them together to have a single filter that will tell you which docs it maps to. Like an inverted index meets a bloom filter.

I'm using it in a totally different domain for an upcoming release in InfluxDB (time series database).

There's also code online here: https://github.com/bingmann/cobs
pauldix
·10 か月前·議論
I've been following this team's work for a while and what they're doing is super interesting. The file format they created and put into the LF, Vortex, is very welcome innovation in the space: https://github.com/vortex-data/vortex

I'm excited to start doing some experimentation with Vortex to see how it can improve our products.

Great stuff, congrats to Will and team!
pauldix
·昨年·議論
Second this one, it's incredible in the level of research and detail.
pauldix
·昨年·議論
Great Fortune, a book about the making of Rockefeller Center is one of my favorites.