HackerTrans
TopNewTrendsCommentsPastAskShowJobs

pauldix

no profile record

Submissions

Duck Hunt: Moving Bauplan from DuckDB to DataFusion

bauplanlabs.com
17 points·by pauldix·8 ay önce·1 comments

comments

pauldix
·8 ay önce·discuss
I believe you could do this effectively with COBS (COmpact Bit Sliced signature index): https://panthema.net/2019/1008-COBS-A-Compact-Bit-Sliced-Sig...

It's a pretty neat algorithm from a paper in 2019 for the application "to index k-mers of DNA samples or q-grams from text documents". You can take a collection of bloom filters built for documents and then combine them together to have a single filter that will tell you which docs it maps to. Like an inverted index meets a bloom filter.

I'm using it in a totally different domain for an upcoming release in InfluxDB (time series database).

There's also code online here: https://github.com/bingmann/cobs
pauldix
·10 ay önce·discuss
I've been following this team's work for a while and what they're doing is super interesting. The file format they created and put into the LF, Vortex, is very welcome innovation in the space: https://github.com/vortex-data/vortex

I'm excited to start doing some experimentation with Vortex to see how it can improve our products.

Great stuff, congrats to Will and team!
pauldix
·geçen yıl·discuss
Second this one, it's incredible in the level of research and detail.
pauldix
·geçen yıl·discuss
Great Fortune, a book about the making of Rockefeller Center is one of my favorites.