HackerTrans
TopNewTrendsCommentsPastAskShowJobs

hashjoiner

no profile record

Submissions

Building a 32 core Kubernetes cluster for less than $1k

andygrove.medium.com
1 points·by hashjoiner·vor 5 Jahren·0 comments

Increasing the level of parallelism in DataFusion 4.0

medium.com
1 points·by hashjoiner·vor 5 Jahren·0 comments

Making joins faster in DataFusion based on table statistics

godatadriven.com
4 points·by hashjoiner·vor 6 Jahren·0 comments

comments

hashjoiner
·vor 2 Jahren·discuss
Very interesting development :)
hashjoiner
·vor 5 Jahren·discuss
DataFusion committee here.

DuckDB can work on Arrow data, so I think it could coexist with DataFusion / Ballista quite well. I am not sure whether having different execution engines for Ballista is on the roadmap, but it's certainly a possibility!
hashjoiner
·vor 5 Jahren·discuss
Yes, that's pretty exciting! There is even support in Polars to execute the dataframe compute in DataFusion (as it can handle larger than memory datasets).
hashjoiner
·vor 5 Jahren·discuss
There is a PR from me (Daniël, committer) with for db-benchmark. For the group by benchmarks, on my machine, it is currently somewhat slower than the fastest (Polars).

https://github.com/h2oai/db-benchmark/pull/182

Also we do support running TPC-H benchmarks. For the queries we can run, those are already finishing faster than Spark. We are planning to do more benchmarking and optimizations in the future.
hashjoiner
·vor 5 Jahren·discuss
Happy for this release, and excited to contribute to this great query engine.