HackerTrans
トップ新着トレンドコメント過去質問紹介求人

hashjoiner

no profile record

投稿

Building a 32 core Kubernetes cluster for less than $1k

andygrove.medium.com
1 ポイント·投稿者 hashjoiner·5 年前·0 コメント

Increasing the level of parallelism in DataFusion 4.0

medium.com
1 ポイント·投稿者 hashjoiner·5 年前·0 コメント

Making joins faster in DataFusion based on table statistics

godatadriven.com
4 ポイント·投稿者 hashjoiner·6 年前·0 コメント

コメント

hashjoiner
·2 年前·議論
Very interesting development :)
hashjoiner
·5 年前·議論
DataFusion committee here.

DuckDB can work on Arrow data, so I think it could coexist with DataFusion / Ballista quite well. I am not sure whether having different execution engines for Ballista is on the roadmap, but it's certainly a possibility!
hashjoiner
·5 年前·議論
Yes, that's pretty exciting! There is even support in Polars to execute the dataframe compute in DataFusion (as it can handle larger than memory datasets).
hashjoiner
·5 年前·議論
There is a PR from me (Daniël, committer) with for db-benchmark. For the group by benchmarks, on my machine, it is currently somewhat slower than the fastest (Polars).

https://github.com/h2oai/db-benchmark/pull/182

Also we do support running TPC-H benchmarks. For the queries we can run, those are already finishing faster than Spark. We are planning to do more benchmarking and optimizations in the future.
hashjoiner
·5 年前·議論
Happy for this release, and excited to contribute to this great query engine.