hashjoiner

2 tahun yang laludiscuss

Very interesting development :)

5 tahun yang laludiscuss

DataFusion committee here.

DuckDB can work on Arrow data, so I think it could coexist with DataFusion / Ballista quite well. I am not sure whether having different execution engines for Ballista is on the roadmap, but it's certainly a possibility!

hashjoiner

5 tahun yang laludiscuss

Yes, that's pretty exciting! There is even support in Polars to execute the dataframe compute in DataFusion (as it can handle larger than memory datasets).

hashjoiner

5 tahun yang laludiscuss

There is a PR from me (Daniël, committer) with for db-benchmark. For the group by benchmarks, on my machine, it is currently somewhat slower than the fastest (Polars).

https://github.com/h2oai/db-benchmark/pull/182

Also we do support running TPC-H benchmarks. For the queries we can run, those are already finishing faster than Spark. We are planning to do more benchmarking and optimizations in the future.

hashjoiner

5 tahun yang laludiscuss

Happy for this release, and excited to contribute to this great query engine.

hashjoiner

Submissions

Building a 32 core Kubernetes cluster for less than $1k

Increasing the level of parallelism in DataFusion 4.0

comments