Relational Operators in Apache Calcite (2021)(querifylabs.com)
querifylabs.com
Relational Operators in Apache Calcite (2021)
https://www.querifylabs.com/blog/relational-operators-in-apache-calcite
15 comments
That's cool, thanks for sharing! Do you know how close they are to their example use cases [1]? So far I've only been able to find a tool for calcite SQL parsing [2] but not the portion connecting to Arrow C++ compute kernel yet.
[1]: https://substrait.io/#example-use-cases [2]: https://substrait.io/tools/producer_tools/
[1]: https://substrait.io/#example-use-cases [2]: https://substrait.io/tools/producer_tools/
I'd check out the Slack, which is where I've seen a few projects integrating it coordinating.
I think DuckDB is one of the projects that has the best support for executing Substrait query plans. I think for most other projects there are some forks with substrait support (e.g. datafusion) but nothing merged to upstream yet.
I'm not sure if there are any systems where it is integrated and yields tangible benefits yet (though there is decent progress on common tooling, so that shouldn't be too far in the future).
I think DuckDB is one of the projects that has the best support for executing Substrait query plans. I think for most other projects there are some forks with substrait support (e.g. datafusion) but nothing merged to upstream yet.
I'm not sure if there are any systems where it is integrated and yields tangible benefits yet (though there is decent progress on common tooling, so that shouldn't be too far in the future).
I recently had to parse source code of views to produce a mapping between the view columns and the underlying table columns - it basically worked with some caveats and was pretty painful to do (most databases expose this as information schema/metadata but that was not an option for us for various reasons).
For anyone who is knowledgable about Calcite features, is this something that Calcite can produce without having to write a ton of code (which is what my current solution is)? I would find it very satisfying to replace hundreds of lines of bespoke code with a few library calls.
For anyone who is knowledgable about Calcite features, is this something that Calcite can produce without having to write a ton of code (which is what my current solution is)? I would find it very satisfying to replace hundreds of lines of bespoke code with a few library calls.
This should be pretty easily doable in Calcite if you have access to the query which defines the view. Calcite has a pretty rich metadata system. Take a look at RelMetadataQuery.getColumnOrigin.
I once heard that Apache is the graveyard for open source software.
Is that really the case?
Is that really the case?
Yes and no.
A lot of projects in Apache control started as one (big) company project, got open sourced then move to Apache, in order to allow collaboration with a "community" without as much conflict of interest.
But then the original sponsor also usually lose interest and/or no community get creates. Hence, graveyard.
Or it got opensource as last ditch try to keep it running after the sponsor stopped wanting it. And then it is usually a graveyard move.
It is not that Apache kills them. It is that they were already dead and Apache is the place that take them in to accompany their demise.
A lot of projects in Apache control started as one (big) company project, got open sourced then move to Apache, in order to allow collaboration with a "community" without as much conflict of interest.
But then the original sponsor also usually lose interest and/or no community get creates. Hence, graveyard.
Or it got opensource as last ditch try to keep it running after the sponsor stopped wanting it. And then it is usually a graveyard move.
It is not that Apache kills them. It is that they were already dead and Apache is the place that take them in to accompany their demise.
Calcite is relatively unknown, but it is the base so many other projects are built on. Among them is Apache Flink, which is quite alive and top of its category (low latency distributed stream processing)
https://calcite.apache.org/docs/powered_by.html
https://calcite.apache.org/docs/powered_by.html
It depends. There are a large number of dead projects. There are also a large number of libraries and tools that are active and widely depended on, but you may not have heard of them.
Then there are quite a few large projects donated by corps with active communities. See this page
https://projects.apache.org/statistics.html
and scroll down to "Largest/Busiest repositories". There's a list of widely used enterprise software in there which are definitely not dead.
Then there are quite a few large projects donated by corps with active communities. See this page
https://projects.apache.org/statistics.html
and scroll down to "Largest/Busiest repositories". There's a list of widely used enterprise software in there which are definitely not dead.
It is and it isn't. There's some good stuff there, but there's ever the risk that you'll get bushwhacked by something that you need to pay an expensive consultancy -- who just so happens to be the team behind the product -- to fix.
Though I feel bad for the person who decides to peruse Apache's list of projects and decide what technology to use based solely on the high-level overviews. Good luck picking the wheat from the chaff!
Though I feel bad for the person who decides to peruse Apache's list of projects and decide what technology to use based solely on the high-level overviews. Good luck picking the wheat from the chaff!
I'll just note that the risk mentioned certainly does not apply to Apache Calcite. (Disclaimer: I am a member of the Apache Calcite PMC, but not terribly active at the moment.)
A lot of open-source Apache projects were backed by commercial companies, and clouds (primarly AWS) forked those projects competing directly with the companies backing the Apache projects. The end result was that the commercial companies mostly released new versions of the software with more restrictive licenses that could not be sold for free by the cloud companies.
That doesn't make Apache software a graveyard. It just makes that go-to-market open-source model leveraging Apache's brand a dead-end.
That doesn't make Apache software a graveyard. It just makes that go-to-market open-source model leveraging Apache's brand a dead-end.
It's certainly not the case for Apache Calcite. Calcite is pretty widely used by a number of companies and there are a number of highly active developers.
No
to me it feels the same, plenty of great projects that essentially died once they got into apache. It has that java feeling.
[0]: https://substrait.io