I benchmarked DuckDB 1.5.2 with the latest Java JDBC driver which now supports user defined functions. This allows very fast modifications https://sqg.dev/blog/java-duckdb-benchmark/
Great article, I always like to structure my queries with CTEs and I was (wrongly) assuming it all gets inlined at the end.
Sometimes it also gets complicated since these intermediate results can't be easily seen in a SQL editor. I was working on a UI to parse CTE queries and then execute them step by step to show the results of all the CTEs for easier understanding of the query (as part of this project https://github.com/sqg-dev/sqg/)
17 years ago we launched the first "Chuck Norris Facts" app for Android (March 2009). It was a big success until end of 2010 when Chuck Norris sent his lawyers after us to get the app removed from the Android market. Chuck Norris won, we took the app down
Great! I will give it a try. I found that using DuckDB to select from the parquet files and using the Apache Arrow API to read the results is also a very fast method
This is great! It would be a cool feature to integrate with SQG. there you currently need to provide the sequence of migration steps to get the full schema.
To get fast access to the query results we use the Apache Arrow interface and generate the code directly from DuckDB SQL queries using the SQG tool ( https://sqg.dev/generators/java-duckdb-arrow/)
I agree this happens quite often. Since Highflux automatically commits changes to the current branch it is extremely easy to switch to a different branch with just one click (or CLI command). (so no stashing / dealing with untracked files is needed)
Then you can implement the side change separately and then switch back again to your first feature branch.
Those two feature branches are then unrelated, we are planning to add stacked changes in a future version.
To me it is most important that the commit is focused only on one thing, and not mix any unrelated changes together.
We develop a Rust based tool HighFlux [1] which simplifies/automates working with git.
In this workflow you create a feature branch for every feature you work on and it automatically commits all changes to this branch.
After code review the branch gets merged as a squashed merge to have it in a single commit in the history.