Ah I understand what you are saying. Not trying to aggressively make you try Fugue out, I appreciate the comments and just want to clarify some stuff.
There is no transpiling, we think that is too magical. It's more of routing to the appropriate functions/method. For example, applying the Pandas function per partition. We use Pandas UDF under the hood when it makes sense. Sometimes, we use map partitions as well, it depends on what the user wants to do. So we simplify usage around these methods with minimal overhead (we benchmarked it).
Fugue can be adopted as minimally as needed so if you feel the need to write native Spark code or tune Spark, we don't block that from the user. The older Fugue interface was bad in allowing this, so we reworked Fugue to have a suite of standalone functions compatible with Spark/Dask/Ray/Pandas DataFrames.
Our opinion (which is totally fine if you disagree), is that most workloads don't really specific features of Spark. There are times when it makes sense to write native Spark code for sure, and for those Fugue won't be a good fit.
Our job is to make sure it's not an all or nothing thing that requires the user to make compromises.
Hi whinvik, we agree that development in Spark is hard, and that is part of the motivation of Fugue. Spark code couples the distributed orchestration and business logic together.
By keeping your code in native Python or Pandas, it will be much easier to develop, debug, and maintain the business logic because your tracebacks will be in native Python. Fugue then takes it to Spark when you are ready to scale.
Hi antman, thanks for the question. I will type some points on differences, but will answer the first question. The Fugue -> Ibis -> DuckDB example is a bit weird. Yes it can be done but it's not practical (as you can tell). There may be some overlap sometime, but I do think the projects differ in scope (more below).
1. We guarantee consistency between backends. NULL handling can be different depending on the backend. For example, Pandas joins NULL with NULL while Spark doesn't. So if you prototype locally on Pandas, and then scale to Spark, we guarantee same results. Fugue is 100% unit tested and the backends go through the same test suite.
2. Ibis is Pythonic for SQL backends. We embrace SQL, but understand its limitations. FugueSQL is an enhanced SQL dialect that can invoke Python code. FugueSQL can be the first-class grammar instead of being sandwiched by Python code. Fugue's Python API and SQL API are 1:1 in capability.
3. Opinionated here, but we don't want users to learn any new language. Ibis is a new way to express things; we just want to extend the capabilities of what people already know (SQL, native Python, and Pandas). Fugue can also be incrementally adopted, meaning it can be used for just one portion of your workflow.
4. Roadmap-wise, we think the optimal solutions will be a mix of different tools. A clear one is pre-aggregating data with DuckDB, and then using Pandas for further processing. Similarly, can we preprocess in Snowflake and do machine learning in Spark? Fugue is working on connecting these different systems to enable cross-platform workloads.
Hi AndrewKemendo, Fugue co-author here. Thanks for the kind words! We do put a lot of effort into our documentation. Always happy to chat potential use cases if you want. My contact info is in my profile.
I am one of the contributors of Fugue. Fugue is an open-source abstraction layer that ports Python/Pandas/SQL code to Spark or Dask. This article covers the programming interface and benefits Fugue provides, specifically:
* Handling inconsistent behavior between different compute frameworks (Pandas, Spark, and Dask)
* Allowing reusability of code across Pandas-sized and Spark-sized data
* Dramatically speeding up testing and and iteration cycles
* Enabling new users to be productive with Spark much faster
* Providing a SQL interface capable of handling end-to-end workflows
There was a previous post on here about our SQL interface that lets you use SQL on top of Pandas, Spark and Dask. This post talks about the broader project.
https://news.ycombinator.com/item?id=28830243
Hello, FugueSQL also supports SQL being the dominant language when describing computation. Current SQL interfaces are normally invoked between predominantly Python code. FugueSQL was created to support the other way around with SQL invoking Python code.
For example, with a python function
```python
# schema: , c:int
def add_new_col(df:pd.DataFrame) -> pd.DataFrame:
df['c'] = 1
return df
query = """SELECT FROM df TRANSFORM USING add_new_col"""