Yes we do. We try to leverage Arrow as much as we can. And different from many other computing frameworks, Fugue doesn't invent new data types, it just uses Arrow data types.
Hello whinvik, I totally agree the debugging experience of Spark is suboptimal.
But why?
I don't think the main reason is too much 'magic', I think it is because of the coupling of user's logic and the complicated backend logic. When there is an issue, not only you have to wait for a long time to get the error back, but also you get the error of the whole execution chain, and you have to find the real error in it, which is challenging even for experienced Spark users.
Not only Spark, if you use Dask or Ray, you will find very similar experience. Can we directly make Spark Dask Ray easier to debug?
I doubt. This is because the complexity of these systems are necessary. And since the problem can happen anywhere either on the user side or on the backend side, the whole stack has to be investigated. It is inevitable, otherwise, people will start to complain the real errors are not captured by the system.
What can we do about it?
Fugue's idea is to let you iterate on your own logic locally before using these complex systems. So you can get quicker and less noisy feedbacks. The challenge is how to unify the semantic so you don't need to change code moving between local and distributed. And how to trim down the system errors so you can focus more on your own problems. These are what Fugue solves.
It's all about good practices
It's common sense that if you can keep your logic agnostic to backends, and if you can iterate locally on small data, the development experience can be a lot better. But why big data developers tend not to follow these practices? Because there is too much discrepancy created by the distributed backends intentionally or unintentionally. We created such abstraction for big data developers to get more rewards than punishment when following good practices, so they are more willing to follow good practices.
Fugue SQL extends standard SQL so you can use SQL-ish language end to end.
For example, let's assume you use duckdb or mysql or other standard sql for data processing, how do you deal with intermediate tables?
You may have to frequently use CREATE TABLE and INSERT or write gigantic WITH statement to avoid the hassle. I guess many people use both.
In Fugue SQL, you can use simple assignment to do the same thing (Fugue may translate the assignment to CREATE TABLE or do something more efficient)
-------
a = CONNECT duckdb SELECT * FROM table1
b = LOAD "/tmp/src.parquet"
c = SELECT * FROM a LEFT SEMI JOIN b ON a.key = b.key
OUTPUT c USING mysql_dumper(table_name="temp", mode="append")
d = SELECT * FROM b UNION SELECT * FROM c
SAVE d OVERWRITE "/tmp/tmp.parquet"
--------
Look at the above Fugue SQL example, it's self-explaining, but how much extra code you have to write to achieve the same semantic without Fugue SQL?
Fugue SQL can provide a unified and minimal way to express your logic in SQL-ish way, while the real executions are done but those wonderful database solutions you mentioned.
Fugue SQL is one way, and it also has functional API. They both can be translated into the underlying runtime. You can choose based your preference and real need.
Fugue is a higher level abstraction compared to Ray. It provides unified and non-invasive interfaces for people to use Spark, Dask and Pandas. Ray/Modin is also on our roadmap.
It provides both Python interface (not pandas-like) and Fugue SQL (standard SQL + extra features). Users can choose the one they are most comfortable with as the semantic layer for distributed computing, they are equivalent.
With Fugue, most of your logic will be in simple Python/SQL that is framework and scale agnostic. From the mindset to the code, Fugue minimizes your dependency on any specific computing frameworks including Fugue itself.
Please let me know if you want to learn more. our slack is in the README of the fugue repo