Fugue: A unified interface for distributed computing(github.com)
github.com
Fugue: A unified interface for distributed computing
https://github.com/fugue-project/fugue
24 comments
Hi AndrewKemendo, Fugue co-author here. Thanks for the kind words! We do put a lot of effort into our documentation. Always happy to chat potential use cases if you want. My contact info is in my profile.
I want to cry... why distributed programming in Python?
I mean, I understand that a lot of people want Pandas to run faster, and distributing computation would help... but come on! There needs to be a line in the sand that a sane person would not cross. It doesn't matter how much effort you put into making Python distributed, it's not going to work unless you have the power to control the language itself.
You know, I just thought of a metaphor for this. Maybe not 100%, but you'll get the idea. Suppose you live in a small country, something close to 10M population. And you want to buy pants. Well, you want to buy them from an offline store, so you don't get all the choices you have by shopping online. You don't care about the price, you can afford to buy one pair of pants for a price of ten. You just want a specific kind. Let's say, you want JNCO jeans.
Tough luck. You go to every store that sells pants, but JNCO is just too niche for a small country to make sense for the retailers to import. So, every store that wants to make profits buys the same exact model of jeans that was advertised in the last year fashion catalogue. You can pay a little more or a little less for better or worse material, but all the pants on the offer have the same trashy design. You just cannot stand them.
This is what's happening with Python. It's trash. But it's the model from the last year fashion catalogue. So any program you write, any problem you solve must be in Python, or else you lose the competition before you even enter into the contest.
I mean, I understand that a lot of people want Pandas to run faster, and distributing computation would help... but come on! There needs to be a line in the sand that a sane person would not cross. It doesn't matter how much effort you put into making Python distributed, it's not going to work unless you have the power to control the language itself.
You know, I just thought of a metaphor for this. Maybe not 100%, but you'll get the idea. Suppose you live in a small country, something close to 10M population. And you want to buy pants. Well, you want to buy them from an offline store, so you don't get all the choices you have by shopping online. You don't care about the price, you can afford to buy one pair of pants for a price of ten. You just want a specific kind. Let's say, you want JNCO jeans.
Tough luck. You go to every store that sells pants, but JNCO is just too niche for a small country to make sense for the retailers to import. So, every store that wants to make profits buys the same exact model of jeans that was advertised in the last year fashion catalogue. You can pay a little more or a little less for better or worse material, but all the pants on the offer have the same trashy design. You just cannot stand them.
This is what's happening with Python. It's trash. But it's the model from the last year fashion catalogue. So any program you write, any problem you solve must be in Python, or else you lose the competition before you even enter into the contest.
The language is not a limiting factor here. Python is an excellent scripting language, and works plenty fine in distributed computation. The Python interface to Spark is a wrapper on the underlying Scala API. You don't lose out on performance when you're building up a lazy chain of computation that's executed by an engine written in a more performant language.
Fugue is a layer to abstract out these distributed computation backends, and it looks like a nice programming interface.
Fugue is a layer to abstract out these distributed computation backends, and it looks like a nice programming interface.
Well said! Python can push down to other languages like Rust and C to speed things up. Python can serve as a great end-user interface.
Trying to wrap my head around Fugue and the comment explaining how Python is a good wrapper.
Does Fugue take advantage of each sublayer that already uses Arrow?
Does Fugue take advantage of each sublayer that already uses Arrow?
Yes we do. We try to leverage Arrow as much as we can. And different from many other computing frameworks, Fugue doesn't invent new data types, it just uses Arrow data types.
Python is not excellent in any domain it's used. But, yes, it's a problem with the language, on which I'll comment later.
First, present day users of Python need to understand how and why Python came under the spotlight. There was always a standoff between programmers who created unimaginative huge programs full of drool and red-taping, and programmers who wanted larger freedom of expression, less strings attached. The later group was usually the more savvy ones.
In a way very similar to how an art student might be spending months studying a model, using a whole bunch of pencils starting from 10B and ending in 10H, various chalks, coal sticks and so on... and would still produce a... "study of a model #80907", which is ugly, anatomically incorrect and just boring. And there's an accomplished artist who can just stick her finger into the chimney, grab some soot, and in a matter of minutes make a great drawing, which will be lively, expressive, you name it.
So... Python, and Perl before it were the soot. The junk languages a more experienced programmer would go to just to show those boring Java programmers "how it's done". But, Python programmers who came in the next wave thought that soot is the good tool to learn how to make good drawings. And, today, we have academies full of students trying very hard to draw models with materials and instruments which are very inappropriate for the task. (Unless you know anything about art education today, the example isn't that big of a stretch of what happened in it around 70s-80s.)
---
I don't care if Python is a glue code for Scala or C or Rust: it doesn't matter. Python, as a language, is inadequate for dealing with concurrency. It needs to remove a bunch of stuff before it can start adding stuff that can be used to that end. It's a language with a lot of mutation semantics which are hard to interpret / implement correctly (what would that even mean?) in distributed context. It's a language with a lot of implicit stuff going on that is somewhat useful (but is not useful enough) if you want to have a quick and dirty "sketch" quality code, but will be devastating in distributed context.
Things like decorators, context managers, imperative loops with break and continue, error handing mechanism, threading -- all of this must go before Python can start to think about becoming a decent language for distributed systems. But, probably more: I would need to research this in much depth to tell for sure if things like method calls would work well for example.
It's a waste of time to try today to fit Python into distributed computation. You will either have to put a humongous effort purging a better half of the language (making all of the famed support libraries useless), or you will end up with a defective hodge-podge mess (which is all those famed support libraries are, including those which aim to do distributed programming in Python).
First, present day users of Python need to understand how and why Python came under the spotlight. There was always a standoff between programmers who created unimaginative huge programs full of drool and red-taping, and programmers who wanted larger freedom of expression, less strings attached. The later group was usually the more savvy ones.
In a way very similar to how an art student might be spending months studying a model, using a whole bunch of pencils starting from 10B and ending in 10H, various chalks, coal sticks and so on... and would still produce a... "study of a model #80907", which is ugly, anatomically incorrect and just boring. And there's an accomplished artist who can just stick her finger into the chimney, grab some soot, and in a matter of minutes make a great drawing, which will be lively, expressive, you name it.
So... Python, and Perl before it were the soot. The junk languages a more experienced programmer would go to just to show those boring Java programmers "how it's done". But, Python programmers who came in the next wave thought that soot is the good tool to learn how to make good drawings. And, today, we have academies full of students trying very hard to draw models with materials and instruments which are very inappropriate for the task. (Unless you know anything about art education today, the example isn't that big of a stretch of what happened in it around 70s-80s.)
---
I don't care if Python is a glue code for Scala or C or Rust: it doesn't matter. Python, as a language, is inadequate for dealing with concurrency. It needs to remove a bunch of stuff before it can start adding stuff that can be used to that end. It's a language with a lot of mutation semantics which are hard to interpret / implement correctly (what would that even mean?) in distributed context. It's a language with a lot of implicit stuff going on that is somewhat useful (but is not useful enough) if you want to have a quick and dirty "sketch" quality code, but will be devastating in distributed context.
Things like decorators, context managers, imperative loops with break and continue, error handing mechanism, threading -- all of this must go before Python can start to think about becoming a decent language for distributed systems. But, probably more: I would need to research this in much depth to tell for sure if things like method calls would work well for example.
It's a waste of time to try today to fit Python into distributed computation. You will either have to put a humongous effort purging a better half of the language (making all of the famed support libraries useless), or you will end up with a defective hodge-podge mess (which is all those famed support libraries are, including those which aim to do distributed programming in Python).
I guess you want to say MPI and C++ are better for distributed computation?
Well... no. MPI is an attempt to fit a saddle on a cow, when it comes to using it with C++. The whole reason it exists is because C or C++ would really, really suck when it comes to distributed programming. So, it tries to save the day with minimal casualties. It's not "the" solution to the problem. Having to use Slurm / PBS and similar my impression from this toolset is... well, pain. But, at least, if you work through pain, you can probably make it go fast. And this is usually what matters to people in this situation.
A much better distributed platform is Erlang. When comparing it to something like Slurm, the benefit is that the programmer has superior access to sharing information between workers, the programmer has programmatic (and superior) control for choosing the workers.
The problem with people in WLM world (that's where Slurm, MPI, OpenMP and friends come from) is that they have never crossed paths with Erlang. They don't even know what they don't know. A lot of the stuff that has to do with the "distributed" part of their computations is perceived outside of their control, something outsourced to infra people: they just need to put a "machine file" somewhere, or use some HTML dashboard to select the nodes where their program needs to run, and hope for the best.
Also, the WLM world has a very peculiar take on distributed programming: they need it to run batches of jobs. Jobs start, finish, and, hopefully, produce results. They don't need to build interactive systems, or deal with upgrades, or create databases etc. So, it's just not an answer to more general problems associated with distributed programming.
A much better distributed platform is Erlang. When comparing it to something like Slurm, the benefit is that the programmer has superior access to sharing information between workers, the programmer has programmatic (and superior) control for choosing the workers.
The problem with people in WLM world (that's where Slurm, MPI, OpenMP and friends come from) is that they have never crossed paths with Erlang. They don't even know what they don't know. A lot of the stuff that has to do with the "distributed" part of their computations is perceived outside of their control, something outsourced to infra people: they just need to put a "machine file" somewhere, or use some HTML dashboard to select the nodes where their program needs to run, and hope for the best.
Also, the WLM world has a very peculiar take on distributed programming: they need it to run batches of jobs. Jobs start, finish, and, hopefully, produce results. They don't need to build interactive systems, or deal with upgrades, or create databases etc. So, it's just not an answer to more general problems associated with distributed programming.
This looks great but as someone who has been doing lots of Spark lately, I feel like this will further worsen the development process.
The problem with Spark is that there's a lot of magic that happens underneath and being able to debug and figure out issues is pretty hairy. If we put this on top then there are 2 levels of magic that we will have to go through to figure out what is going on.
I wish someone works on this aspect.
The problem with Spark is that there's a lot of magic that happens underneath and being able to debug and figure out issues is pretty hairy. If we put this on top then there are 2 levels of magic that we will have to go through to figure out what is going on.
I wish someone works on this aspect.
Hi whinvik, we agree that development in Spark is hard, and that is part of the motivation of Fugue. Spark code couples the distributed orchestration and business logic together.
By keeping your code in native Python or Pandas, it will be much easier to develop, debug, and maintain the business logic because your tracebacks will be in native Python. Fugue then takes it to Spark when you are ready to scale.
By keeping your code in native Python or Pandas, it will be much easier to develop, debug, and maintain the business logic because your tracebacks will be in native Python. Fugue then takes it to Spark when you are ready to scale.
I appreciate your response but that is not what I was getting at. I understand that with this you only have to write Pandas and then not worry about scaling.
First, I think PySpark syntax is much better than the insanity than is Pandas but if you really like Pandas then you can always use Pandas UDF which Spark supports.
But let's say that writing only in Pandas is the preferred way. Now comes the magic part. How do I know that it is using the best join? Will it optimize for spills? Will there be OOM's? These are the things we need to worry about which often lead us needing to go deep inside Spark magic.
Now if there's another level of magic which is Pandas to Spark transpiling as I imagine you do here, then I have even less of an idea how to tune it.
Again I appreciate you are solving a specific problem in a nice way but I feel like we are actually making the problem even more complicated.
First, I think PySpark syntax is much better than the insanity than is Pandas but if you really like Pandas then you can always use Pandas UDF which Spark supports.
But let's say that writing only in Pandas is the preferred way. Now comes the magic part. How do I know that it is using the best join? Will it optimize for spills? Will there be OOM's? These are the things we need to worry about which often lead us needing to go deep inside Spark magic.
Now if there's another level of magic which is Pandas to Spark transpiling as I imagine you do here, then I have even less of an idea how to tune it.
Again I appreciate you are solving a specific problem in a nice way but I feel like we are actually making the problem even more complicated.
Ah I understand what you are saying. Not trying to aggressively make you try Fugue out, I appreciate the comments and just want to clarify some stuff.
There is no transpiling, we think that is too magical. It's more of routing to the appropriate functions/method. For example, applying the Pandas function per partition. We use Pandas UDF under the hood when it makes sense. Sometimes, we use map partitions as well, it depends on what the user wants to do. So we simplify usage around these methods with minimal overhead (we benchmarked it).
Fugue can be adopted as minimally as needed so if you feel the need to write native Spark code or tune Spark, we don't block that from the user. The older Fugue interface was bad in allowing this, so we reworked Fugue to have a suite of standalone functions compatible with Spark/Dask/Ray/Pandas DataFrames.
Our opinion (which is totally fine if you disagree), is that most workloads don't really specific features of Spark. There are times when it makes sense to write native Spark code for sure, and for those Fugue won't be a good fit.
Our job is to make sure it's not an all or nothing thing that requires the user to make compromises.
There is no transpiling, we think that is too magical. It's more of routing to the appropriate functions/method. For example, applying the Pandas function per partition. We use Pandas UDF under the hood when it makes sense. Sometimes, we use map partitions as well, it depends on what the user wants to do. So we simplify usage around these methods with minimal overhead (we benchmarked it).
Fugue can be adopted as minimally as needed so if you feel the need to write native Spark code or tune Spark, we don't block that from the user. The older Fugue interface was bad in allowing this, so we reworked Fugue to have a suite of standalone functions compatible with Spark/Dask/Ray/Pandas DataFrames.
Our opinion (which is totally fine if you disagree), is that most workloads don't really specific features of Spark. There are times when it makes sense to write native Spark code for sure, and for those Fugue won't be a good fit.
Our job is to make sure it's not an all or nothing thing that requires the user to make compromises.
Hello whinvik, I totally agree the debugging experience of Spark is suboptimal.
But why?
I don't think the main reason is too much 'magic', I think it is because of the coupling of user's logic and the complicated backend logic. When there is an issue, not only you have to wait for a long time to get the error back, but also you get the error of the whole execution chain, and you have to find the real error in it, which is challenging even for experienced Spark users.
Not only Spark, if you use Dask or Ray, you will find very similar experience. Can we directly make Spark Dask Ray easier to debug?
I doubt. This is because the complexity of these systems are necessary. And since the problem can happen anywhere either on the user side or on the backend side, the whole stack has to be investigated. It is inevitable, otherwise, people will start to complain the real errors are not captured by the system.
What can we do about it?
Fugue's idea is to let you iterate on your own logic locally before using these complex systems. So you can get quicker and less noisy feedbacks. The challenge is how to unify the semantic so you don't need to change code moving between local and distributed. And how to trim down the system errors so you can focus more on your own problems. These are what Fugue solves.
It's all about good practices
It's common sense that if you can keep your logic agnostic to backends, and if you can iterate locally on small data, the development experience can be a lot better. But why big data developers tend not to follow these practices? Because there is too much discrepancy created by the distributed backends intentionally or unintentionally. We created such abstraction for big data developers to get more rewards than punishment when following good practices, so they are more willing to follow good practices.
But why?
I don't think the main reason is too much 'magic', I think it is because of the coupling of user's logic and the complicated backend logic. When there is an issue, not only you have to wait for a long time to get the error back, but also you get the error of the whole execution chain, and you have to find the real error in it, which is challenging even for experienced Spark users.
Not only Spark, if you use Dask or Ray, you will find very similar experience. Can we directly make Spark Dask Ray easier to debug?
I doubt. This is because the complexity of these systems are necessary. And since the problem can happen anywhere either on the user side or on the backend side, the whole stack has to be investigated. It is inevitable, otherwise, people will start to complain the real errors are not captured by the system.
What can we do about it?
Fugue's idea is to let you iterate on your own logic locally before using these complex systems. So you can get quicker and less noisy feedbacks. The challenge is how to unify the semantic so you don't need to change code moving between local and distributed. And how to trim down the system errors so you can focus more on your own problems. These are what Fugue solves.
It's all about good practices
It's common sense that if you can keep your logic agnostic to backends, and if you can iterate locally on small data, the development experience can be a lot better. But why big data developers tend not to follow these practices? Because there is too much discrepancy created by the distributed backends intentionally or unintentionally. We created such abstraction for big data developers to get more rewards than punishment when following good practices, so they are more willing to follow good practices.
[deleted]
What is the difference with ibis which is also one of its included backends? Also ibis has other backends e.g. duckdb so what is the constraint of accessing duckdb with a python interface as in: fugue->ibis->duckdb?
Seems very interesting but I can't tell what it's scope range is, or its comparative advantages with ibis or dbt or others?
Seems very interesting but I can't tell what it's scope range is, or its comparative advantages with ibis or dbt or others?
Hi antman, thanks for the question. I will type some points on differences, but will answer the first question. The Fugue -> Ibis -> DuckDB example is a bit weird. Yes it can be done but it's not practical (as you can tell). There may be some overlap sometime, but I do think the projects differ in scope (more below).
The Ibis integration is more about accessing data in various data stores already. For example, we use it under the hood also for our recently released BigQuery integration: https://fugue-tutorials.readthedocs.io/tutorials/integration...
On to differences:
1. We guarantee consistency between backends. NULL handling can be different depending on the backend. For example, Pandas joins NULL with NULL while Spark doesn't. So if you prototype locally on Pandas, and then scale to Spark, we guarantee same results. Fugue is 100% unit tested and the backends go through the same test suite.
2. Ibis is Pythonic for SQL backends. We embrace SQL, but understand its limitations. FugueSQL is an enhanced SQL dialect that can invoke Python code. FugueSQL can be the first-class grammar instead of being sandwiched by Python code. Fugue's Python API and SQL API are 1:1 in capability.
3. Opinionated here, but we don't want users to learn any new language. Ibis is a new way to express things; we just want to extend the capabilities of what people already know (SQL, native Python, and Pandas). Fugue can also be incrementally adopted, meaning it can be used for just one portion of your workflow.
4. Roadmap-wise, we think the optimal solutions will be a mix of different tools. A clear one is pre-aggregating data with DuckDB, and then using Pandas for further processing. Similarly, can we preprocess in Snowflake and do machine learning in Spark? Fugue is working on connecting these different systems to enable cross-platform workloads.
There may be more information for you here: https://fugue-tutorials.readthedocs.io/tutorials/integration...
The Ibis integration is more about accessing data in various data stores already. For example, we use it under the hood also for our recently released BigQuery integration: https://fugue-tutorials.readthedocs.io/tutorials/integration...
On to differences:
1. We guarantee consistency between backends. NULL handling can be different depending on the backend. For example, Pandas joins NULL with NULL while Spark doesn't. So if you prototype locally on Pandas, and then scale to Spark, we guarantee same results. Fugue is 100% unit tested and the backends go through the same test suite.
2. Ibis is Pythonic for SQL backends. We embrace SQL, but understand its limitations. FugueSQL is an enhanced SQL dialect that can invoke Python code. FugueSQL can be the first-class grammar instead of being sandwiched by Python code. Fugue's Python API and SQL API are 1:1 in capability.
3. Opinionated here, but we don't want users to learn any new language. Ibis is a new way to express things; we just want to extend the capabilities of what people already know (SQL, native Python, and Pandas). Fugue can also be incrementally adopted, meaning it can be used for just one portion of your workflow.
4. Roadmap-wise, we think the optimal solutions will be a mix of different tools. A clear one is pre-aggregating data with DuckDB, and then using Pandas for further processing. Similarly, can we preprocess in Snowflake and do machine learning in Spark? Fugue is working on connecting these different systems to enable cross-platform workloads.
There may be more information for you here: https://fugue-tutorials.readthedocs.io/tutorials/integration...
Are you suggesting that Snowflake compatibility/integration is coming in the future? If so, do you plan to integrate Snowpark (Snowflake's custom DataFrame APIs) into the mix?
Yes, we plan to integrate with Snowflake and Snowpark.
Fugue is really cool as it makes a complicated thing simple. The learning curve is a few minutes (provided that you have an installation that works, of course)
Thanks a lot for the kind words, cmarschner!
It also looks pretty powerful, though I admit I haven't used it yet and haven't used Dask or Ray but it's making me go look into popping spark back up and learning this.
Kudos on what looks to be a really well built product.