What if Python was natively distributable?(medium.com)
medium.com
What if Python was natively distributable?
https://medium.com/@bzurak/what-if-python-was-natively-distributable-3bfae485a408
66 comments
For one, it's faster than prefect and ray so I can use it for more stuff without having to import a 10000000 line library.
But seriously, I just wanted to have an easy way to parallelize my projects without a bunch of slow, bloated, boilerplate ridden frameworks, and I'm having fun building it. And I must not be the only to have had this desire considering there's way more than one solution to this problem out there already, so f*k it.
Fun is what its all about. You don't a reason other than that.
> Regardless, I love ambitious projects furiously coded by one crazy person.
Me too; the world lost a treasure, once.
RIP Terry Davis. There was so much to be learned from your approach.
Me too; the world lost a treasure, once.
RIP Terry Davis. There was so much to be learned from your approach.
The distributed object fallacy is never going away, is it?
https://martinfowler.com/articles/distributed-objects-micros...
https://martinfowler.com/articles/distributed-objects-micros...
Wasn’t previously aware of this opinion, but having read it, I can tell you I’m not actually committing this fallacy within the framework- I’m not forcing anyone into making their instance methods RPCs- I leave that up to my users. They are grown adults (presumably) and can commit whichever fallacies they desire.
[deleted]
Not on my watch
Every time I see something like this (turn function calls into a network call), I reflect fondly on the list of fallacies in distributed computing [0]. These are issues that largely have to be handled in an application-specific way, and cannot be solved in the general case.
This list alone has saved me many late debugging nights, just by not making or using systems that ignore the list.
[0]: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
This list alone has saved me many late debugging nights, just by not making or using systems that ignore the list.
[0]: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
I ask this question, but instead: I ask it of Lua.
As in, what if there was a Linux distro that focused, primarily, on building a Lua layer on top of everything, system-wise. Just replace all the standard stuff with one single, system-friendly language: Lua. C/C++ everything as it currently is: put Lua on top all the way to the Desktop.
It’s only a thought experiment, except there are cases where I can see a way to use it, and in fact have done it, only not to the desktop, such as of course embedded. Realtime data collection, processing and management. In this case, it is superlative to have a single system/app language on top of C/C++.
So I think there may be a point in the future where this ‘single language for everything’ becomes a mantra in distro land. I see the immense benefit.
Lua table state can be easily transferred, give or take a bit of resource management vis a vis sync’ing state and restoring it appropriately. Lua Bytecode, itself, in a properly defined manner can serve as a perfectly cromulant wire spec. Do it properly and nobody will ever know it isn’t just a plain ol’ C struct on an event handler, one of the fastest, except it’ll be very well abstracted to the application.
Oh, and if things are doing bytecode, may as well have playback and transactions intrinsically… it is all, after all, just a stream.
App state as gstreamer plugin? Within grasp, imho…
As in, what if there was a Linux distro that focused, primarily, on building a Lua layer on top of everything, system-wise. Just replace all the standard stuff with one single, system-friendly language: Lua. C/C++ everything as it currently is: put Lua on top all the way to the Desktop.
It’s only a thought experiment, except there are cases where I can see a way to use it, and in fact have done it, only not to the desktop, such as of course embedded. Realtime data collection, processing and management. In this case, it is superlative to have a single system/app language on top of C/C++.
So I think there may be a point in the future where this ‘single language for everything’ becomes a mantra in distro land. I see the immense benefit.
Lua table state can be easily transferred, give or take a bit of resource management vis a vis sync’ing state and restoring it appropriately. Lua Bytecode, itself, in a properly defined manner can serve as a perfectly cromulant wire spec. Do it properly and nobody will ever know it isn’t just a plain ol’ C struct on an event handler, one of the fastest, except it’ll be very well abstracted to the application.
Oh, and if things are doing bytecode, may as well have playback and transactions intrinsically… it is all, after all, just a stream.
App state as gstreamer plugin? Within grasp, imho…
There used to be a roguelike game called Angband, which was written in C. There was a vibrant community around it, many of whom produced Angband variants by hacking the text config files and the C code. One developer got the idea of making most of the game scriptable in Lua, over a C core; which would, in theory, make even more people be able to hack at the game and produce variants.
What happened was the Angband community imploded, and the number of variants went way down.
I don't know if this is a generalizable example and there may have been other factors at work, but it is a cautionary tale.
Angband is still around btw, and is still excellent. But I believe it's C and text config files again now.
What happened was the Angband community imploded, and the number of variants went way down.
I don't know if this is a generalizable example and there may have been other factors at work, but it is a cautionary tale.
Angband is still around btw, and is still excellent. But I believe it's C and text config files again now.
When I was a wet behind the ears programmer, I learned a cautionary tale about Lisp and macros. Very smart people love macros and are good at them, and will modify things to fit their needs. Problem is, other people think different ways. A flexible language which is easy to mutate results in incompatible dialects.
The sage elders suggested that making the language harder to change (cough Python) is more likely to result in a single widely used dialect, with the differences at the library level rather than in language/macro level.
The sage elders suggested that making the language harder to change (cough Python) is more likely to result in a single widely used dialect, with the differences at the library level rather than in language/macro level.
Arrays starts at one scared everybody away.
Wouldn't you face the same problem as Dotnet on Windows? AFAIK, dotnet based frameworks and apps suffered from huge performance issues. It might have improved in recent times, I am not actually a windows dev.
If just the end user application is in Lua, then maybe it's fine and the high level language slowdown won't matter. If you want to wrap the low level kernel APIs etc in a high level language as the canonical interface, I would be very skeptical.
If just the end user application is in Lua, then maybe it's fine and the high level language slowdown won't matter. If you want to wrap the low level kernel APIs etc in a high level language as the canonical interface, I would be very skeptical.
I’m not sure the ‘language slowdown’ is as significant as one might think, given the common shared libs that would be in place with a one-size-fits-all solution, but its really all just a dream until someone does it, anyway.
Dotnet is pretty fast these days. It has a lot more low level control than something like Java, with value types and manual memory management available (it invented the unsafe{} blocks Rust is famous for).
MS even had a prototype version of Windows where the entire OS from the kernel up was managed code (a little excessive, IMO).
Dotnet GUI apps failed on Windows mostly because their UI toolkits are a mess and Electron won the cross platform war. I avoid this stack like the plague, but I write a lot of web backends in C#/ASP.NET (on Linux deploying to k8s) and it's great!
MS even had a prototype version of Windows where the entire OS from the kernel up was managed code (a little excessive, IMO).
Dotnet GUI apps failed on Windows mostly because their UI toolkits are a mess and Electron won the cross platform war. I avoid this stack like the plague, but I write a lot of web backends in C#/ASP.NET (on Linux deploying to k8s) and it's great!
Sounds kind of like Arcan https://arcan-fe.com/about/
That is pretty much what we've done, but not in as "narrow" a scope as just Desktop but more as 'everything user facing in computing'. Full breakdown will be covered later this year as 'Arcan As OS Implementation' in a section on "Lua at every layer".
Do it. Build on the work of AwesomeWM probably, it's a Lua focused window manager that's quite nice. You can also build up less "minimalist" widgets and whatnot using Lua and claude code, which is very good at unconventional GUI work in Lua. I can attest to this specifically, I've had it build numerous GUIs with mpv Lua userscripts.
Join the EDA industry, and you can have TCL. :D
100%
Pah - all these newcomers with interpreted/bytecode languages for remote functions!
It's easy to send text or bytecode to another instance of your runtime. I did a distributed system sending native code functions to be executed remotely.
Fair enough, it was for academic purposes, but it worked[1].
-------------------------------------
[1] By "worked" I mean got a passing grade on a thesis that got an IMDB number. Probably still got the actual PDF somewhere, or maybe it was indexed by google at some point.
It's easy to send text or bytecode to another instance of your runtime. I did a distributed system sending native code functions to be executed remotely.
Fair enough, it was for academic purposes, but it worked[1].
-------------------------------------
[1] By "worked" I mean got a passing grade on a thesis that got an IMDB number. Probably still got the actual PDF somewhere, or maybe it was indexed by google at some point.
Film studies thesis?
Good Catch :-)
I'm gonna leave that there so that your comment remains funny!
I'm gonna leave that there so that your comment remains funny!
I don't have much experience with this kind of thing, but from here it looks like a program written this way would be nearly impossible to reason about performance when something as simple as a function call in your Python interpreter can have wild fluctuations in predictability just due to underlying network latency, remote host saturation, etc.
It seems like you would need an entire observability framework built and instrumented on top of this to ever really be that useful.
It seems like you would need an entire observability framework built and instrumented on top of this to ever really be that useful.
To be clear, not all of your coroutines become distributed. You explicitly declare what you want executed remotely, and you'll probably want your workers running in a VPC - it's just a nifty way of scaling your app horizontally without sinking too much effort into how you choose to define your distribution boundaries. It wouldn't be any different than scaling any other service horizontally using something like Kubernetes (which is probably how you'd manage wool workers as well).
Answering the question: it would be called "the Erlang VM", and you'd use Elixir to program it.
https://elixir-lang.readthedocs.io/en/latest/mix_otp/10.html
https://elixir-lang.readthedocs.io/en/latest/mix_otp/10.html
Yeah but... who does that?
Meanwhile, in BEAM land, this is a solved problem, and just watching different languages converge into towards the set of primitives and constraints. It is hard to replicate BEAM’s premptive scheduler, but I would not be surprised if someone will end up thinking they invented something novel by adding queues (mailboxes), or failing that, something like go channels.
And even then, once you have a workable set of primitives, it turns out that there are some orchestration patterns that recurs over and over again, so people will converge towards OTP once the primitives are there.
And even then, once you have a workable set of primitives, it turns out that there are some orchestration patterns that recurs over and over again, so people will converge towards OTP once the primitives are there.
Absolutely wild to see none of the long lineage of similar attempts mentioned here. The earliest I could find with a quick search was Pyro which started in 1998 and still seems to be going: https://pyro4.readthedocs.io/en/stable/
RPyC came along in the aughts. There's a long history of "transparent clustering and rpc" efforts in Python that could be used or drawn on.
Sad to see that history ignored here.
RPyC came along in the aughts. There's a long history of "transparent clustering and rpc" efforts in Python that could be used or drawn on.
Sad to see that history ignored here.
I guess I'm not old enough.
to research? you should start. it can be pretty illuminating.
Harsh but fair. :)
I'm a little confused by the article. Everything mentioned seem to be real problems you'll fairly quickly run into that need real solutions (passing data over the network, handling transient errors).
Celery + Redis is already mature, easy, and can be setup quickly. Not my favorite framework by any means but it seems to work, more or less
Celery + Redis is already mature, easy, and can be setup quickly. Not my favorite framework by any means but it seems to work, more or less
Sure, my argument is that those concerns should be composable with the distribution layer, not defined by it. There are plenty of different ways to pass data across a network or between machines, I'm not going to reinvent them all. There are also plenty of ways to manage non-transient errors (and very app-specific), I'm not going to implement them all. I'm also not assuming you need or want object persistence and replayability of your workflows. If you do, then use the right tool for the job.
cloudpickle serializes code without signatures; which is an RCE vuln.
It is much safer to distribute signed code in signed packages out of band and send only non-executable data in messages.
It is more safe to store distributed messages in a page with the NX bit flipped.
A compromise of any client in this system results in DOS and arbitrary RCE; but that's an issue with most distributed task worker systems.
To take a zero trust approach, you can't rely on the shared TLS key or the workers never being compromised.
mDNS doesn't scale beyond a broadcast segment without mDNS repeaters (which don't scale) which are on multiple network segments.
Something centralized like Dask, for example, can log and track state centrally to handle task retries on network, task, and worker failure.
But Dask doesn't satisfy zero trust design guidelines either.
How are these systems distinct from botnets with a shared TLS key and no certificate revocation?
It is much safer to distribute signed code in signed packages out of band and send only non-executable data in messages.
It is more safe to store distributed messages in a page with the NX bit flipped.
A compromise of any client in this system results in DOS and arbitrary RCE; but that's an issue with most distributed task worker systems.
To take a zero trust approach, you can't rely on the shared TLS key or the workers never being compromised.
mDNS doesn't scale beyond a broadcast segment without mDNS repeaters (which don't scale) which are on multiple network segments.
Something centralized like Dask, for example, can log and track state centrally to handle task retries on network, task, and worker failure.
But Dask doesn't satisfy zero trust design guidelines either.
How are these systems distinct from botnets with a shared TLS key and no certificate revocation?
I'm running this in a trusted environment, I'm not so ambitious as to try to make this some sort of whacky trustless distributed Python runtime. Just a fun project that's been marinating for a while, and now I have an army of clankers to do the dirty work of documenting and testing it.
Distributed DAG state with W3C VC and W3C DID (did:tdw did:webvh) microledgers and maybe W3C PROV would probably work, but the opcode cost of the control layer and the opcode cost of each dispatched, signed code object, and the byte cost for data storage, and messages.
Agent isolation with WASM and now wasmtime-mte . Opcode costs with eWASM or similar?
Agent isolation with WASM and now wasmtime-mte . Opcode costs with eWASM or similar?
Basically, there's no good way to sidestep the authentication and authorization and resource quota controls of a resource grid scheduler.
With redundancy and idempotency, distributed computation can work.
In order to run computation distributedly with required keys, sharded distributed ledger protocol nodes cost the smart contracts that they execute in a virtual machine with network access limited to in-protocol messages. Each "smart contract" costs money to run redundantly on multiple nodes, and so it should or must have an account with a verifiably-sufficient balance in order to run.
Smart contracts must be uploaded using a private key with a corresponding public key.
Smart contracts are identified by and so are addressed by a hash of their bytecode.
Wool, Dask, and Celery @task don't solve for smart contract output storage costs with redundancy. They don't set up database(s) with replication large enough for each computation step for you. Dask and Celery model and track the state of each executed DAG centrally with logs as trustable as the centralized nodes which are a single point of failure.
Why isn't Docker Swarm - which L2 bridges amongst all nodes without restriction - appropriate for a given application, given the Access Control Lists and cloud (pod,) configuration necessary for e.g. AWS or GCP to prevent budget overflow? Why quota grid users somehow?
Serverless functions must be uploaded/deployed before being run, too. To orchestrate a bunch of web services is to execute the DAG and handle errors due to network, node, and service failures and latency
But then what protocol do the (serverless function) services all implement, so that we don't have to have a hodgepodge of API clients to use all the services in the grid?
With (serverless) functions bound to /URL routes, cost each function and estimate resource requirements to continue to run a function of that cost. To handle just benign resource exhaustion, say, Scale up the databases and/or redundant block storages, change other distributed computation grid parameters for that pod or those pods of resources which service a named, signed function like /api/v1/helloworld_v2?q=select%20* which is creating contention for the costed resources of the organization
On what signals do you scale up or scale down - within a resource budget - to afford fanning out over multiple actually parallel nodes to compute and sign and store the data?
With redundancy and idempotency, distributed computation can work.
In order to run computation distributedly with required keys, sharded distributed ledger protocol nodes cost the smart contracts that they execute in a virtual machine with network access limited to in-protocol messages. Each "smart contract" costs money to run redundantly on multiple nodes, and so it should or must have an account with a verifiably-sufficient balance in order to run.
Smart contracts must be uploaded using a private key with a corresponding public key.
Smart contracts are identified by and so are addressed by a hash of their bytecode.
Wool, Dask, and Celery @task don't solve for smart contract output storage costs with redundancy. They don't set up database(s) with replication large enough for each computation step for you. Dask and Celery model and track the state of each executed DAG centrally with logs as trustable as the centralized nodes which are a single point of failure.
Why isn't Docker Swarm - which L2 bridges amongst all nodes without restriction - appropriate for a given application, given the Access Control Lists and cloud (pod,) configuration necessary for e.g. AWS or GCP to prevent budget overflow? Why quota grid users somehow?
Serverless functions must be uploaded/deployed before being run, too. To orchestrate a bunch of web services is to execute the DAG and handle errors due to network, node, and service failures and latency
But then what protocol do the (serverless function) services all implement, so that we don't have to have a hodgepodge of API clients to use all the services in the grid?
With (serverless) functions bound to /URL routes, cost each function and estimate resource requirements to continue to run a function of that cost. To handle just benign resource exhaustion, say, Scale up the databases and/or redundant block storages, change other distributed computation grid parameters for that pod or those pods of resources which service a named, signed function like /api/v1/helloworld_v2?q=select%20* which is creating contention for the costed resources of the organization
On what signals do you scale up or scale down - within a resource budget - to afford fanning out over multiple actually parallel nodes to compute and sign and store the data?
I’m not sure why people are assuming I’m pushing for distributed OOP here. Wool is completely unopinionated about how you decide to architect your application - that’s the entire point.
the idea has been around (and implemented) at least since the 1980s.
https://en.wikipedia.org/wiki/Emerald_(programming_language)
The CSP approach is also interesting. fe Limbo
https://en.wikipedia.org/wiki/Limbo_(programming_language)
https://en.wikipedia.org/wiki/Emerald_(programming_language)
The CSP approach is also interesting. fe Limbo
https://en.wikipedia.org/wiki/Limbo_(programming_language)
Erlang is natively distributed. There are no distinctions between local & remote messages & mailboxes.
LLM generated article.
I wonder if an LLM generated article would get the title to use proper English, though: "What if Python were natively distributable?".
It's possible LLMs pick up improper English, of course, since proper is some measure of what used to be a norm, but may presently be perceived as outdated.
It's possible LLMs pick up improper English, of course, since proper is some measure of what used to be a norm, but may presently be perceived as outdated.
Is it possible it's both?
evidently what becomes standard gradually changes. i believe you can see this in construction of the past tense (perfect tense) of verbs in Polish/etc vs Russian, where Russian just uses the grammatical past participle as if it were the simple past tense.
Speakers of English in the Americas make this same substitution, which sounds like a mistake to those who speak in the version of English taught in schools. They will say "i seen that" rather than "i saw that", for example, just as would happen in Russian.
Speakers of English in the Americas make this same substitution, which sounds like a mistake to those who speak in the version of English taught in schools. They will say "i seen that" rather than "i saw that", for example, just as would happen in Russian.
I have a feeling people will begin to purposely use slightly incorrect grammar to give the impression they are indeed human in their writing.
definitely: look at groups choosing their own deviations to signal group membership. american slang groups for instance, including teen kids purposefully using jargon they redefine among themselves so parents are un-cool.
this article comes back 100% ai (for the opening). i will not reveal the tell that made me churn
You mean the cadence where everything is presented in threes? Yeah I hate it too, but beats paying some blogger to do it
[deleted]
> You mean the cadence where everything is presented in threes? Yeah I hate it too, but beats paying some blogger to do it
You could also use your own words instead of an LLM. Would have been more interesting.
You could also use your own words instead of an LLM. Would have been more interesting.
I think LLMs are incredible tools that I will continue to use unapologetically, but I’m also very particular and not going to be putting my name to AI slop. Those are my genuine thoughts on the matter, they just happened to be cleaned up by an automated stochastic parrot.
just disclose please. i churn on non-disclosed ai generated content as a matter of principle. i don't necessarily churn on other variants
Fair enough
[flagged]
"Don't be snarky."
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
You peeps are right - I should have been less snarky.
Having worked with projects like celery I at first was enamored by the relative simplicity of getting a job queue going, but it traps you into celery which is just python or whatever language happens to create a celery compatible message structure (it’s amqp for most shops I’ve worked with).
Thing is in distributed systems it’s annoying when all the different systems have to pepper celery as a layer of your understanding. I found that systems like Argo workflows/events did an awesome job of keeping the message agnostic to anything (even Argo) such that writing the code was boring (the good kind). At the end you’re not even thinking about pubsub or anything other than either what you log out or write to a file. Either of those is fine.
How those different services connect or scale is irrelevant to your service (that’s for another layer handled by your Argo manifests) and it’s contextually liberating.
I am probably not being fair to this project, I just dislike making my software be someone else’s implicitly - there are places that would consider themselves x (celery in my case) shops because of the investment not only in the invasive code presence that is commanded by the library but the necessary mental model. I felt like while it is an investment to learn Argo - the code structure and mindset is pretty portable.
Having worked with projects like celery I at first was enamored by the relative simplicity of getting a job queue going, but it traps you into celery which is just python or whatever language happens to create a celery compatible message structure (it’s amqp for most shops I’ve worked with).
Thing is in distributed systems it’s annoying when all the different systems have to pepper celery as a layer of your understanding. I found that systems like Argo workflows/events did an awesome job of keeping the message agnostic to anything (even Argo) such that writing the code was boring (the good kind). At the end you’re not even thinking about pubsub or anything other than either what you log out or write to a file. Either of those is fine.
How those different services connect or scale is irrelevant to your service (that’s for another layer handled by your Argo manifests) and it’s contextually liberating.
I am probably not being fair to this project, I just dislike making my software be someone else’s implicitly - there are places that would consider themselves x (celery in my case) shops because of the investment not only in the invasive code presence that is commanded by the library but the necessary mental model. I felt like while it is an investment to learn Argo - the code structure and mindset is pretty portable.
So if you wanted to scale, say, a hyperparameter search across a cluster, what would you prefer?
It’s a good point. The Argo workflow/event set up I’m referencing isn’t for everything. It does distribute ELT pipelines rather well but I am not sure it was designed with that particular usecase in mind. Thank you!
Elixir flame?
It’d turn Python from a dev-friendly language into a true deployment powerhouse with no env hell, just ship and run.
Perhaps I am misunderstanding this, but after looking at the code what exactly are we achieving here over other frameworks? The repo is obviously very new (and the author certainly seems busy), so perhaps a better question is what do we aim to achieve? So far it seems like the exact same pattern with some catchy naming.
Regardless, I love ambitious projects furiously coded by one crazy person. And I mean "crazy" in the best sense of the word, not as an insult. This is what open source is all about.
Please prove us all wrong. If you fail, you'll learn a ton!