For those of you unfamiliar with Wool (which I'm assuming is most of you reading this), it's a distributed execution runtime for Python I've been working on that adds parallelism to Python's existing async primitives, allowing you to "await" CPU-bound logic without blocking the caller with a relatively simple service topology compared to some of the other distributed runtimes out there. Concerns like prioritization, queuing, and routing can be expressed in native Python, while Wool handles the inter-process/host transport.
This gist illustrates some of the trials and tribulations I went through while implementing a distributed context system for the runtime. Maybe some of you will find it (i.e., wool, the feature, or the lessons) useful, but in any case I enjoyed the process (I may be a masochist) and learned a lot.
I’m not sure why people are assuming I’m pushing for distributed OOP here. Wool is completely unopinionated about how you decide to architect your application - that’s the entire point.
Wasn’t previously aware of this opinion, but having read it, I can tell you I’m not actually committing this fallacy within the framework- I’m not forcing anyone into making their instance methods RPCs- I leave that up to my users. They are grown adults (presumably) and can commit whichever fallacies they desire.
I think LLMs are incredible tools that I will continue to use unapologetically, but I’m also very particular and not going to be putting my name to AI slop. Those are my genuine thoughts on the matter, they just happened to be cleaned up by an automated stochastic parrot.
Sure, my argument is that those concerns should be composable with the distribution layer, not defined by it. There are plenty of different ways to pass data across a network or between machines, I'm not going to reinvent them all. There are also plenty of ways to manage non-transient errors (and very app-specific), I'm not going to implement them all. I'm also not assuming you need or want object persistence and replayability of your workflows. If you do, then use the right tool for the job.
To be clear, not all of your coroutines become distributed. You explicitly declare what you want executed remotely, and you'll probably want your workers running in a VPC - it's just a nifty way of scaling your app horizontally without sinking too much effort into how you choose to define your distribution boundaries. It wouldn't be any different than scaling any other service horizontally using something like Kubernetes (which is probably how you'd manage wool workers as well).
But seriously, I just wanted to have an easy way to parallelize my projects without a bunch of slow, bloated, boilerplate ridden frameworks, and I'm having fun building it. And I must not be the only to have had this desire considering there's way more than one solution to this problem out there already, so f*k it.
I'm running this in a trusted environment, I'm not so ambitious as to try to make this some sort of whacky trustless distributed Python runtime. Just a fun project that's been marinating for a while, and now I have an army of clankers to do the dirty work of documenting and testing it.
This gist illustrates some of the trials and tribulations I went through while implementing a distributed context system for the runtime. Maybe some of you will find it (i.e., wool, the feature, or the lessons) useful, but in any case I enjoyed the process (I may be a masochist) and learned a lot.