Goblins: A transactional, distributed actor model environment(docs.racket-lang.org)
docs.racket-lang.org
Goblins: A transactional, distributed actor model environment
https://docs.racket-lang.org/goblins/
32 comments
Cool! I hope one day Racket will take off as a serious language for app development.
I know it's not really the point of Racket, but it has all the right attributes^, all it needs is a bigger library ecosystem.
^: including a package manager & build toolchain, an IDE, nicely-formatted web hosting for docs, runtime is reasonably fast, has plenty of batteries included, has cool stuff like contracts, supports multiple programming paradigms.
I know it's not really the point of Racket, but it has all the right attributes^, all it needs is a bigger library ecosystem.
^: including a package manager & build toolchain, an IDE, nicely-formatted web hosting for docs, runtime is reasonably fast, has plenty of batteries included, has cool stuff like contracts, supports multiple programming paradigms.
That would be great but even if doesn’t get there, Racket is still an amazing ecosystem for exploring new ideas and will continue to love it. But of course any improvements are welcome
Kind of weird hearing that when we're posting on a website running atop Racket.
This website also runs on a single server[0], stores user state in closures[1] and uses tables for layout[2], in violation of just about every single modern web development practice.
And it's all the better for it. I wish more sites would follow HN's example.
--
[0] - https://news.ycombinator.com/item?id=16076041
[1] - Mostly, at least - https://news.ycombinator.com/item?id=23374471. It used to be almost for every action, the pg's OG hack, but got optimized away.
[2] - Press CTRL+U.
And it's all the better for it. I wish more sites would follow HN's example.
--
[0] - https://news.ycombinator.com/item?id=16076041
[1] - Mostly, at least - https://news.ycombinator.com/item?id=23374471. It used to be almost for every action, the pg's OG hack, but got optimized away.
[2] - Press CTRL+U.
Hacker News isn't Racket, it's Arc
Arc is built in / runs on Racket: http://www.arclanguage.org/install
[deleted]
I agree. Racketfest Amateur Night happened last Friday and Saturday and many cool projects were show cased.
Oh hey neat, some coverage. Actually I'm right on the verge of the next big release (v0.8) which should make doing the networked programming significantly easier.
But here's some more background: I'm co-author/co-editor of the ActivityPub specification, which might give you some idea that I have some experience with trying to build networked systems. Goblins is part of the Spritely project, or even more accurately, the foundation for it: https://spritelyproject.org/
Spritely has some pretty wild ambitions... I won't go into them in detail, the video near the top of the above site explains better. But achieving those in a feasible timeframe means we need some better tooling. Goblins is the foundation of that; it's implemented as a Racket library, but can/will be ported elsewhere. The docs linked above explain some of the core ideas, but honestly I think the video linked from this section of the site explains it better: https://spritelyproject.org/#goblins
In general, people tend to realize that something interesting is happening when they see a cool shiny demo. So here are two pages that might give you an idea with some cool shiny demos:
- Demonstrating "time travel" in a space shooter game (no special code added for this other than exposing it in the GUI): https://dustycloud.org/blog/goblins-time-travel-micropreview... - "goblin-chat", which doesn't look quite as shiny but is interesting more that the code you see above is "end to end encrypted", but is written in a mere 250 lines of code for the whole protocol http://dustycloud.org/blog/spritely-goblins-v0.7-released/
What? How can the last one be only 250 lines of code? (And a mere 300 more for the GUI!) Well, that's because we're implementing a peer-to-peer distributed object programming protocol called CapTP (which includes wild things like distributed garbage collection over the network (for acyclic references), "promise pipelining", and is object capability secure). The next release will be advancing that work significantly. The Agoric organization is also implementing CapTP, but their stuff is in Javascript instead of Racket; we plan to have our CapTPs to converge. Thus it shouldn't really matter whether you write your code in Javascript or Racket, your code should be able to talk to each other.
Anyway, new release coming out soon. Hope that answers some things since the docs page might not convey what's fully interesting about it (and indeed, neither does this text, but the videos mentioned above do a better job).
But here's some more background: I'm co-author/co-editor of the ActivityPub specification, which might give you some idea that I have some experience with trying to build networked systems. Goblins is part of the Spritely project, or even more accurately, the foundation for it: https://spritelyproject.org/
Spritely has some pretty wild ambitions... I won't go into them in detail, the video near the top of the above site explains better. But achieving those in a feasible timeframe means we need some better tooling. Goblins is the foundation of that; it's implemented as a Racket library, but can/will be ported elsewhere. The docs linked above explain some of the core ideas, but honestly I think the video linked from this section of the site explains it better: https://spritelyproject.org/#goblins
In general, people tend to realize that something interesting is happening when they see a cool shiny demo. So here are two pages that might give you an idea with some cool shiny demos:
- Demonstrating "time travel" in a space shooter game (no special code added for this other than exposing it in the GUI): https://dustycloud.org/blog/goblins-time-travel-micropreview... - "goblin-chat", which doesn't look quite as shiny but is interesting more that the code you see above is "end to end encrypted", but is written in a mere 250 lines of code for the whole protocol http://dustycloud.org/blog/spritely-goblins-v0.7-released/
What? How can the last one be only 250 lines of code? (And a mere 300 more for the GUI!) Well, that's because we're implementing a peer-to-peer distributed object programming protocol called CapTP (which includes wild things like distributed garbage collection over the network (for acyclic references), "promise pipelining", and is object capability secure). The next release will be advancing that work significantly. The Agoric organization is also implementing CapTP, but their stuff is in Javascript instead of Racket; we plan to have our CapTPs to converge. Thus it shouldn't really matter whether you write your code in Javascript or Racket, your code should be able to talk to each other.
Anyway, new release coming out soon. Hope that answers some things since the docs page might not convey what's fully interesting about it (and indeed, neither does this text, but the videos mentioned above do a better job).
Thanks for setting me off on a wonderful trail tonight! After a couple of hours of reading, I finally appreciate how useful, simple and coherent the capabilities+actors part of the story is! I particularly enjoyed the analogy/application [1] with federated social networking -- Seeing how easily they are modeled by ocaps actually makes capabilities much easier to understand! :-)
[1] https://gitlab.com/spritely/ocappub/blob/master/README.org
[2] This talk also has a nice & simple technical perspective: https://www.youtube.com/watch?v=2H-Azm8tM24
[1] https://gitlab.com/spritely/ocappub/blob/master/README.org
[2] This talk also has a nice & simple technical perspective: https://www.youtube.com/watch?v=2H-Azm8tM24
That's neat; the ability to move state backwards is a powerful tool. I'm building my own language around reactivity and state management ( http://www.adama-lang.org/ ), and I'm getting to the point where I could demo replaying a board game backwards.
This looks very cool! And some of the ideas look very interesting!
Can someone familiar with the framework answer a few questions?
Do transactions span distributed actors? Perhaps I missed it, but I see no recovery log.
It implements distributed garbage collection. This is a non-starter for any environment where the network cannot be taken for granted. RMI/DCOM/CORBA had no shortage of problems due to distributed GC alone.
Do transactions span distributed actors? Perhaps I missed it, but I see no recovery log.
It implements distributed garbage collection. This is a non-starter for any environment where the network cannot be taken for granted. RMI/DCOM/CORBA had no shortage of problems due to distributed GC alone.
If you want to understand the tech more you should start with the E language it is based on at erights.org. In the case of distributed GC the system understands that the network can fail and plans for this: http://www.erights.org/history/original-e/dgc/index.html (caveat: I know original E and have no direct knowledge of whether or not this bit has been implemented in Goblins)
I am quite familiar with E. Or used to be during my PhD. As far I recall, neither E's implementation nor spec cover distributed transactions, fault-tolerance or GC issues.
self-plug: you may want to checkout Manikin: it is very similar to Goblins, but with a clear transactional model.
https://odipar.github.io/manikin
https://odipar.github.io/manikin
Hi! So, first of all, I think I need to make the description of Goblins clearer. It says "A transactional, distributed actor model environment", but "transactional" and "distributed" are two separate components. This follows the "vat turn" communicating event loop model of E, where every message handling in an event loop is done in a transaction. It isn't meant to say distributed transactions. You can build distributed transactions a couple of ways though: as an abstraction on top of the distributed object system provided, or as a consensus model where a quorum of nodes agree on the order of messages for a deterministic vat/event loop.
By the way, the "open source version of E" does do distributed garbage collection, but it can't collect cycles across a network barrier. However, the original version of E did collect cycles across the network too: http://erights.org/history/original-e/dgc/
However, you needed hooks into the garbage collector for it to work, since you had to do some introspection of how things were rooted iirc. Mark S. Miller told me that originally they thought this could be fixed because Sun had promised to make Java open source, but they took a damn long time to do it. In the middle period, apparently the E folks said "please! just take this patch... you can have it even!" but Sun didn't merge it.
But Mark also told me that he learned not to push for this, because the cycles-across the network stuff turned out to not be common enough to bother, and you could build distributed acyclic gc using just weakrefs (and weak maps) and gc finalizer hooks, which are much more common than the rooting-introspection stuff.
Not sure if I got all that right, but there you go.
Anyway, I'll fix the tagline for Goblins for next release to make it clearer. Thanks for the feedback.
By the way, the "open source version of E" does do distributed garbage collection, but it can't collect cycles across a network barrier. However, the original version of E did collect cycles across the network too: http://erights.org/history/original-e/dgc/
However, you needed hooks into the garbage collector for it to work, since you had to do some introspection of how things were rooted iirc. Mark S. Miller told me that originally they thought this could be fixed because Sun had promised to make Java open source, but they took a damn long time to do it. In the middle period, apparently the E folks said "please! just take this patch... you can have it even!" but Sun didn't merge it.
But Mark also told me that he learned not to push for this, because the cycles-across the network stuff turned out to not be common enough to bother, and you could build distributed acyclic gc using just weakrefs (and weak maps) and gc finalizer hooks, which are much more common than the rooting-introspection stuff.
Not sure if I got all that right, but there you go.
Anyway, I'll fix the tagline for Goblins for next release to make it clearer. Thanks for the feedback.
What are the best use cases for distributed actor model solutions? Im just slightly familiar with it and it’s on my learning list. Anybody here have any experience with it? How about with Goblins?
The use case for actors is any time you want to distribute work across multiple nodes. In general, actors are a great concurrency model that simplifies a lot of the complexity of distributed programming.
For more info on "why", see:
https://doc.akka.io/docs/akka/current/typed/guide/actors-int...
https://www.microsoft.com/en-us/research/wp-content/uploads/...
For more info on "why", see:
https://doc.akka.io/docs/akka/current/typed/guide/actors-int...
https://www.microsoft.com/en-us/research/wp-content/uploads/...
I don't want to distract from this comment, because I think it's spot on, but I think another interesting use for actors is as one of the few sane concurrency models for mutation. Pony does this - objects are actors by default. Things can act concurrently without fear external to an object, but all mutations within a given object are serialized. Seems like a nice set of defaults.
Thanks, the first link answers some of my questions.
>Use of actors allows us to:
> Enforce encapsulation without resorting to locks.
> Use the model of cooperative entities reacting to signals, changing state, and sending signals to each other to drive the whole application forward.
> Stop worrying about an executing mechanism which is a mismatch to our world view.
Did anyone here use Goblins for any project and would like to share their experience?
>Use of actors allows us to:
> Enforce encapsulation without resorting to locks.
> Use the model of cooperative entities reacting to signals, changing state, and sending signals to each other to drive the whole application forward.
> Stop worrying about an executing mechanism which is a mismatch to our world view.
Did anyone here use Goblins for any project and would like to share their experience?
The author posted some interesting applications using Goblins (a video game and chat app). There is some interest in porting Goblins to Guile Scheme to integrate with GNU Shepherd (a service orchestrator) ... somehow.
I was recently trying to answer this question while playing around with Dapr (amazing project by the way).
So despite it being a solution that has existed for distributed systems since the 70s I was watching a talk from Microsoft on the topic where they would refer to actors as “cloud native objects” and the model itself as an ideal way to do OOP on distributed systems. The marketing feels a little cringe at first but the mental model works well.
I have an object which contains some state. Normally I would be responsible for managing the entire lifecycle of this object including resolving race conditions for DB writes and a million other problems that make distributed systems difficult. With many modern implementations of the actor model however (Microsoft have done a lot of interesting stuff here building Azure and Xbox games like Halo using Actors heavily) you don’t have to worry about any of this. You can just call the object as though you had unlimited RAM and it was in memory for you and race conditions were no longer a thing that you had to worry about.
This paper is a good intro to the topic I found https://www.microsoft.com/en-us/research/wp-content/uploads/...
So despite it being a solution that has existed for distributed systems since the 70s I was watching a talk from Microsoft on the topic where they would refer to actors as “cloud native objects” and the model itself as an ideal way to do OOP on distributed systems. The marketing feels a little cringe at first but the mental model works well.
I have an object which contains some state. Normally I would be responsible for managing the entire lifecycle of this object including resolving race conditions for DB writes and a million other problems that make distributed systems difficult. With many modern implementations of the actor model however (Microsoft have done a lot of interesting stuff here building Azure and Xbox games like Halo using Actors heavily) you don’t have to worry about any of this. You can just call the object as though you had unlimited RAM and it was in memory for you and race conditions were no longer a thing that you had to worry about.
This paper is a good intro to the topic I found https://www.microsoft.com/en-us/research/wp-content/uploads/...
> I was watching a talk from Microsoft on the topic where they would refer to actors as “cloud native objects” and the model itself as an ideal way to do OOP on distributed systems. The marketing feels a little cringe at first but the mental model works well.
Was that my Orleans Deep Dive video? https://youtu.be/R0ODfwU6MzQ
I started referring to Grains as Cloud Native Objects as a way to better convey them to people who aren't familiar already. My view is that Grains are different enough from Erlang/Akka style actors that lumping them under the Actor terminology can cause confusion. In the research publications, the term Virtual Actor is coined, which better describes them.
That video doesn't discuss it, but Orleans also supports distributed cross-actor ACID transactions, largely thanks to Phil Bernstein: https://www.microsoft.com/en-us/research/publication/transac... and more recently https://www.microsoft.com/en-us/research/publication/resurre....
Was that my Orleans Deep Dive video? https://youtu.be/R0ODfwU6MzQ
I started referring to Grains as Cloud Native Objects as a way to better convey them to people who aren't familiar already. My view is that Grains are different enough from Erlang/Akka style actors that lumping them under the Actor terminology can cause confusion. In the research publications, the term Virtual Actor is coined, which better describes them.
That video doesn't discuss it, but Orleans also supports distributed cross-actor ACID transactions, largely thanks to Phil Bernstein: https://www.microsoft.com/en-us/research/publication/transac... and more recently https://www.microsoft.com/en-us/research/publication/resurre....
Looks fun. How novel is it? Can I do these things already in some other language? (Esp. Haskell or Python -- my favorite language and my work language, respectively.)
If the version of CapTP we have is ported to other languages, then you could do some of it, but probably not all. Haskell might be better up for some, but Python won't be able to do some of the more interesting parts (like the time-travel support) quite as easily since it's a much more mutation-oriented environment. Still, you could get systems talking to each other over the network if a compatible CapTP was implemented.
The networked actor-model bit & CapTP goes back to E originally[1]. The other contemporary real-world protocol based on this design is capnproto rpc[2], which has implementations in several languages including both Haskell (of which I am the author) and Python.
The ergonomics of the Haskell implementation could use some work IMO, and I've got some ongoing refactoring work on a branch. But it does work, and folks are using it.
[1]: http://erights.org/
[2]: https://capnproto.org/
The ergonomics of the Haskell implementation could use some work IMO, and I've got some ongoing refactoring work on a branch. But it does work, and folks are using it.
[1]: http://erights.org/
[2]: https://capnproto.org/
[deleted]
--
[0] - https://gitlab.com/spritely/goblin-chat
[1] - https://octodon.social/@cwebber/105992747855007110