Ergo: Erlang/OTP patterns implemented in Golang(cloud.ergo.services)
cloud.ergo.services
Ergo: Erlang/OTP patterns implemented in Golang
https://cloud.ergo.services/#Framework
72 comments
This could be pretty cool for a real time trading system. Go would be pretty solid for the numerical stuff and then BEAM could be used to coordinate all the trading agents. I have written some Elixir modules that call out to C binary executables to do computations that I needed to be faster than what could be achieved with the Erlang/Elixir standard libraries, but had to perform the business logic that called out to these faster implementations in the Elixir code. Will be interesting to watch where this goes.
Would NX help your use case for computations in Elixir?
I haven't looked at it much. is Nx just like the equivalent of pandas DataFrame?
I could have some use for it but I haven't really checked it out.
I'm working on a trading system separately from my initial command on this HN post, and tried to pull in all the new fancy Elixir libraries for machine learning stuff (axon, xla, exla) but had some issues with the protobuf dependency locally.
I could have some use for it but I haven't really checked it out.
I'm working on a trading system separately from my initial command on this HN post, and tried to pull in all the new fancy Elixir libraries for machine learning stuff (axon, xla, exla) but had some issues with the protobuf dependency locally.
Nx is more like Numpy, Explorer is the one that provides data frames (closer to dplyr than pandas though). We provide precompiled assets for both, so hopefully it eases installation. :)
As someone without a lot of knowledge of GO and with only a trivial understanding of OTP/the gen_server impl, how would this add friction? What exactly is wrong with it and what would be a better alternative?
I go over this fairly deeply in https://www.jerf.org/iri/post/2930/ .
The Big Idea is that it's rarely a good idea to transliterate a library. You should translate a library. When I was done translating into Go, I found that what was idiomatic in Go was much simpler than Erlang. To some extent this is because Go simply can't do some things Erlang can do (in particular Erlang is one of the very short list of languages that can safely use "asynchronous exceptions", or to put it another way, can let one thread simply reach out and nuke another thread, so safely it can be incorporated into the core design of systems rather than treated as a dangerous special case)... but if that is the case, why port over the aspects of the Erlang design that are now dangling free in space, unconnected to anything?
The Big Idea is that it's rarely a good idea to transliterate a library. You should translate a library. When I was done translating into Go, I found that what was idiomatic in Go was much simpler than Erlang. To some extent this is because Go simply can't do some things Erlang can do (in particular Erlang is one of the very short list of languages that can safely use "asynchronous exceptions", or to put it another way, can let one thread simply reach out and nuke another thread, so safely it can be incorporated into the core design of systems rather than treated as a dangerous special case)... but if that is the case, why port over the aspects of the Erlang design that are now dangling free in space, unconnected to anything?
Maybe the answer to the challenge of asynchronous exceptions is that “threads” are not well suited to robust parallelism.
Certainly, I reached this conclusion in the past. A better paradigm is the “process”. Unlike a thread a process can be killed arbitrarily (your asynchronous exception), and completely safely (all resources are released etc). Software can be composed of a collection of processes and thus be much more robust (e.g. Chrome). It’s great that Erlang built this into the language itself, but any language can leverage processes for the same effect.
As an aside, Lunatic (leveraging wasm to create “process” like sandboxes inside one process), looks very promising as a way of leveraging the power of process isolation without all the fuss.
Certainly, I reached this conclusion in the past. A better paradigm is the “process”. Unlike a thread a process can be killed arbitrarily (your asynchronous exception), and completely safely (all resources are released etc). Software can be composed of a collection of processes and thus be much more robust (e.g. Chrome). It’s great that Erlang built this into the language itself, but any language can leverage processes for the same effect.
As an aside, Lunatic (leveraging wasm to create “process” like sandboxes inside one process), looks very promising as a way of leveraging the power of process isolation without all the fuss.
I took a pretty good distributed computing course in college, and I loved threads when they were 'new' to most people, my way of representing information in my head was pretty amenable to thinking about concurrent code. But what I found over time is that I was seeing a lot of bugs (and not seeing a few), but I could never manage to explain to other people how to avoid the problem. I could show them the code change that fixed it, and they mostly understood, but I spent way too much time following people around doing cleanup work.
Shared state threads are simply not fit for human consumption, and if you're in the (for argument's sake) 5% who understand them well, you should consider if you really want to inflict them on your coworkers. Even if you agree to write the 2-10% of critical code that needs esoteric language constructs, your coworkers are still likely to have to step through your code and the longer they're in a cloud of unfamiliar code the harder it is for them to connect cause and effect and fix their own bugs.
You end up playing yourself by creating a class of bugs that preempt you from other work you may have been enjoying doing or is potentially blocking other people.
A functional core architecture avoids a lot of these issues because you can get away with a lot more threading when you have a block of code that is side effect free. You still have to watch PR closely for people violating that invariant, though.
Shared state threads are simply not fit for human consumption, and if you're in the (for argument's sake) 5% who understand them well, you should consider if you really want to inflict them on your coworkers. Even if you agree to write the 2-10% of critical code that needs esoteric language constructs, your coworkers are still likely to have to step through your code and the longer they're in a cloud of unfamiliar code the harder it is for them to connect cause and effect and fix their own bugs.
You end up playing yourself by creating a class of bugs that preempt you from other work you may have been enjoying doing or is potentially blocking other people.
A functional core architecture avoids a lot of these issues because you can get away with a lot more threading when you have a block of code that is side effect free. You still have to watch PR closely for people violating that invariant, though.
I like threads. I like threads a lot. I like them a lot better than promises. But I hate shared state threads, as you say. Threads should "own" as much data as they can, as strongly as they can.
Erlang is nice in that it codifies this. I write my Go in this style as much as I can, but enforcement is good. I wish Pony was doing better.
Erlang is nice in that it codifies this. I write my Go in this style as much as I can, but enforcement is good. I wish Pony was doing better.
Shouldn't the patterns transcend the language implementation? The fact that this exists in Go should be proof of that.
Go’s concurrency model and Erlangs concurrency model are fundamentally the same.
Erlangs mailbox and Go’s channels are the same primitive.
And Erlangs processes and Go’s Goroutines are also the same primitive.
The big difference between the languages (from a concurrency perspective, there are other differences) is that idomatic Go is generally built around spinning off many short lived goroutines passing objects back and forth via channels, whereas Erlang generally spins off many long lived processes that represent objects, communicating mutation requests via mailboxes.
But you can do go style concurrency in Erlang, and Erlang style concurrency in go. You’ll just be swimming upstream the entire time, because your concurrency style won’t mesh well with any of the libraries out there which will be written in the idiomatic go/Erlang concurrency style.
So the idea you can create an Erlang style concurrency framework in Go isn’t actually surprising. Go and Erlang concurrency models are just opposite sides of the same coin.
Erlangs mailbox and Go’s channels are the same primitive.
And Erlangs processes and Go’s Goroutines are also the same primitive.
The big difference between the languages (from a concurrency perspective, there are other differences) is that idomatic Go is generally built around spinning off many short lived goroutines passing objects back and forth via channels, whereas Erlang generally spins off many long lived processes that represent objects, communicating mutation requests via mailboxes.
But you can do go style concurrency in Erlang, and Erlang style concurrency in go. You’ll just be swimming upstream the entire time, because your concurrency style won’t mesh well with any of the libraries out there which will be written in the idiomatic go/Erlang concurrency style.
So the idea you can create an Erlang style concurrency framework in Go isn’t actually surprising. Go and Erlang concurrency models are just opposite sides of the same coin.
Idiomatic Erlang and Go are fundamentally different. In Erlang there is one mailbox per process and multiplexing is archived via polymorphic messages. In Go there can be multiple channels and one multiplexes them using the select statement.
Erlang message queue is a priority queue, while in Go a channel is a strict FIFO. In fact implementing a priority queue on top of channels in Go is rather hard. A much better approach is to code a priority queue directly using mutexes.
In Erlang (at least the last time I checked this) the message queue is unbound while Go gives a lot of control over the channel queue size.
As was already pointed, Erlang also supports reliable cancelation of threads, while in Go a thread has to be coded explicitly to support cancelation.
All of this leads to very different style of multithreaded code in the two languages.
Erlang message queue is a priority queue, while in Go a channel is a strict FIFO. In fact implementing a priority queue on top of channels in Go is rather hard. A much better approach is to code a priority queue directly using mutexes.
In Erlang (at least the last time I checked this) the message queue is unbound while Go gives a lot of control over the channel queue size.
As was already pointed, Erlang also supports reliable cancelation of threads, while in Go a thread has to be coded explicitly to support cancelation.
All of this leads to very different style of multithreaded code in the two languages.
> In Erlang (at least the last time I checked this) the message queue is unbound while Go gives a lot of control over the channel queue size.
You can bound the queue by proxy, becsause there is a max_heap_size process flag you can set. If your process is configured to use on-heap messages, and you have a max_heap_size, that provides a bound, although maybe not in the units you'd like.
You can also do things where your process might check it's own queue length (which is very cheap) and discard messages if it's too long. This doesn't work if your process gets stuck, or if discarding is still slower than the incoming message rate.
Another option is a process that lists all processes and finds their queue length (not so cheap) and kills process above a threshold.
You could also modify BEAM to do what you want. When I was at WhatsApp, we had a patch so you could do process_flag(flush_message_queue, N) and it would drop N messages (or all of them if N == 0), and this worked with process_flag/3 if you wanted to drop messages from the queue of another process.
Of course, that sort of thing doesn't fit the BEAM messaging guarantees, so unlikely to be accepted upstream. (See also the message prepending patch, to put a message to another process at the front of its mailbox)
You can bound the queue by proxy, becsause there is a max_heap_size process flag you can set. If your process is configured to use on-heap messages, and you have a max_heap_size, that provides a bound, although maybe not in the units you'd like.
You can also do things where your process might check it's own queue length (which is very cheap) and discard messages if it's too long. This doesn't work if your process gets stuck, or if discarding is still slower than the incoming message rate.
Another option is a process that lists all processes and finds their queue length (not so cheap) and kills process above a threshold.
You could also modify BEAM to do what you want. When I was at WhatsApp, we had a patch so you could do process_flag(flush_message_queue, N) and it would drop N messages (or all of them if N == 0), and this worked with process_flag/3 if you wanted to drop messages from the queue of another process.
Of course, that sort of thing doesn't fit the BEAM messaging guarantees, so unlikely to be accepted upstream. (See also the message prepending patch, to put a message to another process at the front of its mailbox)
It's not about concurrency, it's about failure domains.
The go VM doesn't natively let you take out monitors/links, so if "the right thing to do" when some part of a set of concurrent tasks fails is to "just give up" (suppose task 3/5 reaches out to an external service and comes back with a 500 error)... Handling the teardown of the whole thing (and potentially rollback state) is gnarly in go. In Erlang it's often zero lines of code. In Elixir, this architecture let the anointed db engine roll back a partially completed database transaction by default (0loc) in such a scenario.
The go VM doesn't natively let you take out monitors/links, so if "the right thing to do" when some part of a set of concurrent tasks fails is to "just give up" (suppose task 3/5 reaches out to an external service and comes back with a 500 error)... Handling the teardown of the whole thing (and potentially rollback state) is gnarly in go. In Erlang it's often zero lines of code. In Elixir, this architecture let the anointed db engine roll back a partially completed database transaction by default (0loc) in such a scenario.
There’s a very interesting aspect to what you mention here that is often overlooked. In Erlang every process has its own garbage collection. Every process has a stack and a heap. Stopping a process, or a process dying, automatically cleans up any resources used.
So, the big difference between Go and Erlang style concurrency is that go style concurrency doesn't give you a readily available reliable way to keep an eye on other goroutines, as I understand it, where there are mechanisms in Erlang (links and monitors come to mind) to help accomplish that, and, in fact, it's one of the core parts of the Erlang model of concurrency
It does not give you a way to reliably track arbitrary goroutines that "this" goroutine (for whatever that may be) wants to track, the way an Erlang process can just "link" to anything it is capable of naming the PID for.
However, you can construct a reliable mechanism where one goroutine can start another and know whether or not the one it started has failed by using the available primitives, as I did in https://github.com/thejerf/suture . It's an easier problem since there's no cluster and no network that can get in the way. I've also done the exercise for the network case: https://pkg.go.dev/github.com/thejerf/reign#Address.OnCloseN... but that only functions within the network defined by that library because, again, it just isn't arbitrarily possible.
(I suppose it's relevant to some of my other comments to point out that I've also implemented basically Erlang-style concurrency in Go, with network, but as a relatively idiomatic translation rather than a blind one.)
However, you can construct a reliable mechanism where one goroutine can start another and know whether or not the one it started has failed by using the available primitives, as I did in https://github.com/thejerf/suture . It's an easier problem since there's no cluster and no network that can get in the way. I've also done the exercise for the network case: https://pkg.go.dev/github.com/thejerf/reign#Address.OnCloseN... but that only functions within the network defined by that library because, again, it just isn't arbitrarily possible.
(I suppose it's relevant to some of my other comments to point out that I've also implemented basically Erlang-style concurrency in Go, with network, but as a relatively idiomatic translation rather than a blind one.)
I actually think it’s rather difficult to mimic Erlang message passing in go, but trivial (and not uncommon) to mimic go channels in Erlang.
"Go’s concurrency model and Erlangs concurrency model are fundamentally the same."
See https://news.ycombinator.com/item?id=34564228 .
At the 30,000 foot view, yes. On the ground, no.
See https://news.ycombinator.com/item?id=34564228 .
At the 30,000 foot view, yes. On the ground, no.
I cautiously agree, the only point I'd make from my own perspective is that the other reference point for this would be Akka, which was/is a behemoth of boilerplate and torturous implementation, so it begs the question of whether its a pattern that just has friction and is it worth trying to reinvent again.
We did a book reading of a Scala book with a bunch of senior Java developers, and even they thought Scala was too baroque. If Java developers think something is too complicated you've really, really gone off the deep end.
I've heard that Scala has spent a lot of time since then trying to simplify itself, so I can't speak to now, but I do know that Akka came into being when Scala was at its worst. When in Rome, do as the Romans.
I've heard that Scala has spent a lot of time since then trying to simplify itself, so I can't speak to now, but I do know that Akka came into being when Scala was at its worst. When in Rome, do as the Romans.
> You can use this to join a Go system to your Erlang cluster directly.
That’s amazing
That’s amazing
There was another recent Erlang post which posited the suggestion that the power of erlang was not in lightweight processes and message parsing, but in generic components erlang calls behaviours. These generic patterns are what allows for building reliable distributed systems, and Joe Armstrong wrote a PhD thesis about it.
I gave an anecdote of having built a c++ framework that unwittingly followed the same principles for “big corp” long ago (convergent design), and posited that the same concepts could probably be built in any language. Here it is for go. So is it a good thing?
Previously I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure. The problem I have is that as you start looking at these higher level patterns involving distributed computing, the landscape of frameworks seems to explode. Is this because the erlang/otp concepts are insufficient or outdated?
I gave an anecdote of having built a c++ framework that unwittingly followed the same principles for “big corp” long ago (convergent design), and posited that the same concepts could probably be built in any language. Here it is for go. So is it a good thing?
Previously I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure. The problem I have is that as you start looking at these higher level patterns involving distributed computing, the landscape of frameworks seems to explode. Is this because the erlang/otp concepts are insufficient or outdated?
> it would be good to capture these core concepts in standardized libraries
A lot of Erlang's performance and robustness characteristics are enabled at both the language level and BEAM VM level (e.g., immutability, memory model, processes, supervision trees, lightweight pre-emption). Whole classes of problems like deadlocks, memory management, and GC pauses go away, enabling an ergonomic, expressive language and runtime for concurrent systems.
A lot of Erlang's performance and robustness characteristics are enabled at both the language level and BEAM VM level (e.g., immutability, memory model, processes, supervision trees, lightweight pre-emption). Whole classes of problems like deadlocks, memory management, and GC pauses go away, enabling an ergonomic, expressive language and runtime for concurrent systems.
"Is this because the erlang/otp concepts are insufficient or outdated?"
Insufficient or outdated is a pretty loaded term.
What they are is really, really tied to the Erlang worldview, of immutable functions, mailbox APIs that can receive messages out of order, the way they have "behaviors" as a really limited kind of class system, and other details of Erlang. This isn't a criticism, it's really a good thing. An Erlang library should harmonize with Erlang, after all.
But when you port them into environments that don't have all those characteristics, the optimal library that harmonizes with that environment looks different. In Go, for instance, rather than the whole gen_server thing, the idiomatic approach to this sort of thing tends to look a lot more like:
Another reason is that while Go and Erlang both have fairly weak type systems, they are weak in very different ways. It is idiomatic in Erlang to label what kind of message something is by making it the first atom in a tuple. Then you can use Erlang's pattern matching in functions to implement many different message types based on what you send to the gen_server. This ports into Go terribly. Go does it in very different ways.
"I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure."
No, it really doesn't work. Even two languages in the same basic family can end up with significant differences in patterns (witness Ruby versus Python, for instance, virtually the same language, significant difference in what is idiomatic). Trying to stretch a library from a language that is based on a super weak type system (on purpose; it is part of the Erlang solution to cross-version communication), immutability, pattern matching, and mailboxes into a language with a weak type system in a different way, mutability, conventional function calls, channels (very different than mailboxes), and all the other relevant differences may be theoretically possible, but it will always result in foreign environments in the language.
You can actually see this in the real world by the way that C is the de facto cross-language in-process bus on Linux. It bends everything that has to use it. Personally I consider it one of the biggest impediments to a new language getting off the ground because of the design decisions it forces on a new language at a young age. We are, slowly, getting past this in a variety of ways, but it's hard.
Insufficient or outdated is a pretty loaded term.
What they are is really, really tied to the Erlang worldview, of immutable functions, mailbox APIs that can receive messages out of order, the way they have "behaviors" as a really limited kind of class system, and other details of Erlang. This isn't a criticism, it's really a good thing. An Erlang library should harmonize with Erlang, after all.
But when you port them into environments that don't have all those characteristics, the optimal library that harmonizes with that environment looks different. In Go, for instance, rather than the whole gen_server thing, the idiomatic approach to this sort of thing tends to look a lot more like:
for {
select {
case msg1 := me.Chan1:
me.handleMsg1(msg1)
case msg2 := me.Chan2:
me.handleMsg2(msg2)
// etc.
}
}
Another reason is that while Go and Erlang both have fairly weak type systems, they are weak in very different ways. It is idiomatic in Erlang to label what kind of message something is by making it the first atom in a tuple. Then you can use Erlang's pattern matching in functions to implement many different message types based on what you send to the gen_server. This ports into Go terribly. Go does it in very different ways.
"I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure."
No, it really doesn't work. Even two languages in the same basic family can end up with significant differences in patterns (witness Ruby versus Python, for instance, virtually the same language, significant difference in what is idiomatic). Trying to stretch a library from a language that is based on a super weak type system (on purpose; it is part of the Erlang solution to cross-version communication), immutability, pattern matching, and mailboxes into a language with a weak type system in a different way, mutability, conventional function calls, channels (very different than mailboxes), and all the other relevant differences may be theoretically possible, but it will always result in foreign environments in the language.
You can actually see this in the real world by the way that C is the de facto cross-language in-process bus on Linux. It bends everything that has to use it. Personally I consider it one of the biggest impediments to a new language getting off the ground because of the design decisions it forces on a new language at a young age. We are, slowly, getting past this in a variety of ways, but it's hard.
This is interesting. When I think of a programming language I tend to consider it a tool to express an idea. The ideas from Erlang Behaviours (not being an Erlang developer), seemed to be a fairly generalisable. So I would have thought the language is not so important.
The example of C as the de facto cross-language standard (it’s not just in Linux), is also interesting. Creating a C interface imposes on a library author a set of restrictions in expression, but is anything truely lost in doing this?
I am much more interested in the concepts/ideas being expressed than I am the language being used to express it.
The example of C as the de facto cross-language standard (it’s not just in Linux), is also interesting. Creating a C interface imposes on a library author a set of restrictions in expression, but is anything truely lost in doing this?
I am much more interested in the concepts/ideas being expressed than I am the language being used to express it.
Re the C part. Yes. A lot. You lose all the expressivity of your type systems and the way you manage memory. Plus you have to deal with the lack of specification of the C ABI.
See https://faultlore.com/blah/c-isnt-a-language/
See https://faultlore.com/blah/c-isnt-a-language/
The example for OPEN(2) isn't for Linux, it's for glibc. Languages are free to implement system calls and bypass glibc if they wish. This isn't the case on BSD as the libc implementation is the official interface, but nobody is forcing anyone to write C. The fact is, lot's of interesting software is written in C, and most people want to use it rather than re-implement things that have worked for decades.
Yes and no. In a way, yes we are forced to interact with C. The syscall are in C and the ubiquitiness of C has made it de facto the thing you need to limit yourself to.
syscall's are not in C. They are an interrupt instruction and some parameters in registers, and these can be emitted in a variety of languages. You only need to use C if you want to use libc of the platform.
You loose the expressiveness, no argument there. The poetry is not visible. Still, that’s just a matter for a wrapper in the consuming language.
E.g. beautiful Rust wrapper -> “Ugly C interface” -> “Ugly C interop layer” -> beautiful Go library
I mean, it’s a pain to maintain “ugly C interop”, but as these tend to be built after the fact anyway, other than cost I don’t see anything lost.
E.g. beautiful Rust wrapper -> “Ugly C interface” -> “Ugly C interop layer” -> beautiful Go library
I mean, it’s a pain to maintain “ugly C interop”, but as these tend to be built after the fact anyway, other than cost I don’t see anything lost.
Nope, you cannot because there is information lost in the middle that means that some of what you wanted to express in order to make it beautiful in Go is not there anymore. The medium is lossy and you cannot recover from it.
What is beautiful in one language is often hideous in another. Translation is required.
Much like with human translations of poetry. A good translation is not word for word, the translator captures the essence and re-expresses it in the other language. When done well, it is frankly a work of genius.
So the C interface is just a medium a “translation dictionary” if you will. To do it well requires the author of the wrapper to express the ideas in a form palatable to users of the other language. I get that this is not always done well, requires a bunch of work, and may raise the question “why not re-write?”. I’m happy to consider better alternatives.
Much like with human translations of poetry. A good translation is not word for word, the translator captures the essence and re-expresses it in the other language. When done well, it is frankly a work of genius.
So the C interface is just a medium a “translation dictionary” if you will. To do it well requires the author of the wrapper to express the ideas in a form palatable to users of the other language. I get that this is not always done well, requires a bunch of work, and may raise the question “why not re-write?”. I’m happy to consider better alternatives.
You realise that the medium language is incapable of expressing that?
And yes, there are wip to a superset ABI
And yes, there are wip to a superset ABI
They are fairly generalizable, to a point.
The thing about Erlang behaviors is that they rely on several other pieces of Erlang to work well.
One big one is being able to be notified when another process goes down, or is aborted.
The other big one is being able to reason about state in the face of such failure.
Erlang decided to go with immutable data structures, shared nothing processes, and the OTP behaviors generally expect you to decompose the behavior of the various bits into functions that represent a single atomic step.
The more of those properties that you don't share with Erlang, the harder it will be to adopt OTP style semantics in your system.
The thing about Erlang behaviors is that they rely on several other pieces of Erlang to work well.
One big one is being able to be notified when another process goes down, or is aborted.
The other big one is being able to reason about state in the face of such failure.
Erlang decided to go with immutable data structures, shared nothing processes, and the OTP behaviors generally expect you to decompose the behavior of the various bits into functions that represent a single atomic step.
The more of those properties that you don't share with Erlang, the harder it will be to adopt OTP style semantics in your system.
The other thing about behaviours is... Especially if you have a synchronous API, you can mock it with a stateful system, for example calling out to a GenServer or calling over the network. This is why GenServer.call doesn't emit an error tuple; an network or interprocess error by default is considered to be unrecoverable. In most other systems you'll fumble around with colored functions transitioning between a sync system or async call, or, have to do annoying error handling, or even worse stack unwinding with an exception or get stuck in a panic.
In Erlang, a sync API behavior can safely have a failable async implementation, and that is powerful
In Erlang, a sync API behavior can safely have a failable async implementation, and that is powerful
Can you elaborate on how are Erlang process mailboxes different than channels, please? I view them as hear-identical but I haven't thought deeply about it.
Me too! I don’t know Erlang enough, but Golang channels accept anything, including mutable data, file descriptors etc. If you want to send to different machines you need glue and refactoring in order to get some similar behavior, since channels are in-process only.
My understanding is that Erlang require messages to be self-contained/immutable (even serializable?) which would make it possible to simply move an actor (process) to a different machine, or even multiple machines? If you know Erlang, please fact check!
My understanding is that Erlang require messages to be self-contained/immutable (even serializable?) which would make it possible to simply move an actor (process) to a different machine, or even multiple machines? If you know Erlang, please fact check!
Messages between Erlang processes are simply valid Erlang terms. That is, they can be lists, tuples, integers, atoms, other pids (process identifiiers), strings, and so on.
However, Erlang and Elixir's AST is also composed of valid Erlang terms, so as I understand. you can send entire Erlang/Elixir programs as messages.
To answer your question, Yes, it is possible, but there is no "Move process to node" call. However, if the process is built with a feature for migration, you can certainly do it by sending the function of the process and its state to another node and arrange for a spawn there. To get the identity of the process right, you will need to use either the global process registry or gproc, as the process will change pid.
There are other considerations as well: The process might be using an ETS table whose data are not present on the other node, or it may have stored stuff in the process dictionary (state from the random module comes to mind)
However, Erlang and Elixir's AST is also composed of valid Erlang terms, so as I understand. you can send entire Erlang/Elixir programs as messages.
To answer your question, Yes, it is possible, but there is no "Move process to node" call. However, if the process is built with a feature for migration, you can certainly do it by sending the function of the process and its state to another node and arrange for a spawn there. To get the identity of the process right, you will need to use either the global process registry or gproc, as the process will change pid.
There are other considerations as well: The process might be using an ETS table whose data are not present on the other node, or it may have stored stuff in the process dictionary (state from the random module comes to mind)
A couple of things come to mind. Sending a message to a pid does not block the sender until message is consumed. Each pid has its own mailbox. If you attempt this in go (a channel per a unit comparable to a pid) it will fail pretty fast. Other than that, pretty similar from user‘s perspective. A process can reside on any node of the cluster. You don’t care about hostnames, ports… just use a node name: https://www.erlang.org/doc/reference_manual/distributed.html. You can spawn a process on another node and send anything that’s a valid Erlang term to it as if it was a local process.
Erlang mailboxes:
1. Are tied to processes. They are not first-class values. (The processes are, and the mailboxes go with them.)
2. Can receive messages out of order using pattern matching. This is critically used all over the place for RPC simulations; you send a message out to some other process, and then receive the reply by matching on the mailbox. You may receive other messages in the meantime in that mailbox, but the RPC process is safe because the block waiting for the answer will keep waiting for that message, and the rest will stay there for later reception.
3. Are zero-or-one delivery. Messages going to the same OS process are reliable enough to be treated as reliable exactly-once delivery but you are not really supposed to count on that. (This is one of the ways you can write an Erlang system and you accidentally make it so it can't run on a cluster.) Messages going across nodes may not arrive because the network may eat them. If you're really excited you can examine a process ID to see if it's local but you're generally not supposed to do that.
As part of this, mailboxes are fundamentally asynchronous. You can not wait on "the message has arrived at the other end" because in a network environment this isn't even a well-defined concept. The only way to know that a message arrived is to be explicitly sent an acknowledgement, an acknowledgement that may itself get lost (Byzantine general problem).
4. Send Erlang terms, only Erlang terms, and exactly Erlang terms. Erlang terms are an internal dynamically-typed language with no support for user-defined types. This is important because this is how Erlang is built around upgradeability and having multiple versions of things in the same cluster. Since there are no user-defined types, you never have problems with having mismatched type definitions. (You can still have mismatched data, of course; if a format changes, it changes. But the problem is at least alleviated by not supported complicated user defined types. It is, arguably and in my opinion, a throwing the baby out with the bathwater situation, but I do admit against interest (as the lawyers say) that practically it works out reasonably well. Especially since the architect of the system really ought to know this is how it works up front.)
Because of the dynamically-typed nature, they are effectively untyped. Any message can be sent to a mailbox.
5. Are many-to-one communication, across an entire Erlang cluster. A given mailbox is only visible to one Erlang process; it is part of that process. Again, the process ID is a first-class value that can be passed around but the mailbox is not.
6. There are debug mechanisms that allow you to look into a mailbox live, on the cluster's shell. You can see what is currently in there. You really shouldn't be using these debug facilities as part of your system, but you can use them as a devops sort of thing. (As hard as I've been on Erlang, the devops story is pretty powerful for fixing broken systems. That said, the dynamic types means I had to fix more broken systems live than I ever have for Go; I haven't missed this because my Go systems generally don't break. Still, if they are going to break, Erlang has a lot of tools for dealing with it live.)
Go channels:
1. Are first-class values not tied to any goroutine. One goroutine may create a channel and pass it off to two others who will communicate on it, and they may further pass it around.
2. Are intrinsically ordered, at least in the sense that receivers can't go poking along the channel to see what they want to pull out of it.
However, an aspect of Go channels is that with a "select" statement, a single goroutine can wait on an arbitrary combination of "things I'm trying to send" and "things I'm trying to receive". This is entirely unlike pattern matching on a mailbox and is probably a really good example of the way you need solutions to certain communication problems, but they don't have to be the exact Erlang solution in order to work. Complicated communications scenarios that Erlang might achieve with selective matching can be done in Go with multiple channels multiplexed with a select. There are tradeoffs in either direction and it isn't clear to me one dominates the other at all.
3. Are synchronous exactly-once delivery. This further implies they only work on a single machine, and in Go they only work within a single process. This further implies that there is no such thing as a "network channel" in Go. You can, of course, have all sorts of things that sort of look like channels that work over a network, but it is fundamentally impossible to have a channel (of the "chan" type that can participate in the "select" statement) that goes over the network because no network can maintain the properties required for a Go channel.
It is also in general guaranteed that if you proceed past a send on a channel, that some other goroutine has received the value. This makes it useful for synchronization.
(There are buffered channels that can hold a certain fixed number of values without having actually been sent, but in general I think they should be treated as exceptions precisely because losing this property is a bigger deal that people often realize. A lot of things in Go are built on it. Contrary to popular belief, unbuffered channels are not asynchronous, because they are fixed size. They're just asynchronous, up to a certain point. Erlang mailboxes are asynchronous, until you run out of memory, which is not unheard of or impossible but isn't terribly common, especially if you follow the normal OTP patterns.)
4. Are typed. Each channel sends exactly one type of Go value, though this value can be an interface value meaning it can theoretically send multiple concrete types. (Generally I define the channel with the type I want though there is the occasional use case where I have a channel with a closed interface that basically uses the interface value like a sum type. I am still not sure whether this is better or worse that having a channel per possible type, and I've been doing this for a long time. I'm still going back and forth.)
5. Are many-to-many communication, isolated to a single OS process. It is perfectly legal and valid to have a single channel value that has dozens of producers and dozens of consumers. Performance implications are related to the rate these goroutines are trying to communicate over; if, for instance, at low rates it's no problem at all.
6. Are completely opaque, even within Go. There is no "peek", which would after all break the guarantee that if an unbuffered channel has had a "send" that there has been a corresponding "receive".
Contra another comment I see, no, you can not implement one in terms of the other. You can get close-ish, but there are certain aspects of them that simply do not cross, notably their sync/async nature, their differing network transparencies, and the inability of an Erlang mailbox to be "many-to-many", particularly in the way the "many-to-many" still guarantees "exactly once" delivery. (You can set up certain structures in Erlang that get close to this, but no matter what you do, you can not build any abstraction on a zero-or-one delivery system to create an exactly-one delivery system.)
You can solve almost any problem you have with either of these. You can use either to sort of get close to the other, but in both directions you'll sacrifice significant native capabilities in the process. It's really an intriguing study in how the solution to very similar problems can almost intertwine like an Asclepius staff, twisting around the same central pole while having basically no overlap. And again, it's not clear to me that either is "better"; both have small parts of the problem space where they are better than the other, both solve the vast bulk of problems you'll encounter just fine.
1. Are tied to processes. They are not first-class values. (The processes are, and the mailboxes go with them.)
2. Can receive messages out of order using pattern matching. This is critically used all over the place for RPC simulations; you send a message out to some other process, and then receive the reply by matching on the mailbox. You may receive other messages in the meantime in that mailbox, but the RPC process is safe because the block waiting for the answer will keep waiting for that message, and the rest will stay there for later reception.
3. Are zero-or-one delivery. Messages going to the same OS process are reliable enough to be treated as reliable exactly-once delivery but you are not really supposed to count on that. (This is one of the ways you can write an Erlang system and you accidentally make it so it can't run on a cluster.) Messages going across nodes may not arrive because the network may eat them. If you're really excited you can examine a process ID to see if it's local but you're generally not supposed to do that.
As part of this, mailboxes are fundamentally asynchronous. You can not wait on "the message has arrived at the other end" because in a network environment this isn't even a well-defined concept. The only way to know that a message arrived is to be explicitly sent an acknowledgement, an acknowledgement that may itself get lost (Byzantine general problem).
4. Send Erlang terms, only Erlang terms, and exactly Erlang terms. Erlang terms are an internal dynamically-typed language with no support for user-defined types. This is important because this is how Erlang is built around upgradeability and having multiple versions of things in the same cluster. Since there are no user-defined types, you never have problems with having mismatched type definitions. (You can still have mismatched data, of course; if a format changes, it changes. But the problem is at least alleviated by not supported complicated user defined types. It is, arguably and in my opinion, a throwing the baby out with the bathwater situation, but I do admit against interest (as the lawyers say) that practically it works out reasonably well. Especially since the architect of the system really ought to know this is how it works up front.)
Because of the dynamically-typed nature, they are effectively untyped. Any message can be sent to a mailbox.
5. Are many-to-one communication, across an entire Erlang cluster. A given mailbox is only visible to one Erlang process; it is part of that process. Again, the process ID is a first-class value that can be passed around but the mailbox is not.
6. There are debug mechanisms that allow you to look into a mailbox live, on the cluster's shell. You can see what is currently in there. You really shouldn't be using these debug facilities as part of your system, but you can use them as a devops sort of thing. (As hard as I've been on Erlang, the devops story is pretty powerful for fixing broken systems. That said, the dynamic types means I had to fix more broken systems live than I ever have for Go; I haven't missed this because my Go systems generally don't break. Still, if they are going to break, Erlang has a lot of tools for dealing with it live.)
Go channels:
1. Are first-class values not tied to any goroutine. One goroutine may create a channel and pass it off to two others who will communicate on it, and they may further pass it around.
2. Are intrinsically ordered, at least in the sense that receivers can't go poking along the channel to see what they want to pull out of it.
However, an aspect of Go channels is that with a "select" statement, a single goroutine can wait on an arbitrary combination of "things I'm trying to send" and "things I'm trying to receive". This is entirely unlike pattern matching on a mailbox and is probably a really good example of the way you need solutions to certain communication problems, but they don't have to be the exact Erlang solution in order to work. Complicated communications scenarios that Erlang might achieve with selective matching can be done in Go with multiple channels multiplexed with a select. There are tradeoffs in either direction and it isn't clear to me one dominates the other at all.
3. Are synchronous exactly-once delivery. This further implies they only work on a single machine, and in Go they only work within a single process. This further implies that there is no such thing as a "network channel" in Go. You can, of course, have all sorts of things that sort of look like channels that work over a network, but it is fundamentally impossible to have a channel (of the "chan" type that can participate in the "select" statement) that goes over the network because no network can maintain the properties required for a Go channel.
It is also in general guaranteed that if you proceed past a send on a channel, that some other goroutine has received the value. This makes it useful for synchronization.
(There are buffered channels that can hold a certain fixed number of values without having actually been sent, but in general I think they should be treated as exceptions precisely because losing this property is a bigger deal that people often realize. A lot of things in Go are built on it. Contrary to popular belief, unbuffered channels are not asynchronous, because they are fixed size. They're just asynchronous, up to a certain point. Erlang mailboxes are asynchronous, until you run out of memory, which is not unheard of or impossible but isn't terribly common, especially if you follow the normal OTP patterns.)
4. Are typed. Each channel sends exactly one type of Go value, though this value can be an interface value meaning it can theoretically send multiple concrete types. (Generally I define the channel with the type I want though there is the occasional use case where I have a channel with a closed interface that basically uses the interface value like a sum type. I am still not sure whether this is better or worse that having a channel per possible type, and I've been doing this for a long time. I'm still going back and forth.)
5. Are many-to-many communication, isolated to a single OS process. It is perfectly legal and valid to have a single channel value that has dozens of producers and dozens of consumers. Performance implications are related to the rate these goroutines are trying to communicate over; if, for instance, at low rates it's no problem at all.
6. Are completely opaque, even within Go. There is no "peek", which would after all break the guarantee that if an unbuffered channel has had a "send" that there has been a corresponding "receive".
Contra another comment I see, no, you can not implement one in terms of the other. You can get close-ish, but there are certain aspects of them that simply do not cross, notably their sync/async nature, their differing network transparencies, and the inability of an Erlang mailbox to be "many-to-many", particularly in the way the "many-to-many" still guarantees "exactly once" delivery. (You can set up certain structures in Erlang that get close to this, but no matter what you do, you can not build any abstraction on a zero-or-one delivery system to create an exactly-one delivery system.)
You can solve almost any problem you have with either of these. You can use either to sort of get close to the other, but in both directions you'll sacrifice significant native capabilities in the process. It's really an intriguing study in how the solution to very similar problems can almost intertwine like an Asclepius staff, twisting around the same central pole while having basically no overlap. And again, it's not clear to me that either is "better"; both have small parts of the problem space where they are better than the other, both solve the vast bulk of problems you'll encounter just fine.
> As hard as I've been on Erlang, the devops story is pretty powerful for fixing broken systems. That said, the dynamic types means I had to fix more broken systems live than I ever have for Go; I haven't missed this because my Go systems generally don't break. Still, if they are going to break, Erlang has a lot of tools for dealing with it live.
First off, I don't think you're being hard on Erlang, meerly truthful and clearly informed. Thank you for your balanced presentation.
I think this may be true (but hard to quantify) that things go wrong more often in an Erlang system than a Go system; but failure in Erlang tends to be much less painful than other environments and making changes can be much quicker and less costly. The primary reason to spend effort to avoid failure is because failure is costly; but if failure isn't costlh, you don't need to preemptively avoid it.
Of course, you can deploy your Erlang system with a lengthy process and ignore the flexibility and danger (and fun!) of hotloading, and in that case, finding failures before deployment is more important.
First off, I don't think you're being hard on Erlang, meerly truthful and clearly informed. Thank you for your balanced presentation.
I think this may be true (but hard to quantify) that things go wrong more often in an Erlang system than a Go system; but failure in Erlang tends to be much less painful than other environments and making changes can be much quicker and less costly. The primary reason to spend effort to avoid failure is because failure is costly; but if failure isn't costlh, you don't need to preemptively avoid it.
Of course, you can deploy your Erlang system with a lengthy process and ignore the flexibility and danger (and fun!) of hotloading, and in that case, finding failures before deployment is more important.
Great comment, thank you. Favorited.
One thing I learned randomly poking around after this post is the difference between the https://en.wikipedia.org/wiki/Rod_of_Asclepius and the https://en.wikipedia.org/wiki/Caduceus.
One thing I learned randomly poking around after this post is the difference between the https://en.wikipedia.org/wiki/Rod_of_Asclepius and the https://en.wikipedia.org/wiki/Caduceus.
Ironically, I meant Caduceus, even though the Rod of Asclepius is probably the better metaphor anyhow....
For starters, it's easy to receive messages across a cluster, and if you send a message across the cluster, and it contains a process id, that process id is magically converted to it's node local value on receipt.
Maybe it’s because the frameworks themselves are less important than the *human language used to describe them. To someone who knows what a Genserver is, or has used one before, what does it matter which of the many frameworks they’re using?
It just seems like we reinvent the same things again and again. Maybe that’s because at this level you can’t have a standard and things need to be bespoke.
It could, on the other hand, be an indication that there is a huge amount of wasted effort in revisiting long ago solved problems.
It could, on the other hand, be an indication that there is a huge amount of wasted effort in revisiting long ago solved problems.
I agree, I just think it’s part of the evolutionary process.
We reinvent them because the core concepts are right, but there isn’t enough connective tissue to re-use the implementations directly.
I don’t see it as a bad thing. The concepts persist and gain momentum and with each iteration we get closer to something that can actually be relied on consistently without replication.
There’s wasted effort, but something about the original was missing that would have made it the obvious choice. I think it feels wasted mostly because it isn’t yet obvious what the missing pieces were.
We reinvent them because the core concepts are right, but there isn’t enough connective tissue to re-use the implementations directly.
I don’t see it as a bad thing. The concepts persist and gain momentum and with each iteration we get closer to something that can actually be relied on consistently without replication.
There’s wasted effort, but something about the original was missing that would have made it the obvious choice. I think it feels wasted mostly because it isn’t yet obvious what the missing pieces were.
I just don’t think this is true. Erlang has existed since the early 80s, open source in the late 90s. Joe Armstrong wrote his thesis in 2003. It’s been 2 decades, and yet reliable distributed systems are still something companies fail at consistently. The latest trend with kubernetes and web interfaces everywhere is, in my opinion, an example of these ideas never gaining popularity so we outsourced it to infrastructure (then realised it needed code and made a whole new class of network/developer, devops).
Erlang basically posited that “process isolation” as a building block was fundamental to reliable distributed systems in the presence of software errors. I don’t think that’s ever been really challenged, but the final solution we have today seems almost ludicrously inefficient in comparison. I just hope lunatic gets traction!
Erlang basically posited that “process isolation” as a building block was fundamental to reliable distributed systems in the presence of software errors. I don’t think that’s ever been really challenged, but the final solution we have today seems almost ludicrously inefficient in comparison. I just hope lunatic gets traction!
This is just a tongue-in-cheek response that I have no stock in, but I find it funny that the page says "You don't have to reinvent the wheel" after they rewrite Erlang processes in another language.
Being compatible with what's there is exactly the opposite of reinventing the wheel (mechanisms, protocols, etc.).
That's true, but it'd be even simpler to implement a node in x language that can communicate with Erlang/OTP, and let Erlang work its fault-tolerant and scalable magic.
This is why analogies like the wheel analogy are silly - to stretch it closer to reality, sometimes you are eyeing a wheel in Go and a wheel in Erlang, so you’ve gotta pick which one to port.
I will forever think of Will Ferrell's parody of the Matrix whenever I hear "ergo": https://youtu.be/iRkiyy3EDl4?t=360
Looks cool. However, since this is a paid product… if one wants an actor framework for go without the need to connect to Erlang nodes, this will be a fine choice: https://github.com/asynkron/protoactor-go.
Ergo, heh. I wanted to do something like that and call it GoTP. But, some OTP features are hard to implement in Go. Particularly, Goroutines can clobber each others' data, unlike Erlang processes.
Does anyone know what tool was used to make these diagrams.. I really like the style.
I wasn't impressed but I look at it again .. it could be more pretty with right colors. Anyone has answer?
I originally thought this had something to due with the IRCd written in Go also named Ergo
https://github.com/ergochat/ergo
>Ergo Framework almost 5 times outperforms the original Erlang network messaging
This sounds like a breakthrough? Isn’t it?
This sounds like a breakthrough? Isn’t it?
Anything similar for Ruby?
Update: Found this, https://ruby-concurrency.github.io/concurrent-ruby/master/Co...
Update: Found this, https://ruby-concurrency.github.io/concurrent-ruby/master/Co...
Anything like this for Python?
This included the concurrency primitives (and a little bit more) almost 20 years ago. Probably requires porting to Python 3.
https://candygram.sourceforge.net/
https://candygram.sourceforge.net/
Not for the patterns, but Thespian is a pretty decent actor library.
Thanks, I'd never heard of Thespian for Python:
https://thespianpy.com/doc/
https://thespianpy.com/doc/
dumbo here, what are some use cases for this?
I imagine similar to the use cases for Erlang. Erlang can be a polarizing language, but what it's designed to do, it does exceptionally well, in fact, maybe the best. And that's not concurrency, multicore programming, or distribution. That was all a consequence of the main goal, which was fault tolerance. If you're a Swedish telecom company, you don't want a little bug in the code, that happens during someone's phone call, to knock out a switch, which takes down 50k phone calls, only for some guy to have to trek out into the middle of the forest to restart the computer running everything.
So the model they came up with was high isolation, via processes(no OS processes, but super tiny, like a few hundred bytes processes, all managed by the VM), and distribution(can't have reliability if one node can knock out the system).
The other unique part is the supervision trees that manage those processes. Let's say you have a chat server. You may have a process per device connected to the server, a supervisor for each chat room(group of processes), and a supervisor for all the chat rooms(supervisor of supervisors). Supervisors are themselves processes, but with a specific job.
If your phone's connection/process on the server has some issue, it'll just crash, that's the erlang philosphy "let it crash". You as the user likely can't fix it. The code can't magically fix itself. But presumably through testing we know that initial connections with a clean state are stable, so let it crash, your phone reconnects. Then it crashes again. Reconnects. Crashes again. (This isn't the ideal state obviously, but a worst case for demonstration). Eventually that'll trigger some limit, where the chat group supervisor will notice, somethings wrong here, there's some state that's in a mess, it's time to crash so we can get a fresh try at this. So it'll crash. Usually the first phone reconnection would have been enough to fix it and the group supervisor wouldn't need to restart, but for demonstration purposes we'll say it does, and let's pretend when it comes back up it crashes again. This will propagate up, until enough restarts have fixed the state of the system to the point where it's stable again.
I believe Erlang was the first system to achieve nine 9s of reliability, which is 31.56 milliseconds of downtime a year. Not saying this lib will get you that, but that's the spiel for Erlang.
So the model they came up with was high isolation, via processes(no OS processes, but super tiny, like a few hundred bytes processes, all managed by the VM), and distribution(can't have reliability if one node can knock out the system).
The other unique part is the supervision trees that manage those processes. Let's say you have a chat server. You may have a process per device connected to the server, a supervisor for each chat room(group of processes), and a supervisor for all the chat rooms(supervisor of supervisors). Supervisors are themselves processes, but with a specific job.
If your phone's connection/process on the server has some issue, it'll just crash, that's the erlang philosphy "let it crash". You as the user likely can't fix it. The code can't magically fix itself. But presumably through testing we know that initial connections with a clean state are stable, so let it crash, your phone reconnects. Then it crashes again. Reconnects. Crashes again. (This isn't the ideal state obviously, but a worst case for demonstration). Eventually that'll trigger some limit, where the chat group supervisor will notice, somethings wrong here, there's some state that's in a mess, it's time to crash so we can get a fresh try at this. So it'll crash. Usually the first phone reconnection would have been enough to fix it and the group supervisor wouldn't need to restart, but for demonstration purposes we'll say it does, and let's pretend when it comes back up it crashes again. This will propagate up, until enough restarts have fixed the state of the system to the point where it's stable again.
I believe Erlang was the first system to achieve nine 9s of reliability, which is 31.56 milliseconds of downtime a year. Not saying this lib will get you that, but that's the spiel for Erlang.
Thanks! Love the in deep explanation
probably none for you, but a bullet item on the authors resume.
On the other hand... this is a pure an example of a transliteration as an idiomatic port as I can think of. There is simply no need, in pure Go at least, to blindly copy every last detail of the Erlang implementation for gen_server for general usage, and more than you should copy every last such thing for Javascript or Python or any other language. If you are joining to an Erlang node and working in the Erlang ecosystem, you have no real choice, but if you aren't doing that, this library is going to add a lot of friction to a lot of things in Go, all over the place.
It's a very niche thing. If you need it it's probably worth all the money, but I'd pay to not get it put into a greenfield Go project.