A Brief Guide to OTP in Elixir(serokell.io)
serokell.io
A Brief Guide to OTP in Elixir
https://serokell.io/blog/elixir-otp-guide
99 comments
Elixir already started the process of extending OTP: Task is a hugely important addition to the family of genserver, genstatem, etc.
Isn't Task an entirely different beast to the gen_* behaviours?
To me it looks like a convenience wrapper around Erlang's primitives. I usually use them directly if I need the flexibility to "await" later. If I don't, I have a small function "run_in_subprocess" (mostly to not run into binary gc problems) that spawns and receives within one function.
The only real advantage I see is that it is separately supervised. When avoiding trap_exit, "spawn_link"ing within a process that is itself supervised has a similar effect, though.
To me it looks like a convenience wrapper around Erlang's primitives. I usually use them directly if I need the flexibility to "await" later. If I don't, I have a small function "run_in_subprocess" (mostly to not run into binary gc problems) that spawns and receives within one function.
The only real advantage I see is that it is separately supervised. When avoiding trap_exit, "spawn_link"ing within a process that is itself supervised has a similar effect, though.
1) Task makes it easy to supervise non-complicated async workflows. In prod, you REALLY want to supervise. Don't use spawn_link, it doesn't show up in the supervisin tree.
2) Task also implements the $callers process library key. What this means is that if you spawn a process with Task, it knows which process was responsible for calling it (note that this is in general different from "the process that supervises it"). In tests, you can use the $callers value to shard global state in a concurrency-friendly fashion.
Examples:
1. Make a mock in Mox. Spawn using Task. Call your mock from the task, Mox knows that the parent test is and serves the "correct mock".
2. Check out a database sandbox. Spawn using Task. Use the database in your task, Ecto knows what is parent test (and checkout) and serves the correct db view.
3. Make an HTTP request from your test. Stuff the $callers parameter into an HTTP header with term_to_binary and hex encoding (probably user-agent is a good choice). Use a plug to put $callers into your phoenix connection genserver. Spawn a Task that accesses your DB. Ecto knows what is the parent test (and checkout) and serves the correct DB view. So the cool thing is that YOUR REQUEST LEFT THE VM and it still worked! and this is composable too, if you do it right.
2) Task also implements the $callers process library key. What this means is that if you spawn a process with Task, it knows which process was responsible for calling it (note that this is in general different from "the process that supervises it"). In tests, you can use the $callers value to shard global state in a concurrency-friendly fashion.
Examples:
1. Make a mock in Mox. Spawn using Task. Call your mock from the task, Mox knows that the parent test is and serves the "correct mock".
2. Check out a database sandbox. Spawn using Task. Use the database in your task, Ecto knows what is parent test (and checkout) and serves the correct db view.
3. Make an HTTP request from your test. Stuff the $callers parameter into an HTTP header with term_to_binary and hex encoding (probably user-agent is a good choice). Use a plug to put $callers into your phoenix connection genserver. Spawn a Task that accesses your DB. Ecto knows what is the parent test (and checkout) and serves the correct DB view. So the cool thing is that YOUR REQUEST LEFT THE VM and it still worked! and this is composable too, if you do it right.
1) It shows up in the application view of the observer. Also, if one both processes dies the other one dies too, what more supervision would I want? A Task is also not magically rerun if there is a problem.
2) Every OTP behaviour keeps the ancestor info as well, I could use proc_lib instead of spawning directly.
2) Every OTP behaviour keeps the ancestor info as well, I could use proc_lib instead of spawning directly.
GenServer is also a convenience wrapper around Erlang's primitives. You could just do a loop handling messages appropriately and get something close to a GenServer in very few lines, but once you start thinking about all the edge cases you end up with GenServer.
Task does the same thing but for a usage model that wasn't previously well supported. It basically removes the last reason I ever did spawn_link directly in Erlang.
Task does the same thing but for a usage model that wasn't previously well supported. It basically removes the last reason I ever did spawn_link directly in Erlang.
> In real life, we don’t need to write code with receive do loops. Instead, we use one of the behaviours created by people much smarter than us.
I make more than half of my income from Elixir. That said, the naive receive loop is much easier to understand than any of the GenServer examples. They pollute the module logic with all that handle_* boilerplate. I believe that neither Erlang nor Elixir got the right abstraction there. Method definition in any OO language is easier to understand. At least Elixir should have taken Agent and made it an invisible part of the language. All those calls to Agent in this example https://elixir-lang.org/getting-started/mix-otp/agent.html are still boilerplate.
I make more than half of my income from Elixir. That said, the naive receive loop is much easier to understand than any of the GenServer examples. They pollute the module logic with all that handle_* boilerplate. I believe that neither Erlang nor Elixir got the right abstraction there. Method definition in any OO language is easier to understand. At least Elixir should have taken Agent and made it an invisible part of the language. All those calls to Agent in this example https://elixir-lang.org/getting-started/mix-otp/agent.html are still boilerplate.
What you said is rarely stated in public, but I've often felt it too: OTP obscures the underlying beauty of the Erlang platform.
OTP is well engineered of course, but the basic notion of the spawn -> receive -> loop cycle is so clean and illuminating that I wish newbies would hold out before learning OTP sometimes.
It's natural to think of case-specific abstractions around the primitives that are more germane to the domain at hand than OTP's.
OTP is well engineered of course, but the basic notion of the spawn -> receive -> loop cycle is so clean and illuminating that I wish newbies would hold out before learning OTP sometimes.
It's natural to think of case-specific abstractions around the primitives that are more germane to the domain at hand than OTP's.
OTOH every decent erlang book or tutorial I've ready, from Armstrong's book[1] to Learn You Some Erlang[2] to Erlang and OTP in Action[3] and others all start off by introducing primitives like spawn, message sending , and receive.
They then introduce OTP and explain all the cases it handles.
I've never read a book or tutorial on OTP that just starts with OTP. In fact (a bit tangetially) whenever I encounter abstractions where I don't already understand the primitives I have a very difficult time. Maybe that's just the way my brain is wired though. I have a terrible time with OO programming for that reason. I much prefer a separation of functions and data because it's much easier for me to reason about what is happening.
Anyway, I agree that programmers should start with the primitives, but I've never seen anyone really teach OTP any other way.
Edit: also there are tools such as Erlang.mk[4] by Loïc Hoguin[5] that will handle a lot of the boilerplate for OTP projects and building releases, though it's good to do it manually at least once while learning.
[1] https://pragprog.com/titles/jaerlang2/programming-erlang-2nd...
[2] https://learnyousomeerlang.com/
[3] https://www.manning.com/books/erlang-and-otp-in-action
[4] https://erlang.mk/guide/index.html
[5] https://ninenines.eu/
They then introduce OTP and explain all the cases it handles.
I've never read a book or tutorial on OTP that just starts with OTP. In fact (a bit tangetially) whenever I encounter abstractions where I don't already understand the primitives I have a very difficult time. Maybe that's just the way my brain is wired though. I have a terrible time with OO programming for that reason. I much prefer a separation of functions and data because it's much easier for me to reason about what is happening.
Anyway, I agree that programmers should start with the primitives, but I've never seen anyone really teach OTP any other way.
Edit: also there are tools such as Erlang.mk[4] by Loïc Hoguin[5] that will handle a lot of the boilerplate for OTP projects and building releases, though it's good to do it manually at least once while learning.
[1] https://pragprog.com/titles/jaerlang2/programming-erlang-2nd...
[2] https://learnyousomeerlang.com/
[3] https://www.manning.com/books/erlang-and-otp-in-action
[4] https://erlang.mk/guide/index.html
[5] https://ninenines.eu/
> every decent erlang book or tutorial I've ready, from Armstrong's book[1] to Learn You Some Erlang...
What a coincidence! I just tweeted about how you can get it and 12 other various programming books for just $8 now: https://twitter.com/AlchemistCamp/status/1311796404830892032
What a coincidence! I just tweeted about how you can get it and 12 other various programming books for just $8 now: https://twitter.com/AlchemistCamp/status/1311796404830892032
> spawn -> receive -> loop cycle is so clean and illuminating that I wish newbies would hold out before learning OTP sometimes
Yes it is neat and its fun to play with but it isn't usually something you want to use in production code. In production code it is important to have processes linked correctly so that errors propagate to callers. GenServer.call handles this for you, and also clarifies intent (blocking call to another process that must respond or fail).
Yes it is neat and its fun to play with but it isn't usually something you want to use in production code. In production code it is important to have processes linked correctly so that errors propagate to callers. GenServer.call handles this for you, and also clarifies intent (blocking call to another process that must respond or fail).
I agree. The reason we stick with OTP, is that OTP behaviors like gen_server handle two system-level concerns that "raw" Erlang code doesn't:
1. OTP behaviors integrate with the OTP supervisor lifecycle management system (i.e. the OTP framework offers the supervisors standardized hooks to start up and shut down your process, guaranteeing that the errors generated during such steps will be in a format the supervisor can use);
2. Processes that implement OTP behaviors will react to `sys` messages; and so can be debugged, hibernated, code-upgraded, etc. on a framework level, "between" the times the process runs developer-defined code, without the developer having to write such handlers into their module.
This is all accomplished by passing control over the receive loop to a framework — `proc_lib` in this case — which in turn passes any messages it doesn't recognize back to your process. It's an inversion-of-control: proc_lib "is" your process; gen_server or whatever is a delegate of proc_lib; and your module is the delegate for gen_server. It's like an OOP class hierarchy in a GUI system, where the base class handles some events, subclasses handle others, and then your module only has to handle the few it's interested in.
But, if there was a way to do exactly that — to specialize one receive statement, defined in some other function, with your own additional clauses — then we wouldn't need the inversion-of-control framework of OTP! We could just write a regular receive statement that "points to" another receive statement as its "parent" or "fallback" (sort of like a chain of firewall rules, each pointing to the next as "what to check next if this one didn't handle it.") Ideally, the compiled result would be one fused receive statement that has all the clauses from all its parents.
And if you think about it, that's totally possible, even if Erlang itself doesn't offer a fancy chainable receive statement like that... because, in a language like Elixir tht has hygenic macros, you can use macros to generate such "fused receive statements"!
I'm honestly really surprised nobody has yet tried. I realized the possibility of this a couple years back, and have been waiting with baited breath for somebody to attempt it. If it worked out, this "tech" could be used to build a wholly-different-feeling language.
1. OTP behaviors integrate with the OTP supervisor lifecycle management system (i.e. the OTP framework offers the supervisors standardized hooks to start up and shut down your process, guaranteeing that the errors generated during such steps will be in a format the supervisor can use);
2. Processes that implement OTP behaviors will react to `sys` messages; and so can be debugged, hibernated, code-upgraded, etc. on a framework level, "between" the times the process runs developer-defined code, without the developer having to write such handlers into their module.
This is all accomplished by passing control over the receive loop to a framework — `proc_lib` in this case — which in turn passes any messages it doesn't recognize back to your process. It's an inversion-of-control: proc_lib "is" your process; gen_server or whatever is a delegate of proc_lib; and your module is the delegate for gen_server. It's like an OOP class hierarchy in a GUI system, where the base class handles some events, subclasses handle others, and then your module only has to handle the few it's interested in.
But, if there was a way to do exactly that — to specialize one receive statement, defined in some other function, with your own additional clauses — then we wouldn't need the inversion-of-control framework of OTP! We could just write a regular receive statement that "points to" another receive statement as its "parent" or "fallback" (sort of like a chain of firewall rules, each pointing to the next as "what to check next if this one didn't handle it.") Ideally, the compiled result would be one fused receive statement that has all the clauses from all its parents.
And if you think about it, that's totally possible, even if Erlang itself doesn't offer a fancy chainable receive statement like that... because, in a language like Elixir tht has hygenic macros, you can use macros to generate such "fused receive statements"!
I'm honestly really surprised nobody has yet tried. I realized the possibility of this a couple years back, and have been waiting with baited breath for somebody to attempt it. If it worked out, this "tech" could be used to build a wholly-different-feeling language.
Well, the sys behavior is easy to implement.
I also am not a fan of the complexity and overhead of gen_server so instead I use metal (http://github.com/lpgauth/metal). It's a simple receive loop with an optional init and terminate callback. It implement sys and can be supervised.
I also am not a fan of the complexity and overhead of gen_server so instead I use metal (http://github.com/lpgauth/metal). It's a simple receive loop with an optional init and terminate callback. It implement sys and can be supervised.
Does gen_server have a measurable overhead compared to this? The main event loop of gen_server looks remarkably similar to what you are doing but handles calls, casts, hibernation, debugging, provides stacktraces and doesn't "force" exit trapping. In the "happy-path", I don't see any additional overhead to what you are doing.
I would also like to see "handle_call" and "handle_cast" (maybe even "handle_info" with a blanket noreply-implementation) optional, but until then I can deal with the additional two lines to exit on cast or call if I don't require those.
I would also like to see "handle_call" and "handle_cast" (maybe even "handle_info" with a blanket noreply-implementation) optional, but until then I can deal with the additional two lines to exit on cast or call if I don't require those.
I don't think this approach would result in lower overhead, since `metal` here is still a separate module.
Maybe if it were instead a macro/parse-transform, injecting all the common "framework" code into the module implementing the server callbacks, you could then at least get the optimizations that result from all the calls into and out of the "framework" being local rather than remote calls. Private functions could be inlined/fused, etc.
Further, if HiPE was made to work again on Erlang 25, such a "statically-compiled" server module could also "stay native", with no context-switches to interpreted code, in a way that HiPE can't presently manage for gen_servers due to the code in `gen_server` and `proc_lib` themselves not executing natively.
Maybe if it were instead a macro/parse-transform, injecting all the common "framework" code into the module implementing the server callbacks, you could then at least get the optimizations that result from all the calls into and out of the "framework" being local rather than remote calls. Private functions could be inlined/fused, etc.
Further, if HiPE was made to work again on Erlang 25, such a "statically-compiled" server module could also "stay native", with no context-switches to interpreted code, in a way that HiPE can't presently manage for gen_servers due to the code in `gen_server` and `proc_lib` themselves not executing natively.
Yeah but that's like saying the awkwardness of stl templates obscures the beauty of C++ or the Android app api obscures the beauty of the jvm. The genserver is messy because it deals with a good chunk of messiness around safely managing distributed systems for you. It probably could be improved (Dave thomas critiques comes to mind) but elixir had to be conservative because if it weren't it wouldn't have had buy in from the beam community and wouldn't have become the first truly successful non-erlang language targeting the BEAM.
I finished reading Elixir in Action a few days ago. This was probably the best part of that book. It takes you through building a primitive GenServer with basic processes before moving onto actually using GenServer. It specifically goes over tail recursion optimization before it introduces the receive do loop, which seems like a very important part of the whole thing.
Elixir in Action is fantastic. Its approach is so different from the other introductory books, but the payoff is real.
Would you recommend the book?
Also not the OP, but unequivocally yes (though for full disclosure, I am biased as was one of the technical reviewers for the second edition). It was hugely important to me personally learning the language; the approach is excellent and it's very well written. You take a simple to-do application and write a number of versions of it, each one progressively more complex and featureful, and each one using progressively higher language/OTP abstractions.
Not the OP, but it is a great book, I'd highly recommend it.
> the basic notion of the spawn -> receive -> loop cycle is so clean and illuminating that I wish newbies would hold out before learning OTP sometimes
The frustrating thing is that when I made exactly such a tutorial two years ago, I was immediately chastised (on my old hosted commenting system) for showing something so low-level to start with:
https://alchemist.camp/episodes/simple-process-example
The frustrating thing is that when I made exactly such a tutorial two years ago, I was immediately chastised (on my old hosted commenting system) for showing something so low-level to start with:
https://alchemist.camp/episodes/simple-process-example
> the basic notion of the spawn -> receive -> loop cycle
I'm working on a video series about these concurrency patterns; even recorded the first one but the audio is bad so I'm going to re-record it before I publish it when some "real" audio equipment comes in (hopefully this weekend) - let me know what you think:
https://www.youtube.com/watch?v=eoEcpcLVumc
I'm working on a video series about these concurrency patterns; even recorded the first one but the audio is bad so I'm going to re-record it before I publish it when some "real" audio equipment comes in (hopefully this weekend) - let me know what you think:
https://www.youtube.com/watch?v=eoEcpcLVumc
Pretty good!
I knew most of that stuff already so I can't speak to the pure education value, but it made sense, and I think you hit the key points.
I bet some visuals would help it make sense for total nooberz.
AND! I didn't know that about "v" - thanks! :)
I knew most of that stuff already so I can't speak to the pure education value, but it made sense, and I think you hit the key points.
I bet some visuals would help it make sense for total nooberz.
AND! I didn't know that about "v" - thanks! :)
another protip: you can also call v(n) and it will give you the item at iex(n)>
There are libraries that do this, like https://github.com/sasa1977/exactor but the Elixir community nowadays seems more conservative about macro "magic" than it was a few years ago. Sasa has written an entire book about OTP and if he doesn't recommend using his own library I wouldn't argue with him, but I haven't written a Genserver myself in years despite writing Elixir for a living the past 3.5 years. If you are going to learn how they work though, it makes sense to stick to how Erlang actually implements them.
> There are libraries that do this, like https://github.com/sasa1977/exactor
At the very top of the Extractor readme:
> I don't maintain this project anymore. In hindsight, I don't think it was a good idea in the first place. I haven't been using ExActor myself for years, and I recommend sticking with regular GenServer instead :-)
This both confirms what you said about the Elixir community shying away from macro magic, and makes this library a very bad idea to use in new projects, imo.
At the very top of the Extractor readme:
> I don't maintain this project anymore. In hindsight, I don't think it was a good idea in the first place. I haven't been using ExActor myself for years, and I recommend sticking with regular GenServer instead :-)
This both confirms what you said about the Elixir community shying away from macro magic, and makes this library a very bad idea to use in new projects, imo.
Right, thats why I said he doesn't recommend using his own library. My point is people have been down this road, and moved away from it, not that anyone should use it.
[deleted]
Of course it's easier to understand. It's also much easier to get wrong. Just yesterday I messed around with running a small process on to ship subscribed messages across node boundaries and forgot the recursive call in one of 4 cases, leading to mysterious stopping.
In Elixir you don't even have to define all of this boilerplate, just the functions you actually need. Also, the callbacks are not classical methods, a handle_call doesn't have to reply immediately, for example. Distinguishing info, call and cast is also very relevant.
If you wanted something like
In Elixir you don't even have to define all of this boilerplate, just the functions you actually need. Also, the callbacks are not classical methods, a handle_call doesn't have to reply immediately, for example. Distinguishing info, call and cast is also very relevant.
If you wanted something like
{:ok, agent} = Agent.start_link ...
agent.get()
That's what I was thinking about, client side. Probably the Elixir compiler could detect it and generate the code to bridge the gap. It feels OO-ish but we're dealing with mutable state anyway. I prefer "easy" than "pure".
Seems like you're losing something here -- ISTM that the goodness of a reified massage is now lost. A message, as a piece of data, can be tested, stored, passed on, etc., while a "call" (the .get()) cannot.
Do you do that with proc_lib or is it just a completely otp-unaware process like Joe would do?
> Forget using a million different technologies for things like background jobs, OTP can supply you with everything.
It's true but in practice this usually doesn't pan out.
For example with just background jobs alone there's the idea of queues, tracking failures / successes, exponential back-off retries, guaranteeing uniqueness, draining, periodic tasks and everything else you'd likely want in a production ready app.
Typically you'd use Redis, Postgres or something else to help with this. Fortunately https://github.com/sorentwo/oban exists and uses Postgres as a back-end with close to 10,000 lines of Elixir.
It's true but in practice this usually doesn't pan out.
For example with just background jobs alone there's the idea of queues, tracking failures / successes, exponential back-off retries, guaranteeing uniqueness, draining, periodic tasks and everything else you'd likely want in a production ready app.
Typically you'd use Redis, Postgres or something else to help with this. Fortunately https://github.com/sorentwo/oban exists and uses Postgres as a back-end with close to 10,000 lines of Elixir.
I think these blog posts usually pretend it's simpler than it is. There's usually (almost always) durable storage required somewhere. But to respond to your example specifically, OTP offers quite a bit. To your points:
Queues: a coordinator genserver process per node, which uses poolboy to only run X jobs at a time
Tracking failures/sucesses: postgres
Exponential backoff: a function
Uniqueness across the cluster: stdlib global locking (https://erlang.org/doc/man/global.html#trans-2)
draining: remote console
periodic tasks: a simple library that knows how to cron
I built a sharded cron-like scheduler in a few hundred lines of code. It's not doing a billion transactions per second, but it's been running in production for a few years without much drama. It's not complicated because it's just postgres and some standard erlang/elixir stuff.
Erlang has plenty of drawbacks, but this isn't it.
Queues: a coordinator genserver process per node, which uses poolboy to only run X jobs at a time
Tracking failures/sucesses: postgres
Exponential backoff: a function
Uniqueness across the cluster: stdlib global locking (https://erlang.org/doc/man/global.html#trans-2)
draining: remote console
periodic tasks: a simple library that knows how to cron
I built a sharded cron-like scheduler in a few hundred lines of code. It's not doing a billion transactions per second, but it's been running in production for a few years without much drama. It's not complicated because it's just postgres and some standard erlang/elixir stuff.
Erlang has plenty of drawbacks, but this isn't it.
> There's usually (almost always) durable storage required somewhere.
Despite the name, mnesia can provide durable storage, without using anything outside of OTP and your nodes' (hopefully durable) filesystems.
Despite the name, mnesia can provide durable storage, without using anything outside of OTP and your nodes' (hopefully durable) filesystems.
Despite? It pretty much means "to remember", it doesn't have the alpha privative of amnesia, "to forget"
IIRC original name was Amnesia, but Joe was made to change it for marketing reasons (it's in one of his books or blog posts).
Anyway, I do not think Mnesia is a particularly good way to persist stuff, just use postgres.
Anyway, I do not think Mnesia is a particularly good way to persist stuff, just use postgres.
At WhatsApp we had a bunch of mnesia-backed services. I can't say that it was the best experience. Pretty much anything non-trivial, such as healing a cluster after a partition, or expanding a cluster, or cross-DC replication was a complete PITA to do properly. There's also very few people that have operational experience with it and very little documentation about it compared to pretty much any other storage engine.
The main benefit was the ability to load and store Erlang terms natively, along with the fact that it's just a library. The fact that your compute is colocated with your storage lends itself to a different architecture than when you use, say, a stateless web service in front of a relational database.
(Edit: I realize now that I'm replying to a WA old-timer so I'd be curious to know what your assessment of mnesia is with the benefit of hindsight.)
The main benefit was the ability to load and store Erlang terms natively, along with the fact that it's just a library. The fact that your compute is colocated with your storage lends itself to a different architecture than when you use, say, a stateless web service in front of a relational database.
(Edit: I realize now that I'm replying to a WA old-timer so I'd be curious to know what your assessment of mnesia is with the benefit of hindsight.)
I think we would have found any database to be a PITA at write volumes and data volumes we were using, with our expectations of availability. I don't think we would have been able to get that performance out of MySQL, or Postgres; and anyway, we didn't have a dedicated team for database management to make sure nobody wrote bad queries if we did. Sync, status and reg ran on MySQL and php when I started, and it was a lot easier to run them and scale them on Erlang, IMHO. Repartitioning MySQL like we repartitioned mnesia would be a lot harder, I think.
I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert, and the source code will be your documentation; mnesia is all written in Erlang, and not too hard to follow; most people won't hit the limits of ets, I'd think. If you don't operate at the limits, you might not need an expert, and if you keep up with OTP releases and hardware releases, the limits get bigger every year.
Looking back, I think we should have spent more time earlier on using the hooks mnesia provides to make healing clusters take less effort; and maybe some automation on cluster expansion. Of course, WA never shied away from manual work by server engineers. :)
(Hmmm, now I'm trying to figure out you are, send me an email if you will :)
I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert, and the source code will be your documentation; mnesia is all written in Erlang, and not too hard to follow; most people won't hit the limits of ets, I'd think. If you don't operate at the limits, you might not need an expert, and if you keep up with OTP releases and hardware releases, the limits get bigger every year.
Looking back, I think we should have spent more time earlier on using the hooks mnesia provides to make healing clusters take less effort; and maybe some automation on cluster expansion. Of course, WA never shied away from manual work by server engineers. :)
(Hmmm, now I'm trying to figure out you are, send me an email if you will :)
> I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert.
Which is why I'm happy others have paved the way.
A huge amount of web apps can get by with a Postgres backed queue that won't break a sweat handling a few hundred writes per second on a low end cloud hosted VPS.
While leveraging something like Oban you don't even need to be an expert. You can just use the library, throw work into it with a reasonable amount of foresight / best practices and you can start developing features in your app.
Which is why I'm happy others have paved the way.
A huge amount of web apps can get by with a Postgres backed queue that won't break a sweat handling a few hundred writes per second on a low end cloud hosted VPS.
While leveraging something like Oban you don't even need to be an expert. You can just use the library, throw work into it with a reasonable amount of foresight / best practices and you can start developing features in your app.
Mnesia is first and foremost and in-memory key-value configuration storage. Relying on it for anything else will damage you in the long run.
Yep exactly. We process millions of jobs using Exq, mainly because of the retry, persistence of jobs, etc. I don't want to risk leaving that in elixir only.
We use Exq because I've used Sidekiq for years. I should check out Oban, since it could reduce the complexity in our stack (we wouldn't need redis anymore).
We use Exq because I've used Sidekiq for years. I should check out Oban, since it could reduce the complexity in our stack (we wouldn't need redis anymore).
A lot of people have been moving from Exq to Oban. Come by the #oban channel on Slack if you have any questions.
Phoenix ships with Ecto ORM-style-binding to Postgres underneath (usually) for a good reason! :) And if you need that in memory magic where the serial nature of a genserver won't suffice for your load, you basically have built-in Redis with Erlang's ets. And if you really need insane performance on longer-cached items than ets is optimal for, Discord released https://github.com/discord/fastglobal
And if you don't want to put all those pieces together, like you said there's a rich set of modules people have built, like oban. Long rise and live the elixir monoliths ;) (although admittedly even Discord had to swap out some of their most intense functions with Rust, but how many people are writing systems that need to handle millions of concurrents?)
And if you don't want to put all those pieces together, like you said there's a rich set of modules people have built, like oban. Long rise and live the elixir monoliths ;) (although admittedly even Discord had to swap out some of their most intense functions with Rust, but how many people are writing systems that need to handle millions of concurrents?)
> you basically have built-in Redis with Erlang's ets
The big difference there is if you have a Dockerized app, Redis won't be restart on every deploy but the BEAM will be so you will lose your ETS cache or whatever you're storing.
Now, I know cache is meant to be disposable but this model of using ETS instead of Redis reminds me of a Python or Ruby app storing its cache in memory. I mean, sure you can do it but for the longest time (7-8 years?) the community as a whole kind of landed on keeping that stuff out of your app's process.
The big difference there is if you have a Dockerized app, Redis won't be restart on every deploy but the BEAM will be so you will lose your ETS cache or whatever you're storing.
Now, I know cache is meant to be disposable but this model of using ETS instead of Redis reminds me of a Python or Ruby app storing its cache in memory. I mean, sure you can do it but for the longest time (7-8 years?) the community as a whole kind of landed on keeping that stuff out of your app's process.
This is my experience as well, working with Elixir fulltime for about 5 years.
OTP = Open Telecom Platform [1], though I think the full name is avoided nowadays.
On an unrelated note, I've always been fascinated with the Erlang VM. The idea that you could hot-load modules and have multiple versions of the same module running at once seems really useful. I wonder why other runtimes haven't adopted these features?
[1] https://en.wikipedia.org/wiki/Open_Telecom_Platform
On an unrelated note, I've always been fascinated with the Erlang VM. The idea that you could hot-load modules and have multiple versions of the same module running at once seems really useful. I wonder why other runtimes haven't adopted these features?
[1] https://en.wikipedia.org/wiki/Open_Telecom_Platform
Hot loading in Erlang is not magical. Here is an example. Let's say you want to redeploy some new code. In your deploy, you change some existing function foo in module A to call a new function bar in module B.
If you do this, you'd better make sure that you hotload the new version of B before you hotload the new version of A. Otherwise some process could end up trying to call B:bar through A:foo before the new B has finished loading.
If you rely on hotloading modules as your primary deploy mechanism, you will run into issues like this all the time. So it's not really clear to me that hotloading is a win for your typical stateless web service compared to just deploying a new binary and doing your standard zero-downtime deploy via forking.
It makes more sense when you realize that certain Erlang apps build up a huge amount of state in-memory. In those cases hotloading allows you to load new versions of the changed modules without having to take down the whole VM.
If you do this, you'd better make sure that you hotload the new version of B before you hotload the new version of A. Otherwise some process could end up trying to call B:bar through A:foo before the new B has finished loading.
If you rely on hotloading modules as your primary deploy mechanism, you will run into issues like this all the time. So it's not really clear to me that hotloading is a win for your typical stateless web service compared to just deploying a new binary and doing your standard zero-downtime deploy via forking.
It makes more sense when you realize that certain Erlang apps build up a huge amount of state in-memory. In those cases hotloading allows you to load new versions of the changed modules without having to take down the whole VM.
> I wonder why other runtimes haven't adopted these features?
Probably mainly due to shared memory. When you know your data can be modified only in one place, changing data structure is easier. Try changing a struct when some other code is using it. It would require a lock for every object in memory.
Probably mainly due to shared memory. When you know your data can be modified only in one place, changing data structure is easier. Try changing a struct when some other code is using it. It would require a lock for every object in memory.
One of the great things about Erlang and the BEAM is that by designing a very opinionated language around concepts like immutability and ubiquitous messaging between lightweight actors, you can define the VM to support those.
So there’s this collection of features between the language and VM that support each other, and are generally hard to reproduce outside that ecosystem because they are so interdependent.
So there’s this collection of features between the language and VM that support each other, and are generally hard to reproduce outside that ecosystem because they are so interdependent.
the fact that looping through a process is a tail call which by its very nature creates well-defined boundaries (both in "code segment" and in "data segment") also helps a gazillion. Try doing hot loading in the middle of a while loop... Not saying it's impossible, but you're not gonna have a good time.
Or "One Time Password" if you're an idiot like me.
Gotta love an introductory article on "OTP" which even includes a section entitled "What is OTP?" but doesn't expand the acronym.
Gotta love an introductory article on "OTP" which even includes a section entitled "What is OTP?" but doesn't expand the acronym.
It's no longer considered a correct acronym as erlang is not really a telecoms language anymore. It's been a problem where outsiders are quick to think or say, "I am not writing a telecoms app" and dismiss it.
I understand this reasoning. The conclusion I draw is that "OTP" is truly a miserable name — like "car cdr" level bad.
As a newcomer to Elixir's OTP, I cannot divine that experts hold the opinion that the acronym expansion is misleading when that information is deliberately withheld. The first thing this article did was send me off to search the web.
Folks should use a backronym, like "Opinionated Threading Platform".
As a newcomer to Elixir's OTP, I cannot divine that experts hold the opinion that the acronym expansion is misleading when that information is deliberately withheld. The first thing this article did was send me off to search the web.
Folks should use a backronym, like "Opinionated Threading Platform".
Back in the 90s there were plenty of programs which ditched their acronyms: elm, pine, pico, gnu, etc.
Importantly, all of those have a discernible pronunciation that is not simply stating the letters in order.
a very good point. That goes for GNU, too.
[deleted]
OSGi (Open Services Gateway initiative) [1] provides this for the JVM by modularizing the ClassLoader [2]. If a module is upgraded but classes from the previous version are still referenced by other modules, both modules will run.
An OSGi "service" is not an actor but, like OTP with servers+supervisors, is the logical unit with which you build your systems. "bundles" (Java JARs with additional metadata) then act as the equivalent "application" that contain the services.
Outside of it being Java, I have strong opinions (positive and negative) that I've decided not to include.
[1] https://en.wikipedia.org/wiki/OSGi [2] https://en.wikipedia.org/wiki/Java_Classloader
An OSGi "service" is not an actor but, like OTP with servers+supervisors, is the logical unit with which you build your systems. "bundles" (Java JARs with additional metadata) then act as the equivalent "application" that contain the services.
Outside of it being Java, I have strong opinions (positive and negative) that I've decided not to include.
[1] https://en.wikipedia.org/wiki/OSGi [2] https://en.wikipedia.org/wiki/Java_Classloader
Because why would you do that? Now we have tool and pipeline where testing / deploying ect .. is easy, so I'm not sure why would I risk to do hot load things.
I don't think it's useful and I think it's pretty dangerous is the first place. You have code that can run two different things, wcgw.
I don't think it's useful and I think it's pretty dangerous is the first place. You have code that can run two different things, wcgw.
Addressed in part by dmitriid and dnautics, but:
1. You can use this during development (where the tight feedback loop is very nice).
2. You can use this during deployment (after verifying that the process will work as intended, hopefully).
(1) is something you could do every day if it was your daily driver language. (2) would happen less frequently (well, as frequently as you deploy) but permit you to do partial updates without bringing the whole system down.
It's worth remembering that deploying Erlang (and Elixir) can be more like deploying with microservices or applications to an application server. You don't want to bring the whole thing down just to change out one part of the system, so you can update just the parts that have changed and need to be updated.
1. You can use this during development (where the tight feedback loop is very nice).
2. You can use this during deployment (after verifying that the process will work as intended, hopefully).
(1) is something you could do every day if it was your daily driver language. (2) would happen less frequently (well, as frequently as you deploy) but permit you to do partial updates without bringing the whole system down.
It's worth remembering that deploying Erlang (and Elixir) can be more like deploying with microservices or applications to an application server. You don't want to bring the whole thing down just to change out one part of the system, so you can update just the parts that have changed and need to be updated.
A very erlang demo showed upgrading code running on a telephone switch without dropping the call. That's why you want hot code loading.
Because control plane does not handle anything phone related most likely? It's like upgrading Kubernates masters, your API servers still works and are not affected.
AFAIK Erlang code does not handle any network traffic, all of that is done in C/C++.
AFAIK Erlang code does not handle any network traffic, all of that is done in C/C++.
This is the original video in its 90s glory: https://www.youtube.com/watch?v=uKfKtXYLG78
Regardless of where the call is handled, you can't just deploy a new switch in a telecom network, but you still want to upgrade them. This is where you really want hot code loading.
Regardless of where the call is handled, you can't just deploy a new switch in a telecom network, but you still want to upgrade them. This is where you really want hot code loading.
Real world usage case - I have a server which handles several thousand telematic devices (telemetry+actuation), split into about 20 different device types. It's small device base so doesn't require multiple servers and we can be really cheap. But requiring to disconnect all devices just to make one small change to one device driver would be painful. Now we can have uptime of over half a year and in that time we had changes/fixes to several drivers. Normally that would require scheduled downtime every week and slow us down considerably.
I think (could be wrong) it's used for hot code reloading in phoenix when you're in dev mode. Make a change in your code, hit refresh in the browser, and now the relevant code deltas are applied to your http request.
Someone correct me if I'm wrong here, but afaics dropping the ability to do it would remove a complication from the platform and wouldn't affect the vast majority of users in any way. But [afaik] it has to exist because Ericsson require that feature. Outside of their specific usecases it's doesn't seem a particularly useful feature, or one that is used in the vast majority of contemporary codebases
Then they really need to pick a new name, because OTP is widely understood in programming to be a “One-Time Pad”, and without defining what they think it means in this context this is the least coherent article I’ve read in a while.
Cryptography \neq Programming
> Every time the loop runs, it will check the top of the mailbox (mailbox is last-in-first-out) for messages that match what we need and process them.
Nope, erlang mailbox is fifo with filtering. When a process sends messages, they are delivered in the same order they were sent.
Nope, erlang mailbox is fifo with filtering. When a process sends messages, they are delivered in the same order they were sent.
That sentence was fixed in the article and now reads:
> Every time the loop runs, it will check from the bottom of the mailbox (in order they were received) for messages that match what we need and process them.
> Every time the loop runs, it will check from the bottom of the mailbox (in order they were received) for messages that match what we need and process them.
I'd just like to point out that while knowledge of OTP is a good thing for an Elixir developer, it isn't really needed for most application development. There are excellent libraries and frameworks that leverage OTP to provide a lot of value to their users, and most of us who use Elixir in industry just use those libraries.
I would go so far as to say a beginner should not write genservers, and using genservers that aren't part of a library is usually an antipattern. When you're ready to write a serious library, then you're ready to write genserver.
Of course it's always good to understand genservers.
Of course it's always good to understand genservers.
I'm working on a game backend (which I'm finding Elixir/Erlang pretty good for) and I don't see how I could reasonably do this without using my own GenServers.
I can see not writing my own GenServer for webapps, but when it comes to concurrency webapps tend to be dead simple. Elixir is a fine choice for these but I feel like its killer features don't shine that much over the alternatives.
I can see not writing my own GenServer for webapps, but when it comes to concurrency webapps tend to be dead simple. Elixir is a fine choice for these but I feel like its killer features don't shine that much over the alternatives.
Would you mind linking to a couple of them? I don't know anything about Elixir (but am very comfortable in a lot of other languages), and am interested in the kind of thing more experienced people are using with Elixir. Thanks!
For web applications, Phoenix is very popular: https://github.com/phoenixframework/phoenix; it uses some OTP directly and its dependencies like the web-server Cowboy use it as well.
DB development usually uses Ecto: https://github.com/elixir-ecto/ecto, it has OTP servers for connection pooling and other tasks.
For managing background tasks I use Honeydew: https://github.com/koudelka/honeydew
DB development usually uses Ecto: https://github.com/elixir-ecto/ecto, it has OTP servers for connection pooling and other tasks.
For managing background tasks I use Honeydew: https://github.com/koudelka/honeydew
Your Phoenix link is broken (includes the semi-colon)
for the lazy: https://github.com/phoenixframework/phoenix
for the lazy: https://github.com/phoenixframework/phoenix
Phoenix is a good example. It makes great use of OTP, but you don't really need to touch OTP when using it to build web applications. You can, and your life as an Elixir dev will certainly be easier if you have some understanding of OTP, but it works great as a web framework even if you don't.
depends on the type of project for sure. possessing knowledge of OTP is a superpower. when working on any serious project, it will always help if there is at least one developer in the team who has knowledge about it.
unless you're only building web apps i don't know how you can build systems in elixir without touching otp
Knowing how to use OTP is definitely something worthwhile if you're going to stick with the BEAM platform:
https://ferd.ca/ten-years-of-erlang.html
https://ferd.ca/ten-years-of-erlang.html
IMO, Erlang/Elixir/BEAM/OTP really is the toolset that can be used to build what the mystical 10x/100x programmer can achieve. But that's also the reason why it's hard to push for its adoption into bigger teams. When the value proposition is "you can do all that by only using Erlang/Elixir/OTP...", while the alternatives are based on multiple, but more "traditional" and well-understood systems (which you are easier to hire for) ,"using one thing for all" sounds more like a risk (and potential tech/dev lock-in) than an advantage to the management. Love to hear success stories of how Erlang/Elixir are pushed into "mainstream" within a relative large org.
The problem is that the runtime does too much magic and it's not portable / standard compared to other languages / runtimes. Lot of what BEAM/OTP is doing is done in other tools and it's language neutral.
Not everything is good tbh, when you see the pain it is to deploy an Erlang app in 2020 ...
Not everything is good tbh, when you see the pain it is to deploy an Erlang app in 2020 ...
is it a pain?
mix release
Compiles everything, tar/gz's it, sends it to a private s3 bucket. On your server side, you periodically watch the s3 bucket, and when one of your nodes detects a relup, it downloads from the s3 bucket, kills itself. Systemd then restarts it, kicking into the newest version.
That's it. This is maybe about 50-100 lines of code, one external library (pick your AWS library of choice), and one systemd script. I think there's even a library for managing systemd from within the BEAM now.
If you want to be more sophisticated (a rolling blue-green deploy across an erlang cluster) you could probably do it with transactional locks with the :global module in about an additional 100-ish lines of code, including fully verifying the soundness of newly upgraded nodes using telemetry.
mix release
Compiles everything, tar/gz's it, sends it to a private s3 bucket. On your server side, you periodically watch the s3 bucket, and when one of your nodes detects a relup, it downloads from the s3 bucket, kills itself. Systemd then restarts it, kicking into the newest version.
That's it. This is maybe about 50-100 lines of code, one external library (pick your AWS library of choice), and one systemd script. I think there's even a library for managing systemd from within the BEAM now.
If you want to be more sophisticated (a rolling blue-green deploy across an erlang cluster) you could probably do it with transactional locks with the :global module in about an additional 100-ish lines of code, including fully verifying the soundness of newly upgraded nodes using telemetry.
Regarding portability, I recently came across this project: https://github.com/spawnfest/bakeware
I haven’t gotten around to actually trying it myself, but it advertises that it can compile Elixir projects into a single binary you can copy, paste & run (on Linux & MacOS). It’s not the greatest solution, and the mentioned ~0.5s startup time doesn’t make it great for cli tools when compared to Go/Rust/C.
I’m mostly just happy to see there are people out there trying to make Elixir/Erlang easier to use, as much as I love the language, some of the tooling and deployment methods (releases) make me groan.
I haven’t gotten around to actually trying it myself, but it advertises that it can compile Elixir projects into a single binary you can copy, paste & run (on Linux & MacOS). It’s not the greatest solution, and the mentioned ~0.5s startup time doesn’t make it great for cli tools when compared to Go/Rust/C.
I’m mostly just happy to see there are people out there trying to make Elixir/Erlang easier to use, as much as I love the language, some of the tooling and deployment methods (releases) make me groan.
I assume because every process has its own GC that messages are always copied, even on the same node? AFAICT, there is no shared/copy-on-write messages in the BEAM, except for a few cases where you explicitly pass a shared byte buffer for special circumstances? I would think this is worth the cost in most cases, but for some workloads would be prohibitive.
Some binaries (byte buffers as you called them) are stored outside the process heap, and those are messaged by reference, not by copying; but all other terms are messaged by copying. This is a lot of copying, but it makes the GC very simple, so it's usually a positive tradeoff. Although, there's probably some use cases where it's not great. For those, it might makes sense to force things into the off process heap binaries (RefC), or possibly by doing terrible things with native code (Nifs). You might also put things into the included key-value memory storage (ets), and more selectively pull things out to avoid copying a large chunk data to every process that needs a small piece.
It is possible to create nif objects which are tied into the garbage collector and reference counted, and bound to a destructor function that gets called when the VM is done with it!
> In a sense, supervisors are very similar to Kubernetes, but they work on the application level instead of the cluster level.
Good article, and very worth sharing, but it could have done without that part. That's a terribly unhelpful analogy. For those who Kubernetes it's wildly misleading, and for those who don't it's meaningless.
Good article, and very worth sharing, but it could have done without that part. That's a terribly unhelpful analogy. For those who Kubernetes it's wildly misleading, and for those who don't it's meaningless.
I use both kubernetes and OTP supervisors and don't find it misleading, rather I quite agree but maybe I'm missing something. I'm curious what you mean by that?
You don't find it misleading because you already know what an OTP Supervisor's role is limited to, and you know that Kubernetes (among a thousand other things) also provides that one feature. But its not a good analogy since it can only be understood by someone who doesn't need it to be explained.
Interesting, thanks for the feedback. I find it misleading because Kubernetes is a massive platform that does all manner of things besides just supervising your Pods, and in fact I had to think about it for a while before concluding that the supervisory function is likely what the author was alluding to.
It strikes me as saying that an oxygen mask in a hospital room is like an airplane: sure they both provide a facility for providing oxygen to the user, but the airplane is significantly more complex than simply an oxygen delivery system. Probably a terrible analogy, just thinking off the top of my head.
It strikes me as saying that an oxygen mask in a hospital room is like an airplane: sure they both provide a facility for providing oxygen to the user, but the airplane is significantly more complex than simply an oxygen delivery system. Probably a terrible analogy, just thinking off the top of my head.
I agree with your sentiment. I think a more apt metaphor than Kubernetes in its entirety would be comparing it to a Kubernetes ReplicaSet[1]. I can see some similarities there.
For fun, to expound on your analogy, maybe it's like comparing an airport to a hospital. They both feature a person you check in with (triage/ticket counter) to direct you to an appropriate room/gate depending on your needs, but the core mission of each are rather different.
[1] https://kubernetes.io/docs/concepts/workloads/controllers/re...
For fun, to expound on your analogy, maybe it's like comparing an airport to a hospital. They both feature a person you check in with (triage/ticket counter) to direct you to an appropriate room/gate depending on your needs, but the core mission of each are rather different.
[1] https://kubernetes.io/docs/concepts/workloads/controllers/re...
What a lot of people seem to want is a batteries included framework that does the OTP for you, but doing at a high level of abstraction is difficult. So more commonly you'll see the OTP managed by libraries for more concrete use cases (web servers, database connection pooling, back pressured job processing, etc). Trying to wrap or rebuild OTP from the primitives available in the BEAM is possible, but definitely not an easy task.
Could we potentially have an OTP-like library that cuts down on the boilerplate significantly? Yes. But how much effort would that take and is it much of a win over OTP as-is? Not sure the trade-off is entirely there yet - OTP really isn't that bad. You're still writing less boilerplate than Java in general.
I think we're much better off writing good library-level abstractions over OTP that can be embedded in a supervision tree or used in more specific use cases. If we were to actually attempt some kind of higher level abstraction in the scope of OTP - maybe we can do so with a very different approach like decorated dataflow graphs and runtime property based testing.