Elixir GenServer Explained(papercups.io)
papercups.io
Elixir GenServer Explained
https://papercups.io/blog/genserver
90 comments
Thank you so much for sharing this. I've not yet finished a decade in the industry, nor ever laid an eye on Elixir before, and this made sense pretty quickly. Great job explaining it all!
On a side note, its pretty amazing to see the impact on readability and understanding that native language features have on something like async. I've worked with similar systems to the example in the past in JS/Go/etc, and while you can definitely do the same stuff presented here, it tends to be a far messier callback-hell ime. For lack of better explanation, this Elixir code just "flows" better to my eye, which I know is a completely subjective and non-helpful unsolicited piece of info. Thanks again.
On a side note, its pretty amazing to see the impact on readability and understanding that native language features have on something like async. I've worked with similar systems to the example in the past in JS/Go/etc, and while you can definitely do the same stuff presented here, it tends to be a far messier callback-hell ime. For lack of better explanation, this Elixir code just "flows" better to my eye, which I know is a completely subjective and non-helpful unsolicited piece of info. Thanks again.
It flows better because it is writing synchronous code. The code inside of every process in Erlang is synchronous.
You achieve concurrency by spinning up processes (spawning functions). This isn't that different to firing an async function in Javascript, or starting a goroutine, etc.
And you communicate between the processes by sending async messages.
Those are the primitives. Asynchronously send messages between processes that are synchronously doing stuff. Go is actually not -that- different, except sending messages is synchronous, too, unless you set a buffer on the channel (and then you have to size it). Though I agree, because Erlang/Elixir keep it simpler than that, and are dynamically typed, it's a lot cleaner to write.
You achieve concurrency by spinning up processes (spawning functions). This isn't that different to firing an async function in Javascript, or starting a goroutine, etc.
And you communicate between the processes by sending async messages.
Those are the primitives. Asynchronously send messages between processes that are synchronously doing stuff. Go is actually not -that- different, except sending messages is synchronous, too, unless you set a buffer on the channel (and then you have to size it). Though I agree, because Erlang/Elixir keep it simpler than that, and are dynamically typed, it's a lot cleaner to write.
The formal framework for this is called CSP ("Communicating sequential processes"). I discovered this 15 years ago, and once you understand it, you will never have problems in multithreading again.
https://arild.github.io/csp-presentation/#3
https://arild.github.io/csp-presentation/#3
Not exactly. CSP is more what Golang adheres to (synchronous communication). The Actor Model predates CSP. And understanding it you'll still have problems...they're just different problems. You either won't be able to use a language with message passing as a concurrency primitive (i.e., almost all popular languages other than Go at this point), or you will, and now you'll have the time to spend on new types of problems (i.e., you can still create race conditions and deadlocks, and, in Erlang at least, now you get to start asking yourself what happens when things go wrong to a level you never would have with any other language).
The nice thing about the cleanly independent processes is that by and large, "let that part crash, and keep going" is a perfectly adequate answer to most process states outside the pretty path, which is a real saving grace when working with wildly concurrent systems.
Oh, certainly.
On this particular distinction, in fact, it's one of the benefits the actor model has over CSP. In the actor model (well, Erlang's take on it, but that's one that has been adopted elsewhere such as in Akka), I care about "if this fails, who should be affected by it". I.e., by default, it's just this process, and with a supervisor it'll restart. How do I get it restarting into a good state? If it goes down, who else needs to go down (to keep state consistent)? That's mostly all I have to ask myself; I don't have to ask what all the ways it could go down are (slight caveat there; you still want circuit breakers around external resources).
With CSP, I have to ask "if this goes down, who else IS affected by it". Not who should be, and be intentional about linking them, but determine who accidentally is affected by it. This is everyone who might be sending or receiving across a channel (to borrow Go parlance) this process has access to. Which breaks the abstraction; from the perspective of the current process I shouldn't have to know who is sending or receiving across a channel, but the reality is I -do-. And that's no better than the error model I have in other languages; I have to understand every part of my application to know what may or may not be affected by this one part going down. Go gets 'around' this by just making it so any uncaught exception (err, unrecovered panic) crashes the whole application, which is certainly one way to do it.
On this particular distinction, in fact, it's one of the benefits the actor model has over CSP. In the actor model (well, Erlang's take on it, but that's one that has been adopted elsewhere such as in Akka), I care about "if this fails, who should be affected by it". I.e., by default, it's just this process, and with a supervisor it'll restart. How do I get it restarting into a good state? If it goes down, who else needs to go down (to keep state consistent)? That's mostly all I have to ask myself; I don't have to ask what all the ways it could go down are (slight caveat there; you still want circuit breakers around external resources).
With CSP, I have to ask "if this goes down, who else IS affected by it". Not who should be, and be intentional about linking them, but determine who accidentally is affected by it. This is everyone who might be sending or receiving across a channel (to borrow Go parlance) this process has access to. Which breaks the abstraction; from the perspective of the current process I shouldn't have to know who is sending or receiving across a channel, but the reality is I -do-. And that's no better than the error model I have in other languages; I have to understand every part of my application to know what may or may not be affected by this one part going down. Go gets 'around' this by just making it so any uncaught exception (err, unrecovered panic) crashes the whole application, which is certainly one way to do it.
Oh that's so funny, because I'm an elixir user and we all claim that genserver (internal) organization is terrible.
But yes, it is way better than callback hell... It just could be better still.
But yes, it is way better than callback hell... It just could be better still.
not genserver, what will you use?
There isn't anything. Dave thomas proposed a better way of organizing genservers, but I can't get behind it because his implementation is too magical and clever, but his idea for what the api should look like in the big picture is reasonable... I'm also generally in favor having everyone speak the same idioms, so it would take a very good implementation to get me off of genserver.
That said you shouldn't write genservers if you can help it. Using the frameworks genservers Is the best choice, Task is a better choice for most cases.
That said you shouldn't write genservers if you can help it. Using the frameworks genservers Is the best choice, Task is a better choice for most cases.
[deleted]
I completely relate to the "code that just flows" statement. Cognitive overhead when dealing with code is a thing. I've been trying to explain this to a bunch of junior devs at every opportunity i get but all I get is a blank stare. Seems that the more senior and especially bittervet developers are a lot more sensitive to this. Thanks for sharing!
Since these insert events are being buffered, what happens if the process dies? Are all of those inserts lost?
I feel like Erlang/Elixir is designed to handle cases like this robustly, but it's not clear to me how this code avoids losing data when a process crashes, potentially up to 5s worth of updates!
I feel like Erlang/Elixir is designed to handle cases like this robustly, but it's not clear to me how this code avoids losing data when a process crashes, potentially up to 5s worth of updates!
Short Answer - yes, data will be lost. THAT SAID - this is true of any buffering solution, it's why even the 'sync' style of request can still time out.
Long Answer - What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir. There's a tradeoff that persistent storage is slower than memory, but it's persistent. So do you want to be fast, or do you want to be durable?
However, there is some flexibility to address this at a system level. Namely, you wait for confirmation. A client, be that an actual user, another process, etc, wants to know if the write has been persisted. Whereas many other languages make this sort of caching layer completely transparent to the client (i.e., they return success immediately, and so failures mean invisible data loss as the cache is dropped), Erlang/Elixir's model makes it so you HAVE to think about this. I sent a message to the downstream process; do I care about a response? If I don't get a response for any given interval, I don't know what happened to that message. I can still do useful work in the meantime, but I don't know that my message was fully handled (persisted). I can retry or I can report failure or whatever.
This is true in any distributed system, and in Erlang/Elixir it's expressed very evidently in the language constructs (rather than being hidden from you).
You can build local disk caches if you want (in fact, there are included tools to make this super easy for you; ETS, DETS, and Mnesia allow you to shove Erlang terms into memory storage, disk based storage, and a hybrid of the two with some nice DB-like behaviors, respectively), but you need to choose to slow your message ingestion to the speed of local disk writes in that case (as well as handle synchronization of deletes in the event of multiple writers to your downstream). An depending what your upstream is, that doesn't provide a guarantee (i.e., a write to local disk != a persisted write from a user perspective, because the disk could crash before it ever makes it off the local one).
Long Answer - What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir. There's a tradeoff that persistent storage is slower than memory, but it's persistent. So do you want to be fast, or do you want to be durable?
However, there is some flexibility to address this at a system level. Namely, you wait for confirmation. A client, be that an actual user, another process, etc, wants to know if the write has been persisted. Whereas many other languages make this sort of caching layer completely transparent to the client (i.e., they return success immediately, and so failures mean invisible data loss as the cache is dropped), Erlang/Elixir's model makes it so you HAVE to think about this. I sent a message to the downstream process; do I care about a response? If I don't get a response for any given interval, I don't know what happened to that message. I can still do useful work in the meantime, but I don't know that my message was fully handled (persisted). I can retry or I can report failure or whatever.
This is true in any distributed system, and in Erlang/Elixir it's expressed very evidently in the language constructs (rather than being hidden from you).
You can build local disk caches if you want (in fact, there are included tools to make this super easy for you; ETS, DETS, and Mnesia allow you to shove Erlang terms into memory storage, disk based storage, and a hybrid of the two with some nice DB-like behaviors, respectively), but you need to choose to slow your message ingestion to the speed of local disk writes in that case (as well as handle synchronization of deletes in the event of multiple writers to your downstream). An depending what your upstream is, that doesn't provide a guarantee (i.e., a write to local disk != a persisted write from a user perspective, because the disk could crash before it ever makes it off the local one).
> What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir.
A little over a decade ago I did a project using Erlang. My next project was using Ruby, and I was suddenly _horrified_ by the fact that I had no idea what would happen if the application crashed (and Ruby apps, at least in those days, crashed fairly often). Erlang/Elixir both forces you to think about these things, and gives you tools to address them, where many other languages (or their libraries) simply assume that we will stay on the happy path.
Of course, you can be very productive using languages that just ignore the possibility of very rare failures. Many successful systems work on the basis that sometimes shit just happens and maybe some data does get lost. But, after programming with Erlang or Elixir for a while, that situation starts to feel less acceptable!
A little over a decade ago I did a project using Erlang. My next project was using Ruby, and I was suddenly _horrified_ by the fact that I had no idea what would happen if the application crashed (and Ruby apps, at least in those days, crashed fairly often). Erlang/Elixir both forces you to think about these things, and gives you tools to address them, where many other languages (or their libraries) simply assume that we will stay on the happy path.
Of course, you can be very productive using languages that just ignore the possibility of very rare failures. Many successful systems work on the basis that sometimes shit just happens and maybe some data does get lost. But, after programming with Erlang or Elixir for a while, that situation starts to feel less acceptable!
100%
I consider the couple years I built systems in Erlang to be fundamental for me. It's affected, for better and worse, my entire approach to system design, at every level. It's meant the stuff I or (now that I'm in management) my teams tend to write is incredibly resilient (compared with the other teams in the department), but also meant that I have a really hard time with any Silicon Valley interview.
I consider the couple years I built systems in Erlang to be fundamental for me. It's affected, for better and worse, my entire approach to system design, at every level. It's meant the stuff I or (now that I'm in management) my teams tend to write is incredibly resilient (compared with the other teams in the department), but also meant that I have a really hard time with any Silicon Valley interview.
I 100% agree with this, but would add that actually in my case I feel like it has helped with Silicon Valley style interviews. Architecting a single BEAM application involves a lot of the same thought processes that go into designing any resilient distributed system so I have felt well prepared for any design questions. Additionally, being forced to write algorithms with immutable data structures, recursion and explicit state prepares you ridiculously well for solving the graph questions that FAANG like to give in interviews.
Specific design questions, yes. The "45 minute to design a system" ones, not so much. The hard part of those is not getting too bogged down in details, and this just means I have one more, extremely important, likely unappreciated by the interviewer, detail I can get bogged down in: what happens when things fail.
What happens when things fail is an absolutely critical part of design / architecture. It's disappointing that interviewers works consider that an unimportant detail.
Again, 45 minutes for system design interviews. I could go on a long diatribe of why the FAANG interview strategies are flawed, but everyone knows that, FAANG interviewers included. You play the game if you want the role; if you're just, you know, good, and don't want to learn to play the game, then you're an acceptable false negative.
The article touched on it briefly, but perhaps it wasn't clear.
If the process dies, the Supervisor invokes the `terminate/2` callback so you're still able to process events.
Here are the relevant lines: https://github.com/plausible/analytics/blob/b724def948d51a0f...
Keep in mind you need to trap exit signal to tell the supervisor to invoke the callback, as done so here: https://github.com/plausible/analytics/blob/b724def948d51a0f...
The erlang docs also mentions this: https://erlang.org/doc/design_principles/gen_server_concepts...
If the process dies, the Supervisor invokes the `terminate/2` callback so you're still able to process events.
Here are the relevant lines: https://github.com/plausible/analytics/blob/b724def948d51a0f...
Keep in mind you need to trap exit signal to tell the supervisor to invoke the callback, as done so here: https://github.com/plausible/analytics/blob/b724def948d51a0f...
The erlang docs also mentions this: https://erlang.org/doc/design_principles/gen_server_concepts...
I usually utilize this with an ETS cache to save and recover GenServer state in the event of a crash.
I use this callback to capture the 'data' that caused the crash, so that it can be used to recover later or better understand why the crash happened.
If I have as significantly long transaction, I can restart the transaction from transaction_id + 1 forward and then handle the actual transaction out of band by hand if its something very odd.
I usually insert a logger in these code paths too, but the crash handler is usually after the logger, which makes it easier to reason what has happened if you are dealing with mutable state.
If I have as significantly long transaction, I can restart the transaction from transaction_id + 1 forward and then handle the actual transaction out of band by hand if its something very odd.
I usually insert a logger in these code paths too, but the crash handler is usually after the logger, which makes it easier to reason what has happened if you are dealing with mutable state.
I think you might solve this by writing a "reliable" supervisor process which actually receives the messages, and preserves message or state history to replay for its supervised process in case it crashes. Of course this really just shoves the problem up the stack, but at least you can stick to "let it crash" when writing the supervised process, and take more care when writing the supervisor which has a smaller scope of responsibility.
Yea, they're just in memory, so they're lost. Erlang/Elixir doesn't solve this. There's no getting around the fact that you need an ack that your event was processed and saved to some durable store. A supervisor won't save you from lack of acknowledgment at the app level. Any restart or retry logic within beam itself is useless if a beaver chews through a power cord and the machine halts. And what is an ack, anyway? Is an ack from mongo as good as an ack from postgres? This is all stuff the language can't know. These are all business questions, all the language can do is give you tools handle these cases, which Erlang/Elixir does.
You can add retry logic in your genServer, but for anything but the most simple use cases, you would want to add a Supervisor and define a retry strategy.
https://elixir-lang.org/getting-started/mix-otp/supervisor-a...
https://elixir-lang.org/getting-started/mix-otp/supervisor-a...
Furthermore I have heard it is actually possible for casts to be dropped under load. I am having a hard time confirming this, and had a hard time confirming it back then too. And even worse, I realize most of the time we code as if the casts will be reliably delivered.
Can someone chime in on this?
Can someone chime in on this?
https://medium.com/learn-elixir/message-order-and-delivery-g... is what you want.
I'll chime in explicitly to your question though - messages don't just disappear without a reason, but the reason may not be visible to you. It's possible a receiving process got a message, crashed partway through processing it (possibly after you even sent other messages, so dropping those, too), restarted, and now is handling the next incoming messages, making it look like a message dropped. It's possible a receiving process is on another node (in distributed Erlang) and the packet dropped. Etc etc.
These all look the same. The way to handle them is always the same; accept you have at most once delivery, or include an 'ack or retry' mechanism and write your system to handle at least once delivery. As that article mentions, Erlang makes you start thinking of your system as a distributed system from the get go. This initially feels incredibly inconvenient, but as you go down the distributed system, fault tolerant, CAP-bound rabbit hole, it becomes incredibly helpful. The system doesn't hide the things it can't guarantee from you; it forces you to feel uneasy about them.
(Whereas, for example, in Java, I don't know that that worker thread has terminated. So some other thread's fiddling with some bit of shared memory to communicate something to that thread does nothing. Which I've had happen in production and led to a single instance out of the fleet have stale data, which caused us no end of grief. Erlang forces you to ask, as you've just done, "what happens if this message doesn't get received because something happened to the receiver"?)
I'll chime in explicitly to your question though - messages don't just disappear without a reason, but the reason may not be visible to you. It's possible a receiving process got a message, crashed partway through processing it (possibly after you even sent other messages, so dropping those, too), restarted, and now is handling the next incoming messages, making it look like a message dropped. It's possible a receiving process is on another node (in distributed Erlang) and the packet dropped. Etc etc.
These all look the same. The way to handle them is always the same; accept you have at most once delivery, or include an 'ack or retry' mechanism and write your system to handle at least once delivery. As that article mentions, Erlang makes you start thinking of your system as a distributed system from the get go. This initially feels incredibly inconvenient, but as you go down the distributed system, fault tolerant, CAP-bound rabbit hole, it becomes incredibly helpful. The system doesn't hide the things it can't guarantee from you; it forces you to feel uneasy about them.
(Whereas, for example, in Java, I don't know that that worker thread has terminated. So some other thread's fiddling with some bit of shared memory to communicate something to that thread does nothing. Which I've had happen in production and led to a single instance out of the fleet have stale data, which caused us no end of grief. Erlang forces you to ask, as you've just done, "what happens if this message doesn't get received because something happened to the receiver"?)
I think that casts are guaranteed delivery if the process exists. Problem is, the caster can't know that with certainty (there could be a race between checking, process death and sending). Or the recipient could just ignore the cast altogether. Out if the box it's impossible to know which happened.
Calls by contrast check out a monitor on the counterparty and crash with a timeout so you have guaranteed delivery and acknowledgement, or crash the calling process.
A lot of times people from other plarforms rush to use casts when they "don't need a response" but the actual meanings have more to do with failure domains and rate limiting back pressure; my personal feeling is you should default to call and only use cast when you need failure isolation.
Calls by contrast check out a monitor on the counterparty and crash with a timeout so you have guaranteed delivery and acknowledgement, or crash the calling process.
A lot of times people from other plarforms rush to use casts when they "don't need a response" but the actual meanings have more to do with failure domains and rate limiting back pressure; my personal feeling is you should default to call and only use cast when you need failure isolation.
I don't think that's quite true, at least not of a gen_X cast.
If you use erlang:send/3 with options or erlang:send_nosuspend/2, you can have some cases where messages would be dropped without trying. I think that may be the source of the drop under load you're thinking of?
Without nosuspend, if you send to a node that's connected, dist will queue it to be sent, but there's no guarantee it's received because networking, and it may not even be sent if the other side has gone away and the send queue is already too large; There is a hard to express constraint that if your message doesn't get received in this case, dist will disconnect eventually, but it's important to note that the tick time-out doesn't guarantee timely delivery either; as long as some data is flowing, you can get quite a backlog; I think I've gotten net_adm:ping times above 30 minutes in some cases. Also, if the dist connection is dropped, but both nodes are online, it will likely reconnect shortly, and some messages will have been lost.
If you send to a node that's not connected, and didn't specify no_connect, dist will queue the message while attempting to connect, but if that attempt fails, the messages will be dropped.
It's also possible for code running as a gen_server (or GenServer, I suppose) to check how many messages are queued for it, and run different logic. I've written gen_servers that would drop optional requests if the queue was large. Also, if client timeouts are known, there are ways to approximate the time spent waiting in queue, and drop requests if they are received after the client already timed out; it's a little tricky to do this though.
If you use erlang:send/3 with options or erlang:send_nosuspend/2, you can have some cases where messages would be dropped without trying. I think that may be the source of the drop under load you're thinking of?
Without nosuspend, if you send to a node that's connected, dist will queue it to be sent, but there's no guarantee it's received because networking, and it may not even be sent if the other side has gone away and the send queue is already too large; There is a hard to express constraint that if your message doesn't get received in this case, dist will disconnect eventually, but it's important to note that the tick time-out doesn't guarantee timely delivery either; as long as some data is flowing, you can get quite a backlog; I think I've gotten net_adm:ping times above 30 minutes in some cases. Also, if the dist connection is dropped, but both nodes are online, it will likely reconnect shortly, and some messages will have been lost.
If you send to a node that's not connected, and didn't specify no_connect, dist will queue the message while attempting to connect, but if that attempt fails, the messages will be dropped.
It's also possible for code running as a gen_server (or GenServer, I suppose) to check how many messages are queued for it, and run different logic. I've written gen_servers that would drop optional requests if the queue was large. Also, if client timeouts are known, there are ways to approximate the time spent waiting in queue, and drop requests if they are received after the client already timed out; it's a little tricky to do this though.
Erlang casts are always delivered, or the sender crashes before that point.
To local pids. Remote, no guarantee is made. Ergo, generally better to assume no guarantees. Same as with everything else; you have at most once, or at least once; there is no exactly once (except in marketing speak that uses an identifier to allow at least once to dedupe within a window).
[deleted]
Actually in Erlang/elixir, your own crash is something you can plan for and handle. if you check the code you will see in init() that the author captures exits. then there is a function terminate () which is called when a process exits for some reason (including crashes)
So for example one possible implementation of state is that you catch your own crash, save some state and then restore it when you restart
You aren't supposed to ever crash, however the whole language it's based around designing a framework to handle what happens if you do crash
So you design a whole framework of supervisors, watching processes, that all have a defined startup and shutdown order. You build structures with rules such as: if my cache module fails, them restart this whole chunk of application over here.
it's very cool
So for example one possible implementation of state is that you catch your own crash, save some state and then restore it when you restart
You aren't supposed to ever crash, however the whole language it's based around designing a framework to handle what happens if you do crash
So you design a whole framework of supervisors, watching processes, that all have a defined startup and shutdown order. You build structures with rules such as: if my cache module fails, them restart this whole chunk of application over here.
it's very cool
It doesn't. We read data from the db in the init function when the GenServer starts and make callers persist it before sending it to the GenServer. Our GenServers keep data in memory as the one in the post but we could read it from the db each time it ticks to process a record.
In this case, that's correct—you could lose up to 5s worth of state, plus any buffered messages. That's by design here. The business impact of that is going to depend on use-case, but I personally would find it difficult to believe that there would be any real impact in that event, in this case, under high load.
Obviously, different tolerances to that are going to necessitate different designs. You can configure the size of the message buffer when a GenServer starts, and if you have very low tolerance for lost data, you'd want to use a synchronous message (using call instead of cast, which blocks the sender until it returns) and appropriate error handling.
BEAM has a plethora of features for reliable applications, you just have to apply them appropriately.
Obviously, different tolerances to that are going to necessitate different designs. You can configure the size of the message buffer when a GenServer starts, and if you have very low tolerance for lost data, you'd want to use a synchronous message (using call instead of cast, which blocks the sender until it returns) and appropriate error handling.
BEAM has a plethora of features for reliable applications, you just have to apply them appropriately.
I'd like a solid answer to this too; it's my biggest concern with any sort of "batch online requests" functionality in any language; unsafe shutdowns can/will always happen at some point, and it feels like this is always a data loss risk.
To some degree there's always data loss risk... A backhoe could cut a line, you could lose power to a PSU, rack, data center, a meteor could destroy the willamette valley, etc. What you want your system to provide you with is the framework to understand the risks and what the recovery looks like and when recovery is a lost cause.
Isn't it true of any, in-memory buffer implementation? It is very easy to add some form of recovery/persistence to a GenServer. You can have it write to a DB, ETS or disk. You lose some throughput obviously, it's all about balance.
In case anyone was curious what I used for the diagrams in this article, it was an awesome open source [0] tool called Excalidraw! [1]
[0] https://github.com/excalidraw/excalidraw
[1] https://excalidraw.com/
[0] https://github.com/excalidraw/excalidraw
[1] https://excalidraw.com/
Excalidraw is a gem. I discovered it weeks ago, and cannot but praise it. The online instance is free and works really well, plus you have the possibility of hosting your own copy and be completely independent.
I also saw that there is the possibility of doing collaborative real time drawing, but still did not try it.
I have seen that the exported SVGs could be simplified (lots of repeated markup), and I am thinking about giving a try to do a PR.
Excalidraw is superb.
I also saw that there is the possibility of doing collaborative real time drawing, but still did not try it.
I have seen that the exported SVGs could be simplified (lots of repeated markup), and I am thinking about giving a try to do a PR.
Excalidraw is superb.
This and the GP comment got me curious to look at past threads because I remember previous discussion of that project. In case anyone else is curious:
Show HN: Eraser — Excalidraw-based visual meeting canvas - https://news.ycombinator.com/item?id=26330288 - March 2021 (8 comments)
One Year of Excalidraw - https://news.ycombinator.com/item?id=25608336 - Jan 2021 (19 comments)
Show HN: Create a Slideshow with Excalidraw - https://news.ycombinator.com/item?id=25081914 - Nov 2020 (1 comment)
Excalidraw whiteboard – easily sketch diagrams with a hand-drawn feel - https://news.ycombinator.com/item?id=23525648 - June 2020 (54 comments)
Building Excalidraw's P2P Collaboration Feature - https://news.ycombinator.com/item?id=22719576 - March 2020 (1 comment)
End-to-end encryption in the browser - https://news.ycombinator.com/item?id=22663435 - March 2020 (107 comments)
End-to-End Encryption in the Browser and how we did it in Excalidraw - https://news.ycombinator.com/item?id=22655009 - March 2020 (4 comments)
Show HN: Excalidraw – Sketch Hand-Drawn Like Diagrams - https://news.ycombinator.com/item?id=22146973 - Jan 2020 (6 comments)
Excalidraw – a whiteboard tool to sketch hand-drawn diagrams (excalidraw.com) - https://news.ycombinator.com/item?id=22104502 - Jan 2020 (3 comments)
Excalidraw – a whiteboard tool that lets you sketch hand-drawn diagrams - https://news.ycombinator.com/item?id=22101381 - Jan 2020 (21 comments)
Show HN: Eraser — Excalidraw-based visual meeting canvas - https://news.ycombinator.com/item?id=26330288 - March 2021 (8 comments)
One Year of Excalidraw - https://news.ycombinator.com/item?id=25608336 - Jan 2021 (19 comments)
Show HN: Create a Slideshow with Excalidraw - https://news.ycombinator.com/item?id=25081914 - Nov 2020 (1 comment)
Excalidraw whiteboard – easily sketch diagrams with a hand-drawn feel - https://news.ycombinator.com/item?id=23525648 - June 2020 (54 comments)
Building Excalidraw's P2P Collaboration Feature - https://news.ycombinator.com/item?id=22719576 - March 2020 (1 comment)
End-to-end encryption in the browser - https://news.ycombinator.com/item?id=22663435 - March 2020 (107 comments)
End-to-End Encryption in the Browser and how we did it in Excalidraw - https://news.ycombinator.com/item?id=22655009 - March 2020 (4 comments)
Show HN: Excalidraw – Sketch Hand-Drawn Like Diagrams - https://news.ycombinator.com/item?id=22146973 - Jan 2020 (6 comments)
Excalidraw – a whiteboard tool to sketch hand-drawn diagrams (excalidraw.com) - https://news.ycombinator.com/item?id=22104502 - Jan 2020 (3 comments)
Excalidraw – a whiteboard tool that lets you sketch hand-drawn diagrams - https://news.ycombinator.com/item?id=22101381 - Jan 2020 (21 comments)
It is very interesting to me that components in OTP like GenServer, Register, and Supervisor are surprisingly intuitive if one tries to learn and play Erlang first, but not so obvious and possibly looks artificial if people don't learn Erlang in the first place.
At first, I was surprised that the Erlang processes resemble the Actor model in such a minimalistic interface. Then I play with them for a while, I realized I wanted to extract something similar to GenServers. Then when I played with GenServers, soon I was bothered by a bunch of Pids, and yearning for Registry. In the end, Supervisor is also pretty natural because things in the universe generally cannot revive themselves.
In a retrospective, OTP resembles a lot of concepts we are familiar with, the Registry just looks like how the web works because we need to resolve the address problem. And vice versa: Docker, Service Discovery, and Kubernetes look like GenServer, Registry, Supervisor respectively just at different scales.
It seems to me to let our systems fulfill some properties, some recurring themes are required. We'll reuse them or rediscover them one way or another.
At first, I was surprised that the Erlang processes resemble the Actor model in such a minimalistic interface. Then I play with them for a while, I realized I wanted to extract something similar to GenServers. Then when I played with GenServers, soon I was bothered by a bunch of Pids, and yearning for Registry. In the end, Supervisor is also pretty natural because things in the universe generally cannot revive themselves.
In a retrospective, OTP resembles a lot of concepts we are familiar with, the Registry just looks like how the web works because we need to resolve the address problem. And vice versa: Docker, Service Discovery, and Kubernetes look like GenServer, Registry, Supervisor respectively just at different scales.
It seems to me to let our systems fulfill some properties, some recurring themes are required. We'll reuse them or rediscover them one way or another.
In BEAM circles, a modification go Greenspun's 10th rule is pretty popular:
"Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang."
Erlang's been around long enough at this point that for pretty much any problem in Erlang's area of expertice you find yourself solving twice, there's a damn good existing solution somewhere in the OTP.
"Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang."
Erlang's been around long enough at this point that for pretty much any problem in Erlang's area of expertice you find yourself solving twice, there's a damn good existing solution somewhere in the OTP.
> I'm ashamed to say most of my Elixir education has been through trial and error, figuring things out as I go along.
Oh, gosh. This is how I learned... everything? I feel like I just walked into homeroom without my pants on.
Oh, gosh. This is how I learned... everything? I feel like I just walked into homeroom without my pants on.
> In particular, I was itching to learn more about handling concurrency in Elixir. This, of course, led me to GenServers.
Might be a nitpicking here, but GenServers aren't useful for concurrency. They just manage state, and only process one message at a time. If you're using this as a cache, your reads will be bottlenecked by however quickly the GenServer can handle the read requests.
Might be a nitpicking here, but GenServers aren't useful for concurrency. They just manage state, and only process one message at a time. If you're using this as a cache, your reads will be bottlenecked by however quickly the GenServer can handle the read requests.
If you've got 5 gen_servers and you cast each one a message then that is concurrency. The whole point of a gen_server is that it's a separate process doing it's own thing.
I can dump 1000s of things into its message queue and then do something else. It'll keep working away.
It's like saying threads aren't useful for concurrency because they can only do one thing at a time.
I can dump 1000s of things into its message queue and then do something else. It'll keep working away.
It's like saying threads aren't useful for concurrency because they can only do one thing at a time.
I would advise to not rely on casting too much because it lacks backpressure.
> GenServers aren't useful for concurrency.
Sure, maybe not by themselves but typically you would run many GenServers concurrently in your application as part of a supervision tree. Libraries like Broadway (and the underlying GenStage) are essentially just leveraging GenServer to make it easier to orchestrate concurrency and state synchronization across multiple processes in your application. But you could build a comparable system on your own just using GenServer and a dynamic supervisor.
Sure, maybe not by themselves but typically you would run many GenServers concurrently in your application as part of a supervision tree. Libraries like Broadway (and the underlying GenStage) are essentially just leveraging GenServer to make it easier to orchestrate concurrency and state synchronization across multiple processes in your application. But you could build a comparable system on your own just using GenServer and a dynamic supervisor.
Well explained article. Good job OP!
GenServer is just one of the numerous things in the Elixir ecosystem that just bring joy. Once you I got the concepts behind Ecto and GenServers everything just felt easy to model in my mind. I find it so nice.
Erlang's OTP and Elixir's wrappers on top of that is what is severely lacking from all languages claiming to provide tools for better concurrency.
Maybe a dumb complaint, but I hate the name. I have a really hard time not reading it as the active name "Generate Server". Like, I'd expect it to be a function that generates servers (whatever that means, since what it's dealing with isn't something I'd normally call "a server" either). It bugs the hell out of me every time I read Elixir code. One of those things I have to keep reminding myself: "OK, it's not what it looks like it is, at all... what did these misleading terms actually mean, again?"
It's "generic" server: http://erlang.org/doc/man/gen_server.html
The sooner you start smacking people with tribal knowledge, the sooner many people start to back away slowly.
I used to interview ex-coworkers. I don't do it as much anymore because it gets old/depressing and the answers tend not to change that much.
If, for the purposes of thinking about turnover, you look at a former colleague as an 'aggrieved party', we are generally somewhere in the neighborhood of mediocre at agreeing that certain things were the 'final straw', and work backward through that to actions to increase retention.
One of the things that interested me about these conversations was that people tend to work chronologically backward through many of their complaints. But what was surprising was that some people would go all the way back to their first weeks. To the first straws. The warning signs they ignored. In a way this is not unlike talking to someone who just broke up with a romantic partner.
And while their ex may realize that his/her problems started back with some early inconsiderate behavior that snowballed, and resolve to 'do better next time', I rarely see companies do this, unless I instigate it.
Point is, these petty slights stack up, and can become a big part of someone's narrative for abandoning you (often in a huff). First impressions are important, and you dismiss them at your own peril.
I used to interview ex-coworkers. I don't do it as much anymore because it gets old/depressing and the answers tend not to change that much.
If, for the purposes of thinking about turnover, you look at a former colleague as an 'aggrieved party', we are generally somewhere in the neighborhood of mediocre at agreeing that certain things were the 'final straw', and work backward through that to actions to increase retention.
One of the things that interested me about these conversations was that people tend to work chronologically backward through many of their complaints. But what was surprising was that some people would go all the way back to their first weeks. To the first straws. The warning signs they ignored. In a way this is not unlike talking to someone who just broke up with a romantic partner.
And while their ex may realize that his/her problems started back with some early inconsiderate behavior that snowballed, and resolve to 'do better next time', I rarely see companies do this, unless I instigate it.
Point is, these petty slights stack up, and can become a big part of someone's narrative for abandoning you (often in a huff). First impressions are important, and you dismiss them at your own peril.
What? GP's response is terse but not impolite, and explaining what an abbreviation means is a far cry from "smacking people with tribal knowledge", it's just table stakes explanation in a field which requires precision.
The only reason modern computing got where it is today is by standing on the shoulders of giants. If we go back and re-litigate every naming decision of the past several decades just to make newcomers feel welcome, all progress will slow as we collectively devolve into continuous bikeshedding.
The only reason modern computing got where it is today is by standing on the shoulders of giants. If we go back and re-litigate every naming decision of the past several decades just to make newcomers feel welcome, all progress will slow as we collectively devolve into continuous bikeshedding.
FWIW I thought the response was entirely fine and it wouldn't have occurred me to take offense at it.
(I actually googled what "gen" in "GenServer" was supposed to mean before posting, because I couldn't remember but knew it wasn't "generate", and couldn't find an answer, including in the docs for GenServer)
[EDIT] to clarify, I failed to find the answer in any of the Elixir docs for GenServer. Evidently I should have looked at Erlang.
(I actually googled what "gen" in "GenServer" was supposed to mean before posting, because I couldn't remember but knew it wasn't "generate", and couldn't find an answer, including in the docs for GenServer)
[EDIT] to clarify, I failed to find the answer in any of the Elixir docs for GenServer. Evidently I should have looked at Erlang.
Version 1.0.4, first paragraph:
https://hexdocs.pm/elixir/1.0.4/GenServer.html#content
`The advantage of using a generic server process (GenServer)`
current release (1.11.4), first paragraph;
https://hexdocs.pm/elixir/1.11.4/GenServer.html#content
`The advantage of using a generic server process (GenServer)`
https://hexdocs.pm/elixir/1.0.4/GenServer.html#content
`The advantage of using a generic server process (GenServer)`
current release (1.11.4), first paragraph;
https://hexdocs.pm/elixir/1.11.4/GenServer.html#content
`The advantage of using a generic server process (GenServer)`
TIL: ignore elixir-lang.org, go to hexdocs.
Elixir-lang.org > docs > elixir takes you straight to hexdocs.
I'm talking about the name GenServer, not brobinson's response.
There is the corollary:
Believe it or not, I'm pro-Elixir. It's just that as I learn any new technology, I start filling out proverbial bingo card boxes for the things I can predict the next person will complain about. Is that a bit cynical? It could be, but it also serves as my todo list for when someone makes an offhand comment about how you really don't have to do this dumb thing, you could use this library that does it for you. Now my coworkers either don't have to worry about that or I have a suggestion when they do.
Turns out when people are in pain, they appreciate sympathy a lot more than they appreciate being gaslit about how it's just their poor sense of history. Their blatantly implied selfishness.
GenServer could use a new name. I think we forget sometimes that you can rename old things by giving them a second, better name and phasing out the old one. I don't think there's anything seditious in that statement.
There is the corollary:
If we have not seen farther, it is because giants were standing on our toes.
Believe it or not, I'm pro-Elixir. It's just that as I learn any new technology, I start filling out proverbial bingo card boxes for the things I can predict the next person will complain about. Is that a bit cynical? It could be, but it also serves as my todo list for when someone makes an offhand comment about how you really don't have to do this dumb thing, you could use this library that does it for you. Now my coworkers either don't have to worry about that or I have a suggestion when they do.
Turns out when people are in pain, they appreciate sympathy a lot more than they appreciate being gaslit about how it's just their poor sense of history. Their blatantly implied selfishness.
GenServer could use a new name. I think we forget sometimes that you can rename old things by giving them a second, better name and phasing out the old one. I don't think there's anything seditious in that statement.
I am not very deep on Elixir, but I have heard of some substantive complaints about GenServer as a specific implementation and the cultural effects of the community say "just use GenServer" which shuts down discourse around why it might not be appropriate or at least optimal for all the cases it's recommended. We should absolutely have those discussions. I'm just not sure how many problems are solved by picking a better name.
I think the community typically says "don't use genserver" more than "just use genserver"
I've been using Elixir for 4 years, it pays my bills and I generally like it. GenServer could be a misleading name (it never occurred to me) but the real mess is the bag of handle this / handle that function names. They follow the conventions of Erlang [1] but Elixir's developers could have cast some syntactic sugar on top of it. The real name of those functions is in their first argument. handle_* is mostly noise.
[1] https://erlang.org/doc/man/gen_server.html
[1] https://erlang.org/doc/man/gen_server.html
Syntactic sugar is of course another form of abstraction and so you have to balance developer ergonomics with that added complexity. As you point out, Elixir's `GenServer` is a fairly simple wrapper module over Erlang's own `gen_server`. As new versions of OTP are released, the maintainers of the Elixir language must also update their own abstractions over those underlying Erlang libraries. It also creates a niche disparity between Erlang and Elixir which some Erlang developers might find superfluous. Coming from Erlang, it's easy to appreciate the syntax of Elixir even if it means getting over some old habits. Perhaps changing the names of foundational parts of the standard library would be less appreciated by those already familiar with OTP, and feel as if the mental overhead is being shifted to those developers rather than simply being ameliorated for developers who are new to the platform.
Designing a programming language is hard, especially when building on top of a 35 year old language like Erlang. As is often the case: if engineers could change the past without breaking everything in the present, we would have already done it. :)
Designing a programming language is hard, especially when building on top of a 35 year old language like Erlang. As is often the case: if engineers could change the past without breaking everything in the present, we would have already done it. :)
They did the right thing if their goal was to move Erlang developers to Elixir. Not so much if they aimed to onboard Java or C++ developers. If they did, GenServer had to look like a Class like and they would use { } instead of do end. Strangely it seems that they aimed to Ruby developers (like me). That's why they picked do end and similar names for modules and functions. But Ruby developers wouldn't mind a GenServer looking like a class with proper function names instead of atoms in argument lists.
if you don't mind a bit of boilerplate, I use this pattern, it avoids syntactic sugar and achieves what you are looking for:
https://www.youtube.com/watch?v=HA4h0cajgaA
https://www.youtube.com/watch?v=HA4h0cajgaA
I think it is accepted in the Erlang/Elixir community that some names are pretty poor yeah. GenServer, OTP, Application. Even Supervisor to a degree, I've read a post making the case that it's more a lifecycle manager than a supervisor.
Anyway, it's just about getting used to the terms and what they truly mean.
Anyway, it's just about getting used to the terms and what they truly mean.
When I first came to Elixir & Erlang, their naming convention gave me no trouble at all. Now of course, I haven't done much for decades with languages that would have given me the mental model that would bias me to automatically be thinking of the things you name. I do know that there are certain well-worn development environments where you might gain such a bias.
Remember that Erlang, too, and some of these technologies, names, and conventions, are pretty old and that they may not have been as evocative then as they might be today with some of their conventions. "Process" almost certainly would have been confusing to the neophyte, but appropriately descriptive nonetheless, but "Gen" for Generic.... eh.
Anyway I guess the point is that I have to look inward to see if something like this rubbing me the wrong way is the substandard choice of the project I'm diving into, or if it's me bringing unwarranted biases and assumptions to the table.
Remember that Erlang, too, and some of these technologies, names, and conventions, are pretty old and that they may not have been as evocative then as they might be today with some of their conventions. "Process" almost certainly would have been confusing to the neophyte, but appropriately descriptive nonetheless, but "Gen" for Generic.... eh.
Anyway I guess the point is that I have to look inward to see if something like this rubbing me the wrong way is the substandard choice of the project I'm diving into, or if it's me bringing unwarranted biases and assumptions to the table.
It's a terrible name, anything starting with the abbreviated prefix "Gen" is hopelessly ambiguous. Is it "Generate"? "General"? "Generic"? "Genetic"? "Genesis"? While some would certainly be less frequently encountered, all can easily be found in programming context. When seeing "GenXxxxx" I could well understand someone feeling immediate brain-fog.
That said, this is one of very few criticisms I can level at Elixir, and it's a tiny part of the eco-system - literally, one single name. I wonder would it be easy to alias?
That said, this is one of very few criticisms I can level at Elixir, and it's a tiny part of the eco-system - literally, one single name. I wonder would it be easy to alias?
Yup.
alias GenServer, as: GenericServer
Done.
alias GenServer, as: GenericServer
Done.
You got downvoted, but I agree. "Gen" is very typically used as a shorthand for "generate", whereas I've _only_ seen it used as a shorthand for "generic" in the Erlang/Elixir world.
I have sympathy for gp. But also, naming things is hard. Wait till you dig deeper into the erlang rabbit hole and discover the undocumented gen module.
And in recent computer science (~30 years) we often pick unintuitive names for things.
Examples: WebAssembly(neither web, nor assembly), Serverless (has a server, actually), JavaScript.
Examples: WebAssembly(neither web, nor assembly), Serverless (has a server, actually), JavaScript.
Upvoted, then downvoted to hell, haha.
IDK, I don't live in Elixir and that name trips me up every damn time I need to read/write some. Everywhere else, "gen" is typically a shortening of "generate" (which I don't love either—just write the word—but it's fairly common), and "generic" rarely occurs in code at all (elsewhere related to programming, yes—in code, no)
[EDIT] incidentally, as long as I'm complaining about Elixir, I've done a lot of Ruby and have no clue whatsoever why people act like Elixir is similar to it, yet constantly see "oh yeah, it's so easy for Rubyists because it's so similar". Then again, I haven't done any Phoenix with Elixir, so maybe they just mean Phoenix is Rails-like.
IDK, I don't live in Elixir and that name trips me up every damn time I need to read/write some. Everywhere else, "gen" is typically a shortening of "generate" (which I don't love either—just write the word—but it's fairly common), and "generic" rarely occurs in code at all (elsewhere related to programming, yes—in code, no)
[EDIT] incidentally, as long as I'm complaining about Elixir, I've done a lot of Ruby and have no clue whatsoever why people act like Elixir is similar to it, yet constantly see "oh yeah, it's so easy for Rubyists because it's so similar". Then again, I haven't done any Phoenix with Elixir, so maybe they just mean Phoenix is Rails-like.
GenServer is a (behavior) module name. IMO, it doesn’t make a lot of sense to expect it to mean “generate server”. If it was a gen_server() function then sure. Or a ServerGen module - that generates servers.
The name makes me read it as "generate server". Nothing else about it fits with that. That's the problem.
This line in the opening paragraph really rubs me the wrong way:
> “I'm ashamed to say most of my Elixir education has been through trial and error, figuring things out as I go along“
This attitude is so prevalent in software as if we are all supposed to be divined with programming knowledge the moment our IDE spins up. In every other industry that’s exactly how you learn: Get your hands dirty, make mistakes, and fix them.
There’s a lot of things wrong with software engineering - harboring this attitude that self-taught learning is bad or shameful makes it unnecessarily worse
Great write-up otherwise
> “I'm ashamed to say most of my Elixir education has been through trial and error, figuring things out as I go along“
This attitude is so prevalent in software as if we are all supposed to be divined with programming knowledge the moment our IDE spins up. In every other industry that’s exactly how you learn: Get your hands dirty, make mistakes, and fix them.
There’s a lot of things wrong with software engineering - harboring this attitude that self-taught learning is bad or shameful makes it unnecessarily worse
Great write-up otherwise
There's a middle ground between (relatively) blind trial and error and being divinely granted insight. Deliberate reading of documentation (be it API, language standard, books, tutorials (better ones), etc.) lets you learn without just trying things or piecing a theory together from examples (underdocumented ones that don't explain the why of their choices).
you could also pay for education, attend workshops, code projects on a team, etc.
OP here -- I know what you mean. "Ashamed" is probably too strong a word. I think the feeling I was trying to convey was just that I'm not exactly an authority on Elixir, so take everything I say with a grain of salt, and nitpick away :)
I know the feeling though, sometimes it feels like other people's code just comes in to being immaculate and pure, while I just beat my head against the keyboard until one of us gives up.
Self-learning is one thing, but uninformed trial and error based on a lack of research is another beast entirely.
> There’s a lot of things wrong with software engineering - harboring this attitude that self-taught learning is bad or shameful makes it unnecessarily worse
This is an interesting question.
There's nothing wrong with being an autodidact, per se, but in my experience in hiring people and working with autodidacts: Autodidacts are not very good at recognizing their own limitations[0] and they usually do not have a lot of breadth of knowledge outside their area of interest. This can even go so far as not knowing that there is such a thing as "Entire Field of Study Foo". (Let's say Foo is Discrete Mathematics.)
Will they be bad hires? It depends on what you need -- and they can absolutely perform as well or better than any "more educated" person. It very much depends on the actual person... as almost all work does.
[0] This explicitly ties into the second point about breadth of knowledge. It's about knowing what you don't know and are willing to research further. The problem with autodidacts is often that there's such an insane leap from "oh, i know this" to a seemingly totally unrelated field which has deep connections to what you're doing. That's what a "broad" education in CS is there to teach you. Are you going to use all of that? Hell no! But just maybe that 5% you do end up using makes up for the 95% you don't.
This is an interesting question.
There's nothing wrong with being an autodidact, per se, but in my experience in hiring people and working with autodidacts: Autodidacts are not very good at recognizing their own limitations[0] and they usually do not have a lot of breadth of knowledge outside their area of interest. This can even go so far as not knowing that there is such a thing as "Entire Field of Study Foo". (Let's say Foo is Discrete Mathematics.)
Will they be bad hires? It depends on what you need -- and they can absolutely perform as well or better than any "more educated" person. It very much depends on the actual person... as almost all work does.
[0] This explicitly ties into the second point about breadth of knowledge. It's about knowing what you don't know and are willing to research further. The problem with autodidacts is often that there's such an insane leap from "oh, i know this" to a seemingly totally unrelated field which has deep connections to what you're doing. That's what a "broad" education in CS is there to teach you. Are you going to use all of that? Hell no! But just maybe that 5% you do end up using makes up for the 95% you don't.
It's not about being self taught, its about the phenomena of dabbling around a conceptual component in aOS/Runtime/Framework/Language that you aren't quite ready to commit to fully understanding and you just want to know enough to get something done. Quite often, this is a pragmatic choice, but if the thing you are dealing with is a central part of your system, then this becomes untenable long term. Unfortunately this is actually semi common and some very strange software gets written because things weren't properly understood. This article is the authors commitment to learn one of these concepts.
[deleted]
Do you think that you can install a gas turbine the wrong way start it up and fix it later?
Like most things, you build up to it.
You start software development with "Hello, world!" and then you do more, like getting input from the user, storing it in a variable, then eventually you're using classes and working with objects, then you're tying together a bunch of classes and working with APIs.
No one just "studied" how to create a gas turbine and wah-la, it was made. The entire process of literally everything was one learning exercise after another. We started by using rocks as tools. Here we are, having built better tools from experience and a lot of trial and error and fooling around with things.
Hell, Lego are the same exact idea.
You start software development with "Hello, world!" and then you do more, like getting input from the user, storing it in a variable, then eventually you're using classes and working with objects, then you're tying together a bunch of classes and working with APIs.
No one just "studied" how to create a gas turbine and wah-la, it was made. The entire process of literally everything was one learning exercise after another. We started by using rocks as tools. Here we are, having built better tools from experience and a lot of trial and error and fooling around with things.
Hell, Lego are the same exact idea.
https://learnyousomeerlang.com/clients-and-servers