Twenty Years of Open Source Erlang: A Retrospective from the trenches(erlang-solutions.com)
erlang-solutions.com
Twenty Years of Open Source Erlang: A Retrospective from the trenches
https://www.erlang-solutions.com/blog/twenty-years-of-open-source-erlang.html
69 comments
Is Elixir more than just a ruby syntax of erlang? Because I prefer Erlang’s syntax.
Elixir is more than syntax. Or rather, it is not even about the syntax :) This is a common question, so I even gave a talk about it some time ago: https://www.youtube.com/watch?v=V5fMQcSy3y8
Elixir features meta-programming, structs and protocols, first-class documentation, strong focus on the tooling, some abstractions that make concurrency more accessible (such as tasks and streams), etc.
However, Erlang and Elixir share a lot! They have the same data-types (with the same names and even the same syntax for almost all of them), the runtime is the same, and the same foundation about processes, fault-tolerance and distribution.
My suggestion is to go with whatever "touches your heart" because, even if Elixir has its own features, they are more alike than they are different and most concepts you learn for one will also apply to the other.
Elixir features meta-programming, structs and protocols, first-class documentation, strong focus on the tooling, some abstractions that make concurrency more accessible (such as tasks and streams), etc.
However, Erlang and Elixir share a lot! They have the same data-types (with the same names and even the same syntax for almost all of them), the runtime is the same, and the same foundation about processes, fault-tolerance and distribution.
My suggestion is to go with whatever "touches your heart" because, even if Elixir has its own features, they are more alike than they are different and most concepts you learn for one will also apply to the other.
> They have the same data-types (with the same names and even the same syntax for almost all of them),
As someone who mostly has just used Erlang, but has looked at Elixir some, I think one of the things I appreciate about Elixir is some of the Erlang warts that you've managed to work around, like having strings be Erlang binaries.
As someone who mostly has just used Erlang, but has looked at Elixir some, I think one of the things I appreciate about Elixir is some of the Erlang warts that you've managed to work around, like having strings be Erlang binaries.
I agree with davidw, 100%
Can we please have one BEAM thread where this isnt brought up? Its such a minor thing. I went from erlang to elixir and whilst at first I really missed atoms that start with a colon, you get used to it. Its 90% the same ideas, and Elixirs pipe operator is worth any small syntax niceties erlang has. No language is completely without its quirks even erlang (unless you LIKE endless repituons of the same variable into var_1, var_2, etc - and no, this doesnt reduce bugs http://blog.plataformatec.com.br/2016/01/comparing-elixir-an... )
That's mostly it.
Personally I find Elixir's syntax confusing. For example:
Losing the ; and . function endings from Erlang you can put same-named functions throughout your module. I tried doing
def create def handle(:create)
def update def handle(:update)
But the compiler warns. So that loss isn't helpful.
Atoms require a : because variable names are lowercase.
Uppercase variable names and lowercase module names is easier to read in Erlang.
The syntactic sugar is too clever for readability imo.
The package management through mix is decent. I used to use an erlang.mk file, looking at hex it looks like the Erlang ecosystem is quite evolved.
Phoenix + Ecto seem to be very actively maintained, useful if you're writing web apps.
When I tried Cowboy a few years ago some of the documentation was out of date, so that's a point in Elixir's favor.
Losing the ; and . function endings from Erlang you can put same-named functions throughout your module. I tried doing
def create def handle(:create)
def update def handle(:update)
But the compiler warns. So that loss isn't helpful.
Atoms require a : because variable names are lowercase.
Uppercase variable names and lowercase module names is easier to read in Erlang.
The syntactic sugar is too clever for readability imo.
The package management through mix is decent. I used to use an erlang.mk file, looking at hex it looks like the Erlang ecosystem is quite evolved.
Phoenix + Ecto seem to be very actively maintained, useful if you're writing web apps.
When I tried Cowboy a few years ago some of the documentation was out of date, so that's a point in Elixir's favor.
> Uppercase variable names and lowercase module names is easier to read in Erlang
This is ENTIRELY subjective and not a rational argument. It's in fact the exact opposite for me, but my prior language was Ruby.
This is ENTIRELY subjective and not a rational argument. It's in fact the exact opposite for me, but my prior language was Ruby.
You have macros in Elixir
Erlang has macros...
http://www1.erlang.org/documentation/doc-4.8.2/doc/extension...
...but I think they work differently, or are more limited in scope.
http://www1.erlang.org/documentation/doc-4.8.2/doc/extension...
...but I think they work differently, or are more limited in scope.
Erlang macros are basically text substitution. It's similar to C pre-processor directives.
Elixir macros are a completely different beast, they can manipulate the AST Lisp-style. A very elegant example is Elixir's unicode module, which reads unicode data tables and turns them into function definitions at compile time [1].
Compare this to the equivalent module in plain old Erlang, which has to be generated by a separate script, in a pre-compilation step in the Makefile, that literally prints blocks of code [2].
[1] https://github.com/elixir-lang/elixir/blob/v1.7.4/lib/elixir...
[2] https://github.com/erlang/otp/blob/OTP-21.2/lib/stdlib/uc_sp...
Elixir macros are a completely different beast, they can manipulate the AST Lisp-style. A very elegant example is Elixir's unicode module, which reads unicode data tables and turns them into function definitions at compile time [1].
Compare this to the equivalent module in plain old Erlang, which has to be generated by a separate script, in a pre-compilation step in the Makefile, that literally prints blocks of code [2].
[1] https://github.com/elixir-lang/elixir/blob/v1.7.4/lib/elixir...
[2] https://github.com/erlang/otp/blob/OTP-21.2/lib/stdlib/uc_sp...
Ernie was my first foray into Erlang (run many production elixir apps now). It's still humming along in one of our legacy services. Never had an issue or downtime event
Elixir's syntax may look more aesthetically pleasing than Erlan's, but Erlang's is way more consistent and simpler once you get used to it/understand the rules.
What makes it interesting is that the syntaxes have (most likely) been designed with different goals. I can say that Elixir was designed with extensibility and meta-programming in mind. So there are some cases where I prefer the Erlang syntax (for example, I still find defining multiple function clauses each with a single expression more elegant and concise in Erlang) while there are other cases I prefer Elixir's syntax exactly because we were able to extend the language (such as the `with` special form for handling nested case expressions).
The funny thing is, if you try to add those features to the other language, they are usually refused. Erlang wants to avoid new syntactical expressions to keep it simple while Elixir's extensibility is bound to a set of limited AST rules (and we don't want to add new ones).
The funny thing is, if you try to add those features to the other language, they are usually refused. Erlang wants to avoid new syntactical expressions to keep it simple while Elixir's extensibility is bound to a set of limited AST rules (and we don't want to add new ones).
I both admire Elixr for making Erlang “cool” and detest it for ruining my single most favorite part about the language, it’s simplicity and clarity.
There’s only one way to do a given thing and you can learn each of the things the syntax can do in a weekend.
It was my first non-C-like, non-assembler language and I loved it.
There’s only one way to do a given thing and you can learn each of the things the syntax can do in a weekend.
It was my first non-C-like, non-assembler language and I loved it.
When I give talks about Erlang I usually say I can teach the syntax in about 5 minutes. An exaggeration, but not a dramatic one.
I too vastly prefer Erlang’s syntax to Elixir’s. It helps me think like Erlang, and most of my functions are just a few lines long. Thanks Garrett[0].
[0]: http://www.gar1t.com/blog/solving-embarrassingly-obvious-pro...
I too vastly prefer Erlang’s syntax to Elixir’s. It helps me think like Erlang, and most of my functions are just a few lines long. Thanks Garrett[0].
[0]: http://www.gar1t.com/blog/solving-embarrassingly-obvious-pro...
My sentiment exactly :)
I'm grateful that Elixir is putting Erlang under the spotlights. That it is doing a lot to conveying all the Erlang/OTP concepts and practices. But each time I use it - and I know I may be the odd duck based on slack, blog posts, etc - I feel like the syntax is so complex, with 4 different manners to write the same thing. And I'm not talking about "ways to do things", but literally syntactic ways to do the same thing.
I'm grateful that Elixir is putting Erlang under the spotlights. That it is doing a lot to conveying all the Erlang/OTP concepts and practices. But each time I use it - and I know I may be the odd duck based on slack, blog posts, etc - I feel like the syntax is so complex, with 4 different manners to write the same thing. And I'm not talking about "ways to do things", but literally syntactic ways to do the same thing.
> literally syntactic ways to do the same thing.
Yup! They are not that many though (6 rules) and they are all documented here: https://hexdocs.pm/elixir/syntax-reference.html#syntactic-su...
Yup! They are not that many though (6 rules) and they are all documented here: https://hexdocs.pm/elixir/syntax-reference.html#syntactic-su...
Does the shorthand lambda belong on this list?
Thanks for all the work!
Thanks for all the work!
Oh, that's a very good question! It doesn't because the list linked above is about syntactical equivalence and the shorthand `&` has its own AST. But now you made me wonder if we should add it to the list anyway (with a caveat) as it would make the list more complete.
I think those more complicated things are there to make things easy for developers. The standard library is not that big, and the extra syntax is there for a reason - for example `with` is pretty fantastic for reducing `case` clutter (a real thing with erlang), and pipes are also great.
I don't love that there are two ways to make lambdas but it's nit the worst thing in the world.
I don't love that there are two ways to make lambdas but it's nit the worst thing in the world.
Thanks, somehow I’d missed Reia in my list of BEAM languages.
https://gist.github.com/macintux/6349828
https://gist.github.com/macintux/6349828
(Incidentally, when did Github break gist editing? I can no longer paste using my soft keyboard on my iPad.)
The article is underselling Erlang's meteoric success. "Adoption was slow during the first few years." - After 5 years there was an international conference devoted to Erlang, a global community around it and the language enjoyed commercial success from the beginning.
It just goes to show that Erlang fills a real niche that is ill served by most other programming languages. Programming distributed systems remains painful in 2018 - not because there aren't any theoretical solutions to make it easier, but because there are astonishingly few practical systems that offer built-in support. Erlang is such a practical system and if you didn't already look into the language it is well worth your time to pick it up. :)
It just goes to show that Erlang fills a real niche that is ill served by most other programming languages. Programming distributed systems remains painful in 2018 - not because there aren't any theoretical solutions to make it easier, but because there are astonishingly few practical systems that offer built-in support. Erlang is such a practical system and if you didn't already look into the language it is well worth your time to pick it up. :)
What an amazing language. I am working on a Elixir project, with a lot of Erlang peaces used directly. Integration is flawless and beautiful to observe. I do have many languages in my toolbox but only few of them were truly exciting for me, Haskell six or so years ago and Elixir/Erlang/OTP this year.
I've really wanted to dig into functional languages like this, because I love the theoretical benefits they provide, but haven't really sunk my teeth into one in a way that would let me understand the flavor.
Let me ask, were you immediately drawn to Haskell and Erlang, or was it a feeling that grew while working with them? Would you recommend working with Haskell or Erlang directly before trying out a language like Elixir, or should I go high level down?
Thanks
Let me ask, were you immediately drawn to Haskell and Erlang, or was it a feeling that grew while working with them? Would you recommend working with Haskell or Erlang directly before trying out a language like Elixir, or should I go high level down?
Thanks
I did initial plunge into Haskell when I started using xmonad, looking at config file was visibly 'something else'. I would probably recommend starting with Haskell first -- if you want to properly blow your mind that's the way to go. All the principles are common so diving into Elixir took very little time afterwards. Some principles are getting their way into imperative languages and it is much easier to grasp them on pure functional level first. Understanding and using map, filter and reduce, in JS project, was much easier to grasp immediately because they borrow that logic from functional languages.
Elixir is beautiful but probably few notches easier to understand and write than Haskell. Reason I would start with Haskell is that it doesn't hide functional logic with syntactic sugar. For example variables in Elixir are not mutable but since you can assign to the variable already used in function, it can prevent programmer from understanding what immutable really means.
Elixir is beautiful but probably few notches easier to understand and write than Haskell. Reason I would start with Haskell is that it doesn't hide functional logic with syntactic sugar. For example variables in Elixir are not mutable but since you can assign to the variable already used in function, it can prevent programmer from understanding what immutable really means.
Before learning Elixir, I used functional-like functionality in C# (LINQ) and Javascript (particularly Rambda and Bacon.js) before learning Elixir, and I found that was plenty of preparation to understand Elixir. There's no need to start off with another functional language prior to learning Elixir.
I'm currently learning Elixir and writing about it so that anyone interested in Elixir can learn with me as I learn it. The goal is to help other learn Elixir as well.
I make some comparisons to features in C# and Javascript, and the occasional reference to other languages like Python, but you don't have to know any of those in order to benefit.
If you're interested, come check it out: https://inquisitivedeveloper.com/
I'm currently learning Elixir and writing about it so that anyone interested in Elixir can learn with me as I learn it. The goal is to help other learn Elixir as well.
I make some comparisons to features in C# and Javascript, and the occasional reference to other languages like Python, but you don't have to know any of those in order to benefit.
If you're interested, come check it out: https://inquisitivedeveloper.com/
In my case I found I was already starting to write Ruby in a very functional way (PORO which is like POJO, only setting properties at object instantiation, focusing on data expressed in built-in classes instead of custom classes, etc.) and that seemed to ease the transition quite a bit.
I've found that "immutability all the way down" has eliminated an entire class of bugs related to unexpected mutation, deadlocking/concurrency issues, etc.
I've found that "immutability all the way down" has eliminated an entire class of bugs related to unexpected mutation, deadlocking/concurrency issues, etc.
I had never watched Erlang the Movie before, this is truly a gem:
https://www.youtube.com/watch?v=uKfKtXYLG78
This is a great way to explain the benefits of Erlang. Anyone who has tried to write robust, distributed systems should immediately see the benefits. Even all of these years later.
https://www.youtube.com/watch?v=uKfKtXYLG78
This is a great way to explain the benefits of Erlang. Anyone who has tried to write robust, distributed systems should immediately see the benefits. Even all of these years later.
I use Erlang the Movie and its Sequel https://www.youtube.com/watch?v=rRbY3TMUcgQ&t=9s whenever I teach my classes at Oxford Uni. #studentlife :-)
I dearly wish he'd make more videos. I also loved 'MongoDB is web scale' - https://www.youtube.com/watch?v=b2F-DItXtZs
You caught the advancing of the movie to the 9 second mark in your URL. (I can't stand when youtube does this automatically, such as when adding videos to playlists...)
Genius.
I've been engineering software for almost 30 years and no languages have ever been more enjoyable for me to work with than Erlang and now Elixir. I'd like to thank the teams behind both languages for making my career interesting again.
Specifically, I love working with pattern matching, OTP, and the functional nature of both languages. Thanks Erlang (and Elixir) guys!
Specifically, I love working with pattern matching, OTP, and the functional nature of both languages. Thanks Erlang (and Elixir) guys!
Not specifically Erlang but I have used Elixir and it was one of the rare languages that blew my mind when it comes to new ways of thinking when implementing things.
How not to love some of these features!
How not to love some of these features!
[first, second | remaining_of_people] = list_of_people
greeting = fn
%{name: name} -> "Hello, #{name}!"
%{} -> "Hello, Anonymous Stranger!"
endFull time Elixir dev here. This was a very neat walk down memory lane. Thanks for sharing! Since Elixir is so new, and I wasn't that aware of Erlang prior to it, it's sometimes easy to forget just how long Erlang has been around!
I didn't hear about Erlang as much as I did since Elixir came out and showed me the power of the VM and its ecosystem. As a ruby developer it's easy to pick up Elixir and I'm learning so much more about functional programming and systems than I did ever before.
The Elixir core oddly was my first taste of an Erlang project. I'm amazed at how the ecosystem and language has expanded since 2011.
Jose Valim is a very personable person and thriving code ecosystems are built around individuals like that.
I really like Erlang and Elixir, but I'm really worried when I see a tech stack that is still trying to figure out deployment and runtime configuration of systems.
I'm not trying to be inflammatory here. As per many resources such as "Phoenix in Action" deployment is still a major culprit. And Dockyard themselves (one of the major Elixir shops, where McCord works actually) have a full time guy on the payroll to try to solve the problem of runtime configuration.
It just sucks, because these are 2 major concerns of any production-ready stack.
I'm not trying to be inflammatory here. As per many resources such as "Phoenix in Action" deployment is still a major culprit. And Dockyard themselves (one of the major Elixir shops, where McCord works actually) have a full time guy on the payroll to try to solve the problem of runtime configuration.
It just sucks, because these are 2 major concerns of any production-ready stack.
While I agree that deployment can still improve, it is mostly because it doesn’t have the same polish as everything else Elixir ships with. In any case, it has already improved considerably over the last year.
My first application was deployed to Heroku and it just worked. For AWS, we had to do some research but once we got Distillery running, everything worked without major concerns. And Distillery uses Erlang releases which have been around for decades so we can rely on its stability.
Btw, runtime configuration works like a breeze in Distillery 2.0, which is the result of Paul’s amazing work at Dockyard (among other things).
My first application was deployed to Heroku and it just worked. For AWS, we had to do some research but once we got Distillery running, everything worked without major concerns. And Distillery uses Erlang releases which have been around for decades so we can rely on its stability.
Btw, runtime configuration works like a breeze in Distillery 2.0, which is the result of Paul’s amazing work at Dockyard (among other things).
Deployment for Erlang needs to be a lot more flexible than other systems, because you have the opportunity to hotload code as a core feature [1], having your deployment solution invoke code loads is totally scriptable, but seems more likely to want human eyes watching than a traditional push, stop, start cycle.
OTP does provide the application construct which is supposed to provide some amount of configuration and start/stop/update support; although my team doesn't use the update support, we just push code and hotload it or stop and start beam depending on the changes.
[1] of course, you can hotload code in most languages if you try hard enough, I've done it in C and almost did it in perl; as long as you can load code and get a function pointer at runtime, you should be able to make it work.
OTP does provide the application construct which is supposed to provide some amount of configuration and start/stop/update support; although my team doesn't use the update support, we just push code and hotload it or stop and start beam depending on the changes.
[1] of course, you can hotload code in most languages if you try hard enough, I've done it in C and almost did it in perl; as long as you can load code and get a function pointer at runtime, you should be able to make it work.
I am not sure what you mean with production-ready here.
Erlang has certainly been used in production, not only Whatsapp but also have a lot of production systems in telecom where requirements on up-times etc are pretty high.
If you mean that there are no ready made off-the-shelf standard way to handle runtime configuration and deployment I guess it is because the big telcom products usually roll their own per product so it haven't made it into the OTP backlog.
Erlang has certainly been used in production, not only Whatsapp but also have a lot of production systems in telecom where requirements on up-times etc are pretty high.
If you mean that there are no ready made off-the-shelf standard way to handle runtime configuration and deployment I guess it is because the big telcom products usually roll their own per product so it haven't made it into the OTP backlog.
It’s not so much an issue as it is something they want to make cleaner for new users. If you are coming for a background in other languages, the process doesn’t behave like you’re already used to.
So much of the work that Jose and Chris put into Elixir and Phoenix is striking the balance between what’s possible and what the onboarding experience looks like for new users.
So much of the work that Jose and Chris put into Elixir and Phoenix is striking the balance between what’s possible and what the onboarding experience looks like for new users.
For one datum, as an inexperienced Phoenix user, it took me 1 day to get a Phoenix app deployed and running on Google App Engine flex, including DB and some other details.
For managed deployment (with a free tier) check out https://www.gigalixir.com/
Yeah but an Erlang deployment is roughly equivalent to a Kubernetes deploy (more powerful in some ways), are you similarly worried about the amount of effort it takes to set up a k8s cluster?
Fantastic tutorial: http://learnyousomeerlang.com/book
Kudos to Francesco and the guys at Erlang Solutions for their outstanding work in the open source high concurrency field!
Thanks, but it is not only us at ESL. Thank Joe, Mike, Robert and Jane, the OTP team and Ericsson first and foremost, and the community as a whole. It has been an amazing journey, looking forward to the next 20 years :-)
My limited understanding of erlang/elixir is that it helps easily run actor model programs (fail-safe loosely defined as something with retries built-in) in a grid of long-running interconnected instances. With more and more focus on "serverless" development that I am being exposed to, I am curious to know if it is a good fit in this space. Say if you are constrained to use AWS serverless solutions - sqs, sns, dynamodb, lambda, cognito, api-gateway ... etc, with event-sourcing (for event-drivenness) with retries (automatic/configurable), (why) would using erlang/elixir vs say nodejs/typescript/go/haskell/java be a good choice (let's assume you equally love these languages for the sake of not getting in to 'but X is better than lang Y')?
From my perspective, I see it as a different way of achieving the same end goal. With "serverless", someone still has to maintain servers somewhere and if your company is big enough, AWS is actually the more expensive option.
So if you are running your own infrastructure on bare metal, application development in erlang/elixir lets you better take advantage of the operations staff that you already have because running that infrastructure is much less complicated than the alternatives.
So if you are running your own infrastructure on bare metal, application development in erlang/elixir lets you better take advantage of the operations staff that you already have because running that infrastructure is much less complicated than the alternatives.
The modern serverless paradigm is a way to give these more traditional computing runtimes/languages a lot of the features of Erlang/OTP.
The downside to having all of these things in services on the network and outside of the runtime is latency. Imagine a runtime where everything is a message, everything goes into inboxes/queues but passing many of those messages is zero-copy, doesn't hit a network interface, and going from the calling code to the called code isn't even a OS level context switch -- so the message is written to and read from a core/thread's local cache lines.
The downside to having all of these things in services on the network and outside of the runtime is latency. Imagine a runtime where everything is a message, everything goes into inboxes/queues but passing many of those messages is zero-copy, doesn't hit a network interface, and going from the calling code to the called code isn't even a OS level context switch -- so the message is written to and read from a core/thread's local cache lines.
To me, the bigger downside is system complexity. If everything is an BEAM process, then testing, debugging and generally understanding is much easier. Things like spinning up a dev 'deployment' become trivial and things are going to behave in production as I expect them to.
For fault-tolerance in case the box is unresponsive for whatever reason, I think you would have to have a network behind a load balancer. So the issue of network latency still exists, but probably to a lesser extent?
Yes, obviously some things will need to be persisted, and some of those things might be some of your queue entries -- so a persisted network queue (and network things to dequeue from it) can definitely be a necessity for that reason as well.
No. Kubernetes & friends implement most of the features of Erlang in a language-independent manner.
Erlang truly is something. It's amazing how much effort is expended trying to achieve even a semblance of its functionality in other languages/ecosystems.
I forget where I read that there were estimated to be only about 1K professional Erlang coders in the world. That seems absurdly low. Does anyone believe this is actually true?
If true, it's astounding so few programmers wrote systems that handle such astounding amounts of data.
That's what people used to think of Symbolics Lisp machines. Customers would assume Symbolics had a team of thousands of programmers, when really it was about a dozen.
One of my favorite CS profs had one of those. I remember visiting his office hours so he could show it off to me.
I didn't do well in the class (it was a bit out of my league at the time) but I ended up in a career in software dev anyway
I didn't do well in the class (it was a bit out of my league at the time) but I ended up in a career in software dev anyway
Incorrect, it probably got confused with the number of Erlang developers at Ericsson. Not all use it on a daily basis, but estimates on their developers, testers and system integrators is in that range.
I was delighted when Jose started getting involved in the space and released Elixir. Thanks Jose!
[0] http://reia-lang.org
[1] https://gilesbowkett.blogspot.com/2007/05/erlectricity-erlan...
[2] https://github.com/mojombo/ernie