Ask HN: Erlang and the telecom market
Hello folks, I was a telecommunications engineering student when I came across the Erlang language.
I was seduced and it looked like a good career for me.
By the description it was a niche language with a high demand on the market.
But it was 3 years ago and and I still did not find an appropriate Erlang position (I applied to all of the very few offers on the internet and it was not only the telecom offers)
How do one find an Erlang job?
PS: screw Elixir by the way
58 comments
The generalist view is that nobody is using Erlang for modern telco workloads anymore.
(yes, there are exceptions. Bear with me)
I learned Erlang when we were programming Ericsson voice switches some twenty years ago, and even though Nokia also adopted OTP and did some pretty amazing things with it (including a Hadoop analogue that used Erlang to coordinate Python workers across nodes), I have only come across Erlang when looking at legacy workloads that needed to be moved to the cloud.
NFV and CNF efforts (i.e., virtualized and containerized network functions) I've come across over the past few years seem to be mostly Go or C++. Jaeger is a common tool. I see a lot of Postgres. I also see really weird CNI approaches (check out multus, because telcos still can't get over having dedicated network interfaces for things even though it's all just a virtualization sandwich and they really should just learn to use network policies).
The technicalities run deeper than web apps (because some thing are very, very finely tuned), but the key point is that the telco "bleeding edge" landscape is now indistinguishable from "web scale" K8s discussions, except that telco workloads demand fixed resource allocations nd we still rely _a lot_ on CPU pinning of specific functions due to latency/jitter sensitive workloads.
It's almost like you took low-latency, pseudo-real-time stuff from embedded systems and shoved it into K8s. No, wait, it's _exactly_ like that.
Source: I work in Azure for Operators and have been in the telco industry since the mid-90s.
(yes, there are exceptions. Bear with me)
I learned Erlang when we were programming Ericsson voice switches some twenty years ago, and even though Nokia also adopted OTP and did some pretty amazing things with it (including a Hadoop analogue that used Erlang to coordinate Python workers across nodes), I have only come across Erlang when looking at legacy workloads that needed to be moved to the cloud.
NFV and CNF efforts (i.e., virtualized and containerized network functions) I've come across over the past few years seem to be mostly Go or C++. Jaeger is a common tool. I see a lot of Postgres. I also see really weird CNI approaches (check out multus, because telcos still can't get over having dedicated network interfaces for things even though it's all just a virtualization sandwich and they really should just learn to use network policies).
The technicalities run deeper than web apps (because some thing are very, very finely tuned), but the key point is that the telco "bleeding edge" landscape is now indistinguishable from "web scale" K8s discussions, except that telco workloads demand fixed resource allocations nd we still rely _a lot_ on CPU pinning of specific functions due to latency/jitter sensitive workloads.
It's almost like you took low-latency, pseudo-real-time stuff from embedded systems and shoved it into K8s. No, wait, it's _exactly_ like that.
Source: I work in Azure for Operators and have been in the telco industry since the mid-90s.
Are you saying BEAM introduces too much latency and/or jitter for telco workloads? This surprises me. I would have thought that if anything BEAM should be pretty efficient for network I/O. Also what would be the causes of unacceptable jitter? The runtime itself is designed to avoid things like long gc pauses - is it that you end up having to migrate elixir processes between BEAM scheduler cores?
I don't think latency and jitter are big issues. If you're imagining VOIP packets flowing through the Erlang app, it's more likely that they are routed by hardware at a lower level. The Erlang app controls the lower level switches. I don't know about Ericsson switches in particular, but I've worked on telecom stuff (not programmed in Erlang, unfortunately) and that's how it worked. The low level routing was done by FPGA's controlled by a C program. The C program could have been written in Erlang instead, and that would have made life a lot nicer for the programmers.
You'd be surprised exactly how much functionality is written in vanilla C and pushed down the stack to raw packet handling. There is hardware acceleration in NICs, but voice is actually a tiny, almost insignificant amount of the workloads involved in some parts of the world.
Think hyperscaler-style SDN, but largely on-premises.
Think hyperscaler-style SDN, but largely on-premises.
I said no such thing. Merely that there is a higher reliance on hardware features even if the workloads are virtualized - more so than on your vanilla K8s cluster in any other industry (except perhaps trading).
I used to experiment a bunch with GPU passthrough to VMs and also found that CPU pinning was required to avoid stutters. I always found it unfortunate as pinning works directly against what one tries to achieve with virtualization (i.e. sharing of resources).
I had never considered telecom suffering from the same issue. Do you know if there's been any work to improve scheduling to mitigate this requirement?
I had never considered telecom suffering from the same issue. Do you know if there's been any work to improve scheduling to mitigate this requirement?
There are other issues at play, such as the need for SR-IOV support and sometimes direct mapping of PCI lanes to some workloads. There is only so much virtualization can do when you're letting millions of sessions through.
Excellent points. I work in telco though mainly on business web services side but what you said is about same I heard in here on network side.
Are there any good reading on modern work being done in telecom networks. I enjoyed this comment and have had a passing interest in the industry, but have been unable to understand modern networks so far.
You can try digging into modern 5G stacks. Search for VNF/CNF, be wary of the fact that it is a much deeper industry acronym-wise (heck, we have 4 and 5 letter acronyms that are not jokes), and hang on to your hat...
Thank you for your input, I was thinking that with all the 5g decoupling,network functions and messages passing Erlang would go stronger and need more workforce
> screw Elixir by the way
This is 100% what is keeping you from getting hired. Folks who use BEAM don’t care if someone is using Erlang or Elixir or LFE or whatever… we care about the problems. This is why we are all on BEAM to begin with. Your immaturity shows you lack the experience to meaningfully understand this. You’re still obsessing with the superficial instead of the problem. Almost everywhere using BEAM is going to be dominated by senior+ engineers who just don’t want to put up with someone who is so assuredly lacking experience.
Your hubris is stunning by the way, to not see your attitude is the problem, I can say I would absolutely never want to work with you… and I’ve been writing for BEAM professionally for 7 years, software development for 15+ years.
This is 100% what is keeping you from getting hired. Folks who use BEAM don’t care if someone is using Erlang or Elixir or LFE or whatever… we care about the problems. This is why we are all on BEAM to begin with. Your immaturity shows you lack the experience to meaningfully understand this. You’re still obsessing with the superficial instead of the problem. Almost everywhere using BEAM is going to be dominated by senior+ engineers who just don’t want to put up with someone who is so assuredly lacking experience.
Your hubris is stunning by the way, to not see your attitude is the problem, I can say I would absolutely never want to work with you… and I’ve been writing for BEAM professionally for 7 years, software development for 15+ years.
That's exactly why I added the last part : to focus the discussion on Erlang and not the BEAM.
I also would never want to work with you. If you read the first part carefully I'm indeed lacking experience and I was asking a question about the erlang jobs market and what's in use in the telecom industry
This is the problem I have with Erlang developers. They only want to work on Erlang. Pragmatism be damned. It is never about the product or the company. It is only about which of the remaining 20 Erlang expert personalities in the world do they get to work alongside with. If there needs to be a special library, all other existing ones are garbage unless written by one of the 20 above, so they will just write their own. I hate to stereotype here, but I would never ever (again) bet my company on Erlang. It is insanely difficult to recruit for. They are truly some of the best developers in the world, but lack pragmatism for problems and every one problem can only be solved with functional programming using Erlang. In my experience, Erlang developers are Zealots for the language and nothing else. If your company decides to pivot away from Erlang to another more accessible language, say Go, you are likely to lose them all.
"Yes, we are going to make your job six times less efficient and substantially more difficult, while simultaneously removing your main incentive for working here. No, you will not receive a raise or any other form of olive branch from us, management, in exchange for the fact that we have measurably reduced your quality of life. No, we are not negotiating this."
If any job I ever had did this, technical or otherwise, I'd leave too. Especially if you were going to rub salt in the wound and pick Go. It's akin to making someone mine coal with a rock instead of a machine.
If any job I ever had did this, technical or otherwise, I'd leave too. Especially if you were going to rub salt in the wound and pick Go. It's akin to making someone mine coal with a rock instead of a machine.
I think this comment actually unintentionally is reinforcing his point.
Other programming languages than Erlang exist for a reason, they're not just fun toy languages for low IQ folks. There are tons of reasons why Erlang may not be a good fit for some project, and one of these other lesser languages would be a better fit, and he doesn't want to work with people who wouldn't even consider something like that.
Other programming languages than Erlang exist for a reason, they're not just fun toy languages for low IQ folks. There are tons of reasons why Erlang may not be a good fit for some project, and one of these other lesser languages would be a better fit, and he doesn't want to work with people who wouldn't even consider something like that.
Two possibilities here:
1. either the person who asked this question doesn't understand that most of the utility in this ecosystem comes from the BEAM VM, the OTP, and the community around them, not just from the Erlang-the-programming-language.
2. or he/she employs the sunk cost fallacy here: I already invested time into learning Erlang and its Prolog-inspired syntax, so "screw Elixir". Disclaimer: I mistakenly avoided learning Elixir for this reason, simply because I already had lots of Erlang/OTP code already written over the years.
1. either the person who asked this question doesn't understand that most of the utility in this ecosystem comes from the BEAM VM, the OTP, and the community around them, not just from the Erlang-the-programming-language.
2. or he/she employs the sunk cost fallacy here: I already invested time into learning Erlang and its Prolog-inspired syntax, so "screw Elixir". Disclaimer: I mistakenly avoided learning Elixir for this reason, simply because I already had lots of Erlang/OTP code already written over the years.
Yeah it's the point 2 , I guess but I really tried to get into Elixir but I really don't like its syntax
I don't even like Erlang. I'm just pointing out that removing the main incentive for being there for an employee without giving them something in return is naturally going to cause them to bail. You don't spit in a person's face and expect them to be happy about it.
> There are tons of reasons why Erlang may not be a good fit for some project, and one of these other lesser languages would be a better fit
But that's not what they said. The statement was literally "to pivot away from erlang". Imagine you are working with, say, python (which I don't really like but it's a popular example) and were forced to switch to Visual Basic.
But that's not what they said. The statement was literally "to pivot away from erlang". Imagine you are working with, say, python (which I don't really like but it's a popular example) and were forced to switch to Visual Basic.
I'd count myself as an Erlang developer. But where I worked, we didn't hire many people who had used Erlang before; and I didn't either. It's a small language, and it doesn't take too long for someone who's done a couple languages before to be productive and/or dangerous in it. Yes, immutable is a change; yes, recursion instead of loops is a mindfuck; Yes, a smart person can figure it out and deal. Personally, I loved the promise of Elixir --- BEAM with syntax that's better than Erlang, but I was hoping for different syntax, so I'm out; but it's fine, it seems to be a gravity well drawing people into BEAM, which is a good thing, IMHO.
I'm not going to reach for libraries in Erlang, because mostly I've seen them not be there, and a lot of stuff is almost the same amount of code and fuss to use a library as to build the portion of the library that's actually needed in the moment. Any code that you bring in is code that you're running and responsible for, so it's got to be worth it. I've pulled in libraries that needed a lot of rewriting, and sometimes that's better than starting from scratch, and sometimes it's not. There's a fair amount of stuff out there where someone scratched their itch and left it as is; which is fine and thank you, but it might need a lot of help to be run in a production capacity.
I'm working a new job now and there's probably no Erlang in it. Which is sad, but I'll deal. That said, if I was working in Erlang and management said we had to switch, I would be out. It's one thing to work without the benefits of Erlang, it's another to be working with them and then have it taken away.
I'm not going to reach for libraries in Erlang, because mostly I've seen them not be there, and a lot of stuff is almost the same amount of code and fuss to use a library as to build the portion of the library that's actually needed in the moment. Any code that you bring in is code that you're running and responsible for, so it's got to be worth it. I've pulled in libraries that needed a lot of rewriting, and sometimes that's better than starting from scratch, and sometimes it's not. There's a fair amount of stuff out there where someone scratched their itch and left it as is; which is fine and thank you, but it might need a lot of help to be run in a production capacity.
I'm working a new job now and there's probably no Erlang in it. Which is sad, but I'll deal. That said, if I was working in Erlang and management said we had to switch, I would be out. It's one thing to work without the benefits of Erlang, it's another to be working with them and then have it taken away.
Or let's say you take a job writing Go and they switch to Erlang? I'm curious how you would react?
If it makes you feel better, a lot of COBOL programmers actually wish it was extinct, because some of the systems they're maintaining are multiple decades old by now.
I suspect that if it wasn't for the virtualization drive behind 5G, Erlang would follow suit as a highly paid legacy language.
As it is, since many legacy telco systems are actually being replaced entirely by virtualized solutions, I see older companies still developing in it, but all hedging their bets on other things (I already mentioned C++ in another thread, but Go and Rust, backed by suitable frameworks that employ Raft and other sync/HA protocols, seem to be in the forefront).
I suspect that if it wasn't for the virtualization drive behind 5G, Erlang would follow suit as a highly paid legacy language.
As it is, since many legacy telco systems are actually being replaced entirely by virtualized solutions, I see older companies still developing in it, but all hedging their bets on other things (I already mentioned C++ in another thread, but Go and Rust, backed by suitable frameworks that employ Raft and other sync/HA protocols, seem to be in the forefront).
> If your company decides to pivot away from Erlang to another more accessible language, say Go, you are likely to lose them all.
I was sympathizing until that final sentence. I'm not an Erlang programmer, but from what I know about both Erlang and Go, that seems like a terrible jump to make. Of course it depends on other circumstances in this hypothetical as well, but I probably would at least take this as an opportunity to reevaluate my current working situation as well.
I was sympathizing until that final sentence. I'm not an Erlang programmer, but from what I know about both Erlang and Go, that seems like a terrible jump to make. Of course it depends on other circumstances in this hypothetical as well, but I probably would at least take this as an opportunity to reevaluate my current working situation as well.
> screw Elixir by the way
If OP is reading… care to elaborate in a more objective fashion?
As an Elixir neophyte, one of the things that is a small frustration for me is that you’ll get to a problem and there just is no solution, and when you ask on the wonderful slack channel, someone will pipe up with a bit Erlang you use for that. It’d be like learning Japanese, and for sizeable pieces of your communication, your instructed to go next door to the Latin class to figure it out.
As near as I can tell, there is no such thing in the real world as a “just elixir” application. Any shop/product of size would probably love to have your Erlang expertise, provided they didn’t feel you were hostile.
If OP is reading… care to elaborate in a more objective fashion?
As an Elixir neophyte, one of the things that is a small frustration for me is that you’ll get to a problem and there just is no solution, and when you ask on the wonderful slack channel, someone will pipe up with a bit Erlang you use for that. It’d be like learning Japanese, and for sizeable pieces of your communication, your instructed to go next door to the Latin class to figure it out.
As near as I can tell, there is no such thing in the real world as a “just elixir” application. Any shop/product of size would probably love to have your Erlang expertise, provided they didn’t feel you were hostile.
I learnt Erlang when it was still cool. And more recently (but not that recently) dabbled with learning Elixir.
Most resources are targeting those who don't come from Erlang (mostly it was from rails). And my word are they being deceived.
I don't think the Japanese/Latin analogy quite works it's like elixir is rhyming slang and Erlang is english.
But everyone insists you can get by with rhyming slang alone. Errr. Nope!
Most resources are targeting those who don't come from Erlang (mostly it was from rails). And my word are they being deceived.
I don't think the Japanese/Latin analogy quite works it's like elixir is rhyming slang and Erlang is english.
But everyone insists you can get by with rhyming slang alone. Errr. Nope!
I think a better analogy is Erlang being like Latin and Elixir being like Portuguese or Spanish.
But I digress, you can learn how the BEAM and OTP work as an Elixir dev without being great at reading or writing Erlang code.
But I digress, you can learn how the BEAM and OTP work as an Elixir dev without being great at reading or writing Erlang code.
There's definitely "just Elixir" applications, I've worked on more than one. And I don't think any of my colleagues really knew Erlang.
(I don't consider using :timer or :crypto as having Erlang code)
But mostly that's more because it has been unnecessary rather than a dab at the language.
(I don't consider using :timer or :crypto as having Erlang code)
But mostly that's more because it has been unnecessary rather than a dab at the language.
> screw Elixir by the way
Not the attitude I'd be hiring for honestly.
Not the attitude I'd be hiring for honestly.
I'd first ask them why they hold the opinion. You might be surprised by the reasoning they have for saying this. Having strong opinions isn't necessarily a bad thing.
You could say, "I don't want to work for a company using other languages on the BEAM VM" or "I considered Elixir and I don't like it because of X"
It just seems completely out of context and aggressive. ¯\_(ツ)_/¯
It just seems completely out of context and aggressive. ¯\_(ツ)_/¯
The fact that it's out of context and the tone is so different is what made me take it as not so serious and just a playful jab at Elixir.
I think the interpretation everyone seems to be taking (which is surprising to me personally!) is such that it's like Elon requiring people to put "Parody" in their name.
I think the interpretation everyone seems to be taking (which is surprising to me personally!) is such that it's like Elon requiring people to put "Parody" in their name.
There's a time and a place, though, no? "Screw Elixir" might be fine as an opinion on its own, but in the same breath as "I would like a job"?
Right? I could easily list a few companies hiring Erlang developers, including the one I currently work for but that attitude is a big no-no for me.
Exactly.
Having strong opinions is a great thing, expressing them in such a way is not. Especially if you expect to be taken seriously.
Yeah, the effects are not going as what I wanted
I was just trying to focus the talks on the Erlang language because I'm already aware of the success and opportunities of the Elixir language.
By not using the word Telecom.
Discord, fintech, and sportsbetting also use BEAM pretty heavily.
Also 100% would avoid hiring you with the elixir comment.
Discord, fintech, and sportsbetting also use BEAM pretty heavily.
Also 100% would avoid hiring you with the elixir comment.
Could you work near UTC+1 timezone? ["info",$@,"treesat.io"]. Otherwise, like other commenter said, get lucky, in Europe there may be more job postings than USA.
Have you considered these guys yet? https://www.2600hz.com/careers
@fantarina If you look for a job, can you please send me your CV? we use Erlang. https://www.thesignallingcompany.com/
[email protected]
[email protected]
I wouldn't be too quick to be down on Elixir. The entire BEAM ecosystem (Erlang, Elixir, Gleam, Luerl, LFE etc) are great! I think one could potentially sneak some Erlang into an Elixir job by building common libraries you want to share with other programming languages across the BEAM.
I don't get the hate for Erlang, either. It's a fun language with a community very willing to help folks out.
I don't get the hate for Erlang, either. It's a fun language with a community very willing to help folks out.
I really like Erlang as a language and OTP as an idea, but have never used it professionally. I'd assume that there exists a relatively close-knit community of practitioners in the world, similar to other "niche" languages like Clojure, so I would start there. It could be a mailing list, a website, a discord server...but I'm sure its out there.
Demand for Erlang is non-existant even within Ericsson and that's the company that invented the language.
Could try for a vFabric/RabbitMQ support roll at NetSuite, Red Hat, or Blackfriars Group.
Also, Elixir subsystems in Erlang are common, so projecting ones preferences onto an employer may earn less utility or more ire.
Best of luck =)
Also, Elixir subsystems in Erlang are common, so projecting ones preferences onto an employer may earn less utility or more ire.
Best of luck =)
Unpopular opinion but yeah screw Elixir but that's not the topic here.
To put it bluntly you won't find an Erlang job, better forget it and go the python/js/php way
Quite curious indeed that both OP and _all_ the commenters who agree with "screw Elixir" are green users...
[deleted]
try athonet, they do telco and use erlang, they are moving more towards elixir nowadays so that might be a show stopper for you though
[deleted]
miafina(3)