Lumen – Elixir and Erlang in the Browser(underjord.io)
underjord.io
Lumen – Elixir and Erlang in the Browser
https://underjord.io/lumen-elixir-in-the-browser.html
87 comments
What I find the most interesting about it is that it doesn't try to rework the BEAM to fit in the browser but rather does the much more ambitious work of trying to compile something fairly optimized that won't bloat the frontend.
Agreed, I was happy to see this.
I like Javascript, a lot, but I would very happily see a future where people have more choices about languages to use on the front-end. My only fear was that everything was going to morph into a kind of Java Applets 2.0, where everything would just be a giant VM on top of another VM.
For the most part, languages aren't doing that though, they're putting in the work to do proper integration with "native" browser systems like the DOM/CSS. I'm optimistic about where language efforts like Rust/Erlang/C#/C are headed, I think these projects are going to be really good for the web overall.
I like Javascript, a lot, but I would very happily see a future where people have more choices about languages to use on the front-end. My only fear was that everything was going to morph into a kind of Java Applets 2.0, where everything would just be a giant VM on top of another VM.
For the most part, languages aren't doing that though, they're putting in the work to do proper integration with "native" browser systems like the DOM/CSS. I'm optimistic about where language efforts like Rust/Erlang/C#/C are headed, I think these projects are going to be really good for the web overall.
> everything would just be a giant VM on top of another VM
this concern is real which is why we didn't just run BEAM c source through a WASM compiler and call it a day. The implementation here is to optimize compilation to WASM and avoid as much overhead as possible. We have footprint requirements for the compiled asset sizes in mind too.
this concern is real which is why we didn't just run BEAM c source through a WASM compiler and call it a day. The implementation here is to optimize compilation to WASM and avoid as much overhead as possible. We have footprint requirements for the compiled asset sizes in mind too.
[deleted]
I bet that in about 5 years time the browser plugins would have had their revenge, unless the browser vendors pull the plug on WebAssembly.
Wasm is the most transformative tech to come along since Linux.
Not really, we already had WASM in 2011.
https://adobe-flash.github.io/crossbridge/
> Combine the power, familiarity, and high-performance of C/C++ with the unparalleled reach of the web. With CrossBridge you can bring your native C/C++ from consoles and PCs to over a billion people on the web – across browsers, with no additional install.
"Unreal Engine 3 Support for Adobe Flash Player"
https://www.youtube.com/watch?v=UQiUP2Hd60Y
https://adobe-flash.github.io/crossbridge/
> Combine the power, familiarity, and high-performance of C/C++ with the unparalleled reach of the web. With CrossBridge you can bring your native C/C++ from consoles and PCs to over a billion people on the web – across browsers, with no additional install.
"Unreal Engine 3 Support for Adobe Flash Player"
https://www.youtube.com/watch?v=UQiUP2Hd60Y
WASM is very explicitly designed not to be another Flash Player.
My concern was that WASM would turn into another kind of Flash Player because devs would just cross-compile everything and spit out frames to a canvas instead of engaging with the web as a platform.
I'm happy that's not what Elixir, or Rust, or C++ are doing. Their approach means that the potential benefits of WASM even today are a lot higher than the potential benefits of Flash ever were. And future to plans to do things like share object and function references between WASM and Javascript push that potential even more.
If you're just concerned about taking a native C++ app and treating the web like a compile target, you've missed the point. The point is to take these languages and make them into first-class web citizens.
My concern was that WASM would turn into another kind of Flash Player because devs would just cross-compile everything and spit out frames to a canvas instead of engaging with the web as a platform.
I'm happy that's not what Elixir, or Rust, or C++ are doing. Their approach means that the potential benefits of WASM even today are a lot higher than the potential benefits of Flash ever were. And future to plans to do things like share object and function references between WASM and Javascript push that potential even more.
If you're just concerned about taking a native C++ app and treating the web like a compile target, you've missed the point. The point is to take these languages and make them into first-class web citizens.
One thing I wasn't able to ascertain from the link: would this run your Erlang code with the semantics of a single-scheduler BEAM node (i.e. concurrently, but not in parallel); or does this use Web Workers to have multiple parallel schedulers?
Initially we’re not planning to support multiple schedulers via WebWorkers, but the runtime has support for multiple schedulers generally speaking. Since the compiler can target other platforms, we didn’t want to artificially limit all targets due to any particular target limitations. As far as Wasm is concerned though, once browser support for SharedArrayBuffer returns, then we’d be looking to take advantage of multiple threads.
Then, could a Lumen user do what pre-SMP Erlang did for parallelism, and
1. run multiple nodes on one machine (i.e. manually create Web Workers each with their own copy of Lumen); and then
2. write an erl_dist alternative-carrier protocol driver that uses worker.postMessage()?
1. run multiple nodes on one machine (i.e. manually create Web Workers each with their own copy of Lumen); and then
2. write an erl_dist alternative-carrier protocol driver that uses worker.postMessage()?
Sure, I can't imagine any particular reason why that wouldn't work. That said, we don't have distribution implemented yet, since we are focused on a single node in the browser initially; so it depends on which comes first - SharedArrayBuffer being stabilized again in the browsers so we can enable multiple schedulers, or getting distribution implemented thoroughly enough to support the approach you mentioned as a workaround.
I think you can get more info from the video linked there. I don't recall the details but if it isn't Web Workers there might be something in Wasm.
I'm pretty sure it's intended to be able to run multiple schedulers. But I'm not the one building it so not sure.
I think this project is a great idea but I think it's overly ambitious to be reasonably feasible in a near-term timeframe as it's basically talking about re-implementing 20+ years and thousands of man-hours put into BEAM dev into the browser.
It's also competing time-wise with simply "compiling Erlang itself to emscripten/WASM", since computing capability will still scale over time in speed and bandwidth and memory capacity to the point that simply "being able to get ALL of Elixir in the browser" will outvalue "being able to run some subset of Elixir in a single-threaded context only in the browser" (assuming performance is not an order of magnitude different).
I was just reading another HN article discussing the fact that OLAP is going away simply because computers have gotten enough memory and databases have gotten fast enough at querying columnar data stores that it's no longer necessary to add the "optimization complexity" of OLAP to get reasonable data analysis performance.
AMD has the Ryzen Threadripper now, 32 cores with 64 threads... If CPU development continues in that direction, this will be a BIG win for Elixir but only if it can take advantage of more than one thread.
That all said, I'm sure a naïve recompilation of Erlang BEAM to emscripten/WASM might not automatically take advantage of those threads properly out-of-the-box due to the abstraction layer, so there'd also have to be some work there to massage the scheduler to use "real" threads "on the metal".
It's also competing time-wise with simply "compiling Erlang itself to emscripten/WASM", since computing capability will still scale over time in speed and bandwidth and memory capacity to the point that simply "being able to get ALL of Elixir in the browser" will outvalue "being able to run some subset of Elixir in a single-threaded context only in the browser" (assuming performance is not an order of magnitude different).
I was just reading another HN article discussing the fact that OLAP is going away simply because computers have gotten enough memory and databases have gotten fast enough at querying columnar data stores that it's no longer necessary to add the "optimization complexity" of OLAP to get reasonable data analysis performance.
AMD has the Ryzen Threadripper now, 32 cores with 64 threads... If CPU development continues in that direction, this will be a BIG win for Elixir but only if it can take advantage of more than one thread.
That all said, I'm sure a naïve recompilation of Erlang BEAM to emscripten/WASM might not automatically take advantage of those threads properly out-of-the-box due to the abstraction layer, so there'd also have to be some work there to massage the scheduler to use "real" threads "on the metal".
Am I understanding this right that it's actually implementing the actor model just like Erlang? It seems to have processes and you can send them messages.
The threads seem really lightweight, but how would some typical frontend benefit from this?
The threads seem really lightweight, but how would some typical frontend benefit from this?
> how would some typical frontend benefit from this
At ElixirConf I built a simple supervisor DOM tree model. When we are ready to implement it for real we're going to experiment with mapping the DOM to a supervisor. Each node being an element. This way events and rendering take place in each node in the supervisor tree. Rendering would be diff based and merged its way back up. Events would be captured as messages in the idiomatic Erlang/Elixir way.
This is all guess work right now and we don't yet know what will be practical vs what sounds good in theory.
However, we are dedicated to the idea that the programming model of the BEAM is one that complex client-side applications will benefit from over what JavaScript current offers.
At ElixirConf I built a simple supervisor DOM tree model. When we are ready to implement it for real we're going to experiment with mapping the DOM to a supervisor. Each node being an element. This way events and rendering take place in each node in the supervisor tree. Rendering would be diff based and merged its way back up. Events would be captured as messages in the idiomatic Erlang/Elixir way.
This is all guess work right now and we don't yet know what will be practical vs what sounds good in theory.
However, we are dedicated to the idea that the programming model of the BEAM is one that complex client-side applications will benefit from over what JavaScript current offers.
That sounds pretty close to what Scenic does for OpenGL-rendered UI.
Scenic has some interesting properties in that if a specific component has a code problem and fails/crashes, the supervision tree will bring it back to life. I'm curious to see what similar things one could do with Lumen for the web.
Components are already a very popular level of abstraction with Vue and React, probably others. So maybe there are useful things to bring from Scenic.
Scenic has some interesting properties in that if a specific component has a code problem and fails/crashes, the supervision tree will bring it back to life. I'm curious to see what similar things one could do with Lumen for the web.
Components are already a very popular level of abstraction with Vue and React, probably others. So maybe there are useful things to bring from Scenic.
Scenic is one area that I've looked to for inspiration on what a GUI can be like in Elixir
Oh wow sounds like a pretty fundamental overhaul of the current DOM model. Then again, various virtual DOM diff implementations have shown they can improve working with the DOM a lot.
Yeah, the undertaking would be quite extraordinary if they could get it working. Something tells me that this is going to take a few years to develop (preemptively giving a "bravo" if it could be done sooner ;) )
This sounds like an interesting experiment, do you have a repo or anything one could follow?
https://github.com/lumen/db_mon_demo
There is a db_mon-esque implementation within the source and ultimately this is where the demo will live but for now (and the purposes of that demo at ElixirConf) I went in a different direction
There is a db_mon-esque implementation within the source and ultimately this is where the demo will live but for now (and the purposes of that demo at ElixirConf) I went in a different direction
For me, the main benefit would be the ability to use a concurrency model other than the event loop.
In Elixir, I can reason about the code being run in one process as though it were synchronous, because it kind of is. But I can do that without blocking anything else, which is also pretty important.
In Elixir, I can reason about the code being run in one process as though it were synchronous, because it kind of is. But I can do that without blocking anything else, which is also pretty important.
It will run Erlang and Elixir. So anything you like or dislike about working with those languages should carry over. I find them delightful, mostly have experience with Elixir.
Asynchronous and event-driven code is already a fact in the browser so working with an ecosystem that is designed for that may prove a better experience.
I'm hopeful. Also, concurrency, staying of the render thread if possible. There's lots of fun to be had there.
Asynchronous and event-driven code is already a fact in the browser so working with an ecosystem that is designed for that may prove a better experience.
I'm hopeful. Also, concurrency, staying of the render thread if possible. There's lots of fun to be had there.
Supervisors, Genservers etc. are OTP patterns, not specifically Erlang or Elixir language features.
Lumen seems to implement OTP patterns so that's why those things work on it.
Lumen seems to implement OTP patterns so that's why those things work on it.
Yeah, I'm aware of the distinction. But I honestly haven't settled on how to refer to it all as a whole. Do you have a preference?
this is fantastic work. thank you and keep up the great work! The day JS becomes irrelevant will be joyous.
[deleted]
Interesting project... I wonder does it have same kind of processes preemption based on reductions like BEAM?
Is that the thing where after a number of reductions by a single process the BEAM will make sure another process gets its time in the sun? I believe so but I'm not familiar enough with the internals. They definitely spoke about reductions in the meeting I half-listened to.
Correct. It makes it so for(true) in other languages doesn't hijack the cpu and strive for the low latency goals of Erlang.
To be fair you can go about it by just filling up a process mail box with tons of messages.
To be fair you can go about it by just filling up a process mail box with tons of messages.
Yes, Lumen has a scheduler and does preemption the same way the BEAM does.
Yes - preemptive multitasking is core to the BEAM process model.
I’d love to try using a wasm interpreter to run this on microprocessors! Is the project at a point that’d be feasible to test?
There is also nerves which specifically targets embedded applications: https://nerves-project.org/
microprocessor/embedded systems is a compilation target after the initial browser implementation. We're very aware of how awesome it would be to have the Erlang programming model without having to require an OS
Something like this thing? https://github.com/wasm3/wasm3
The sample I tried in this post is only an Erlang interpreter. The whole compile chain is not quite ready from what I gather. It should be feasible at later point. I don't see why not.
The sample I tried in this post is only an Erlang interpreter. The whole compile chain is not quite ready from what I gather. It should be feasible at later point. I don't see why not.
[deleted]
darkstar999(1)
evilduck(5)
fraktl(3)
Uh, Lumen is already in use by the Laravel team for their micro-framework, maybe they're in time for a name change?
Elixir has had Mix since 2012, but Laravel made Laravel Mix in 2016 and hasn't changed it.
Unless Laravel team trademarked the name and are ready to go to court to defend it, I don't see why.
Mandatory word of warning [1].
Don't use wasm magic like this in production until its ready.
[1] https://www.reddit.com/r/csharp/comments/ea5dnb/the_nightmar...
[1] https://www.reddit.com/r/csharp/comments/ea5dnb/the_nightmar...
That guy is using server side blazer which doesn't use wasm. And his experiences aren't normal.
or just ignore the warning. Go for it, deploy it to production.
this is just FUD spreading. Having a bad experience with Blazor doesn't translate to anything on the BEAM.
So you write it in elixer and web assembly converts it to Js that browsers understand. Cool. Just write it in JS and it will probably be faster.
That's not accurate.
WebAssembly is a different beast than JS. As far as I've understood it is executed differently and can do some different things. I don't see any reason why it couldn't be optimized more than JS either, though that's bound to take some time considering the amount of tuning that's gone into JS engines.
More accurately, you write Elixir, it compiles down to Wasm. Wasm is run by your browser as a more machine-friendly format than JS. It may well be faster.
More accurately, you write Elixir, it compiles down to Wasm. Wasm is run by your browser as a more machine-friendly format than JS. It may well be faster.
That’s not how WebAssembly works, at all, it is handled directly by browser JIT compilers.