LFE: Lisp Flavored Erlang(lfe.io)
lfe.io
LFE: Lisp Flavored Erlang
https://lfe.io/books/tutorial/index.html
32 comments
Lisp has two big ideas. The first big idea is that a regular syntax allows the trivial implementation of macros. Just have a separate compilation where the AST is passed in as a list to different macros and then compile the result.
The second idea (not shared by the some Lisps like Scheme) is that of a system image which is modified in real time. This allows on the fly debugging, adding of new features, etc with no downtime.
Macros have worked there way into languages like Julia or Nim, while the system image idea is mostly constrained to Smalltalk and Common Lisp.
The best example of the power of macros is Racket, which has world class meta programming facilities and is probably the best language in existence for creating new languages, DSLs, and doing experimental PL research.
I've used both CL and Racket professionally, and they both shine in certain situations.
That being said, I've grown tired of the relative verbosity of both languages (minimal syntax has a high cost) and the performance and productivity cost of dynamic typing.
For most new engineering projects, I'd much rather use something like OCaml or Scala than CL or Racket. For scientific computing, I usually go with Julia or kdb+/q.
That being said, I think Racket especially shines in the development of internal business or research tools. The large number of high quality libraries (especially for GUIs) makes it a great choice for desktop apps.
The second idea (not shared by the some Lisps like Scheme) is that of a system image which is modified in real time. This allows on the fly debugging, adding of new features, etc with no downtime.
Macros have worked there way into languages like Julia or Nim, while the system image idea is mostly constrained to Smalltalk and Common Lisp.
The best example of the power of macros is Racket, which has world class meta programming facilities and is probably the best language in existence for creating new languages, DSLs, and doing experimental PL research.
I've used both CL and Racket professionally, and they both shine in certain situations.
That being said, I've grown tired of the relative verbosity of both languages (minimal syntax has a high cost) and the performance and productivity cost of dynamic typing.
For most new engineering projects, I'd much rather use something like OCaml or Scala than CL or Racket. For scientific computing, I usually go with Julia or kdb+/q.
That being said, I think Racket especially shines in the development of internal business or research tools. The large number of high quality libraries (especially for GUIs) makes it a great choice for desktop apps.
Thanks!
I haven't really experienced the image feature, as I used Scheme. It doesn't sound like it would be consequential to how I program, but that might just be my ignorance of it.
My experience with the macro stuff is that they enable in-house implementation of language features like lazy evaluation (not possible in other languages without a lot of extra code), which we implemented during the course. But implementing features like that is not really something I need to do for my everyday programming.
I haven't really experienced the image feature, as I used Scheme. It doesn't sound like it would be consequential to how I program, but that might just be my ignorance of it.
My experience with the macro stuff is that they enable in-house implementation of language features like lazy evaluation (not possible in other languages without a lot of extra code), which we implemented during the course. But implementing features like that is not really something I need to do for my everyday programming.
The power of macros goes beyond just core language features. For example, Racket has a prolog implementation, an OOP implementation, contracts, and much more, all developed with macros. You can generate HTML, do templating, or write documentation, all with macros. There's a certain magical feeling as a library designer when you can decide upon whatever syntax you want, with absolutely no constraints. Leads to the development of some very cool and clean stuff.
For example, I used Racket macros at my old job as a bio informatics researcher at a hospital to design a DSL to specify bio informatics processing pipelines. You could either write it yourself, or you could use a GUI to manipulate a visual graph of operations and write out the pipeline using the DSL to a file.
These sort of DSL use cases are a lot more common than people think, mostly because they don't think of an API as a new language, merely as some code written in an existing one. When you realize that macros can be used to eliminate all repetition and all patterns, you start to think in a different way.
Designing proper interfaces with macros can be challenging though: you want to provide a nice syntax that is both composable with other macros, and clean.
Also macros allow the core language to be paired down and very simple. Stuff like Scheme, in which all desired features (OOP, actors, async, generators, etc) can be implemented by external packages.
But like I said, I don't use Lisp much anymore. Functional programming with higher kindred types goes a long way in matching the power of macros, albeit in a more structured way. Of course, type systems will always fall short of the unlimited power of macros, but for a lot of problems it's more than sufficient.
Think of Lisp as the prototypical programming language, the essence of transforming symbols into other symbols. Lisp forms the base, on top of which all other features can be implemented.
For example, I used Racket macros at my old job as a bio informatics researcher at a hospital to design a DSL to specify bio informatics processing pipelines. You could either write it yourself, or you could use a GUI to manipulate a visual graph of operations and write out the pipeline using the DSL to a file.
These sort of DSL use cases are a lot more common than people think, mostly because they don't think of an API as a new language, merely as some code written in an existing one. When you realize that macros can be used to eliminate all repetition and all patterns, you start to think in a different way.
Designing proper interfaces with macros can be challenging though: you want to provide a nice syntax that is both composable with other macros, and clean.
Also macros allow the core language to be paired down and very simple. Stuff like Scheme, in which all desired features (OOP, actors, async, generators, etc) can be implemented by external packages.
But like I said, I don't use Lisp much anymore. Functional programming with higher kindred types goes a long way in matching the power of macros, albeit in a more structured way. Of course, type systems will always fall short of the unlimited power of macros, but for a lot of problems it's more than sufficient.
Think of Lisp as the prototypical programming language, the essence of transforming symbols into other symbols. Lisp forms the base, on top of which all other features can be implemented.
I had many epiphanies as a CS student in all sorts of CS areas. It was because I was always taking things out of their educational context, and finding applications in programming, or at least seeing what the "big deal" is.
E.g. learning about red-black trees inspired me to write a decent C implementation of them, as something I could use. That ended up used in production all over the place, like in the ext2fsck program.
Undoubtedly, some of my classmates likely also had the reaction "when would I, in everyday programming, be rotating some nodes to balance a tree ..."
E.g. learning about red-black trees inspired me to write a decent C implementation of them, as something I could use. That ended up used in production all over the place, like in the ext2fsck program.
Undoubtedly, some of my classmates likely also had the reaction "when would I, in everyday programming, be rotating some nodes to balance a tree ..."
>Image
The best video I've come across thwt demonstrates images is https://youtu.be/3GEAINRCbJ4
TBH, one can use emacs to get a similar workflow in single file python projects. There are also julia-snail and slime variants in emacs for providing it for julia; none of which I've tested to comment much.
The best video I've come across thwt demonstrates images is https://youtu.be/3GEAINRCbJ4
TBH, one can use emacs to get a similar workflow in single file python projects. There are also julia-snail and slime variants in emacs for providing it for julia; none of which I've tested to comment much.
> The first big idea is that a regular syntax allows the trivial implementation of macros.
Note that Lisp existed for quite a while before macros. The regular syntax allowed for the evaluation algorithm of Lisp to be described as if it were a Lisp procedure itself, and it allowed the language to be extended with "FEXPR" routines.
Note that Lisp existed for quite a while before macros. The regular syntax allowed for the evaluation algorithm of Lisp to be described as if it were a Lisp procedure itself, and it allowed the language to be extended with "FEXPR" routines.
It's well worth looking at interlisp's file/image hybrid approach if you haven't already. I think there's a lot to be said for it.
* Hiccup (https://github.com/weavejester/hiccup) - DSL for generating html. You can easily mix it with other language constructs. There are similar libraries for Common Lisp and Scheme.
* Compojure-api (https://github.com/metosin/compojure-api) - for building web applications. Mix json-like, hiccup and clojure constructs for web applications and autogenerate Swagger documentation.
* LOL book (https://letoverlambda.com/). Extreme examples what you can do with macros and code modifications during compile time.
* Seesaw (https://gist.github.com/daveray/1441520) - library for building GUI apps with Clojure and Swing. Express GUI elements through declarative syntax. Qt and other libraries has similar feature, but is usually preprocessed with external tools.
* Scheme - MiniKanren (https://docs.racket-lang.org/minikanren/index.html)
Most of these things in regular languages would require modifying language parser or compiler, or adding external tool that will parse that code and generate new one; e.g. like React is doing with html chunks. Also, many Scheme/CL/Clojure implementations provide functions to modify syntax table in runtime, allowing you to alter how things are parsed. That is extremely hard in regular languages due unregular syntax constructs.
* Compojure-api (https://github.com/metosin/compojure-api) - for building web applications. Mix json-like, hiccup and clojure constructs for web applications and autogenerate Swagger documentation.
* LOL book (https://letoverlambda.com/). Extreme examples what you can do with macros and code modifications during compile time.
* Seesaw (https://gist.github.com/daveray/1441520) - library for building GUI apps with Clojure and Swing. Express GUI elements through declarative syntax. Qt and other libraries has similar feature, but is usually preprocessed with external tools.
* Scheme - MiniKanren (https://docs.racket-lang.org/minikanren/index.html)
Most of these things in regular languages would require modifying language parser or compiler, or adding external tool that will parse that code and generate new one; e.g. like React is doing with html chunks. Also, many Scheme/CL/Clojure implementations provide functions to modify syntax table in runtime, allowing you to alter how things are parsed. That is extremely hard in regular languages due unregular syntax constructs.
Pretty sure you're writing on a good example now!
Hacker news is written in Arc, which is a dialect of Lisp. There's an old copy of code base here:
https://github.com/wting/hackernews
Hacker news is written in Arc, which is a dialect of Lisp. There's an old copy of code base here:
https://github.com/wting/hackernews
I think you will have to program with it to get it. It's not so much about a final, static copy of code that you can look at and be amazed, but more about how fluid and well oiled the journey there was.
Try to build something with Clojure (bit easier to get started with than LFE, Common Lisp or others, except maybe Racket) and try to focus on a REPL heavy flow (with integration with your favorite editor) to see the full potential of Lisp.
Try to build something with Clojure (bit easier to get started with than LFE, Common Lisp or others, except maybe Racket) and try to focus on a REPL heavy flow (with integration with your favorite editor) to see the full potential of Lisp.
> Can someone direct me to some code where I can see the full potential of Lisp?
There's exists no code that shows the full potential of Lisp because its potential lies not in the code itself, but in the way of working with it.
The language is interactive and image-based, meaning that you do not program in it by passing code through a compiler into executables, but instead you modify the running Lisp image until it contains the program you want. This is usually done by connecting the editor to the running Lisp image and sending commands to it: usually that is to compile-and-load (or compile-and-replace) individual forms/functions/variables, or whole files, or whole systems, or to work with the read-eval-print loop, or to interactively inspect objects, or to interactively resolve signaled errors via the debugger. This provides a lot of introspection into how the Lisp image and the program contained within works, and drastically reduces the feedback loop since the overhead of compiling-and-loading individual functions or even whole files is unnoticeable.
Disclosure: I am a Common Lisp programmer.
There's exists no code that shows the full potential of Lisp because its potential lies not in the code itself, but in the way of working with it.
The language is interactive and image-based, meaning that you do not program in it by passing code through a compiler into executables, but instead you modify the running Lisp image until it contains the program you want. This is usually done by connecting the editor to the running Lisp image and sending commands to it: usually that is to compile-and-load (or compile-and-replace) individual forms/functions/variables, or whole files, or whole systems, or to work with the read-eval-print loop, or to interactively inspect objects, or to interactively resolve signaled errors via the debugger. This provides a lot of introspection into how the Lisp image and the program contained within works, and drastically reduces the feedback loop since the overhead of compiling-and-loading individual functions or even whole files is unnoticeable.
Disclosure: I am a Common Lisp programmer.
Peter Norvig’s Paradigms of Artificial Intelligence: case studies in Common Lisp shows how Common Lisp can be used “in anger” to create the compact powerful code people always say Common Lisp is good for.
Despite this being obvious hn catnip, I’m interested.
I loved the little and seasoned schemer books, yet found both racket and CL somehow divergent and unappealing.
Does anyone here use it? I can’t find if it deals with continuations well. I’ve never used many languages that allow eloquent use.
I loved the little and seasoned schemer books, yet found both racket and CL somehow divergent and unappealing.
Does anyone here use it? I can’t find if it deals with continuations well. I’ve never used many languages that allow eloquent use.
no openbsd port for lfe yet?
LFE is really cool, but I think the biggest thing that LFE has going against it in terms of mainstream/industry adoption is that Elixir is _roughly_ a Lisp (pretending it's not). LFE is competing against a language that has some _very_ well polished edges, and IMO it doesn't feel like LFE brings to much to the table for industry over the Lisp-y features that Elixir already has[0][1][2][3][4].
What I'd really like to see is a Mix LFE compiler[7] that lets you write LFE modules in a bigger Elixir project[5], plus a set of mix tasks like `mix lfe.repl` that integrates with the other BEAM components of your Mix projects.
Long term I think it would be cool if the BEAM community comes together around the excellent tooling (mix, ExUnit, Iex) that the Elixir Team is bringing, I'd love to see other BEAM languages (LFE, Gleam[6]) with top notch integrations that let a mix project include them side by side.
I think the BEAM is a really cool piece of technology, and many of these new BEAM languages offer pretty neat advantages, I'm excited to see the interop story grow over time
[0] https://elixir-lang.org/getting-started/meta/macros.html
[1] https://elixir-lang.org/getting-started/comprehensions.html
[2] https://elixir-lang.org/getting-started/protocols.html
[3] https://elixir-lang.org/getting-started/basic-types.html#lin...
[4] https://elixir-lang.org/getting-started/basic-types.html#ano...
[5] This exists, but looks unmaintained https://hex.pm/packages/mix_lfe
[6] https://gleam.run/
[7] https://hexdocs.pm/mix/1.10.2/Mix.Task.Compiler.html
What I'd really like to see is a Mix LFE compiler[7] that lets you write LFE modules in a bigger Elixir project[5], plus a set of mix tasks like `mix lfe.repl` that integrates with the other BEAM components of your Mix projects.
Long term I think it would be cool if the BEAM community comes together around the excellent tooling (mix, ExUnit, Iex) that the Elixir Team is bringing, I'd love to see other BEAM languages (LFE, Gleam[6]) with top notch integrations that let a mix project include them side by side.
I think the BEAM is a really cool piece of technology, and many of these new BEAM languages offer pretty neat advantages, I'm excited to see the interop story grow over time
[0] https://elixir-lang.org/getting-started/meta/macros.html
[1] https://elixir-lang.org/getting-started/comprehensions.html
[2] https://elixir-lang.org/getting-started/protocols.html
[3] https://elixir-lang.org/getting-started/basic-types.html#lin...
[4] https://elixir-lang.org/getting-started/basic-types.html#ano...
[5] This exists, but looks unmaintained https://hex.pm/packages/mix_lfe
[6] https://gleam.run/
[7] https://hexdocs.pm/mix/1.10.2/Mix.Task.Compiler.html
I completely agree. There's a higher abstraction level than the BEAM itself that it should be possible to integrate a new BEAM language at.
I'd like to invite you to the Other Side Of The Force, Grant.
http://www.petecorey.com/blog/2017/08/07/what-if-elixir-were...
It's much more pretty over there.
I haven't looked at Elixir but I have read a bit about Supervisors which I think are an excellent idea for fault tolerant systems.
However, the phrase "self healing" is thrown around a lot in talks about Elixir, as if it's a magical feature of the language, without much explanation as to what it means to universally heal from faults (I think maybe they mean self healing strategies that the programmer authors and implements, not some AGI that rewrites the code on the fly to fix it, lol, but that's my gripe, mostly the hype around that... and "coding for the happy path" which is not really a rigorous statement either... maybe someone can tell me what is the happy path for an algorithm in my head without knowing the algorithm, because that somebody is me, and I don't really know my algorithms until I have thought through the not-so-happy path... I mostly code to explore what I need to specify rather than describe some specification I already have.)
Maybe Elixir is not for me. Or maybe it is. I need to learn more, and go past the hype.
For me, Lisp is an alien technology with great power that will never be matched, except by another Lisp.
So it was great to find out about LFE! I even appreciate the separate var/fn namespaces, like nouns and verbs in natural language, e.g. fight the good fight, and I can see how it comes in handy, yet it's a new PL concept for me, so I'm still processing it.
http://www.petecorey.com/blog/2017/08/07/what-if-elixir-were...
It's much more pretty over there.
I haven't looked at Elixir but I have read a bit about Supervisors which I think are an excellent idea for fault tolerant systems.
However, the phrase "self healing" is thrown around a lot in talks about Elixir, as if it's a magical feature of the language, without much explanation as to what it means to universally heal from faults (I think maybe they mean self healing strategies that the programmer authors and implements, not some AGI that rewrites the code on the fly to fix it, lol, but that's my gripe, mostly the hype around that... and "coding for the happy path" which is not really a rigorous statement either... maybe someone can tell me what is the happy path for an algorithm in my head without knowing the algorithm, because that somebody is me, and I don't really know my algorithms until I have thought through the not-so-happy path... I mostly code to explore what I need to specify rather than describe some specification I already have.)
Maybe Elixir is not for me. Or maybe it is. I need to learn more, and go past the hype.
For me, Lisp is an alien technology with great power that will never be matched, except by another Lisp.
So it was great to find out about LFE! I even appreciate the separate var/fn namespaces, like nouns and verbs in natural language, e.g. fight the good fight, and I can see how it comes in handy, yet it's a new PL concept for me, so I'm still processing it.
> coding for the happy path
It's not that complicated. A relatable example would be writing an API endpoint that receives data and then you do something with that data.
1. Write the endpoint so it works for the expected data.
2. You're done
That's it. You don't need to worry about anyone sending you malformed payloads or fuzzing your API. You can ignore it. They cannot exploit anything, they cannot crash BEAM. It will keep working.
Maybe they'll run you out of memory or CPU or bandwidth, but those are problems solved at other layers (unless you include rate limiting in your application).
Ignore all errors you do not need to explicitly handle. That's the whole point. It's wonderful.
Maybe you can do this in other languages with worker pools and supervisors, but it will be very expensive and high latency due to the cost of OS process forking and you're also wasting a lot of resource on context switches with that design anyway.
It's not that complicated. A relatable example would be writing an API endpoint that receives data and then you do something with that data.
1. Write the endpoint so it works for the expected data.
2. You're done
That's it. You don't need to worry about anyone sending you malformed payloads or fuzzing your API. You can ignore it. They cannot exploit anything, they cannot crash BEAM. It will keep working.
Maybe they'll run you out of memory or CPU or bandwidth, but those are problems solved at other layers (unless you include rate limiting in your application).
Ignore all errors you do not need to explicitly handle. That's the whole point. It's wonderful.
Maybe you can do this in other languages with worker pools and supervisors, but it will be very expensive and high latency due to the cost of OS process forking and you're also wasting a lot of resource on context switches with that design anyway.
My 'A-ha' moment with Elixir was exactly your example.
Both me and a friend were building an app that ingests cryptocurrency data and stores it in a database for later analysis. The important part was that we wanted data at regular intervals. I'm not much into all this but this is what was asked of us. He wrote the app in Python; I wrote it in Elixir.
Turns out the API's of the various exchanges are (or were) atrocious. We'd get responses ranging from weird error codes to malformed data to timeouts. API's would randomly change. It was a mess.
My app just chugged along, a process per request per api endpoint. When some of these endpoints 'misbehaved' the others just kept going.
His app kept crashing, restart, and so one misbehaving endpoint would cause trouble for all the other ones. He had to add try...catch statements and fix the problems. I would just look at my logs, update the happy-path code, push it, and recompile() in the REPL.
I also have a bunch of personal projects chugging along on my VPS. They actively handle requests a few times per hour at least, by me, and they're doing fine despite the fact that I was quick about it and only wrote the 'happy path'. There's crashes and errors all over the place, and yet when one part crashes the rest just keeps going. It's been a rare occasion where the entire supervision tree failed and the app gave up.
Both me and a friend were building an app that ingests cryptocurrency data and stores it in a database for later analysis. The important part was that we wanted data at regular intervals. I'm not much into all this but this is what was asked of us. He wrote the app in Python; I wrote it in Elixir.
Turns out the API's of the various exchanges are (or were) atrocious. We'd get responses ranging from weird error codes to malformed data to timeouts. API's would randomly change. It was a mess.
My app just chugged along, a process per request per api endpoint. When some of these endpoints 'misbehaved' the others just kept going.
His app kept crashing, restart, and so one misbehaving endpoint would cause trouble for all the other ones. He had to add try...catch statements and fix the problems. I would just look at my logs, update the happy-path code, push it, and recompile() in the REPL.
I also have a bunch of personal projects chugging along on my VPS. They actively handle requests a few times per hour at least, by me, and they're doing fine despite the fact that I was quick about it and only wrote the 'happy path'. There's crashes and errors all over the place, and yet when one part crashes the rest just keeps going. It's been a rare occasion where the entire supervision tree failed and the app gave up.
So, both of you guys talk about API request handling.
How about some computational task where coding only the happy path means accurate calculated results for 80% of the cases?
I think my problem is the marketing hype with Elixir around "happy path" and "self healing" are too broad and resemble hype more than feature description. I'd be happy if those terms weren't used outside of marketing literature, until we have an AGI that can manage complexity on its own :)
How about some computational task where coding only the happy path means accurate calculated results for 80% of the cases?
I think my problem is the marketing hype with Elixir around "happy path" and "self healing" are too broad and resemble hype more than feature description. I'd be happy if those terms weren't used outside of marketing literature, until we have an AGI that can manage complexity on its own :)
I dunno. I'd say it's rather obvious that it's not a magical solution to everything. For one, Elixir is too slow for computation-heavy stuff.
I just gave an example of how happy path programming works in a use case that is pretty common to my day to day. A lot of other stuff I do is similar enough that the same applies, and that's just my experience, not marketing hype.
Some articles might be a little over-excited, but I don't think I've generally read articles that claim Elixir is God's gift to programmers ;). And in situations where Elixir does help, it can be very exciting.
I just gave an example of how happy path programming works in a use case that is pretty common to my day to day. A lot of other stuff I do is similar enough that the same applies, and that's just my experience, not marketing hype.
Some articles might be a little over-excited, but I don't think I've generally read articles that claim Elixir is God's gift to programmers ;). And in situations where Elixir does help, it can be very exciting.
Fair enough. Could you please describe other happy path scenarios? Is there an educational resource (blog post maybe?) that talks in depth about "happy path programming" in the general sense? Same applies to "self healing" I really want to know what the folks who came up with these phrases meant, in both specifics and the bigger picture. Thank you for the enlightenment so far!
FWIW, I wasn't aware that Elixir is not optimal for CPU-bound tasks. I use Go for highly concurrent servers, but it uses goroutines not a process per request, and mutexes and/or persistent channels to synchronize shared memory access.
FWIW, I wasn't aware that Elixir is not optimal for CPU-bound tasks. I use Go for highly concurrent servers, but it uses goroutines not a process per request, and mutexes and/or persistent channels to synchronize shared memory access.
That is seems to be more about the underlying infra (BEAM) than the language. I thought it was something specific to Elixir. I thought they meant it at the level of program logic, which is why it seemed very confusing.
I guess Lisp is about beauty and poetic justice and it shall remain in that realm.
I'll look into Elixir. Seems to have a healthy and growing ecosystem.
I guess Lisp is about beauty and poetic justice and it shall remain in that realm.
I'll look into Elixir. Seems to have a healthy and growing ecosystem.
The underlying infra is not decouplable from the language. The beam is first and foremost designed to get stuff done easy and without error, not to be beautiful and poetic. Doesn't mean it can't be beautiful! And programming is fun because it feels like programming with training wheels on.
Supervisors are only the beginning of the journey to fault tolerance in the BEAM... There's so much you can do; one way of thinking about it is that you get very smart garbage collection on arbitrary system resources, tied to concurrent threads, and you can link threads so they fall in the same failure domain.
Also related is clojerl which is Clojure running on Erlang VM: https://github.com/clojerl/clojerl (http://clojerl.org/)
Some past submissions of clojerl:
- https://news.ycombinator.com/item?id=21965469
- https://news.ycombinator.com/item?id=20219563
Some past submissions of clojerl:
- https://news.ycombinator.com/item?id=21965469
- https://news.ycombinator.com/item?id=20219563
There are quite a few languages targeting the BEAM by now, with LFE one of the oldest.
I've been trying to keep track of them all: https://gist.github.com/macintux/6349828#alternative-languag...
Gleam is one of the more interesting recent additions:
- https://gleam.run
- https://news.ycombinator.com/item?id=22902462
I've been trying to keep track of them all: https://gist.github.com/macintux/6349828#alternative-languag...
Gleam is one of the more interesting recent additions:
- https://gleam.run
- https://news.ycombinator.com/item?id=22902462
I have programmed in Scheme before while taking a course using SICP. Final project was a Scheme interpreter. But I didn't have any epiphany/awakening like many others seems to have.