Static Executables with SBCL v2(timmons.dev)
timmons.dev
Static Executables with SBCL v2
https://www.timmons.dev/posts/static-executables-with-sbcl-v2.html
31 comments
Single executables make distribution really easy. Sure, a lot of languages like Python don't have this, but it's also easier to tell someone to download and install Python to run a script than the SBCL process. The entire thing from emacs/slime to the package manager and so forth was just very complicated to me. Things like Portacle didn't help much either.
Yeah, I heard that argument — but seriously, how many people run or "install" single executables these days? What actually gets distributed as a single executable? I honestly can't think of anything, on any system, except perhaps ngrok which I use for development.
In my case:
$ find ~/bin /usr/bin/ /bin/ -executable | xargs -I XXX file XXX | rg "statically linked" | awk '{ print $1 }'
/home/t/bin/floki:
/home/t/bin/yj:
/home/t/bin/helm:
/home/t/bin/fzf:
/home/t/bin/kubectl:
/usr/bin/containerd-shim-runc-v1:
/usr/bin/hyperfine:
/usr/bin/docker-init:
/usr/bin/containerd-shim:
/usr/bin/containerd-shim-runc-v2:
/bin/busybox:Chicken and egg. "many people" on Linux don't download executables and run them even if they're perfectly trusting of the source, because they'd often be linked against missing or incompatible dynamic libraries, which can lead to other problems during use. So then users are hesitant of them, so there's less of a desire to produce them in the first place (because users don't want it.) This is exacerbated by the fact static linking on Linux is complicated by historical factors and design decisions in some components like glibc.
Go and Alpine Linux especially have caused a resurgence of interest in distributing static binaries. The vast majority of tools in the Go ecosystem encourage this, for instance.
Really the whole conflation of and "executable application" as a "single file" is a historical mistake that we made but haven't fully let go of yet, and it's caused huge issues for users. For example, the distinction between "static linking" and "dynamic linking" largely isn't exposed to users on macOS, because they use .app bundles -- for all intents and purposes every macOS .app is "statically linked" because it can never resolve against other (possibly incompatible) dynamic libraries and it's a single file the user can manage. Then static/dynamic linking is just a technical aspect of how some amounts of code get loaded into the process, it's not a factor which impacts distribution or anything.
At the end of the day, an "application" is NOT a file. It's an abstract object from the users perspective. Maybe there is a single file hiding behind the curtain, or it's a collection of files, or maybe something else entirely, but they aren't the same.
Even Windows is also now moving in this direction (you can now have 'Runtimes' in Windows so that, say, a .py file and its data files can appear collectively like an .exe and be managed like a single file, with a single python runtime in the background), and Linux is too with things like AppImage, etc.
Static linking is dead, long live static linking, etc etc.
Go and Alpine Linux especially have caused a resurgence of interest in distributing static binaries. The vast majority of tools in the Go ecosystem encourage this, for instance.
Really the whole conflation of and "executable application" as a "single file" is a historical mistake that we made but haven't fully let go of yet, and it's caused huge issues for users. For example, the distinction between "static linking" and "dynamic linking" largely isn't exposed to users on macOS, because they use .app bundles -- for all intents and purposes every macOS .app is "statically linked" because it can never resolve against other (possibly incompatible) dynamic libraries and it's a single file the user can manage. Then static/dynamic linking is just a technical aspect of how some amounts of code get loaded into the process, it's not a factor which impacts distribution or anything.
At the end of the day, an "application" is NOT a file. It's an abstract object from the users perspective. Maybe there is a single file hiding behind the curtain, or it's a collection of files, or maybe something else entirely, but they aren't the same.
Even Windows is also now moving in this direction (you can now have 'Runtimes' in Windows so that, say, a .py file and its data files can appear collectively like an .exe and be managed like a single file, with a single python runtime in the background), and Linux is too with things like AppImage, etc.
Static linking is dead, long live static linking, etc etc.
> but seriously, how many people run or "install" single executables these days?
Pretty much everyone who's ever used an AppImage on Linux, for one.
This is also still the norm for macOS, from the user's point of view (yes, the application is technically a folder, but as far as most end users are concerned it's a single "thing" that gets dropped into their Applications folder). Still the norm for Windows, too, at least for installers (and "portable" apps, while not as common as ones installed to Program Files, certainly ain't obscure, even in this day and age).
Pretty much everyone who's ever used an AppImage on Linux, for one.
This is also still the norm for macOS, from the user's point of view (yes, the application is technically a folder, but as far as most end users are concerned it's a single "thing" that gets dropped into their Applications folder). Still the norm for Windows, too, at least for installers (and "portable" apps, while not as common as ones installed to Program Files, certainly ain't obscure, even in this day and age).
Almost every single tool in the k8s ecosystem is a simple statically linked go-based executable. Core stuff like kubectl, helm, etc. Read any k8s guide and step 1 is usually "go download all theses exes from github releases" (or more in vogue now, install a global NPM package that has the executable in its bin dir).
The previous version of the article (https://www.timmons.dev/posts/static-executables-with-sbcl.h...) explains a few contexts where static binaries are desirable.
In my eyes, that's part of the problem. A single executable is a lot easier to deal with.
Having someone download Anaconda Python (massive install) to run a script is a huge fail in my eyes.
Having someone download Anaconda Python (massive install) to run a script is a huge fail in my eyes.
This is a standing trend that goes many ways.
A common detour you will hit, is that the ease you had in distribution of the program can be erased by the complication in tracking what statically linked lib is active in that program.
Such that static versus dynamic linking is just a show swinging of a trend back and forth.
A common detour you will hit, is that the ease you had in distribution of the program can be erased by the complication in tracking what statically linked lib is active in that program.
Such that static versus dynamic linking is just a show swinging of a trend back and forth.
I help develop a CL application that is delivered as a static executable built using SBCL, it doesn't need any other shared libraries so save-lisp-and-die is enough.
> why other languages didn't have to do it, but CL absolutely did
Possibly a feedback loop on lack of popularity :). Other languages do have to do this in some way or another, with respect to their runtimes. The dependencies can be dynamically loaded, but the entry point - from the user's POV - still has to be invocable by double-clicking or by ./, or you'll lose audience. Also size considerations; people are now more willing to download a 100+ MB trivial application than they were just a few years ago.
On the runtime end, C/C++ solve this by having the runtime being part of the system, Java by somehow managing to make itself required everywhere, so everybody has a JRE already. Rust also compiles down to regular executables. Python got popular enough that at least Linux users are likely to have several runtimes already installed - but then Windows users don't, and any serious app will ship the runtime with it, etc.
In my experience, it always boiled down to this: how to ship a portable Lisp application in a single package (by "portable", I mean no installation step), where the user just has to double-click on something and have it run, and at no point they have to know or care it's written in Lisp? This, IIRC, was meaningfully non-trivial, especially if you wanted to interop with external dependencies by FFI. I remember that at one point, I shipped my code to a non-Lisper in form of a Dockerfile which handled installing SBCL for them. Not only they were grateful, they've managed to pass on that program to another friend who (back then) was not much of a coder at all, and they both used it. That's the qualitative difference easy deployment makes.
Possibly a feedback loop on lack of popularity :). Other languages do have to do this in some way or another, with respect to their runtimes. The dependencies can be dynamically loaded, but the entry point - from the user's POV - still has to be invocable by double-clicking or by ./, or you'll lose audience. Also size considerations; people are now more willing to download a 100+ MB trivial application than they were just a few years ago.
On the runtime end, C/C++ solve this by having the runtime being part of the system, Java by somehow managing to make itself required everywhere, so everybody has a JRE already. Rust also compiles down to regular executables. Python got popular enough that at least Linux users are likely to have several runtimes already installed - but then Windows users don't, and any serious app will ship the runtime with it, etc.
In my experience, it always boiled down to this: how to ship a portable Lisp application in a single package (by "portable", I mean no installation step), where the user just has to double-click on something and have it run, and at no point they have to know or care it's written in Lisp? This, IIRC, was meaningfully non-trivial, especially if you wanted to interop with external dependencies by FFI. I remember that at one point, I shipped my code to a non-Lisper in form of a Dockerfile which handled installing SBCL for them. Not only they were grateful, they've managed to pass on that program to another friend who (back then) was not much of a coder at all, and they both used it. That's the qualitative difference easy deployment makes.
With some languages the problem is solved by simply being so ubiquitous that the runtime is already on your machine. I think this is true of C, C++, Python, Java... The second best thing is a fully self-contained executable like you get with Go or Rust. (I do not think it is a coincidence that both Go and Rust have been massively successful, though I think statically linked binaries are only a minimal part of that.)
The third best thing is shipping the binary alongside the app. The worst thing I can think of is just not shipping a runtime.
(Of course when it comes to package maintainers, it may still be desirable to ship a separate runtime package that everything links to, so as long as the language provides ABI stability for the runtime... but I like to think of package maintainer use cases as being separate from normal usage.)
And of course, for developers who primarily write apps that don't "ship" to external users, it would be normal to feel puzzled about this.
The third best thing is shipping the binary alongside the app. The worst thing I can think of is just not shipping a runtime.
(Of course when it comes to package maintainers, it may still be desirable to ship a separate runtime package that everything links to, so as long as the language provides ABI stability for the runtime... but I like to think of package maintainer use cases as being separate from normal usage.)
And of course, for developers who primarily write apps that don't "ship" to external users, it would be normal to feel puzzled about this.
Static executables seems like one of the key selling points driving adoption of Go, for example.
PSA: SBCL stands for Steel Bank Common Lisp - http://www.sbcl.org
PSA: PSA stands for Public Service Announcement. Just in case.
PSA: PSA stands for Public Service Announcement. Just in case.
> The name "Steel Bank Common Lisp" is a reference to Carnegie Mellon University Common Lisp from which SBCL forked: Andrew Carnegie made his fortune in the steel industry and Andrew Mellon was a successful banker.
https://en.wikipedia.org/wiki/Steel_Bank_Common_Lisp
https://en.wikipedia.org/wiki/Steel_Bank_Common_Lisp
Warning - rant ahead:
I've been in this field for 15 years now. At some point the puns are a bit overbearing. We have enough cognitive overhead anyway, it's inherent in our field.
They could have named it Carnegie Mellon Common Lisp, but hey, someone wanted to feel clever.
Though this is super minor compared to Ruby (gems & co.) and especially to Chef (where you work with cookbooks, recipes, etc.) And of course, the granddaddy of them all, Unix. Because of course, less is more (all of my non-techie friends roll their eyes when I tell them about that one).
I've been in this field for 15 years now. At some point the puns are a bit overbearing. We have enough cognitive overhead anyway, it's inherent in our field.
They could have named it Carnegie Mellon Common Lisp, but hey, someone wanted to feel clever.
Though this is super minor compared to Ruby (gems & co.) and especially to Chef (where you work with cookbooks, recipes, etc.) And of course, the granddaddy of them all, Unix. Because of course, less is more (all of my non-techie friends roll their eyes when I tell them about that one).
Well it’s a fork of cmucl. Carnegie Mellon university Common Lisp. So they couldn’t have named it the same as the thing it was derived from.
I think Steel Bank Common Lisp is a good name and the pun makes me like it even more. How is "Steel Bank" any different from a made up name? It's not.
Do names based on wordplay really cause you cognitive overhead? How are they any worse than totally arbitrary names? How is a pun a more distracting etymology than any other kind? Does it bother you that New York is named for King James II, as he was the Duke of York at the time?
Musl is tired and Cosmopolitan wired, since it takes the fully static binary concept to a whole new level. SBCL should consider using it, since it lets your binaries run on Windows, Mac, FreeBSD, OpenBSD, and NetBSD too. https://github.com/jart/cosmopolitan See https://justine.lol/ape.html which talks about how the executable format is also a valid zip file, which was used to create a JavaScript interpreter that loads sources from itself in a single file. Another option for SBCL is to just use Cosmopolitan's research to implement the same thing from scratch. See files like https://github.com/jart/cosmopolitan/blob/master/libc/sysv/s... and https://github.com/jart/cosmopolitan/blob/master/libc/sysv/c...
"tired and Cosmopolitan wired" This poster needs to be banned. Only relevant if you want an interpreter which runs an secondary interpreter, encapsulated in a .zip file.
Cosmopolitan binaries aren't interpreted. They're just flat x86 binaries. It needs a 12kb runtime that's written in C. All the runtime does is (a) polyfills magic numbers for the SYSCALL instruction and (b) boot-load from BIOS to long mode on metal w/ linux kernel style negative memory page tables. So basically, it's just glue that takes care of the ugly low-level system details so you can bolt on whatever high level language interpreter you want. For example, the Wasm3 engine recently adopted it.
Haven't used SBCL in a while, so just to confirm: is this a better alternative to SB-EXT:SAVE-LISP-AND-DIE?
`save-lisp-and-die` is still how you do the image dump, but the process here is:
1. compile & load all lisp code
2. dump list of external symbols
3. save-lisp-and-die into a .core file
4. build new SBCL w/ symbols from step #2 linked in (using sbcl.o, which is the SBCL runtime)
5. run new SBCL, using the core from step #3
6. save-lisp-and-die into an executable
edit. fixed formatting
1. compile & load all lisp code
2. dump list of external symbols
3. save-lisp-and-die into a .core file
4. build new SBCL w/ symbols from step #2 linked in (using sbcl.o, which is the SBCL runtime)
5. run new SBCL, using the core from step #3
6. save-lisp-and-die into an executable
edit. fixed formatting
Without looking at the details, this work probably still uses something like that, under the hood. In mainline sbcl, dumping an executable with save-lisp-and-die will create an executable that depends on a few dynamic libraries and, if you use the ffi, whatever dynamic libraries you're using though that. With the already existing static-program-op in one of the cffi packages (grovel, maybe?), you can get an executable with all of the libraries linked through the cffi linked in staticly (you won't need those .so files when you distribute the executable) but it will still need a few dynamic libraries when run. Glibc and libm at least, I think.
As I understand it, this work is attempting to remove even those last few dynamic dependencies so you can distribute the dumped executable and be done.
So, better if you want simpler distribution at the cost of larger executables.
As I understand it, this work is attempting to remove even those last few dynamic dependencies so you can distribute the dumped executable and be done.
So, better if you want simpler distribution at the cost of larger executables.
First hit on google: SBLC == Stand By Letter of Credit.
Clearly, Common LISP is not important/relevant according to google. It seems like "Common Lisp" implementations may be stuck in the mid 1980s and the implementors do not have any idea about how to use shared libraries. So sad
Most people who actually used CL for building stuff were somewhat amused, and tried to find out what was so important about encapsulating everything into a single executable and why other languages didn't have to do it, but CL absolutely did.
I thought this was a red herring, but indeed for some reason it did affect acceptability. Glad to see a solution that looks fairly good.