Depending in Common Lisp – Using the CLOS dependent maintenance protocol (2022)(stevelosh.com)
stevelosh.com
Depending in Common Lisp – Using the CLOS dependent maintenance protocol (2022)
https://stevelosh.com/blog/2022/08/depending-in-common-lisp/
32 comments
Nice to see more LISP posts.
We need to at least try to get rid of the bad (old) perception; Common Lisp (with SBCL and some others) is a very viable environment for most things people want to do besides hiring bags of people (for now anyway).
Image-based development is a dead end. Whatever one's feelings are about interactive development, any system that relies on information that was typed into a REPL three-months ago and is now only stored in the binary state of that REPL and nowhere else is totally insane as a production product.
And this is unfortunately how most Lisp tooling still works. Quicklisp and friends assume you're fulfilling external dependencies by evaluating things in the REPL. This is without mentioning that the entire ecosystem of external dependencies ships monthly as one monolithic group. Versions? Lockfiles? Nah dog, give me "October 2023".
The only person trying to fix this if Fukamachi with Roswell and qlot, and those are very recent projects (in the scale of Common Lisp), that have gotten irrational amounts of resistance from lispers. C now has project-local dependency management thanks to vcpkg/conan/Spack/etc, but lispers still insist that ~/common-lisp is good enough for every single project they will ever work on.
And this is unfortunately how most Lisp tooling still works. Quicklisp and friends assume you're fulfilling external dependencies by evaluating things in the REPL. This is without mentioning that the entire ecosystem of external dependencies ships monthly as one monolithic group. Versions? Lockfiles? Nah dog, give me "October 2023".
The only person trying to fix this if Fukamachi with Roswell and qlot, and those are very recent projects (in the scale of Common Lisp), that have gotten irrational amounts of resistance from lispers. C now has project-local dependency management thanks to vcpkg/conan/Spack/etc, but lispers still insist that ~/common-lisp is good enough for every single project they will ever work on.
It largely is fine though.
Dependency management is Lisp-like languages without a separate binary-compilation stage typically only breaks when you have external dependencies.Pip would be fine if everything was in Python; it's not because numpy/protobuf/jax/pytorch/... breaks everytime you look look at it.
CL libs are nowhere near that level of cross-dependency and all of them get built in a local cache when you first start anyway, so you can pretty much "lock" the versions by git sub-module-ing them. It's very nix-like way of working (IME things only ever break on the CL-<outside world> barrier).
Dependency management is Lisp-like languages without a separate binary-compilation stage typically only breaks when you have external dependencies.Pip would be fine if everything was in Python; it's not because numpy/protobuf/jax/pytorch/... breaks everytime you look look at it.
CL libs are nowhere near that level of cross-dependency and all of them get built in a local cache when you first start anyway, so you can pretty much "lock" the versions by git sub-module-ing them. It's very nix-like way of working (IME things only ever break on the CL-<outside world> barrier).
> any system that relies on information that was typed into a REPL three-months ago and is now only stored in the binary state of that REPL and nowhere else
Implemenations like SBCL can produce executables. Source code is stored in files and can be versioned.
> The only person trying to fix this [is] Fukamachi
There are other ways to get dependencies, such as ultra lisp https://ultralisp.org/ and OCICL https://github.com/ocicl/ocicl.
Implemenations like SBCL can produce executables. Source code is stored in files and can be versioned.
> The only person trying to fix this [is] Fukamachi
There are other ways to get dependencies, such as ultra lisp https://ultralisp.org/ and OCICL https://github.com/ocicl/ocicl.
Yes, of course, that's why I brought up Fukamachi's work to point out that things can be better. It is possible to forego the mechanisms of interactive-development that intersect with image-based development, but that requires lispers to change how they've done things for generations. Neither the ecosystem or culture is aligned to it.
Ultralisp + Roswell + qlot produces something resembling a modern development workflow, but it is swimming upstream. ASDF is totally inadequate as a build system, it makes CMake appear terse and elegant by comparison (why on God's green earth is the present working directory not searched for the package I'm trying to build, like every other build system to ever exist?). Documentation and tutorials assume you're pulling things down with quicklisp and evaluating in the REPL. /r/Common_Lisp posters wonder why you would even want to use dependencies at all when you could just use [incomprehensible, brittle, unhygienic macro].
It has become possible, in the last two years or so, to produce sane common lisp workflows that operate somewhat like other modern environments. I very, very rarely encounter that usage in the wild. I would like nothing more than for the Common Lisp infrastructure to produce environments and workflows as obvious, intuitive, and correct as what every undergrad using Python in VSCode has today. I do not think the broader language ecosystem is there yet.
Ultralisp + Roswell + qlot produces something resembling a modern development workflow, but it is swimming upstream. ASDF is totally inadequate as a build system, it makes CMake appear terse and elegant by comparison (why on God's green earth is the present working directory not searched for the package I'm trying to build, like every other build system to ever exist?). Documentation and tutorials assume you're pulling things down with quicklisp and evaluating in the REPL. /r/Common_Lisp posters wonder why you would even want to use dependencies at all when you could just use [incomprehensible, brittle, unhygienic macro].
It has become possible, in the last two years or so, to produce sane common lisp workflows that operate somewhat like other modern environments. I very, very rarely encounter that usage in the wild. I would like nothing more than for the Common Lisp infrastructure to produce environments and workflows as obvious, intuitive, and correct as what every undergrad using Python in VSCode has today. I do not think the broader language ecosystem is there yet.
> why on God's green earth is the present working directory not searched for the package I'm trying to build, like every other build system to ever exist?
What if you want to share packages (aka CL systems) between different CL systems / applications?
What if you want to share packages (aka CL systems) between different CL systems / applications?
Do it the same way every language environment not named Common Lisp does?
There might be people using cl like that, I don't know these people. We just use files and versioning and execute saves and versioned snippets in slime which works fast and well but is not image based development at all.
>any system that relies on information that was typed into a REPL three-months ago and is now only stored in the binary state of that REPL and nowhere else is totally insane as a production product.
Nobody does that.
Nobody does that.
Yes, I am confused where people get that idea. 30 years ago in irc maybe?
(not dissing on irc: i was one of these people 30 years ago)
(not dissing on irc: i was one of these people 30 years ago)
I could very well be unlucky, and my experience is mostly with academics (not much industry usage of CL anymore).
But if I had a nickel for everytime I got emailed a core file I would not have had to be working in academia.
But if I had a nickel for everytime I got emailed a core file I would not have had to be working in academia.
I use OCICL, it's pretty great. That is, as long as you ignore it's basically one guy and his GitHub profile. It's unfortunate clpm died, that could've also been interesting.
First step would be to support SLIME in editors other than Emacs.
see: https://lispcookbook.github.io/cl-cookbook/editor-support.ht... Atom/Pulsar (good to very good support), VSCode, Sublime, Jetbrains, Jupyter notebooks, vim, Geany (experimental), Lem (built in CL)…
vim has 2 implementations of slime.
Lem is a new editor written in Common Lisp that has slime "built-in".
Lem is a new editor written in Common Lisp that has slime "built-in".
> vim has 2 implementations of slime.
Which are both buggy and one of them is abandoned.
> Lem is a new editor written in Common Lisp that has slime "built-in".
"Lem" is not (neo)vim, has Emacs RSI bindings and even if by some chance it included some "vi mode", it would simply be an emulation, not an actual vi editor. Same issue with Emacs using "evil mode".
Which are both buggy and one of them is abandoned.
> Lem is a new editor written in Common Lisp that has slime "built-in".
"Lem" is not (neo)vim, has Emacs RSI bindings and even if by some chance it included some "vi mode", it would simply be an emulation, not an actual vi editor. Same issue with Emacs using "evil mode".
> even if by some chance it included some "vi mode", it would simply be an emulation, not an actual vi editor
Lem supports vim key bindings, according to its README.
btw, I don't have any issues with emacs keybindings, but I chuckled at the "Emacs RSI bindings" :)
In fact, I think vim keybindings are superior for editing - you can't beat hitting "." to repeat an action - but after using emacs for ~5 years I find myself in it a lot more than I expected when I started using it. The whole thing with key bindings is that for whichever one(s) you use your muscle memory catches up.
Lem supports vim key bindings, according to its README.
btw, I don't have any issues with emacs keybindings, but I chuckled at the "Emacs RSI bindings" :)
In fact, I think vim keybindings are superior for editing - you can't beat hitting "." to repeat an action - but after using emacs for ~5 years I find myself in it a lot more than I expected when I started using it. The whole thing with key bindings is that for whichever one(s) you use your muscle memory catches up.
I despise SLIME. It made perfect sense in 2003 and was way ahead of the game compared to every other language for decades, but today we have the language server protocol and every effort should be made to support that instead.
Anecdotally, the AliveLSP works fine although it suffers from the same "Why would you want a standalone program? Load it into the REPL" problems I derided in response to the parent.
Anecdotally, the AliveLSP works fine although it suffers from the same "Why would you want a standalone program? Load it into the REPL" problems I derided in response to the parent.
I guess SLIME just works well for most as I would say it is still ahead every time I see python devs using their 'great tools' including a lsp in pain. I like a lsp where, during development, my core runs and I can see live feedback of it running when I make changes. Kind of like, you know, an image. That does not say you need to depend on that image outside dev (like an lsp, reset it when you reload a project, quit the editor etc), but during dev it's just superior imho.
Ya, this is the core of the disagreement that I would run into with the old school lispers.
I see it, I do, my third eye is open. I get the entire flow that the interactive development environment brings to the table and how it is infinitely superior than what existed in every other language for decades.
But I don't like it more than the modern tooling. That's pure personal preference. I don't like this weird stateful thing hanging out in the background of my dev environment. I don't find value in sending random expressions into the weird stateful blob. If I wanted to drop into a debugger, I'll ask, don't hijack my stacktraces.
I write tight test loops, modify my code, run the tests from scratch in the same one-key press the lispers use to evaluate their s-expressions. I don't need to go back and verify my build or application works from scratch, it never breaks, I'm constantly rebuilding.
I would run into situations not infrequently where some post-doc wasn't quite sure how they achieved the state they did in the image, had a bug, but no tests to reproduce the bug or anything else. Maddening. An anti-pattern I never saw accomplished to the same degree with the Python boys (not to say Jupyter Notebooks aren't their own stateful fucking mad house of bad software engineering).
I see it, I do, my third eye is open. I get the entire flow that the interactive development environment brings to the table and how it is infinitely superior than what existed in every other language for decades.
But I don't like it more than the modern tooling. That's pure personal preference. I don't like this weird stateful thing hanging out in the background of my dev environment. I don't find value in sending random expressions into the weird stateful blob. If I wanted to drop into a debugger, I'll ask, don't hijack my stacktraces.
I write tight test loops, modify my code, run the tests from scratch in the same one-key press the lispers use to evaluate their s-expressions. I don't need to go back and verify my build or application works from scratch, it never breaks, I'm constantly rebuilding.
I would run into situations not infrequently where some post-doc wasn't quite sure how they achieved the state they did in the image, had a bug, but no tests to reproduce the bug or anything else. Maddening. An anti-pattern I never saw accomplished to the same degree with the Python boys (not to say Jupyter Notebooks aren't their own stateful fucking mad house of bad software engineering).
> Jupyter Notebooks
Well said. My favorite is the notebook that can only get into the state of interest by executing the cells out of order, and skipping the ones with syntax errors. Jupyter is fine for messing around with data, but saving a notebook is a bad idea and sharing one is a terrible idea. The fact that there are software delivery workflows built around Jupyter notebooks is just sheer lunacy.
Well said. My favorite is the notebook that can only get into the state of interest by executing the cells out of order, and skipping the ones with syntax errors. Jupyter is fine for messing around with data, but saving a notebook is a bad idea and sharing one is a terrible idea. The fact that there are software delivery workflows built around Jupyter notebooks is just sheer lunacy.
But that is 'the modern way'... It is what all python people are doing. It is terrible, I agree, however people make out stuff is somehow better now, while making this nightmare. While most CL people moved on a long time ago.
We can be more precise, I'm talking about declarative build systems and dependency tracking, project-local builds, tight IDE integration with automatic test suite detection, debug adapters, and language servers.
Those are the kinds of things I'm addressing when I say "modern". I'm not talking about Jupyter Notebooks. Notebooks are as bad or worse than anything I'm whinging about in the CL ecosystem.
Those are the kinds of things I'm addressing when I say "modern". I'm not talking about Jupyter Notebooks. Notebooks are as bad or worse than anything I'm whinging about in the CL ecosystem.
I still see those as different things; if there is no 'state' besides running your code as in files as you edit it and it kills the image when you stop, how did you get to :
> how they achieved the state they did in the image
Because in my view, there is no image beyond your debug session.
But yeah, it would be nice to sit together at a meetup and you showing me what's so great about this modern tooling, because the modern tooling is terrible in my opinion. Sure, a debugger is ok-ish, when it works (which is sketchy at best if I see my friends trying to debug nextjs/react in vscode and breakpoints not hitting or in the wrong place etc etc or simply completely not working etc, but let's say it works perfectly); you still cannot just quickly test different path etc. I have been programming for well over 40 years and I did lisp/prolog in uni, then I went to the usual, Java, C#, JS/TS, Python, C/C++, Pascal/Delphi etc before trying lisp (&prolog) again and finding the modern tooling fairly lacking and annoying.
What you do, as you describe it, has simply nothing to do with modern test tools; you could do TDD with tight test loops in 1993 and it was a good way to do things. There is nothing in Coomon Lisp or SLIME preventing you from doing that, so I don't understand the issue here. The way you work is not incompatible with what I say (what I suggest doesn't exist fully, mind you); I work in the same way with Lisp as you suggest. I work like that in other languages too; I literally need 0 modern tooling for that. With 'modern' virtually worthless stacktraces, my tests are much faster at finding issues than a debugger.
Anyway; I think we are not far removed, just for some reason you keep hanging in that image based thing while I didn't suggest that, at least not in any traditional sense. And you don't have to use it; it seems it needs to be clearer that there is choice and traditional image based execution is considered bad practice; that's fine and a good plan.
> how they achieved the state they did in the image
Because in my view, there is no image beyond your debug session.
But yeah, it would be nice to sit together at a meetup and you showing me what's so great about this modern tooling, because the modern tooling is terrible in my opinion. Sure, a debugger is ok-ish, when it works (which is sketchy at best if I see my friends trying to debug nextjs/react in vscode and breakpoints not hitting or in the wrong place etc etc or simply completely not working etc, but let's say it works perfectly); you still cannot just quickly test different path etc. I have been programming for well over 40 years and I did lisp/prolog in uni, then I went to the usual, Java, C#, JS/TS, Python, C/C++, Pascal/Delphi etc before trying lisp (&prolog) again and finding the modern tooling fairly lacking and annoying.
What you do, as you describe it, has simply nothing to do with modern test tools; you could do TDD with tight test loops in 1993 and it was a good way to do things. There is nothing in Coomon Lisp or SLIME preventing you from doing that, so I don't understand the issue here. The way you work is not incompatible with what I say (what I suggest doesn't exist fully, mind you); I work in the same way with Lisp as you suggest. I work like that in other languages too; I literally need 0 modern tooling for that. With 'modern' virtually worthless stacktraces, my tests are much faster at finding issues than a debugger.
Anyway; I think we are not far removed, just for some reason you keep hanging in that image based thing while I didn't suggest that, at least not in any traditional sense. And you don't have to use it; it seems it needs to be clearer that there is choice and traditional image based execution is considered bad practice; that's fine and a good plan.
I don't think I would have minded working with you and your workflow!
There's nothing bad about CL interactive development when used responsibly (and this extends to CL generally, nothing wrong with macros when used responsibly). It enables some behaviors that are still totally unavailable in other language environments. The REPL is remains unique to lisp and its descendants.
The problem is it allows irresponsible users the ability to create scenarios that are nightmarish for the technical experts tasked to support them. I've come to realize I care a lot more about it being hard to do bad things than enabling occasional convenience for advanced users.
And for me personally, I've never cared for or used the features of the REPL that are unique to it so I never miss them. My workflow, my editor, UI, keybindings, test reporting, all of it, for C++, Lisp, Python, JS, Fortran, etc, is mostly identical across the board and I find immense value in that consistency.
There's nothing bad about CL interactive development when used responsibly (and this extends to CL generally, nothing wrong with macros when used responsibly). It enables some behaviors that are still totally unavailable in other language environments. The REPL is remains unique to lisp and its descendants.
The problem is it allows irresponsible users the ability to create scenarios that are nightmarish for the technical experts tasked to support them. I've come to realize I care a lot more about it being hard to do bad things than enabling occasional convenience for advanced users.
And for me personally, I've never cared for or used the features of the REPL that are unique to it so I never miss them. My workflow, my editor, UI, keybindings, test reporting, all of it, for C++, Lisp, Python, JS, Fortran, etc, is mostly identical across the board and I find immense value in that consistency.
Saying you despise it only shows how ignorant you are.
Can someone confirm if this is an accurate summary of the article: "CLOS dependent maintenance protocol" allows you to run code when a class definition (or one of its ancestors or descendants) changes?
I would say "it establishes a protocol to support recognizing and propagating changes the way you would expect throughout an inheritance hierarchy if something related to a meta object in that hierarchy changes, even while working interactively in the REPL".
this is cool, but holy crap that's a lot of work for a seemingly-simple type of feature!
All in the service of interactivity, a key feature of CL.
Even if some stuff could be simpler, I'll hazard a guess and say it's the cost of flexibility and "correctness".