Shows one-file examples of using Datastar for reactive web development in Common Lisp. Covers the basics but also related topics like CQRS an fat morphing.
This is one of the demos included in the Datastar SDK for Common Lisp I've been working on. The ideas behind it are also present in the interactive guide here https://lambda-combine.net/hyper-guide - essentially a complement to the Datastar guide in Common Lisp.
This demo specifically shows Datastar being used, with multiple POST commands and one single GET stream - CQRS, as per the Datastar Tao - plus a persistent store using BKNR datastore. The map is based on SVG, which uses morph ("fat" or perhaps with a slightly above average BMI) for updates.
The work around this led me to places I didn't necessarily expect, but a side-effect is that I made a Brotli library for Common Lisp (https://lambda-combine.net/cl-brotli), and the SSE library has support for something I added to Woo (perhaps poorly, but I'll follow that up upstream) to support user channels, and thus allow the event pool to be used for CQRS; in short, making it possible to use Clack+Woo with notifications without blocking the event loop.
This is all WIP in the sense that needs to be tested more. If it breaks, I'll fix it, or do my best at it.
Hope you find it interesting though. I'm out driving down to the Algarve for holidays in hours, so apologies if any possible questions are left unanswered more than they should, I wanted to get this out there since I was in a perpetual improvement loop.
An offshoot of my work in SSE and Datastar, and in anticipation of a new rmuch expanded release of the SDK. Brotli brings very good compression rates for the SSE use-case. Comes with a Hunchentoot example (but also works with Clack).
In short: I've replace the Common Lisp loop (that works for Hunchentoot since it opens threads, but doesn't for Woo since it blocks) with a deeper integration into the event loop:
> And that was the main change: looking at the innards of it, there are some features available, like woo.ev:evloop. This was not enough, and access to the libev timer was also needed. After some work with lev and CFFI, the SDK now implements a Node.js-style approach using libev timers via woo.ev:evloop and the lev CFFI bindings (check woo-async.lisp).
This is likely (almost surely) not perfect or even ideal, but it does seem to work, and I've been testing the demo app with 1 worker and multiple clients.
The CL-SPICE library I used, that wraps the SPICE C library through CFFI, doesn't cover the type of SPICE kernel that I wanted to use for the Comms module. I could try and add it, but it could be more involved than what I expected and put the thing on hold.
So I used the FORTRAN SDK for SPICE, since I had used it before, and it's reasonably small and easy. The alternative coud be using the C SDK, but I went with FORTRAN since I already had most of the code from a previous project.
I haven't tried Wookie, since adding Clack+Woo was already a substantial change. Reading https://fukamachi.hashnode.dev/woo-a-high-performance-common... , where it compares with Wookie, I'm not sure if it would make a difference: it might be wrong, but "it says:
> Of course, this architecture also has its drawbacks as it works in a single thread, which means only one process can be executed at a time. When a response is being sent to one client, it is not possible to read another client's request.
... which for SSE seems to be similar to what the issue is with Woo. I wrote a bit more on it in https://github.com/fsmunoz/datastar-cl/blob/main/SSE-WOO-LIM... , and it can be more of a "me" problem than anything else, but to keep a SSE stream open, it doesn't play well with async models. That's why I added a with-sse-response macro that, unlike with-sse-connection, sends events without keeping the connection open.
This is my attempted at something that makes using Common Lisp with Datastar easier. To test the SDK I made this demo that shows the simulation of the Cassini-Huygens mission using the NASA SPICE toolkit and JPL Horizons API: https://dataspice.interlaye.red/
The Datastar API itself is very simple, 3 functions or so, I ended up wasting a lot more time on stuff like leeping the SSE stream open, compression support (zstd only atm), and trying to use CLOS in a way that would fit both Hunchentoot and Clack (not always easy).
Depends on what you want, I touch upon that somewhat: to replicate this specific pattern, you can replace Squid with something that fills in the gap without any major changes - so, nginx or Caddy for example -- but you would have to make sure the feature set is adequate: I see Squid as being egress-first, where others are ingress-first (nginx being used a an ingress controller, recently discontinued but still...), so I do think that for this specific purpose it works quite well.
As for Envoy and others, I think this would fit in a different architecture that I sort of point to near the end, one that includes using a service mesh: Istio for example uses Envoy for Egress Gateway, Cillium also has an Egress Gateway, etc. This to me would be a separate pattern though.
Yes! And this can be partially a limitation that helps, in the sense that it forces you to add that. In this example, I had to spent some time with the Common Lisp dexador approach to make it work. I've added a "PROXY: " UI hint in the page at https://horizons.interlaye.red/ , you will see that it says "-- PROXY: http://squid.egress-proxy.svc.cluster.local:3128 --". This was actually something from my debugging that I decided to keep.
A next article will likely address this limitation though, and look into transparent proxying. This will involve nftables, sidecars, etc, and the more we go into this direction, the more installing a CNI that comes with this by default starts to make sense.
Depending on what want for "lock down", this or something like this could work: you are essentially defining a single outbound communication path. In a way, your scenario was one of the reasons behind this experiment.
I'll take a look a the overflow thing, although I'm not sure if I will be able to fix it: I do have an image at the start which is an alternative to the text-based drawing, so nothing is lost. I use my own blogging solution that is essentially Texinfo (https://interlaye.red/Texiblog.html) so these blocks are the result of using an @example block (which is then converted into a preformatted block). I'm not sure this can be improved, apart from (as you said) using alternative images.
You can certainly use the Squid ACLs to limit the egress for agents. One of the current shortcomings (I explicitly mentioned it near the end) is that there's no per-namespace granularity, so you wouldn't be able to determine it on a per-agent level -- but you would be able to generally establish that all agents would only have access to a global whitelist.
I guess you have just described what I was hinting at here:
>Linked with several of the above (mainly the centralised configuration) is that when using ACL rules to limit communication to external domains, these are cumulative: all namespaces will be able to communicate with all whitelisted domains, even if they only need to communicate with some of them.
> These limitations point toward why more sophisticated solutions exist, after all; a follow-up article will explore using Squid’s include directive to enable per-namespace configuration, and in doing so, show why you’d eventually want a controller or operator to manage the complexity.
... which is actually a good thing. More than making something "new", it's great to hear that the overall approach is sound.
I'm not bothered by restart times but that's mostly because that has never been a priority... but one thing I have half-done is a controller that gathers per-namespace configs, and with that reload times will become more of an issue.
Part of the reason I chose Squid here was precisely because I found it interesting to reuse something that was such a staple of web architecture patterns.
That's a more elegant approach. I usually just plow through obstacles, and the end result is not always ideal -- I like your approach better than the sidecar, I guess that I was using sidecars for other things and it sort of influenced my approach.
I'll try it your suggestions out and update the article, and thank you for your comment, already made sharing this worth it.