During my first LLM experiments in Emacs using gptel, I also found that the LLM has considerable difficulties changing source code files with the Unix patch tool.
As Emacs has a built-in tree-sitter package, I implemented this same idea. I created gptel tools like tree_sitter_list_nodes, tree_sitter_get_nodes, tree_sitter_update_nodes, tree_sitter_insert_before_node and tree_sitter_insert_after_node. The "list" tool returns a list of AST nodes with first line number, first line content and node hash. The LLM can then use "get" to collect interesting nodes in their entirety and "update" to update a list of nodes identified by hash with new content (var/function bodies).
This is an AST-walking interpreter for my personal LISP dialect written in C. Once it's ready, I would use it to implement a low-level, statically typed language (Schnell) as a Langsam library. The goal is to gain the ability to JIT-compile Schnell code (sexps of a statically typed language) from Langsam. Once this works, I would rewrite Langsam in Schnell so that it becomes a fast bytecode interpreter. With the faster Langsam (and the Schnell built into it) I could build a little OS called "Oben". The OS would first run on top of Linux, then I would attempt to bootstrap the entire stack on bare-metal. I already have a Forth dialect implemented in assembly language (Grund/Boden). The idea is to implement Langsam in Grund and then bootstrap the entire Grund -> Langsam -> Schnell -> Oben chain on something like the qemu q35, later on a Raspberry Pi Zero 2W and maybe even my own hardware (ie. an FPGA board like what Wirth et al. created for Project Oberon).
This is a TUI MIDI tracker written in Go. Not too user-friendly: one has to enter raw MIDI messages in hex into the tracks. Can be connected to synths like Fluidsynth or Surge XT via JACK MIDI. Unfortunately it takes a lot of CPU time, probably due to the use of BubbleTea (and no time spent on optimization).
Beginnings of a programmable, non-realtime audio sample generator/manipulator written in Go with an OpenGL GUI. I was thinking about how people in the old times cut up the magnetic tape which contained the sound bites and rearranged them to build something new. What if I'd implement a data type called "tape" which is basically a piece of sound and then provide operators in a Forth-like language to create and manipulate such tapes. Each tape could be a sound and then these could be stitched together to form songs. Who knows maybe an entire song could be represented as a hierarchy of these tapes. Each sound or song section could be its own file (*.tape), these could be loaded from each other, maybe even caching the WAV generated from the code of a tape to speed things up when there is a huge hierarchy of tapes in a project. Lots of interesting ideas are brewing in this one.
I am fascinated by the idea of building something like the Lisp Machines or Smalltalk 80 from scratch. Build a Forth in assembly, build a Lisp in Forth, build an OS and computing environment in Lisp. AOT-compile only the Forth interpreter, load and compile the rest from source during system boot, maybe with later stages optimizing the previous stages as the system is assembling itself.
I imagine two languages - Langsam and Schnell - intertwined in some sort of yin-yang fashion. Langsam is slow, dynamic, interpreted, Schnell is fast, static, compiled. Both would be LISPs. Schnell would be implemented as a library in Langsam. If you said (define (add x y) (+ x y)) in Langsam, you would get a Langsam function. If you said (s:define (add (x int) (y int)) (+ x y)) in Langsam, you would get a Langsam function which is a wrapper over a JIT-compiled Schnell function. If you invoke it, the wrapper takes care of the FFI, execution happens at C speed. Most of the complexity typical of a low-level compiled language could be moved into Langsam. I could have sophisticated type systems and C++ template like code generation implemented in a comfortable high level language.
This latter part I managed to partially implement in Clojure and it works (via LLVM), it would be just too much effort to get it completed.
As we understood it, Kopf lets you build an entire operator in Python, with the watch/update/cache/expansion logic all implemented in Python. But the first operator we wrote in it just didn't feel right. We had to talk to the K8S API from Python to do all the expansions. It was too complex. We also had aesthetic issues with the Kopf API.
Metacontroller gave us a small, Go binary which takes care of all the complex parts (watch/update/cache). Having to write only the expansion part in Python felt like a great simplification - especially now that we have Pydantic.
Some of the resources are short-lived, including jobs and dev containers. The corresponding CRs are created/updated/deleted directly in the cluster by the project users through a REST API. For these, expansion of the CR into child resources must happen dynamically.
Other CRs are realized through imperative commands executed against a REST API. Prime example is KeycloakRealm and KeycloakClient which translate into API calls to Keycloak, or FSXFileSystem which needs Boto3 to talk to AWS (at least for now, until FSXFileSystem is also implemented in ACK).
For long-lived resources up-front (compile time?) expansion would be possible, we just don't know where to put the expansion code. Currently long-lived resource CRs are stored in Git, deployment is handled with Flux. When projects want an extra resource, we just commit it to Git under their project-resources folder. I guess we could somehow add an extra step here - running a script? - which would do the expansion and store the children in Git before merging desired state into the nonprod/prod branches, I'm just not clear on how to do this in a way that feels nice.
Currently the entire stack can be run on a developer's laptop, thanks to the magic of Tilt. In local dev it comes really handy that you can just change a CRs and the children are synced immediately.
Drawbacks we identified so far:
If we change the expansion logic, child resources of existing parents are (eventually) regenerated using the new logic. This can be a bad thing - for example jobs (which expand into Argo Workflows) should not change while they are running. Currently the only idea we have to mitigate this problem is storing the initial expansion into a ConfigMap and returning the original expansion from this "expansion cache" if it exists at later syncs.
Sometimes the Metacontroller plugin cannot be a pure function and executing the side effects introduces latency into the sync. This didn't cause any problems so far but maybe will as it goes against the Metacontroller design expressed in the docs.
Python is a memory hog, our biggest controllers can take ~200M.
At work we are using Metacontroller to implement our "operators". Quoted because these are not real operators but rather Metacontroller plugins, written in Python. All the watch and update logic - plus the resource caching - is outsourced to Metacontroller (which is written in Go). We define - via its CompositeController or DecoratorController CRDs - what kind of resources it should watch and which web service it should call into when it detects a change. The web service speaks plain HTTP (or HTTPS if you want).
In case of a CompositeController, the web service gets the created/updated/deleted parent resource and any already existing child resources (initially none). The web service then analyzes the parent and existing children, then responds with the list of child resources whose existence and state Metacontroller should ensure in the cluster. If something is left out from the response compared to a previous response, it is deleted.
Things we implemented using this pattern:
- Project: declarative description of a company project, child resources include a namespace, service account, IAM role, SMB/S3/FSX PVs and PVCs generated for project volumes (defined under spec.volumes in the Project CR), ingresses for a set of standard apps
- Job: high-level description of a DAG of containers, the web service works as a compiler which translates this high-level description into an Argo Workflow (this will be the child)
- Container: defines a dev container, expands into a pod running an sshd and a Contour HTTPProxy (TCP proxy) which forwards TLS-wrapped SSH traffic to the sshd service
- KeycloakClient: here the web service is not pure - it talks to the Keycloak Admin REST API and creates/updates a client in Keycloak whose parameters are given by the CRD spec
So far this works pretty well and makes writing controllers a breeze - at least compared to the standard kubebuilder approach.
Implementation: https://github.com/cellux/dotfiles/blob/master/.emacs.d/rb-t...