I've been thinking if this could be done in Nix and toyed around with the idea just a little bit using nano-gpt.
Hermeticity always seems to mean isolation, but depending on who you ask it does not always mean computing some sort of hash over all build input as the 'identity' of a particular step in the pipeline, like Nix does.
If you do that hash-based identity part, looking up intermediary results and resuming from there happens using this sort of hash.
Does savanah do that, or will it resume where it left off based on a less strict notion of identity?
A consequence of universal healthcare that people don't talk about much is that it turns unhealthy citizens from an individual cost into more of a collective one. So it makes sense that countries with universal healthcare regulate in favor of their citizens as opposed to their food industry, because they're paying for the consequences more directly.
I would blame how Austria, a very small country, is organized into 9 provinces that actually have their own budget and can pass their own laws on some topics.
Rail service is funded at the federal level, so there's less arguing about who pays for what. Bus service, however, is managed by regional transport associations funded by the provinces. This creates disincentives for cross-province bus routes because no single province wants to pay more than its 'fair' share for a service that primarily benefits voters in another province.
Similar dynamics play out at the city/province level. Take Linz, the provincial capital of Upper Austria: the city has had a social democratic (SPÖ) mayor continuously since 1945, while the province has had a conservative (ÖVP) governor for exactly the same period of 80 years. This disincentivizes the province government from helping to fund public transport within or into the city, because it's a win for social democratic city voters, while the more conservative rural voters would rather take the car anyway since they often can't do the whole trip by public transport.
Arguably the reason for the excellent public transport in the city of Vienna is that they are also their own province. Their mayor/governor, who has been a social democrat as well for the last 80 years, always controls both levels of funding.
If you make a conventional AI agent do packaging and configuration tasks, it has to do one imperative step after the other. While it can forget, it can't really undo the effects of what it already did.
If you purpose-build these tools to work with Nix, in the big picture view how these functional units of composition can affect each other is much more constrained.
At the same time within one unit of composition, you can iterate over a whole imperative multi-step process in one go, because you're always rerunning the whole step in a fresh sandbox.
LLMs and Nix work together really well in that way.
Yes, the cycle times are bad and some ecosystems and tasks are a real pain still.
I also agree with you when it comes to the task of auditing every line of Nix code that factors into a given system. Nix doesn't really make things easier there.
The benefit I'm seeing really comes from composition making it easier to share and direct auditing effort.
All of the tricky code that's hard to audit should be relied on and audited by lots of people, while as a result the actual recipe to put together some specific package or service should be easier to audit.
Additionally, I think looking at diffs that represent changes to the system vs reasoning about the effects of changes made through imperative commands that can affect arbitrary parts of the system has similar efficiency gains.
I think as AI gets smarter, defenders should start assembling systems how NixOS does it.
Defenders should not have to engage in an costly and error-prone search of truth about what's actually deployed.
Systems should be composed from building blocks, the security of which can be audited largely independently, verifiably linking all of the source code, patches etc to some form of hardware attestation of the running system.
I think having an accurate, auditable and updatable description of systems in the field like that would be a significant and necessary improvement for defenders.
I'm working on automating software packaging with Nix as one missing piece of the puzzle to make that approach more accessible:
https://github.com/mschwaig/vibenix
(I'm also looking for ways to get paid for working on that puzzle.)
I mentioned another alternative to adding flake-specific metadata to data structures that are transferred over the network, as part of the signed traces or otherwise, in a comment on that PR Eelco linked.
It's keeping flake-specific data locally, to guarantee that it matches how the user ended up with the data, not how the builder produced it. I think otherwise from the user POV such data could again look misleading.
Hermeticity always seems to mean isolation, but depending on who you ask it does not always mean computing some sort of hash over all build input as the 'identity' of a particular step in the pipeline, like Nix does.
If you do that hash-based identity part, looking up intermediary results and resuming from there happens using this sort of hash.
Does savanah do that, or will it resume where it left off based on a less strict notion of identity?
I could see arguments for either approach.