It feels like a weird tension: we worry about AI alignment but also want everyone to have unrestricted local AI hardware. Local compute means no guardrails, fine-tune for whatever you want.
Maybe the market pricing people out is accidentally doing what regulation couldn't? Concentrating AI where there's at least some oversight and accountability. Not sure if that's good or bad to be honest.
I actually find that nixpkgs being a monorepo makes it even better. The code is surprisingly easy to navigate and learn if you've worked in large codebases before. The scaling issues are good problems to have, and git has gotten significantly better at handling large repos than it was a decade ago, when Facebook opted for Mercurial because git couldn't scale to their needs. If anything, it's GitHub issues and PRs that are probably showing its cracks.
Not yet! But I will make sure to link here once it's up in a few days (or post to HN? not sure what the etiquette around self-promotion is these days). It's somewhat functional but not usable by anyone other than me at this point most likely (:
Out of curiosity, what would be an ideal UX for you? I'm working on a Rust library for this exact problem (CLI and language bindings should be easy to add).
It uses KVM directly on Linux and Virtualization.framework on macOS, with a builder API for VM configuration. For AI sandboxing specifically, it has a higher-level "sandbox" mode with a guest agent for structured command execution and file I/O over vsock. You get proper exit codes and stdout/stderr without console scraping.
Also supports pre-warmed VM pools for fast startup and shared directories via virtio-fs.
I'm planning to support OCI images, but not sure if that's important to people. I typically just build my own root disks with Nix.
RAM requirements stay the same. You need all 358B parameters loaded in memory, as which experts activate depends on each token dynamically. The benefit is compute: only ~32B params participate per forward pass, so you get much faster tok/s than a dense 358B would give you.
yes, completely agree. having some sort of guardrails for the LLM is extremely important.
in the earlier models I would sometimes write tests for checking that my coding patterns were being followed correctly. basic things like certain files/subclasses being in the correct directories, making sure certain dunder methods weren't being implemented in certain classes where I noticed models had a tendency to add them, etc.
these were all things that I'd notice the models would often get wrong and would typically be more of a lint warning in a more polished codebase. while a bit annoying to setup, it would vastly improve the speed and success rate at which the models would be able to solve tasks for me.
nowadays many of those don't seem to be as necessary. it's impressive to see how the models are evolving.
I wonder if LLMs can help here to some extent. I agree with others that cooldowns aren't helpful if everyone is doing it.
I've been working on automatic updates for some of my [very overengineered] homelab infra and one thing that I've found particularly helpful is to generate PRs with reasonable summaries of the updates with an LLM. it basically works by having a script that spews out diffs of any locks that were updated in my repository, while also computing things like `nix store diff-closures` for the before/after derivations. once I have those diffs, I feed them into claude code in my CI job, which generates a pull request with a nicely formatted output.
one thing I've been thinking is to lookup all of those dependencies that were upgraded and have the LLM review the commits. often claude already seems to lookup some of the commits itself and be able to give a high level summary of the changes, but only for small dependencies where the commit hash and repository were in the lock file.
it would likely not help at all with the xz utils backdoor, as IIRC the backdoor wasn't even in the git repo, but on the release tarballs. but I wonder if anyone is exploring this yet?
this looks really polished, congrats! in your opinion, how does it compare with alternatives like actualbudget [0]? I've been using Quicken for a long time and might be in the market for a subscription-less alternative that is ideally self-hosted. Quicken has been running into lots of issues syncing some of my accounts lately (mostly duping assets).
I've been working on something similar, a typed shell scripting language called shady (hehe). haven't shared it because like 99% of the code was written by claude and I'm definitely not a programming language expert. it's a toy really.
but I learned a ton building this thing. it has an LSP server now with autocompletion and go to definition, a type checker, a very much broken auto formatter (this was surprisingly harder to get done than the LSP), the whole deal. all the stuff previously would take months or a whole team to build. there's tons of bugs and it's not something I'd use for anything, nu shell is obviously way better.
the language itself is pretty straightforward. you write functions that manipulate processes and strings, and any public function automatically becomes a CLI command. so like if you write "public deploy $env: str $version: str = ..." you get a ./script.shady deploy command with proper --help and everything. it does so by converting the function signatures into clap commands.
while building it I had lots of process pipelines deadlocking, type errors pointing at the wrong spans, that kind of thing. it seems like LLMs really struggle understanding race conditions and the concept of time, but they seem to be getting better. fixed a 3-process pipeline hanging bug last week that required actually understanding how the pipe handles worked. but as others pointed out, I have also been impressed at how frequently sonnet 4.5 writes working code if given a bit of guidance.
one thing that blew my mind: I started with pest for parsing but when I got to the LSP I realized incremental parsing would be essential. because I was diligent about test coverage, sonnet 4.5 perfectly converted the entire parser to tree-sitter for me. all tests passed. that was wild. earlier versions of the model like 3.5 or 3.7 struggled with Rust quite a bit from my experience.
claude wrote most of the code but I made the design decisions and had to understand enough to fix bugs and add features. learned about tree-sitter, LSP protocol, stuff I wouldn't have touched otherwise.
still feels kinda lame to say "I built this with AI" but also... I did build it? and it works? not sure where to draw the line between "AI did it" and "AI helped me do it"
anyway just wanted to chime in from someone else doing this kind of experiment :)
I think specialized hardware will emerge for specific proven workloads (transformer inference, for example), but GPUs won't become obsolete. They'll remain the experimentation platform for new architectures. You need flexibility to discover what's worth building custom silicon for.
Think 3D printers versus injection molds: you prototype with flexibility, then mass-produce with purpose-built tooling. We've seen this pattern before too. CPUs didn't vanish when GPUs arrived for graphics. The canal analogy assumes wholesale replacement. Reality is likely more boring: specialization emerges and flexibility survives.
Keyboards were replaced with a touch screen alternative that effectively does the same job though. What is the alternative to a camera? Cameras are way too useful on a mobile device for anyone to even consider dropping them IMO.
This is awesome. Thank you for sharing. I have been working on a small interpreted language for shell scripts with lots of help from Claude Code. The main idea is to automatically generate the cli interface for the scripts based on function definitions. However, I'm far from a programming languages expert, so I've been a bit hesitant to share my work. Going to give this book a read this week to see how far I am. Thank you!
Yeah, from discussing this with friends in academia it seems like this is the biggest hurdle at the moment. Someone on Reddit reached out to arXiv [0] and their response was very negative towards it, which seems unfortunate:
> At this point, we’re more likely to add support for .docx than a markdown language.