Funnily enough - I built this (delegation) over the weekend with Fable for a local voice chat running 100% on local LLMs, Parakeet and Kokoro. I say "...ask the thinking model..." and that redirects it to Qwen 3.6 27B on vLLM.
Can't claim originality though - it was inspired by Sesame - where their models will invoke a search, or check the weather etc, and make a vocalisation to keep you engaged.
Turn taking is one of the hardest things to get right for the exact reasons mentioned - but does seem to be the way that Claude.ai's voice works - in a very obvious way.
Anthropic + OpenAI both rug-pulled voices I liked and got used to and OpenAI really dumbed down their voices at the same time - Arbor went from Estuary English and almost "jack the lad" to some generic English accent. Claude had a Birmingham accent and said things like "shit", ending sentences like "So you're telling me that they asked for a 90% discount yeah?" - then it changed overnight to a mock Derbyshire accent with a dull tone.
ChatGPT's voice also gaslights me for conventional opinions - "my Eastern European neighbour helped me lift a wardrobe upstairs - something you just can't ask your typical neighbour neighbour"... then you get a full on left-leaning lecture from the safety layers rather than a head nod or "what luck!"
Claude + Sesame are nowhere near as overbearing.
In both cases - from edgy and engaging to something that just didn't gel.
The point of making my own assistant is that I can talk for as long as I want, episodic memory is personal and private, there's no "trust me bro, we're a big corporation" vibes.
This was not my first attempt - when I had a bunch of Opus credit around Jan/Feb - I tried really hard and created something that was not good enough. What I have now, is working, and each session is training Claude/Codex on what to tune, and to fix.
"Just had a convo - can you look into what happened?" And if it's one I don't mind sharing with the model - I'll say, "and what did you think of the questions I asked?" Sometimes it'll give a lovely commentary on how the model did.
af_heart is probably the smoothest voice - but yes it's more like another commented - more "StarTrek" than "telesales assistant that pauses and laughs at your jokes".
If you're on a similar path and want something full duplex - the go to solution is PersonaPlex from Nvidia based upon Moshi.
For self-hosting, have a look at what we're building with SlicerVM.com (disclosure: I'm the founder). Also runs just as well on Apple Silicon.
We run quite a few Slicer instances on mini PCs and Ryzen builds - also on Hetzner (and yes ouch 120 EUR / mo up to ~ 550 EUR / mo for 16core / 128GB RAM feels almost unfair)
This is clever work, especially given that Proxmox is already a very viable VMware replacement and wasn’t originally designed around microVMs as the primary abstraction. I’m glad this is working well for you.
We’ve been on a similar journey, but came at it from the opposite direction. We started SlicerVM in 2022 after seeing how slow Multipass felt when launching more than one Linux VM, even though it is relatively lean. Tearing them down was slower.. we made it seconds either way for a 30 node cluster and kept it internal until August last year.
With Slicer, microVMs are the native primitive: API launch, guest-agent exec/shell/cp/forward workflows, isolated networking, and agent sandboxes are built into the control plane.
That was not our first use case. Back then we were standing up Kubernetes clusters quickly for OpenFaaS e2e testing and customer scale-out support across multiple machines. The agent/sandbox workflows came naturally after that.
We do see people come over from Proxmox when they want something more directly driven from code, especially with a deeper guest-agent model: exec, file copy, port forwarding, fs watches, etc. When you string it all together it becomes very powerful and what we've gradually dogfooded for our code review bot that started out by using SSH/SFTP to completely native SDK (Go/TS).
One thing I’d separate in the benchmarks is in-guest boot time vs. actual time-to-interactive/useful. For agent-style workloads, the number that tends to matter is: API request made -> VM created/cloned -> network policy applied -> guest agent reachable -> exec/shell/cp/forward works. Snapshot cloning, network device setup, and control-plane readiness all show up there.
TTI can also be moved around depending on tradeoffs: no real init system, snapshot resume, CrosVM-style lower-level primitives, or a VMM built for one narrow job. We use systemd in the guest, so we’re intentionally carrying some weight there.
I also liked that you retained module support for Docker. Supporting Docker, Kubernetes-ish workloads, and eBPF tends to add a lot of useful weight back in.
There’s room for several tools here. The space is moving quickly, and I’m looking forward to seeing which approaches consolidate.
If folks are looking to scratch that microVM, or programmable / bash / agent / SDK driven primitive, you're welcome to check us out and join the Discord.
Thanks for the comment ZDR is mentioned in the post - in particular many the coding plans that are not from the two major leaders have questionable IP/ownership claims on inputs/outputs :)
And ZDR is still data sharing with a third party. This is the essence of an enterprise agreement, it's not allowed, even if they pinkie promise not to store it.
If your customers allow you to share their data with third parties, then ZDR may be an option for you. I am not a laywer.
Where I see ZDR as being more relevant is in protecting your employer's IP - not allowing a missed setting to mean AI labs can train, retain, and publish/resell your work. It's what we'll consider when the subsidies stop being available - open-router, ZDR - but for coding - not for customer data. Very important distinction.
The cache only makes generation fast, it doesn't influence what gets chosen next. The loops that hurt the most (point 2 below) are when the model re-decides to do the same thing in different words, which is much harder to detect automatically. We're experimenting with repetition penalty and turning thinking off to solve for the 1st kind of looping (below)
2. On "why is looping a problem" for us
Practical example, which I covered in the post: "add --json to every command that does a get or list in faas-cli" - this was a small-ish, open source CLI written with Cobra a very common framework.
If I send that to Claude (any of their models) or Codex (GPT), I would have a fully working solution the next time I opened that terminal - a few seconds - a few minutes.
With the local model, when it loops, you get some progress and start working on something else. Come back, maybe even 30 minutes later and see it's been printing the same 5 lines over and over constantly.
Trust is important for a tool like this, that eroded it.
The other type of loop I mention in the blog post is "unable to solve it" loop - Han ran into that more.
"Oh I need to fix the indent from 8 to 5 characters in main.py"
"Wait I don't know how to write Python code"
"Oh now it's broken and I don't know what to do, maybe I should stop"
"Let me edit ... "
etc, etc
vLLM is great at continuous batching and model serving in production, but it's a very different beast and much less versatile for the prosumer category (where we sit for our usage)
Dismissed is a strong term, but let me give you some more details.
It took a good 4 minutes plus to load up on the 2x 3090 rig, and served a single request 3 tokens/second slower.
And the worst bit? With all that work - setting it up and tuning it - it still looped. I was hoping "use just vLLM" advice that we get touted everywhere was the silver bullet.
The only thing I'd caution here is that we don't start bashing on llama.cpp like people did with Ollama. It's a very capable tool and for the use-cases we actually want the card for makes more sense.
For a large team replacing their Claude Subs perhaps vLLM is the only option, but you really need to add about 5 more RTX 6000 cards into the mix, so you can load something like GLM 5.2.
We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning.
It's the right call for concurrent batched serving (barrkel's point downthread is spot on), but for how we use it llama.cpp is still better for us.
The Spark/GX10 route is a genuinely different bet though and appreciate you sharing your numbers. At the time (several months ago) the consensus was that GX10s were for fine-tuning only, and the numbers were severely low.
..and the card was never about replacing a Claude Max sub. For the workloads we actually bought it for, it's giving us 140-200 tok/s (which matters).
Ha, you underestimate how dogged you need to be to get this stuff working well.
The RTX 3090 in question was used from eBay, no way to return it. The RTX 6000 Pro is the "new card" in question here. The 3090s remain an interesting playground for testing things like VFIO passthrough for SlicerVM and other models whilst not interrupting people on the newer card.
In the end, the most stable fix I've found is to install the older proprietary driver and disable the GSP firmware. Have had no issues since.
So "clearly defective hardware" seems like it may not be quite correct. And the thing that kept me coming back - along with not having a suitable replacement - or having to gamble on eBay again was the reliability once it showed up in nvidia-smi.
The important thing about MoEs which I mention in the conclusion is that they carry fewer (way fewer) active tokens during inference/generation.
35B-A3B is what we started out with in the days of only having the 3090, but the quality is not as good, and the speed from the cards we have now can blaze at 130-200 tokens per second of generation with q5 and a full context in fp16.
Not to say that MoEs don't have their place. For people running on unified RAM, they're sometimes the only viable option due to the slowness of dense models.
Why is a dense model slower? All model weights have to be loaded and exercised. Passing through 27B vs 3B (active) is maths. So yes you will always get more tokens per second of generation.
You must (just as we did) evaluate on your own products and daily work. If the MoE gives the results you need with only 3B parameters then you have your answer.
Not prescriptive at all. This is experience based, from the trenches of a actual software business so hopefully a different perspective for folks than "Ran Qwen on my macbook, generated a great python script for me"
> Local models can quickly read and explain codebases, even if they can't write them - this is a superpower
Might have been buried lower down.
And yes latency of local on a fast card with MTP enabled can be blistering 130-200 tokens per second sustained at full context on Q5. About 100+ on Q8.
On tool calling
> Agent Skills can help immensely - we had a local agent set up Slicer completely from scratch on a new mini PC. It even gave feedback on the usability of slicer CLI which we integrated
There's a link to a post showing some examples.
Occasionally, we'll also have the local model _review_ the changes of GPT/Opus - and it can return duds, but also insights the larger model overlooked, or was too intelligent to pick out.
So yes - absolutely blazing fast at understanding a codebase, very good at running skills "cheaply" and could be used with larger models as a "helper" / sub-agent.
Author here. Thanks for the question. I'll answer assuming this is a question you have for me.
As explained in the post - the 3090s were what were the test bed that proved the investment was worth it. Customer support, architecture reviews, telemetry to check license compliance. None of that could be done with online models. The amount of time we can spend going backwards and forth with enterprise customers over email can really amplify costs to our team. A few actual issues we found and fixed were listed on the linked blog post: https://www.openfaas.com/blog/painless-support-with-diag/
Having recovered revenue using it in an airgap, to preserve data agreements was more of a cherry on the cake. No need to worry about the investment, it's covered itself.
Hi - the author of the post here. I wanted to write up something that was a bit more than "Qwen is the goat" or "Cancelled Claude, run everything local now" or even "The model organised my CD collection, so it's great" - this is real from the trenches stuff. And what the model/harness did achieve may surprise you - along with how it ended up paying for itself.
I was thinking about the RPi 6 yesterday whilst realising I couldn't set up my RPi Zero 2W anymore - the OS has become burdensome - tied strictly to an imager, that gives me an allergic reaction. Yes - they did all this for the uninitiated - but for Raspberry Pi OS Lite - bring back this experience: dd the image, write ssh into the boot drive, SSH in - change password, fully set up in almost zero fuss or effort.
Then I actually couldn't set the thing up because of the mini HDMI connection - I have a mini to HDMI cable, but to use my portable screen with it I need mini HDMI to MINI HDMI. Don't get me started on micro HDMI - almost everyone of of those connectors I've bought slips off or breaks in the device. Every time I go to set up an RPi5 I end up having to order another one of those tiny connectors.
Full HDMI for all new devices please. Even if the second display can't be connected.
These days a 175 GBP N95 from a no-name Chinese OEM on Amazon, with 16 GB of RAM and a 500GB SATA SSD is way better value and performance - and importantly - zero fuss - standard setup.
If you look at https://slicervm.com you'll see he's copied our terminal animation from the top of the website. Took out a monthly subscription for 1x month, cloned the majority of the UX/DX and way the guest agent works.
Had people reach out and flag it to me and I'm like "yes there's a reason for that"..
I think this is just par for the course in an AI slop world. Nothing to stop people imitating, copying, cloning with a good prompt and partial source / detailed docs available.
It's interesting to see this one launch (yes yet another sandbox.. I was getting worried we'd not seen one for a few days)
SlicerVM (est. 2022) is already used for prime time, not "free as in beer" but has pretty reasonable individual plans that include all features. Shares the core code with actuated. (Creator of both speaking here)
Feel free to take a look and see if gives you a little more than the others you mentioned. If not no problems, I realise some folks prefer free stuff.