I built some BigQuery workflows on 2.0 and 2.5 flash lite that are something like 6x more expensive with 3.1 flash lite.
I tried 3 flash for months and it didn’t work using Googles own vertexai integration because it’s been in preview mode for months.
Not wanting to pay significantly more and do a bunch of rework isn’t a smell.
They left a large gap in their new pricing vs the prior generation, and if you had a working use case that sucks. The model is >99% reliable for my use case so there’s nothing to gain from a smarter model.
Lemonade proxies your request to llama.cpp that it installs and manages. Is out of the box all local LLMs, but you can connect it to an API endpoint (supports OpenAI compatible or Anthropic compatible).
I think it stores basic logs but if you wanted to monitor you’re probably going to want to proxy it with an LLM gateway.
I used llama swap for a while before leaning into lemonade. The UI has improved a lot, but be careful as the most of the models default to very small 4K context windows by default.
They’re doing some nice things with their Halo models, which load an ensemble of different types of model at the same time. With high vram it’s easy to keep them all in memory, so even though the compute is limited the context switching is fast.
You do lag a bit on the upstream engine releases, the llama.cpp/sd.cpp/whisper libraries are downloaded from inside the app.
vLLM is in experimental mode, I haven’t tested it. It’s limited in the models they suggest, but you can download anything from huggingface with a two click install.
Tmux with vim as a 50% split, I’ll run an agent in a quarter split.
I have neovim hooked up to a local llama.cpp (through lemonade) for completion, but I have not found a plugin that I’m happy with. Avante is probably the closest because you can give inline instructions, but it both got very very heavily and stopped working with small LLMs so I uninstalled it. I’m using minuet currently.
I built a small one of these as a calculator back in the day, it was fun and I always wanted to see someone really run with the idea more, so your project is super cool!
Did you ever look at symbolic or exact operators instead of purely monte-Carlo?
I remember reading the paper for distr, they used a mix of Fourier transforms for convolution and symbolic reduction to build a probabilistic computing library in R. I attempted building a small python library for this, but for my problems the CLT ended up sufficient to approximate the results faster, so I went with that.
You may enjoy reading the paper, it’s not groundbreaking but is a nice presentation of relationships/operations. Maybe it’ll inspire some features for you.
AMD ROCm has come a long long long ways since last year, but you'll probably be happier not dealing with AMD's software.
I posted another comment with my experience with AMD 395+, I am overall happy and it's usable now, but it's only useful for models under 64gb of vram due to the active parameter counts on larger MoEs.
If you add 2x 5090s, do you actually need the base system?
Today I am pretty happy with it. LLMs are finally good enough (fast enough with MTP+MoE, but also just much better in capability) that I can fit local ones into real tasks, and I've used image generation with invokeAI to do some genuinely useful things like rendering concepts for a renovation.
I mostly use lemonade-server and invokeAI for my workloads, previously I used llama-swap, but lemonade is just an easier to manage system. ROCm is finally usable.
Up until end of Q1 2026 it felt like a total waste of money largely due to AMD. ROCm was unusable all of last yera; there was an entire month where PyTorch crashed just trying to multiply two matrices due to AMD Linux driver issues. kyuz0's toolboxes were the only way to do anything really on the machine.
Thankfully things are in a good state now, finally.
I probably actually only need ~64gb of ram. There aren't a ton of high parameter count MoE with a small enough active set that it feels nice to use. But it is nice I can have many models or different modalities in memory at the same time, which is what the LMX Omni "models" do.
The numbers in the article for gptOSS feel a little irrelevant now. Prompt processing is definitely an issue, and diffusion is very very slow. PP speed hits hard you if you run an agent and try to compact context. Realistically most files are not large enough that it's a huge deal, but it does make large-scale agentic work slow.
When I started self hosting last year it seemed insane that hosting a service required separately managing the imperative firewall state, vpn, container runner, systemd, and reverse proxy.
Nixos is nice because it just works. When I do a rebuild switch it will take down my affected services gracefully, apply the updates, update my firewall, and start the container all automatically and declaratively.
Plus the owning account and file permissions are explicit so it’s impossible to forget you ran chmod/crown on some script and now your service is over-permissioned.
It’s not magic but it feels like magic because it works so easily and reliably.
I don’t know how anyone can look at the innovation going on at DeepSeek and come to the conclusion that China can only copy.
Distillation and copying are how they’ve bootstrapped their models, but that feels not so different than Anthropic and Meta torrenting millions of pirated books.
The Chinese labs are solving problems for a different set of constraints.
The calculus is changing for non US, non Chinese users.
Hypothetically if the US continues to restrict their frontier models and adds a ban on Chinese/open models then it would to obliterate services like open router. American cloud companies would presumably be blocked from selling capacity to run banned models in this situation.
That causes a shortage of compute/gpu resources internationally and an oversupply of non-revenue generating hardware in the US.
If that happens then what percent of your salary is worth securing this compute worth? How much does the cost of a data centre chip change? It’s difficult to say.
The US is really shooting itself in the foot here.
The restrictions on LLM models like Fable has created a viable international LLM market where it was difficult to justify investment two weeks ago.
As a non-US citizen Opus 4.8 is the best American LLM I will ever have access to. That's no longer up for debate or question. Each month that I pay Anthropic is now a depreciating value -- I'm paying for models I'll never be able to access, while other models are able to catch up.
Adding US based identity verification through Persona is also incredibly off-putting. I think it's sufficient to kill my use of Claude altogether.
So the question I have to live with is what do I do instead.
I installed Mistral Vibe last week and I've been experimenting with offloading work to it. I won't pretend that Mistral-medium is close to state of the art. It isn't. It still writes incorrect tool calls.
From the last week about 50% of my LLM tasks actually reduced to "take this work and write about it" and Mistral excels there -- it definitely beats Opus at writing. Mistral nails it, and when it doesn't its so fast to iterate.
There's another say 30% of tasks that's writing queries against a data warehouse. I updated my semantic layer MCPs and Vibe uses them, but it struggles with ambiguity here. It's not a replacement, it's maybe where Opus was a year ago.
The rest of my work involves writing code. That's going to be harder to replace for now. My next step is exploring OpenRouter and other models. I can't decide if I was ever actually happy with Opus's work on this front though -- the understanding tradeoffs when you trust LLMs with decisions stack non-linearly and negatively. I did like Fable on these tasks, I won't lie, I will miss it, but not by any choice of my own.
That’s on Wikipedia, it’s not PII, it’s also not going to be relevant to any meaningful IRL work.
I challenge the assumption you can do meaningful work in this field without blatant disregard for intellectual property.
The idea that it’s all down to training size is clearly incorrect, as every expert human learned their craft without nearly the sum total information of the internet. Clearly there are architectural wins to be found.
Besides that, why would everyone just be fine with Opus level AI at best, as that’s all the US is willing to export, and I doubt China will share beyond that.
Sovereign AI is more important than ever after Friday.
I find devstral (even though it’s weak generally) much better at writing and documentation than Opus. I’m actually now delegating all documentation to devstral and away from Claude, which makes a mess.
> For years I could ask Google what song was playing and it would identify it. It was one of the most useful features it had. You'd hear a song in a store, on television or drifting over from a neighbour's backyard and Google would identify it in seconds.
A huge pet peeve of mine is when I’m in the car and want to know what song is playing on the radio. I run Shazam and my phone mutes the stereo to activate a microphone. I have to disconnect from CarPlay then run Shazam, then reconnect — it’s a passenger only operation.
Song recognition is built into both iOS and Android, the device should always use the internal mic instead of a CarPlay/Android Auto microphone over Bluetooth.
Side note: is there a good “dumb smart speaker” I can have run with a wake word connected to my own API? Speech to Text and Speech to Speech are fairly well supported for local AI workflows now, it would be great to have my own Home device without worrying about where the audio goes.
I’m sure it’s a very niche audience today, but I imagine giving this thing MCP for Wikipedia, a music app, and my recipes would be perfect.
I tried 3 flash for months and it didn’t work using Googles own vertexai integration because it’s been in preview mode for months.
Not wanting to pay significantly more and do a bunch of rework isn’t a smell.
They left a large gap in their new pricing vs the prior generation, and if you had a working use case that sucks. The model is >99% reliable for my use case so there’s nothing to gain from a smarter model.