llama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed)
Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?
> So... maybe we can still use third party harnesses with Claude Code subscriptions... for now?
The way I read this is: yes, if the third party harness uses Anthropic's Agent SDK. Most of them do not, AFAIK, and are still against ToS (though maybe its not enforced for now)
> Anthropic just provides a subscription - which Enterprise usually doesn't want you to use because everything you're submitting through that will be trained on / becomes part of their model.
My Pro account very clearly has a toggle for "Help improve our AI models:
Allow the use of your chats and coding sessions to train and improve Anthropic AI models."
I did my first ESP32 project recently and was amazed you can get a system that starts up Micropython, then a Wifi AP, DNS, and Web Server in a second or two total and uses less than 512kB RAM. And thats with a high level programming language.
You could subscribe to Anthropic/OpenAI for the rest of your life for the cost it would take to host GLM5.2 locally - you need 1.5TB of VRAM just for the weights
I get this, though the pace of Chinese releases is relentless. Qwen3.7 Plus/Max (closed variants) feel notably better than Qwen3.6, and Minimax M3 is a big jump from 2.7 in capability as well. Both of these families had their previous major release less than 90 days ago.
Anthropic must have Sonnet 5 either waiting or cooking though, they said smaller and larger models than Opus were coming and we already briefly had the larger model.
> I'm curious what the downside for this speed is here
"DiffusionGemma's speedup is designed for local and low-concurrency inference. In high-QPS cloud serving, autoregressive models can be deployed to saturate compute efficiently, so DiffusionGemma's parallel decoding offers diminishing returns and can result in higher serving costs"
According the the X post, each 150kW satellite is a bit over 2 tons, so a full year of launches would get you 75MW. Thats not much when AI datacenters are GW-class.