Who is their right minds would be wedded to an identity of saying "No"? Code quality puritans are annoying but if they do their job right they actually speed-up the development process because they don't let technical debt accumulate. Ultimately saying "No" is protecting your codebase. In the era of LLMs, saying "No" is much easier because you don't have to worry about the author feeling bad.
Fair enough. I agree with you - although DS4 Pro is a GPT 5 class model which scores 46% on ARC-AGI-2[^1]. It's behind by maybe 9 months, I think it's still good enough for a lot of complex tasks as well. They definitely need to work on a "just fucking works" harness like CC/Codex. Also thanks!
People doing economics with the cloud GPUs, of course cloud GPUs are going cheaper. But also, is generating tokens all you do with your computer? I can play games on DGX spark and also do LLM inference, so sometimes the economics work out, apart from having fun with it.
Thank you, there are two things I would like to point out:
1) Google releasing something probably means they don't see it as important. 4-bit KV-cache quantization has been known for a long time. The fact there is almost a mass hysteria about this paper makes me think there is a lack of skepticism in this AI mania, even in relatively tech-savvy crowd.
2) But prices for memory companies are crashing! look around, the whole market is crashing.
There are techniques which already achieve great compression of the cache at 4 bit, eg using hadamard transforms. Going from
4 bit to 3 bit isn’t the great leap people expect this to be. It’s actually slower to run and is generally worse in practice.
Working in open source, I've now heard a wide variety of disabilities that people have and they have to be aided by an LLM for writing even descriptions of their PRs.
You can still run larger MoE models using expert weight off-loading to the CPU for token generation. They are by and large useable, I get ~50 toks/second on a kimi linear 48B (3B active) model on a potato PC + a 3090