Forgot to mention, but it was after the 4.7 release when I was still using 4.6 that I saw those loops too... Before that, 4.6 had been a pretty seamless experience.
They would quantize the model. That'd make it cheaper to run, and have slightly worse output but it would still generate outputs with a similar feel, derived from a compressed version of the same knowledge base etc.
They wouldn't even need to do this uniformly, quantized versions of the model could be routed only a subset of the requests. They could do this to nerf the old model, or more likely just to give themselves more hardware to run the new one on by handling more requests on less hardware. Or to handle increased request volume as traffic ramps up faster than hardware can be provisioned.
Playing with local models at various quants, the degradation can be hard to spot. Sometimes it's only noticeable in aggregate. And even then, you never really know if you just got unlucky with a bad response due to RNG.
I've had Opus 4.6 fall into some weirdly incoherent loops that I rarely see from even Sonnet, that felt like the kind of thing I got frequently with Qwen3.5 9B on local. And the above applies... Was that just bad RNG? Or was my request to Opus routed to some lower quality variant? There's no great way for me to tell for any given request, nor any way to guarantee Anthropic _didn't_ do that.
This is a great post that covers a lot of the recent ground. I have a very similar setup after a very similar journey, minus the RTX6000. Worth noting though that a lot of the recent changes make a single 3090/4090 much more viable here too. MTP and the recent improvements to kv quantization in particular, as well as model-specific template & quant fixes. I run a 4090 with the 4-bit quantized variant of the same model now and have had a great experience. Qwen3.5 was already a big step up, but with 3.6 and the rest of the improvements it's substantially more reliable as a daily use tool and I find myself reaching for hosted models a lot less. Feels like I could work entirely without them if they were to disappear without going back to typing every line of code myself.
To make 4-bit fit on one card with reasonable (100k+) context needs a bit more care though. And tuning can be highly specific to your machine, gpu and use-case. But I use a headless server, offload multi-modal to CPU, use fit-target to reduce wasted memory and use q8_0 kv since the 4090 performs well with it... In addition to most of the same config as the author elsewhere. I get 50-60tps generation with a power limit of 275W (450W is default), more than enough to offer a roughly an Opus-speed feedback loop.
I haven't seen many of the issues with looping the author mentions. But I did with Qwen3.5 and in particular other 4-bit quants in the past. But the difference is probably a mix of the improvements above, as well as habits changing to avoid cases where models will loop. For what I'm doing, it seems like I loop Qwen3.6 on the same kind of prompts I'll make Haiku or Sonnet loop on (the latter hide some of their existential loops behind "thinking"). Usually it's cause I was too vague about some aspect of what I'm wanting them to do or I forgot to include some context that smaller models just don't have access to in their smaller knowledge base. But at least for what I'm doing (Rust, React, kubernetes) it's not been a notable problem at all with the latest iteration of this whole stack. And knowledge of standard libraries and default k8s resource kinds has been almost flawless.
There's still plenty of more complex stuff where I'll choose to jump straight to Claude or GLM-5.2, but if it's not worth that jump I've stopped paying for the middle ground as it's usually not much better than just one more iteration through qwen.
All this to say, if you have a 3090/4090, feel free to give the same setup a go. It's come a long way in recent weeks.
I've got a similar setup with a k3s homelab and a bunch of small projects that need basic data storage and caching. One thing worth considering is that if someone wants to run both redis and postgres, they need to allocate enough memory for both including enough overhead that they don't suddenly OOM.
In that sense, seeing if the latency impact of postgres is tolerable is pretty reasonable. You may be able to get away with postgres putting things on disk (yes, redis can too), and only paying the overhead cost of allocating sufficient excess RAM to one pod rather than two.
But if making tradeoffs like that, for a low-traffic service in a small homelab, I do wonder if you even need a remote cache. It's always worth considering if you can just have the web server keep it's own cache in-memory or even on-disk. If using go like in the article, you'd likely only need a map and a mutex. That'd be an order of magnitude faster, and be even less to manage... Of course it's not persistent, but then neither was Redis (excl. across web server restarts).
Sure, but if you insist it's like a tide you downplay the risk of the initial hit of the wavefronts and the potential for it to slam up the coast or a seawall becoming a larger local wave. And if you insist it's like a wave, you downplay the persistent risk of both follow-up waves and ongoing flooding that won't subside quickly.
So saying it's not waves is dangerous, and saying it's not a sea level rise is dangerous. It's not useful to try and delineate between a tsunami being one of the two when it's in reality an event that consists of both.
(Ignoring that a sea level rise and a long-wavelength wave are the same thing)
It's a distinction without value I think. There are waves, and many of them. There is a rise in the sea level. For anywhere affected, both certainly matter. Like you mentioned, tsunami isn't a brief event. And here in Japan, they are talking about tsunami waves, not a singular tsunami. And talking about sea level rise and checking the local power poles for sea level indicators from previous tsunami events and floods.
There are planes, buoys and other things being mentioned on the news here in Japan as ways things are being tracked. Maybe not what you meant, but tracking the wave isn't necessarily correct. There are many waves, and the initial wave is often (in this case also) not the largest.
The news mentioned a previous similar event where the largest wave was 4 hours later.
Not true. As the news reporters here in Japan are repeating every few minutes, there will be many waves and they can get bigger over time. They already have, 20-30cm initial waves had 40-60cm later waves.
Waves can get bigger due to earthquakes not being instantaneous or necessarily a single movement, due to amplification by geography, by reflections, by aftershocks, and many other things. The news is suggesting waves lasted about a day for a previous event in a similar area.
If you haven't tried OpenAI's advanced voice mode, it's a mind blowing version of exactly what things like Siri really ought to become with a little more development. If that's what you mean by LLM Siri, I totally agree.
Being able to chat casually with low latency, correct yourself, switch languages mid-sentence, incorporate context throughout a back-and-forth conversation etc. turns talking to these kinds of systems from a painful chore into something that can actually add value.