Understanding KV Cache: The Hidden Memory Cost of Serving LLMsmelchi.me3 points·by colescodes·2 เดือนที่ผ่านมา·0 comments
colescodes·27 วันที่ผ่านมา·discussI'd try Qwen 3.6 35B-A3B or 27B with a temperature of around 0.6. You'd need to offload some of it to RAM though. I've had okay luck with Gemma 4 12B with a temperature of 1.0 but it will miss things a lot :(
https://coles.codes