wanna chime in, recently tried vLLM to consume a NVFP4 Gemma4 safetensor model and see how the batching can show up in nice t/s numbers.
it's slow to start, it's Linux only, it doesn't like WSL much, ended up with either old or nightly container builds, I more or less have given up.
Appreciate how llama.cpp simply works and does things fast and obvious
my latest experiments with local LLM (mistral coder variations) fitting in older 6 GB GTX1060 were disappointing as long as you try to hook Copilot (CLI or VScode) to it and are used to provide a lot tooling. this seems to bloat initial prompt to 20k and more which seems the bottleneck if I did not completely misconfigured things. output tokens/s are more than fine, but PP is frustrating / unusable.
e.g. if on an annual plan? 0x will be gone, but there are okay 1x and 0.3x models left. I am pretty much curious how the early may test invoicing will look like. current setup of tools etc. is way too chatty eats up 1+M token per PRU easily. not sure how much is cached.
not sure how much resources it took from my WinARM Win11 as store app, but it's buggy in either release or beta since months as it loses network connectivity after some time...
I think it's more Dave that "sucks"...click baiting titles, no need to and ever and ever repeating that he was at MS.
He should do Podcasts as there is no value in videos only speaking (feels like 1.5x btw)