The reason speculative decoding shows diminishing returns in batched workloads is because the principle of both is the same.
Speculative decoding predicts a group of tokens and verifies this group using the main model in one pass instead of decoding each token separately. Eg. for this group, the weights are loaded from RAM per group instead of per token: roughly the same computation is performed but not the same memory movement (and other overhead like kernel launches).
Batching utilizes the same mechanism, so speculative decoding is essentially an attempt to batch a single stream using prediction. An attempt, because the verification may reject some tokens if the prediction was inaccurate.
>LLMs are still bad at low-level hardware optimizations, but really good at high-level composition.
I disagree. While yes they don't have all the architectural quirks of every GPU memorized, they are able to extract such optimizations from ISA docs and online guides. Now with 1M context available on frontier models, they can even fit the whole ISA definition in context (RDNA 3.5 here specifically) and spit out swathes of optimizations to try. The rest is just bruteforcing a single goal which they are extremely good at.
Or that's how simple it'll look until you have subtle bugs to solve somewhere deep in your stack.
Anyways, low-level hardware optimized GPU kernels has been an exceptionally good use case for agents in my opinion. They have far more trouble in other domains like doing GUI.
For me writing manpages as a spec worked well. While the format is unintuitive and tedious (at least to me), I used AI to format the manpages from my descriptions and rendered to PDF to review. Though it's been mainly useful for CLI utilities.
Once I got the manpages ready and the way I wanted, I just stored those in an empty project folder and told the agent to implement as specified in a language of my choosing.
Maybe for other development scenarios like GUI there is some "native" way to spec the project in a common format like this.
The GPU has INT4, INT8, BF16 and FP16. Notably no FP8 or FP4.The official GPTQ-Int4 release from Qwen is a great quant for this but custom kernels are still rare for this hardware.
>Genuine question, in what scenario is the self hosting setup and maintenance worth it?
Maybe if you're a huge org with a dedicated security team and so on, which could easily handle managing such service. I guess at a certain point it would bring cost savings at a scale in comparison to using Bitwarden, where it costs per team member or seat. Inhouse team has fixed costs in comparison.
Of course for smaller orgs or individuals there is little sense in hosting security software yourself. No way you're going to have enough time to manage the service and keep it secure, which is where almost all of such software's value is derived from.
Speculative decoding predicts a group of tokens and verifies this group using the main model in one pass instead of decoding each token separately. Eg. for this group, the weights are loaded from RAM per group instead of per token: roughly the same computation is performed but not the same memory movement (and other overhead like kernel launches).
Batching utilizes the same mechanism, so speculative decoding is essentially an attempt to batch a single stream using prediction. An attempt, because the verification may reject some tokens if the prediction was inaccurate.