Adding to the prior comments as my intuition matched yours, there’s a nice Reddit thread that gives some context into how it can be faster even if you require exact matches: https://www.reddit.com/r/LocalLLaMA/s/ARxHLqRjdM
The TLDR/key (from my understanding) is that verifying N tokens can be faster than generating N tokens.
Do you think this would scale to larger transformer models with more parameters per layer?
How would this work with MOE models or sparse models?