It's sort of amazing that people still say these sorts of things today.
I don't think it's at all inconceivable where people in the future are streaming high resolution multi modal personal sensor arrays to AI that is running in a data center...and ditto we are streaming ever more content back.
That's just the in vogue answer of 2026. There are undoubtedly endless innovations that I can't envision that will require ever more resources.
My point is just that every single "X ought to be good enough" has been proven false. And the only times where we see consumption really plateau are due to other reasons than desire (like cost).
> Venture capitalists & private investors are sucking all of the possible growth and future upside from these companies and then dumping them on retail investors when there's nothing left.
A lot of the money that is deployed by VCs comes from pension funds and asset managers that ultimately manage money for the average Joe.
I haven't read TFA yet but a common technique is speculative decoding where a fast draft model will generate X tokens, which are then verified by the larger target model. The target model may accept some Y <= X tokens but the speedup comes from the fact that this can be done in parallel as a prefill operation due to the nature of transformers.
So let's say a draft model generates 5 tokens, all 5 of these can be verified in parallel with a single forward pass of the target model. The target model may only accept the first 4 tokens (or whatever) but as long as the 5 forward passes of the draft model + 1 prefill of the target model is faster than 4 forward passes of the target, you will have a speedup while maintaining the exact output distribution as the target.
> With prompt caching, verbose context that gets reused is basically free.
But it's not. It might be discounted cost-wise, however it will still degrade attention and make generation slower/more computationally expensive even if you have a long prefix you can reuse during prefill.
> Tradition warrants a negotiation phase when one party wishes to change the terms of an agreement, or becomes cognizant that the counterparty may wish to do the same.
They didn't change the agreement. One party violated it, and the other party withdrew as a result.
This is so vanilla. But people will moan because they want subsidized tokens.
I would guess you haven't done this in practice. Yes, of course inference is memory bound at low batch sizes. This is why we run larger batch sizes!
Also there does not exist any batch size > 1 where per-request throughput is equal to bs=1. Doing any batching at all will slow all intra-batch requests down.
Yes this article is full of misunderstanding. The main explanation of bottleneck is wrong: it’s the model weights which dominate memory bandwidth (and hence why batching multiple requests in a single pass increases total throughput). If copying user tokens was the bottle neck, batching would not achieve any speed up.
When an author is confused about something so elementary, I can’t trust anything else they write.
This is necessarily a question that requires thought and speculation about the future.
You can't just flip a switch and do it overnight if it turns out there is demand.
And if there isn't demand, you are going to lose a lot of money.