MTP can give you small batches, but it is still WAY smaller than the batches you can get with prefill, which is limited only by the number of input tokens you have (but has diminishing returns on performance).
But:
1) It still makes no sense to upload the weights to the GPU with MTP as you are still bottlenecked by the weight upload.
For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU.
For decode, you are bandwidth bound, and it is difficult to batch multiple output tokens together. There is no benefit to sending your weights to the GPU because even if it internally has insane bandwidth, you are still bottlenecked by system RAM (and adding a pcie5 upload would bottleneck it further). This is the number people usually talk about when they say they are getting a certain tk/s.
Sindorov, who won the candidates, and will be playing for the world championship, has also had accounts banned on chess.com for (presumably) cheating as a teenager. Do you think it would be ok if a large part of the chess world conspired to keep him out of tournaments. Do you think it would be ok if the world made jokes that he cheated using a sex toy? Do you think it would be ok for the largest youtube chess channel (GothamChess) to publish a video titled "Chess cheating device" featuring a thumbnail showing his face next to a butt plug? And all this while he was 19?
What happened to Hans is FAR worse than anything Kramnik did. But maybe you think it's ok because you find him to be an asshole?
My experience of working in the industry is that many of my peers have had great animosity and little sympathy for blue collar workers who lost their livelihood to globalization.
edit: Only one of 3 victims was killed in his home:
> In 1994, Burson-Marsteller executive Thomas J. Mosser was killed after opening a mail bomb sent to his home in New Jersey. In a letter to The New York Times, Kaczynski wrote he had sent the bomb because of Mosser's work repairing the public image of Exxon after the Exxon Valdez oil spill.
But:
1) It still makes no sense to upload the weights to the GPU with MTP as you are still bottlenecked by the weight upload.
2) I'm not sure MTP helps much with MoE models.