> For decode, (...) it is difficult to batch multiple output tokens together.
I think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of next-token predictions sitting there on the GPU goingh through the layers makes better use of the bandwidth.
At least that's what I observed on a Strix Halo, batching 4 predictions yields like 2-3x the total tps.
I think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of next-token predictions sitting there on the GPU goingh through the layers makes better use of the bandwidth.
At least that's what I observed on a Strix Halo, batching 4 predictions yields like 2-3x the total tps.