Speculative pre-positioning: off-path decode for stateful inference sessions(arxiv.org)
arxiv.org
Speculative pre-positioning: off-path decode for stateful inference sessions
https://arxiv.org/abs/2606.29565
1 comments
With native support for sessions in an inference engine we can make use of idle GPUs... doing work!