Just curious, can prefill be run on a machine with a mighty GPU and decode on Big Unified Memory Box?
If the prefill produces results the size of several GBs, that is probably not realistic.
If the prefill produces results the size of several GBs, that is probably not realistic.