HackerTrans
TopNewTrendsCommentsPastAskShowJobs

sorrow17

no profile record

Submissions

VLLM: The High-Throughput and Memory-Efficient Serving Engine for LLMs

vllm.ai
1 points·by sorrow17·il y a 6 mois·0 comments

comments

sorrow17
·il y a 7 mois·discuss
Yeah, for a lot of users who control the exported source code, rewriting model to use control flow ops, or simply removing the control flow code is a viable option and solvable. For some other users who want to export the model as-is, the option is either using the (deprecated) torchscript, or just move on and use torch.compile and run your model in Python.