Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale(daft.ai)
daft.ai
Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale
https://www.daft.ai/blog/cutting-llm-batch-inference-time-in-half-dynamic-prefix-bucketing-at-scale
1 コメント
Part of the Daft team here! Happy to answer any questions