HackerTrans
TopNewTrendsCommentsPastAskShowJobs

george_123

no profile record

Submissions

Loading Llama-2 70B 20x faster with Anyscale Endpoints

anyscale.com
3 points·by george_123·vor 3 Jahren·0 comments

How continuous batching improves LLM inference throughput 23x

twitter.com
1 points·by george_123·vor 3 Jahren·0 comments

comments

george_123
·vor 3 Jahren·discuss
1) when the blog was released, TGI didn’t support paged attention, 2) many people don’t even know about TGI to reduce inference costs.
george_123
·vor 3 Jahren·discuss
A lot of AWS services (especially SageMaker) aren’t very good in customer experience. People buy them for nominal capabilities and AWS core bread and butter — short-term and long-term reliability.

Most of these startups (AI and others) have to offer a compelling product before even being notable.

Besides, AWS top level doesn’t care if you use sagemaker or not. There’s a premium but if you’re still using EC2 via another startup, they’re still capturing lions share of value.
george_123
·vor 3 Jahren·discuss
this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..