HackerTrans
TopNewTrendsCommentsPastAskShowJobs

immortal3

no profile record

Submissions

Improving LLM Inference with Continuous Batching: Orca Through Tinyorca

junupark.xyz
2 points·by immortal3·vor 3 Monaten·0 comments

[untitled]

1 points·by immortal3·vor 4 Monaten·0 comments

Bits-per-Byte (BPB): a tokenizer-agnostic way to measure LLMs

dipkumar.dev
1 points·by immortal3·vor 9 Monaten·0 comments

comments

immortal3
·vor 9 Monaten·discuss
There's another angle to this comparison. Groq and Cerebras use custom chips, but I'm not sure about Together. In this case, Together is sharing results based on the B200 GPU. Another important point is the accuracy of these speed-ups compared to the baseline model. It's known that such tricks reduce accuracy, but by how much? Kimi has already benchmarked several providers. https://x.com/Kimi_Moonshot/status/1976926483319763130