HackerTrans
TopNewTrendsCommentsPastAskShowJobs

immortal3

no profile record

Submissions

Improving LLM Inference with Continuous Batching: Orca Through Tinyorca

junupark.xyz
2 points·by immortal3·3 mesi fa·0 comments

[untitled]

1 points·by immortal3·4 mesi fa·0 comments

Bits-per-Byte (BPB): a tokenizer-agnostic way to measure LLMs

dipkumar.dev
1 points·by immortal3·9 mesi fa·0 comments

comments

immortal3
·9 mesi fa·discuss
There's another angle to this comparison. Groq and Cerebras use custom chips, but I'm not sure about Together. In this case, Together is sharing results based on the B200 GPU. Another important point is the accuracy of these speed-ups compared to the baseline model. It's known that such tricks reduce accuracy, but by how much? Kimi has already benchmarked several providers. https://x.com/Kimi_Moonshot/status/1976926483319763130