Why do output tokens cost 5x more than input tokens?(anirudhsathiya.com)

3 pointsby ani173 เดือนที่ผ่านมา2 comments

Why do output tokens cost 5x more than input tokens?

https://www.anirudhsathiya.com/blog/transformer

2 comments

ani173 เดือนที่ผ่านมา

Author here. I wanted to understand what vLLM and llama.cpp are actually doing under the hood, but the codebases are massive. So I wrote a stripped down version from scratch to see the core ideas without the production complexity.

Code: https://github.com/Anirudh171202/WhiteLotus

lazyMonkey693 เดือนที่ผ่านมา

I think the paged attention part is a bit oversimplified. Nice read otherwise!