Generating one token at a time is a blessing in disguise(kachkach.com)

3 pointsby halflingsvor 3 Monaten1 comments

Generating one token at a time is a blessing in disguise

https://kachkach.com/blog/generating-one-token-at-a-time-is-a-blessing-in-disguise

1 comments

LLMs generate their output one token at a time. The first thought when you learn this is that this is a huge performance bottleneck, as we are used to highly parallelized systems.

However, a large part of what makes LLMs feel so magical comes from this bottleneck.