crashocaster

há 3 anosdiscuss

I always find evals of this flavor offputting given that 3.5 and 4 likely share preference models (or at least feedback data)

crashocaster

há 3 anosdiscuss

Actually, the only numbers every LLM developer should know are their accelerator specs. For example:

A100 specs:

- 312e12 BF16 FLOPS

- 1555e9 GB/s HBM bandwidth

H100:

- 1000e12/2000e12 BF16/INT8 FLOPS

(apply ~0.7 flops efficiency multiplier because h100s power throttle extremely quickly)

- 3000 GB/s HBM bandwidth

---

For a 13B model on an A100, this nets:

13e9 * 2 bytes per param = 26 GB HBM required (at bf16)

26e9/1555e9 = 17ms / token small-batch latency (~60 tokens / second)

What about large batches?

latency for some batch size B is 13e9 * 2 FLOP per param * B / 312e12

We want B such that we're just about no longer HBM bound: 26e9/312e12 * B = 17ms

<=> 17e-3/(26e9/312e12)

giving a batch size of 204.

At that batch size (and all larger batch sizes), the a100 delivers a throughput of B * 1/17ms = 12000 tokens / second

---

KV caching, multi-gpu and -node comms and matmul efficiencies left as an exercise to the reader :)

crashocaster

há 5 anosdiscuss

I’d suggest to revisit voice recognition — it works quite well for me in the same usecase.

I also like to take walks, sometimes listening to podcasts. The stock iOS voice recognition (the microphone button on the keyboard, not Siri) is quite good, I usually just talk into the phone without looking at the output. After the walk, I format and clean up the notes to fix any errors.

comments