We actually profiled one of the models, and saw that the last GeMM, which is completely memory bound, is taking a lot of time, which reduces the token speed by a lot.

yvbbrjdr

·9 miesięcy temu·discuss

I see! Do you know what's causing the slowdown for ollama? They should be using the same backend..

yvbbrjdr

·9 miesięcy temu·discuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that none of my friend has this device.

yvbbrjdr

·9 miesięcy temu·discuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

yvbbrjdr

·w zeszłym roku·discuss

We use DeepSeek v3 in prod. Works even better than GPT-4o.

yvbbrjdr

·w zeszłym roku·discuss

Join the Discord server to see Athena in action! https://discord.gg/X38GnhdTH8