We actually profiled one of the models, and saw that the last GeMM, which is completely memory bound, is taking a lot of time, which reduces the token speed by a lot.

yvbbrjdr

·há 9 meses·discuss

I see! Do you know what's causing the slowdown for ollama? They should be using the same backend..

yvbbrjdr

·há 9 meses·discuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that none of my friend has this device.

yvbbrjdr

·há 9 meses·discuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

yvbbrjdr

·ano passado·discuss

We use DeepSeek v3 in prod. Works even better than GPT-4o.

yvbbrjdr

·ano passado·discuss

Join the Discord server to see Athena in action! https://discord.gg/X38GnhdTH8