yvbbrjdr

9 months agodiscuss

We actually profiled one of the models, and saw that the last GeMM, which is completely memory bound, is taking a lot of time, which reduces the token speed by a lot.

yvbbrjdr

9 months agodiscuss

I see! Do you know what's causing the slowdown for ollama? They should be using the same backend..

yvbbrjdr

9 months agodiscuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that none of my friend has this device.

yvbbrjdr

9 months agodiscuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

yvbbrjdr

last yeardiscuss

We use DeepSeek v3 in prod. Works even better than GPT-4o.

Submissions

NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

LMRouter: Run Claude Code on Any LLM with One Command

Athena – An open source production-ready general AI agent

comments