yvbbrjdr

9 tháng trướcdiscuss

We actually profiled one of the models, and saw that the last GeMM, which is completely memory bound, is taking a lot of time, which reduces the token speed by a lot.

yvbbrjdr

9 tháng trướcdiscuss

I see! Do you know what's causing the slowdown for ollama? They should be using the same backend..

yvbbrjdr

9 tháng trướcdiscuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that none of my friend has this device.

yvbbrjdr

9 tháng trướcdiscuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

yvbbrjdr

năm ngoáidiscuss

We use DeepSeek v3 in prod. Works even better than GPT-4o.

Submissions

NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

LMRouter: Run Claude Code on Any LLM with One Command

Athena – An open source production-ready general AI agent

comments