yvbbrjdr

We actually profiled one of the models, and saw that the last GeMM, which is completely memory bound, is taking a lot of time, which reduces the token speed by a lot.

yvbbrjdr

9 mesi fadiscuss

I see! Do you know what's causing the slowdown for ollama? They should be using the same backend..

yvbbrjdr

9 mesi fadiscuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that none of my friend has this device.

yvbbrjdr

9 mesi fadiscuss

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

Submissions

NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

LMRouter: Run Claude Code on Any LLM with One Command

comments