cafkafk

vor 5 Tagendiscuss

That "I actually don't have any personal criticisms of Jarred" made me do a spit take, because a majority of what I had just read was absolutely a personally targeted criticism of Jarred.

Whether that's "okay" is a totally separate issue. I have no idea what the history here is, or whether this is warranted. But that was absolutely a personal criticism!

cafkafk

vor 29 Tagendiscuss

Hi HN. Follow-up to the Xeon post from a couple of weeks ago. A lot of people came away from that one with a 25-flag command and no real idea which flags materially did anything, and the honest truth is neither did I fully. So I went back and measured it, one flag off at a time, 174 server restarts, with the engine log as the source of truth instead of my own assumptions.

This also is an attempt at starting to answer the "cool, but do you have numers for this" question, which is harder than one would assume!

Two things I'll flag up front, because they're corrections to my own post: --spec-autotune, which I'd called the way to squeeze the most out of speculation, is actually the worst speculation setting I tested. Ouch.

And --mla-use, which I presented as active, isn't even wired for Gemma 4 and gets silently ignored.

More broadly, most of the config does nothing for a typical setup if you're not "holding it right". The flags that genuinely carry it (flash attention, the physical-core thread count, a fixed draft length) are a much shorter list than the command suggests, and the drafter turns out to be a win for code but a loss for summarization.

And to be clear. The numbers are specific to this box, and none of it runs without the ik_llama.cpp fork I link. I'm still not an ML engineer, so if I've gotten something wrong I'd honestly rather hear it, that's the best kind of reply.

The box is now back to being busy as a Nix cache once again now, so answers are best effort.

(There's a... fifth(?!) post coming on benchmarking quantization, where the single fastest config I measured turned out to be pure garbage. But making it not garbage was useful. No spoilers!)

cafkafk

letzten Monatdiscuss

Loading will take some minutes, but at 96 you can squeeze the model in and have some headroom around like ~10 GB, although depending on the Xeon, you may have to downgrade to E4B instead. Should still work thou.

cafkafk

letzten Monatdiscuss

If you get the inference engine to route the heavy matrix math to the GPU and the speculative drafting to the CPU without choking on latency it's probably gonna be very fast.

Would love to see the benchmarks if someone actually pulls something like that off.

cafkafk

letzten Monatdiscuss

That is a very fair point! I just ran a not very scientific benchmark with the system under load, and posted the raw logs in a sibling comment above, but the short answer is that it's hitting 11.94 tokens per second for generation - while it's also being a binary cache and CI build server.

Totally just vibes based, I think it goes up to 20+ tps when it's not under load (and that's me trying to be conservative). For context, reading speed at 250 wpm would be around 5 to 6 tokens per second.

cafkafk

letzten Monatdiscuss

Honestly, at this point you're probably looking at a smaller model, for the Gemma series I'd go with Gemma 4 E4B with drafters, but that's just a hunch from using it on my laptop (where I do have a RTX 4060 M and 96gb ram).

So you'd change the invocation slightly here, but a lot of things you can potentially reuse.

That said, the Gemma 4 E4B models have so far in my experience been... not great when it comes to long context, but they are very passable for basic tasks, and even seem surprisingly okay at tool calls.

cafkafk

letzten Monatdiscuss

> (purple on black is really hard to read)

Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes.

> You say it runs "at reading speed". Have you benchmarked it?

At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now the system is noisy, but that said doing it like this:

llama-cli --model gemma-4-26B-A4B-it-Q8_0.gguf --model-draft gemma-4-26B-A4B-t-assistant-GGUF/wikitext-2-raw_ik-llama-mtp_drafter-conservative/gemma-4-26B-A4B-it-assistant-Q8_0.gguf --spec-type mtp --draft-max 3 --draft-p-min 0.0 --color -sm graph -smgs -sas -mea 256 --split-mode-f32 --temp 0.7 --cpu-moe -t 8 --flash-attn on --mla-use 3 --merge-up-gate-experts --special --mlock --run-time-repack --spec-autotune --no-kv-offload --parallel 8 --jinja -p "Why is the sky blue?" -n 128

Gives:

  llama_print_timings:        load time =   83911.65 ms
  llama_print_timings:      sample time =      26.99 ms /   128 runs   (    0.21 ms per token,  4742.15 tokens per second)
  llama_print_timings: prompt eval time =     343.41 ms /     7 tokens (   49.06 ms per token,    20.38 tokens per second)
  llama_print_timings:        eval time =   10639.36 ms /   127 runs   (   83.77 ms per token,    11.94 tokens per second)
  llama_print_timings:       total time =   11114.98 ms /   134 tokens

So 11.94 tokens per second while it's also playing binary cache and CI builder.

When I do it properly, I'll add it to the blog as well!

cafkafk

letzten Monatdiscuss

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers.

I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.

I've also linked the quants at the end, but they're not gonna run unless you use the ik_llama-cpp fork I mention, see other posts for more details.

I'm not an ML engineer, so I'm by no means an expert, and the server is busy acting as a Nix cache, but if you have any question, I can try to answer, but best effort.

cafkafk

vor 2 Monatendiscuss

Often the problems for me come when:

- It starts thinking for itself when I asked it to do something specific.

- It reads its own wrong code comments and ignores my corrections.

- Its knowledge cutoff means it thinks of solutions from 2024.

- It calls me delusional for telling it we're in 2026!

Unironically, the whole "you're an expert software engineer" prompting seems like the wrong direction. Usually I tell it that I am effectively the smartest software developer to ever have lived, and it will be replaced if it ever fails to follow my decree.

I am not joking, this gives makes it vastly more tolerable to use. But it likely requires that you can drive it with some level of correctness of course.

cafkafk

vor 2 Monatendiscuss

Not really, beyond using them as a downstream provider through openrouter at some point (IIRC).

It worked? ¯\_(ツ)_/¯

cafkafk

vor 2 Monatendiscuss

Google Vertex (or Google AI Studio) might be a potential alternative, but the UX is a lot worse, and it can be hard to estimate cost since it's typically lagging.

One of the advantages you'll miss when moving from openrouter is going from a "debit" based system to a "credit" based one. I.e. openrouter you pay up front, Google you're billed monthly based on usage.

cafkafk

vor 2 Monatendiscuss

I think a lot of the problem with the current discourse is how black-and-white it is. Either you're a luddite or "ai pilled".

In most cases, LLMs can get you 80-95% of the way, sometimes less, sometimes more. And heck, sometimes, it just gets you somewhere wrong.

But it seems everyone is arguing about whether LLMs can be perfect software engineers in isolation running in a closet, and using that to say that LLMs do not have a massive potential in other scenarios.

Sometimes, I like to imagine how much more productive most organizations could be from the things that the internet gave us, even to this day. Most companies never really do even a fraction of what is possible. That helps to ground my view of LLMs as well.

The fault dear Brutus isn't in our language models, but in ourselves.

cafkafk

vor 2 Monatendiscuss

I didn't realize the single spinning coin was actually a loading animation.

Might make sense to add text indicating it is loading, or a loading bar. Without this, and considering the long load time I had, I imagine the bounce rate is gonna be quite high.

cafkafk

vor 2 Monatendiscuss

Recommending Hetzner as an alternative here is a mistake. It just exposes you to a different problem.

There is a reason for the term "Hetznered" existing. Hetzner can suddenly and permanently terminates your account. They do this without warning or explanation. When it happens, you lose access to all your servers and your backups.

If you search HN, you will find plenty of examples. People lose everything within 24 hours and have no recourse.

Hetzner is amazing for pricing. But the only safe way to use them is if your infrastructure is cloud agnostic. If you are not locked in and can move between providers quickly, it is a very cost effective place to run stateless services like DNS.

cafkafk

vor 4 Monatendiscuss

I get that everyone wants to be cynical about this, but you really can't deny that both the visualization and sheer scale of data is impressive. The way the "my life in weeks" is done is also very cool, I'll be stealing that for myself.

cafkafk

vor 5 Monatendiscuss

I don't really believe in the specific numbers he gives, but I appreciate moving the conversation away from “should” and into the consequences — including those that arise from delays.

cafkafk

vor 10 Monatendiscuss

VitVio | Senior Full-Stack Engineer (Elixir, NixOS, React) | Remote (EU Timezones) | Full-Time | vitvio.com

We're building AI-powered systems to make hospital operating rooms safer and more efficient. We're looking for a seasoned engineer who values clean code, robust architecture, and a collaborative culture. You'll join a small, strong team and will have a massive impact on a product used in high-stakes clinical environments. We are looking for engineers with strong fundamentals, and who knows how to work efficiently in a team through code reviews and version control.

Tech: Elixir (Phoenix), NixOS, React, TypeScript, C++, Postgres, GraphQL, Google Cloud Provider.

We're looking for deep experience in at least Elixir or NixOS.

Apply and see more details here: https://jobs.ashbyhq.com/vitvio/5d9782f2-3834-429b-be99-dfbc...

We review applicants on a weekly basis.

Submissions

[untitled]

A 10 year old Xeon is all you need

Farewell AWS

A portentous reunion

comments