Good question. I agree with what I think you're implying, which is that local generation is not the right choice if you want to maximize results per time/$ spent. In my experience, hosted models like Claude Opus 4.6 are just so effective that it's hard to justify using much else.
Nevertheless, I spend a lot of time with local models because of:
1. Pure engineering/academic curiosity. It's a blast to experiment with low-level settings/finetunes/lora's/etc. (I have a Cog Sci/ML/software eng background.)
2. I prefer not to share my data with 3rd party services, and it's also nice to not have to worry too much about accidentally pasting sensitive data into prompts (like personal health notes), or if I'm wasting $ with silly experiments, or if I'm accidentally poisoning some stateful cross-session 'memories' linked to an account.
3. It's nice to be able solve simple tasks without having to reason about any external 'side-effects' outside my machine.
Supporting all the various devices does sound quite challenging.
I wouldn't expect a perfect single measurement of "quality" to exist, but it seems like it could be approximated enough to at least be directionally useful. (e.g. comparing subsequent releases of the same model family)
This (+ llmfit) are great attempts, but I've been generally frustrated by how it feels so hard to find any sort of guidance about what I would expect to be the most straightforward/common question:
"What is the highest-quality model that I can run on my hardware, with tok/s greater than <x>, and context limit greater than <y>"
(My personal approach has just devolved into guess-and-check, which is time consuming.) When using TFA/llmfit, I am immediately skeptical because I already know that Qwen 3.5 27B Q6 @ 100k context works great on my machine, but it's buried behind relatively obsolete suggestions like the Qwen 2.5 series.
I'm assuming this is because the tok/s is much higher, but I don't really get much marginal utility out of tok/s speeds beyond ~50 t/s, and there's no way to sort results by quality.
Thanks for the response! I hear what you're saying, and I apologize for my joke.
I re-read your essay, and what you say about needing sophistication reminds me of the concept of proof-of-work -- "sophistication" could be a way to convey that effort was spent by a writer, even if it doesn't add meaning. That is kind of inherently annoying, because it implies a lack of trust between the author and reader, and in the thesaurus example, the reader would be rightfully annoyed to spent time parsing a sentence only to find that the "proof of effort" was actually just a "facade of effort".
It's clever that the the author provides both his essay and an example at the same time! Sorry, that joke felt obligatory.
Miscellaneous reactions, in an elegant bulleted list:
- "Simple" sentences are certainly expressive, but "elegant" wording expands the set of meanings that can be conveyed. And vice-versa
- I think a lot of the meat of a sentence is conveyed in the connotations of words and not their literal meaning. "Simple" wording is necessarily more common, and therefore will necessarily have a less specific or reliable connotation. This is a blessing and a curse.
- More subjectively, I think ideal writing is also a window into the author's experience of the world (or moreso whatever topic they're writing about), and as a reader, I want that to come through in an authentic way that matches the author's experience. So, using a thesaurus and agonizing over sentence structure might end up 'elegant' but still vaguely bad, but on the other hand if you agonize over a sentence and come up with something more "sophisticated" that ultimately rings truer to you, then go for it.
- ^ The above points aren't direct rebuttals to TFA, but I think they relate to why elegance can be appealing.
I might be misunderstanding your point, but quantization can have a dramatic impact on the quality of the model's output.
For example, in diffusion, there are some models where a Q8 quant dramatically changes what you can achieve compared to fp16. (I'm thinking of the Wan video models.) The point I'm trying to make is that it's a noticeable model change, and can be make-or-break.
Nevertheless, I spend a lot of time with local models because of:
1. Pure engineering/academic curiosity. It's a blast to experiment with low-level settings/finetunes/lora's/etc. (I have a Cog Sci/ML/software eng background.)
2. I prefer not to share my data with 3rd party services, and it's also nice to not have to worry too much about accidentally pasting sensitive data into prompts (like personal health notes), or if I'm wasting $ with silly experiments, or if I'm accidentally poisoning some stateful cross-session 'memories' linked to an account.
3. It's nice to be able solve simple tasks without having to reason about any external 'side-effects' outside my machine.