My first real programming achievement was building a website for an Ultima Online shard. I wrote some really terrible PHP and HTML, but it worked for 20+ years afterwards. Great memories!
I was surprised that there is still an active community around UO!
In any case, this is very cool.
Thanks for sharing!
V4 is definitely a step-up from V3.2 on our multilingual benchmarks.
Two caveats:
- when inferring through Openrouter, we've had a lot of issues with very slow speeds (TPS) and an occasional instability. I just checked and it's still 10-30 TPS on all available providers, which is not a lot for a model that likes to think as much as DeepSeek does.
- the official DeepSeek API makes no guarantees of data privacy even for paying users.
Both points could be moot with using it through Azure AI foundry (the latter is, afaik); I have yet to test that.
In any case, happy to see more open-weights models that are somewhat competitive with SOTA models!
MoE is mostly an optimization of the active parameters and therefore lowering the compute requirements, but it can provide some performance improvements over dense models in some cases.
I would not describe reasoning as optimization: In fact, it's typically doing the opposite, as models spend way more tokens (and therefore compute) on responding to the same prompt. Some of the smartest models these days use ridiculous amounts of reasoning before they ever respond. Try Deep Research in Gemini or Claude and you'll see what I'm talking about.
>> But this seems to be a very clear path to be "taking the car to the carwash by foot" for a long time, isn't it?
I thought the progress was plateauing sometime last year too, but then some new models got released and we saw that the multilingual capabilities improvements are real. And if you want something more tangible and reported on, consider the Opus 4.5/4.6 coding revolution (Claude Code explosion) a few months back.
LLMs being stochastic and statistical machines, there will always be funny things that people will come up with that will trick them, be it R's in strawberry or the carwash by foot. At the same time, I can tell you from my experience that a lot of the Misguided attention ( https://github.com/cpldcpu/MisguidedAttention ) type of stump questions work at a much lower rate with newer models. Progress is being made, it's just not in visible areas.
BTW, you can come up with many trick questions that will stump even humans with PhDs. They will be of different kind than the ones for LLMs, but this is not a flaw unique to LLMs.
If you're asking whether the progress to AGI isn't taking too long, then I personally think LLMs, at least with their current architecture, are not the foundation of AGI, and will always have inherent limitations. But we're fully in the "that's just like, your opinion, man" territory now :)
There are architectural changes (such as reasoning or mixture of experts) that measurably improve how well models perform. So the improvements are definitely not just from data.
I can speak for my area of expertise - multilingual capabilities. Some SOTA models are making huge strides in their support of various languages, and increasingly they understand and can produce text in languages where GPT-4 era models were absolutely lost. These are probably from a combination of richer training dataset and architectural improvements (more parameters?).
Now that doesn't necessarily mean that models are also getting substantially better at English or other major languages. They likely are to some degree, but we've reached a point with major languages where core linguistic proficiencies are covered, and what's left is the more squishy part: style, tone of voice, ability to use different registers naturally, or what some people would call linguistic taste. But that's much harder to measure and therefore trickier to provide evidence for.
Just saw your thinking edit! That's a great question and one I wanted to study in depth, but these days you don't really get access to the raw thinking data. It's usually summarized and you can't even be sure what language the model thought in unless you have access to the logits (so only viable for open-weights models).
I am fairly convinced that there's a certain polyglot snowball effect: once the LLM is fluent in 20 languages, it can pick up on similarities in vocabulary, syntax etc. and learn the 21st language with much less effort (and training data). This might be difficult to actually study in an isolated way, but it's a real effect for humans and it makes sense the the pattern matchers that LLMs are would find these shortcuts.
Using similar words should land you in similar places in the latent space, even if they actual word or their order is slightly different. Where it gets interesting is how well English words map to their counterparts in other languages, and what practical differences it makes. From various studies, it seems that the gravitational pull of English language/culture training data is substantial, but an LLM can switch cultures and values when prompted in different languages.
One more thing: we're working on a multilingual benchmark that will evaluate core linguistic proficiency in 30 languages. We already have a lot of data internally and I can tell you that:
- Gemini 3 Pro is a multilingual monster.
- GPT-5.4 is a really good translation model, big improvements over previous subversions in the 5 family.
- Opus 4.6 is good but usually third place.
- Somehow, Grok 4.20 is surprisingly good at some long-tail languages? Its performance profile is really odd. Unlike all the other models.
Disclosure: I work at RWS/TrainAI, we did this study. Recently I alluded to it in a comment and was encouraged to share it, so here it is!
We focus on multilingual proficiency, which tends to be understudied: most benchmarks are English-heavy or even English-only and don't tell you much about how models actually perform across languages.
This is our second iteration of the study. 120 linguists, 8 models, 8 languages, 4 tasks, every output blind-reviewed by 3 native speakers.
Some notable insights:
- GPT-5 is strong at text normalization and translation but regressed on content generation vs GPT-4o. Chinese outputs had spacing/punctuation issues, Polish read like "translationese" even with no source text.
- Gemini 2.5 Pro scored 4.56/5 on Kinyarwanda. In our first study (late 2024), no model could produce coherent text in that language.
- Top LLMs outscored humans working under realistic constraints (time-limited, single pass, no QA). Humans didn't rank 1st in any language. (We're now planning a follow-up to zoom in on that.)
- Tokenizer efficiency matters again: reasoning models burn 5-10x more tokens thinking. Claude Sonnet 4.5 encodes Tamil at 1.19 chars/token vs Gemini's 4.24 — ~3.5x cost difference for the same output. There has been a lot of talk about the Opus 4.7 tokenizer, this is the same issue, just in multilingual setting.
Yes, but post training cannot possibly account for all possible use cases. Sane defaults are fine, you can't really do much about sampling parameters in chatbots and coding harnesses anyway. And when making an API call, you have to actively change the parameter in your payload. I don't believe there's any real risk.
There's been quite a few threads about Opus 4.7 but none of them seems to have discussed some breaking changes on the API side, particularly removal of sampling parameters.
From the migration guide:
>> Sampling parameters removed: Setting temperature, top_p, or top_k to any non-default value on Claude Opus 4.7 returns a 400 error.
Let's set aside that this should probably be a deprecation warning and not a 400. Not having these dials limits utility for cases like synthetic data generation, natural language QA and many more. Even though temp=0 does not guarantee determinism, getting 99 identical responses out of 100 is reasonably close to determinism for most practical use case. Default temp gives you wild swings in performance which temp=0 almost perfectly eliminates. And there are valid use cases for using temp=0 or experimenting with different values.
The writing was on the wall since even earlier Opus versions would override temperature setting and reset to default when thinking was enabled. Now there is no way to control it at all. It is a bit disappointing.
I understand most people around here will be using Opus in Claude Code or in another harness for coding, and in that case you are not really affected. But for those of you building products and using the API in different ways, how are you dealing with this change?
If anyone from Anthropic is reading this, any insights into why it was removed would be great. I am struggling to believe this is because of distillation concerns. Thanks!
Claude's tokenizers have actually been getting less efficient over the years (I think we're at the third iteration at the least since Sonnet 3.5). And if you prompt the LLM in a language other than English, or if your users prompt it or generate content in other languages, the costs go higher even more. And I mean hundreds of percent more for languages with complex scripts like Tamil or Japanese. If you're interested in the research we did comparing tokenizers of several SOTA models in multiple languages, just hit me up.
Thanks for sharing! Have been begrudgingly using Darktable since that seems to be your best option on Linux, but the UI/UX never really clicked with me. I wish this was opensource but I will give this a shot (pun intended) for sure.