AI2: Open Coding Agents(allenai.org)
allenai.org
AI2: Open Coding Agents
https://allenai.org/blog/open-coding-agents
48 comments
They either updated it or you quoted it wrong but the article says Devstral is open-weights now.
Yeah, they’ve updated it. Here’s the old version: https://web.archive.org/web/20260128034831mp_/https://allena...
Yes! We updated the blog, thanks for flagging the mistake.
AFAIK gpt-oss-20b on high reasoning has SWE score of just over 60. It is smaller than all comparable models. Maybe I am missing something, but it is still state of the art all the way up to 50B parameters vs all models released after.
At least https://huggingface.co/facebook/cwm team had balls comparing to it directly (sort of, see TTS).
What does this model do that gpt-oss-20b does not? AFAIU the base model it was finetuned from is not reproducible, and if I flip a single bit in gpt-oss-20b and tell you how (instruction under MIT) that would satisfy "fully open finetuning" they claim as advantage. But that "open" fine-tuned gpt-oss-20b is probably going to beat their model.
Am I missing something?
At least https://huggingface.co/facebook/cwm team had balls comparing to it directly (sort of, see TTS).
What does this model do that gpt-oss-20b does not? AFAIU the base model it was finetuned from is not reproducible, and if I flip a single bit in gpt-oss-20b and tell you how (instruction under MIT) that would satisfy "fully open finetuning" they claim as advantage. But that "open" fine-tuned gpt-oss-20b is probably going to beat their model.
Am I missing something?
Great work! Really respect AI2. they open source everything. The model, the weights, the training pipeline, inference stack, and corpus
Claims in the article are incorrect. They conveniently ignore Meta CWM models, which are open-sourced [1] and open-weight [2] and are at 65% SWE-bench verified (with TTS) and 54% pass@1 and the same size (32B dense). So claims like "surpassing prior open-source state-of-the-art coding models of comparable sizes and context lengths" and conveniently leaving out the previous OSS SOTA out of your eval tables are ... sketch.
[1]https://github.com/facebookresearch/cwm [2]https://huggingface.co/facebook/cwm
[1]https://github.com/facebookresearch/cwm [2]https://huggingface.co/facebook/cwm
Hey! These are great observations. So first, while TTS can improve performance, we wanted to evaluate the raw capability of our model. This meant generating only one rollout per evaluation instance, which follows other papers in the space like SWE-smith and BugPilot. In addition, TTS adds extra inference cost and is reliant on how rollouts are ranked, two confounding factors for deployable models where memory and inference speed are extremely important.
Following that line of reasoning, context length is another very large confounding factor. Longer context lengths improve performance - but also result in enormous increases in KV cache size and memory requirements. We decide to control for this in our paper and focus at the 32K context length for 32B size models, a context length that already pushes the bounds of what can be "deployable" locally.
Still, we evaluate at 64K context length using YARN and are able to outperform CWM's 54% performance (non TTS), which it achieves using 128K context, a substantial increase over what we use. This is also pretty significant because we only ever train at 32K context, but CWM trains for a full 128K.
Following that line of reasoning, context length is another very large confounding factor. Longer context lengths improve performance - but also result in enormous increases in KV cache size and memory requirements. We decide to control for this in our paper and focus at the 32K context length for 32B size models, a context length that already pushes the bounds of what can be "deployable" locally.
Still, we evaluate at 64K context length using YARN and are able to outperform CWM's 54% performance (non TTS), which it achieves using 128K context, a substantial increase over what we use. This is also pretty significant because we only ever train at 32K context, but CWM trains for a full 128K.
The difference is that the Allen Institute models have open training data, not just open code and weights. Meta doesn't share the training data you would need to reproduce their final models. For many uses open-weight models are nearly as good, but for advancing research it's much better to have everything in the open.
Reading their paper, it wasn't trained from scratch, it's a fine tune of a Qwen3-32B model. I think this approach is correct, but it does mean that only a subset of the training data is really open.
The linked open weight disallows commercial, and is only licensed for research purpose
An interesting shift I’ve seen over the past few weeks, is we’re starting to refer to bare LLMs themselves as “agents”.
Used to be that agent = LLM + scaffold/harness/loop/whatever.
Used to be that agent = LLM + scaffold/harness/loop/whatever.
I think some of the distinction here is that the more recent "bare LLMs" have been more purpose built, augmented with "agent" specific RL, and in general more fine tuned for the requirements of "agents". Things such as specific reasoning capabilities, tool calling, etc.
These all make the "bare LLMs" better suited to be used within the "agent" harness.
I think the more accurate term would be "agentic LLMs" instead of calling them "agents" outright. As to why its the case now, probably just human laziness and colloquialisms.
These all make the "bare LLMs" better suited to be used within the "agent" harness.
I think the more accurate term would be "agentic LLMs" instead of calling them "agents" outright. As to why its the case now, probably just human laziness and colloquialisms.
Yes, the post training is the special sauce.
GPT 5.2 in a simple while loop runs circles around most things right now. It was released barely a month ago and many developers have been on vacation/hibernating/etc. during this time.
I give it 3-4 more weeks before we start to hear about the death of agentic frameworks. Pointing GPT5+ at a powershell or C#/Python REPL is looking way more capable than wiring up a bunch of domain-specific tools. A code-based REPL is the ultimate tool. You only need one and you can force the model to always call it (100% chance of picking the right tool). The amount of integration work around Process.Start is approximately 10-15 minutes, even if you don't use AI assistance.
I give it 3-4 more weeks before we start to hear about the death of agentic frameworks. Pointing GPT5+ at a powershell or C#/Python REPL is looking way more capable than wiring up a bunch of domain-specific tools. A code-based REPL is the ultimate tool. You only need one and you can force the model to always call it (100% chance of picking the right tool). The amount of integration work around Process.Start is approximately 10-15 minutes, even if you don't use AI assistance.
Yes this “REPL/CLI is all you need” realization is exactly what’s behind the wild success of Claude Code and derivative CLI coding agents.
My definition of agent has always been an LLM with "effectful" tools, run in a loop where the LLM gets to decide when the task is complete. In other words, an LLM with "agency".
This is exactly how I think of it. An agent has three elements: intelligence (LLM), autonomy (loop) and tools to do anything interesting/useful.
Whats the practical benefit of fine tune training on a local repo, vs putting the summary of local infomation in the context? i.e every team has their own style and preference for coding patterns that could be generalized - but i imagine a large scale model has seen fhem all so they could be described in the context, or are there specific domain level patterns that can be generalized that would never be seen outside an org so are difficult for a model to infer without fresh tunning?
I work on the biggest codebase in the world. We have a fine-tuned model on our codebase. I've not been impressed with it. It does not produce better code than the non-tuned model.
Maybe there's certain problems that it excels at but probably 99% of what I throw it at can be gleaned from the context/nearby code anyway, like you said. Even if I'm using some in-house library (pretty much all of our code), the models are good enough to dig into that library and read the headers if they need to.
Maybe it can help with speed? If it needs to do less research before it can start coding.
Maybe there's certain problems that it excels at but probably 99% of what I throw it at can be gleaned from the context/nearby code anyway, like you said. Even if I'm using some in-house library (pretty much all of our code), the models are good enough to dig into that library and read the headers if they need to.
Maybe it can help with speed? If it needs to do less research before it can start coding.
How many lines of code is there in the biggest codebase in the world?
Fine-tuning coder models is not nearly as effective as intelligently managing the context with frontier models (opus, gpt-5.2-codex).
I don't think it's even a question. A 32b model will not compete with SotA for years to come (if ever). The idea behind this release is to fine-tune on your codebase and compare to non-finetuned open models from the same class (or one higher). So if you need local processing, without access to SotA (security, compliance, whatever) then this is an interesting avenue for you. And the cost is fairly low. They are releasing the method to do this on your own codebase / docs / processes.
Prove it's the biggest codebase in the world. No way do you know that for sure!
"Hey Claude, please scaffold me the biggest codebase in the world"
Is this how you say "I work at Google" without explicitly saying that?
Awesome stuff. Output speed looks crazy fast too.
I wonder if this indeed will start prompting more language specific work.
Afaik training still requires not just looking at sample code but also being able to write loss functions being able to have problems the AI can work at. That seems hard.
One random thought, are there training styles of just deleting some code from "good" projects then making the AI make it work again?
I wonder if this indeed will start prompting more language specific work.
Afaik training still requires not just looking at sample code but also being able to write loss functions being able to have problems the AI can work at. That seems hard.
One random thought, are there training styles of just deleting some code from "good" projects then making the AI make it work again?
The technique people use is to capture PR diffs from public repos and extract the tests then use that to see if agents can reconstruct the patch that satisfies the tests.
One claim in article is definitely very wrong or at least needs to be narrowed. Claude is the only closed agent harness and there are about two dozen open ones. Many models may be closed, but when people say agent they are generally referring to the harness, not the underlying model.
Hey this looks great? Is it available on Openrouter.
I wish if AI2 could release a more denser model on Openrouter for free than the 8B model as I was using Devstral model for agentic purposes.
If we can get an agentic good 32B like model on openrouter for ~free, then I feel like it will be very interesting to see how things would go imo.
Good luck with AI2! The premise of truly open source models is really interesting and I feel like it could help bring more innovation in the space imo!
I wish if AI2 could release a more denser model on Openrouter for free than the 8B model as I was using Devstral model for agentic purposes.
If we can get an agentic good 32B like model on openrouter for ~free, then I feel like it will be very interesting to see how things would go imo.
Good luck with AI2! The premise of truly open source models is really interesting and I feel like it could help bring more innovation in the space imo!
Note that this is also a super interesting technique for specialising consumer facing apps like Lovable that need to generate code that matches your API very well.
It's also a great approach for building custom languages.
It's also a great approach for building custom languages.
For low cost tuning wouldn't something like LoRa via ie. unsloth on ie. GLM-4.7-Flash be the way to go?
it's great to see this kind of progress in reproducible weights, but color me confused. this claims to be better and smaller than Devstral-Small-2-24B, while clocking in at 32B (larger) and scoring more poorly?
Hey! We are able to outperform Devstral-Small-2-24B when specializing on repositories, and come well within the range of uncertainty with our best SERA-32B model. That being said, our model is a bit larger than Devstral 24B. Could you point out what in the paper gave the impression that we were smaller? If theres something unclear we would love to revise
"SERA-32B is the first model in Ai2's Open Coding Agents series. It is a state-of-the-art open-source coding agent that achieves 49.5% on SWE-bench Verified, matching the performance of much larger models like Devstral-Small-2 (24B)" from https://huggingface.co/allenai/SERA-32B
Ah great catch I don't know how we missed that. Thanks! Will fix.
So this "open" system still requires you to use Claude to actually use it?
No. You can point e.g. Opencode/Cline/Roo Code/Kilo Code at your inference endpoint. But CC has high install base and users are used to it, so it makes sense to target it.
augusteo(2)
Devstral Small 2 is an open-weights model: https://huggingface.co/mistralai/Devstral-Small-2-24B-Instru...