SWE-Bench Pro(github.com)
github.com
SWE-Bench Pro
https://github.com/scaleapi/SWE-bench_Pro-os
29 comments
Sir, we've already ingested 503,377 copyleft licensed codebases, I don't think the training set can take anymore!
Snark is against the rules but sometimes a good one-liner says more than a whole paragraph.
I happen to have worked on exactly this at Google. No, we don't train on restrictively-licensed code to the best of our abilities.
Out of curiosity, and IANAL, what is it in a GPL/copyleft license that would make it undesirable to train LLMs on projects with such license? Or are there stronger yet copyleft licenses that you had in mind?
(FWIW, and not directly related to my question, I always thought of GPL as the less (not more) restrictive license from the perspective of a user, because I could always ask for the source code and debug a problem on my own.)
(FWIW, and not directly related to my question, I always thought of GPL as the less (not more) restrictive license from the perspective of a user, because I could always ask for the source code and debug a problem on my own.)
I feel like public datasets are something we're holding onto with LLM benchmarks for historical reasons, but need to move on from.
Older, non-instruction tuned models needed post-training on public datasets to even reliably produce meaningful answers.
Now we're testing tasks that are so complex that the LLM should reasonably be expected to answer without additional post-training.
Once you have a public dataset, even feeding those examples to an LLM and producing synthetic variations is enough to let you game the benchmark. And the worst part is you don't need to be unethical to do this: some people would say it's just a good way to expand your training data even though it incidentally allows you to overfit on the task, without overfitting on the public dataset.
So everyone's doing stuff like that, and we're getting models that are increasing overfit to a few narrow tasks.
-
The alternative is just giving detailed plain english descriptions of the tasks in question. Those can be used to generate synthetic tasks, but won't result in matching the benchmark's "shape" perfectly (as long as the questions stay hidden), and that alone is enough to ensure some level of generalization takes place.
Older, non-instruction tuned models needed post-training on public datasets to even reliably produce meaningful answers.
Now we're testing tasks that are so complex that the LLM should reasonably be expected to answer without additional post-training.
Once you have a public dataset, even feeding those examples to an LLM and producing synthetic variations is enough to let you game the benchmark. And the worst part is you don't need to be unethical to do this: some people would say it's just a good way to expand your training data even though it incidentally allows you to overfit on the task, without overfitting on the public dataset.
So everyone's doing stuff like that, and we're getting models that are increasing overfit to a few narrow tasks.
-
The alternative is just giving detailed plain english descriptions of the tasks in question. Those can be used to generate synthetic tasks, but won't result in matching the benchmark's "shape" perfectly (as long as the questions stay hidden), and that alone is enough to ensure some level of generalization takes place.
https://scale.com/leaderboard/swe_bench_pro_commercial
I definitely trust the totally private dataset more.
I definitely trust the totally private dataset more.
One of the authors here -- we should clarify that strong copyleft license is a best attempt at decontamination for the public set. It's part of the tradeoff for having an open source set -- true decontamination is available with the private commercial set, but we can't release these, and if we did they'd be immediately susceptible to future contamination.
This is a key question in my opinion. It's one of the things that make benchmarking the SWE capabilities of LLMs difficult. It's usually impossible to know whether the LLM has seen a problem before, and coming up with new, representative problem sets is time-consuming.
You can just fuzz names and switch to a whitespace compact representation.
If you fuzz the names they won’t mean the same thing anymore, and then it’s no longer the same test. If you remove the whitespace the LLM will just run a formatter on the code. It’s not like the LLM just loads in all the code and then starts appending its changes.
I've never had a LLM try to run a formatter on my code with probably a few thousand hours logged driving agents (driving 4+ agents at once in most of those). Fuzzing makes the semantics slightly less immediately obvious, but LLMs are more robust to this than you or I, the biggest difference is the reduction in memorization carryover. If it feels like too different of a test for you, not sure what to tell you, but I know the world would appreciate a better way to test for training set contamination if you can figure one out.
And your basis for saying this is...?
I've done it? I have a benchmark called scramblebench that will do rewriting to evaluate model performance degradation with symbol replacement and layers of indirection.
If courts find model training and inference to be fair use of data sets, licenses mean nothing.
It looks like one court did in a non-precedent binding case, but I might be remembering incorrectly.
It looks like one court did in a non-precedent binding case, but I might be remembering incorrectly.
Not a chance. Even if American companies did abide by it, there is no reason Chinese companies would. And good luck definitely proving that a model trained on it.
Unless this is actually made by the SWE Bench team, and I see no evidence it is, this name is incredibly poor form. Just adding "Pro" to someone else's name not only is infringing on their mark, but implying yours is superior.
Looks like the associated article is: https://scale.com/research/swe_bench_pro (link in the repo is wrong)
> Larger models (e.g., Opus 4.1) often fail on semantic or
algorithmic correctness in large, multi-file edits, whereas smaller models (e.g., Qwen 3 32B) more frequently fail
due to issues in syntax and formatting, tool use, or context management.
While I haven’t dug into the details of this benchmark, this absolutely matches my personal experience.
Assuming “semantic correctness” is in the sense of Rice and runtime behavior.
While syntactic correctness has dramatically improved, security and architectural erosion and other long term issues have not.
Unfortunately Rice’s theorem also applies to finite programs in finite time too.
Actually it can apply to total functions in the general case.
I am still optimistic that coding agents will provide value long term in some fashion.
But the open domain frame problem simply reduces to the halting problem, yes and humans struggle with it too.
But fundamentally, PAC learning has to be reduced to _trivial_ problems, with only T/F.
We have found clever ways to work within these s limitations, but they still exist.
Hopefully we find clever ways to keep humans engaged with the code, while gaining the potential force multiplier that ML may offer.
The long tailed problems are particularly important, and while human SREs make mistakes and organizations often have constraints that add to the problem, SREs do a lot more to avoid those long tailed problems than they are given credit for.
IMHO that has always been one of the hardest parts of the industry and a true measure for what makes great team members.
Unfortunately the metrics and incentives often don’t capture that value.
While I haven’t dug into the details of this benchmark, this absolutely matches my personal experience.
Assuming “semantic correctness” is in the sense of Rice and runtime behavior.
While syntactic correctness has dramatically improved, security and architectural erosion and other long term issues have not.
Unfortunately Rice’s theorem also applies to finite programs in finite time too.
Actually it can apply to total functions in the general case.
I am still optimistic that coding agents will provide value long term in some fashion.
But the open domain frame problem simply reduces to the halting problem, yes and humans struggle with it too.
But fundamentally, PAC learning has to be reduced to _trivial_ problems, with only T/F.
We have found clever ways to work within these s limitations, but they still exist.
Hopefully we find clever ways to keep humans engaged with the code, while gaining the potential force multiplier that ML may offer.
The long tailed problems are particularly important, and while human SREs make mistakes and organizations often have constraints that add to the problem, SREs do a lot more to avoid those long tailed problems than they are given credit for.
IMHO that has always been one of the hardest parts of the industry and a true measure for what makes great team members.
Unfortunately the metrics and incentives often don’t capture that value.
Recently it was pointed out that models were sometimes finding SWE-Bench verified cheats by scanning parts of the repo not meant to be visible.
Hope they’re addressing that at the same time.
Hope they’re addressing that at the same time.
Silly, if you are going to come up with a new benchmark, then add capable models, they have Opus, Gemini Pro, and then Qwen3-32B.
Why not qwen3-coder-480b, qwen3-235b-instruct, deepseek-v3.1, kimi-k2, GLM-4.5, gpt-oss-120b?
Why not qwen3-coder-480b, qwen3-235b-instruct, deepseek-v3.1, kimi-k2, GLM-4.5, gpt-oss-120b?
Frankly, several repos and tools from Google/DeepMind look a lot better.
https://github.com/google-deepmind/bbeh?tab=readme-ov-file
https://github.com/google/lmeval
I hesitate to say this lest folks adapt, but does anyone else immediately distrust a repo when it has a bunch of emojis in the README? It is often a giveaway that they were LLM-generated.
https://github.com/google-deepmind/bbeh?tab=readme-ov-file
https://github.com/google/lmeval
I hesitate to say this lest folks adapt, but does anyone else immediately distrust a repo when it has a bunch of emojis in the README? It is often a giveaway that they were LLM-generated.
The public dataset only contains 3 or 4 languages.
go-280
python-266
js-165
ts-20
I hope in future the benchmark can cover other widely used languages, such as c++, java, swift, rust etc.
I hope in future the benchmark can cover other widely used languages, such as c++, java, swift, rust etc.
Is it possible to benchmark the GPT-5-Pro model?
would be nice to finally see multi-turn coding benchmarks. everything we have so far is single-turn and that's clearly not a realistic scenario.
why should we trust this benchmark more than an other for coding? geniune question, there are so many out there
Does it actually work? Isn’t AI training so far simply ignores all license and copyright restrictions completely?