This was extremely impressive to me. AtCoder has the hardest problems these days, usually the human onsite final round contestants can't solve more than 2 or 3 problems. This year the problem setter sets the round in a way that maximizes humans chance of winning. Then OpenAI just comes in and solves all problems..
No point for them to even go to IOI or ICPC this year anymore, those are all much easier than this Atcoder contest. And given the mathematical nature of AtCoder problems, not sure if there's any value to do IMO either, except for publicity of course.
SWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors.
But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.
And they'd be right, it's an almost saturated benchmark where even some subpar open source models score very well on. And most models are clustered within a small range so it really doesn't tell you much.
SWE-Bench pro is pretty much useless now even though many ppl still look at it. OpenAI published a report yesterday saying so as well. Only look at DeepSWE and FrontierCode right now for coding imo.
GPT usually performs better on DeepSWE while Claude does better on FrontierCode. These two coding benchmarks are pretty much the only ones right now that's still worth taking a look at imo.
Most programming is that, but most music and literature are probably uninspired junks as well. But there are many beautiful algorithms (such as the ones in Knuths books) that are more beautiful than any music for me personally.
I think at the current stage of LLM, it just doesn't make sense to ever have an annual sub. Things change too quickly that you really don't want to be stuck with one model.
Hmm your last sentence seems to exactly agree that it's a class of algos that parallelize well? What does sped up arbitrarily mean? It's still polynomial speed up right?
Even as a GLM z.ai fan, I wouldn't pay for their plans. They are just way worse values than gpt or anthropic plans, in terms of both usage and capabilities.
No point for them to even go to IOI or ICPC this year anymore, those are all much easier than this Atcoder contest. And given the mathematical nature of AtCoder problems, not sure if there's any value to do IMO either, except for publicity of course.