Schema Harness Achieves ~99% on Arc‑AGI‑3 Public(schema-harness.github.io)
schema-harness.github.io
Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
https://schema-harness.github.io/
85 comments
> To be clear, we’ll want to see how this performs against the hold-out set.
they could take open weight model, and check what will be impact from that harness on hold-out
they could take open weight model, and check what will be impact from that harness on hold-out
[deleted]
I think I'd typify it as "ARC-AGI doesn't matter" more than "harness matters". Or maybe "harness matters for some very specific tasks".
ARC-AGI is a bit of a joke at this point. The first version was supposed to be hard for neural nets because it had few examples and a super secret test set, with a different distribution than the public set no less. It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation, thanks to data augmentation techniques that increased the available training examples to the point that the private test set was eventually overcome.
ARC-AGI 2 went the same way because it was basically the same kind of dataset except this time with some attempt to further defend it against LLMs with restrictions on the compute budget. And now ARC-AGI 3 is saturated within ... what is it, weeks? since its release. The fact that it's the public set that's beaten doesn't matter, when the score is 99%. Systems that can score ~90% on the public sets of the previous ARC's can comfortably reach 70-80% on the corresponding private test sets, as far as my eyballing of results suggests.
It is time to accept that the whole idea of ARC is for the dustbin. It does not measure what it's supposed to measure -fluid intelligence, reasoning, whatever it is today. Its original premise, that a system could only beat ARC if it possessed human-like core knowledge systems (a-la Elizabeth Spelke's theory) has been comprehensively refuted: none of the systems that have ever performed well on any version of ARC has made any attempt to represent core knowledge systems in any way, shape or form.
Ultimately, if your machine intelligence (let alone AGI) test relies on tricks like only giving a few examples or keeping a secret test set to defend itself against the dominant approach to machine intelligence... then it's not a useful machine intelligence test. Or it just doesn't measure machine intelligence but... something else. Who knows what.
ARC-AGI 2 went the same way because it was basically the same kind of dataset except this time with some attempt to further defend it against LLMs with restrictions on the compute budget. And now ARC-AGI 3 is saturated within ... what is it, weeks? since its release. The fact that it's the public set that's beaten doesn't matter, when the score is 99%. Systems that can score ~90% on the public sets of the previous ARC's can comfortably reach 70-80% on the corresponding private test sets, as far as my eyballing of results suggests.
It is time to accept that the whole idea of ARC is for the dustbin. It does not measure what it's supposed to measure -fluid intelligence, reasoning, whatever it is today. Its original premise, that a system could only beat ARC if it possessed human-like core knowledge systems (a-la Elizabeth Spelke's theory) has been comprehensively refuted: none of the systems that have ever performed well on any version of ARC has made any attempt to represent core knowledge systems in any way, shape or form.
Ultimately, if your machine intelligence (let alone AGI) test relies on tricks like only giving a few examples or keeping a secret test set to defend itself against the dominant approach to machine intelligence... then it's not a useful machine intelligence test. Or it just doesn't measure machine intelligence but... something else. Who knows what.
Benchmarks are meant to measure something, so can't be too hard else all measurements will be 0. At the same time the systems being tested - LLMs - are getting larger and more capable, at least in the narrow areas most benchmarks are focusing on, so all benchmarks will continually become saturated and need to be revised.
So, are you against all benchmarks or specifically ARC AGI? At least ARC AGI is trying to test for something a bit different and not play to the text prediction strength of LLMs. It should go without saying that no single test, or type of test, can claim to test for AGI or human level intelligence, which would require a suite of tests as broad and varied as the generality of intelligence you are trying to test for.
So, are you against all benchmarks or specifically ARC AGI? At least ARC AGI is trying to test for something a bit different and not play to the text prediction strength of LLMs. It should go without saying that no single test, or type of test, can claim to test for AGI or human level intelligence, which would require a suite of tests as broad and varied as the generality of intelligence you are trying to test for.
>It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation
No it didn't. People tried big data memorization, and it didn't work. Base LLMs (even with millions of synthetic examples) never solved ARC-AGI-1.
It took a real algorithmic advancement - reasoning models - to solve it.
No it didn't. People tried big data memorization, and it didn't work. Base LLMs (even with millions of synthetic examples) never solved ARC-AGI-1.
It took a real algorithmic advancement - reasoning models - to solve it.
>> It took a real algorithmic advancement - reasoning models - to solve it.
You could only claim that if the numbers of parameters and training tokens remained constant while "reasoning" was added to the base models, which is not the case.
So if you look at the ARC-AGI-1 leaderboard (https://arcprize.org/leaderboard), you can clearly see that the bigger a model the better it performs, and that's for the "reasoning" models, e.g. looking at the graph, Claude Opus 4 is at ~30%, Opus 4.5 is between ~60% and ~80% and Claude 4.7 is at ~90% [1].
Not surprising: LLMs continue to improve in performance as long as more resources are spent to train them. "Algorithmic" advances would show the trend line going the other way, i.e. tokens and parameters decreasing steadily while performance either staying the same or improving.
If you've observed something like that I'll be happy to be corrected but I haven't.
____________________
[1] Incidentally, Opus 4.6 slightly outperforms 4.7 and 4.8 with their -alleged- reduced total parameter count. 4.6 is at 94.0% while 4.7 is at 93.5% and 4.8 at 92.5%.
You could only claim that if the numbers of parameters and training tokens remained constant while "reasoning" was added to the base models, which is not the case.
So if you look at the ARC-AGI-1 leaderboard (https://arcprize.org/leaderboard), you can clearly see that the bigger a model the better it performs, and that's for the "reasoning" models, e.g. looking at the graph, Claude Opus 4 is at ~30%, Opus 4.5 is between ~60% and ~80% and Claude 4.7 is at ~90% [1].
Not surprising: LLMs continue to improve in performance as long as more resources are spent to train them. "Algorithmic" advances would show the trend line going the other way, i.e. tokens and parameters decreasing steadily while performance either staying the same or improving.
If you've observed something like that I'll be happy to be corrected but I haven't.
____________________
[1] Incidentally, Opus 4.6 slightly outperforms 4.7 and 4.8 with their -alleged- reduced total parameter count. 4.6 is at 94.0% while 4.7 is at 93.5% and 4.8 at 92.5%.
I don't agree with your definition.
The point of reasoning models is that some tasks fundamentally require a certain number of serial steps. Base models are limited to learning parallel algorithms because of their parallel training, and so struggle on inherently-serial tasks like solving logic puzzles.
The advancement from reasoning is that it allows LLMs to learn a broader class of algorithms.
The point of reasoning models is that some tasks fundamentally require a certain number of serial steps. Base models are limited to learning parallel algorithms because of their parallel training, and so struggle on inherently-serial tasks like solving logic puzzles.
The advancement from reasoning is that it allows LLMs to learn a broader class of algorithms.
Sorry, which definition do you mean?
The problem with "reasoning" is that for the most part it happens outside of training e.g. as CoT. The base model knows what it knows, it knows what it's trained on, and it can't really go much farther than that.
Now, starting with o3, "reasoning" models are probably (who knows exactly) trained on traces of reasoning, either from automated systems or from human experts, but that doesn't mean they learn any kind of algorithm, just that they learn to reproduce the behaviour of "algorithms" (or of reasoning humans). That can improve performance on certain kinds of task (the ones in the training set) up to a point, but you're not going to make a tiny model perform like one ten times larger just by that.
On the other hand, I do think that LLMs can be made smaller without losing a commensurable amount of performance, like e.g. the Tiny Recursion Model (TRM) which did OK at ARC 1 (45% I think). But then you lose a lot of functionality also. Basically the larger models probably have more parameters than they really need and that's something the industry seems to have realised, but you still need huge parameter counts to reach top performance anyway.
And then there's the training tokens, which aren't getting any fewer.
The problem with "reasoning" is that for the most part it happens outside of training e.g. as CoT. The base model knows what it knows, it knows what it's trained on, and it can't really go much farther than that.
Now, starting with o3, "reasoning" models are probably (who knows exactly) trained on traces of reasoning, either from automated systems or from human experts, but that doesn't mean they learn any kind of algorithm, just that they learn to reproduce the behaviour of "algorithms" (or of reasoning humans). That can improve performance on certain kinds of task (the ones in the training set) up to a point, but you're not going to make a tiny model perform like one ten times larger just by that.
On the other hand, I do think that LLMs can be made smaller without losing a commensurable amount of performance, like e.g. the Tiny Recursion Model (TRM) which did OK at ARC 1 (45% I think). But then you lose a lot of functionality also. Basically the larger models probably have more parameters than they really need and that's something the industry seems to have realised, but you still need huge parameter counts to reach top performance anyway.
And then there's the training tokens, which aren't getting any fewer.
>but you're not going to make a tiny model perform like one ten times larger just by that.
Small reasoning models do indeed outperform base models that are 10x larger, at these logic/reasoning tasks that require serial computation.
They do not outperform at tasks that rely more on world knowledge or memorization.
In most cases the base model cannot complete logic tasks at all, or only for very small instances; it's reasoning or nothing.
> but that doesn't mean they learn any kind of algorithm
They do indeed learn algorithms and can step through them with CoT. This is what allows reasoning models to, e.g. reliably multiply large numbers by applying the grade-school multiplication algorithm.
Small reasoning models do indeed outperform base models that are 10x larger, at these logic/reasoning tasks that require serial computation.
They do not outperform at tasks that rely more on world knowledge or memorization.
In most cases the base model cannot complete logic tasks at all, or only for very small instances; it's reasoning or nothing.
> but that doesn't mean they learn any kind of algorithm
They do indeed learn algorithms and can step through them with CoT. This is what allows reasoning models to, e.g. reliably multiply large numbers by applying the grade-school multiplication algorithm.
DeepSeek V3.2 was tried without reasoning and it got 57% on ARC AGI 1. It's a 7 month model, so I'm pretty confident that base LLMs would be able to solve ARC AGI 1 without reasoning/CoT.
No, you are misunderstanding the paper.
https://arxiv.org/abs/2607.06764
The base model got 15%. They built an elaborate looping harness that allows it to burn 100k tokens "thinking" about the problem, which got the 57%. This is just an alternative approach to reasoning.
https://arxiv.org/abs/2607.06764
The base model got 15%. They built an elaborate looping harness that allows it to burn 100k tokens "thinking" about the problem, which got the 57%. This is just an alternative approach to reasoning.
I'm not referring to this paper, I'm referring to this leaderboard: https://arcprize.org/leaderboard. Set it to "arc agi 1", "base LLM" and you'll see deepseek at 57%. Submitted 2025-12-01, $0.120 per task. The paper you linked was later than that, and also says "We do not report an official ARC Prize leaderboard score".
So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and wasn't even tested on the verified set.
So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and wasn't even tested on the verified set.
I think this is an error in the leaderboard. Looking at the test logs, they had reasoning effort set to 'high'. So it should be in the CoT category instead of the base LLM category.
https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e...
https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e...
"kwargs": {
"max_tokens": 100000,
"stream": true,
"reasoning_effort": "high",
"rate_limit": {
"rate": 2,
"period": 60
}
}
Interesting, good find! Yeah I may be wrong and this may be an error in the leaderboard. Weirdly it shows no reasoning cost and no reasoning tokens used, but for example here https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e... the answer is super short but it says "4945" completion tokens.
Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required.
Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
I think there is a chasm to cross. the model's training to be aware of the harness it is in, at the same time building probes and observation tool can help it to cross that chasm.
Still seem too soon for a model to have that ability to build a harness on its own, and swap its session to another harness in the same environment. Like a snake shedding its skin, but in this case its harness.
Still seem too soon for a model to have that ability to build a harness on its own, and swap its session to another harness in the same environment. Like a snake shedding its skin, but in this case its harness.
How is custom engineering the tooling from scratch for every task the best path forward? There will always be some engineered tools that are better than others even when re-made by the same models - not to mention the cost of starting from scratch every single time just seems wasteful with current token spend.
Does "don't re-invent the wheel" not apply to agents for some reason?
Does "don't re-invent the wheel" not apply to agents for some reason?
> Does "don't re-invent the wheel" not apply to agents for some reason?
Correct, the purpose of agents is to become the ultimate wheel so we humans can stop needing to reinvent the wheel. For it to do that it needs to know how to reinvent wheels.
And, just to be clear, "don't re-invent the wheel" is just for small teams, as a whole humanity needs to re-invent the wheel all the time to adapt to new situations. For every wheel your team shouldn't reinvent some other teams main job is to reinvent that wheel.
Correct, the purpose of agents is to become the ultimate wheel so we humans can stop needing to reinvent the wheel. For it to do that it needs to know how to reinvent wheels.
And, just to be clear, "don't re-invent the wheel" is just for small teams, as a whole humanity needs to re-invent the wheel all the time to adapt to new situations. For every wheel your team shouldn't reinvent some other teams main job is to reinvent that wheel.
Humans (exhibiting "general intelligence") are tool builders; virtually all our capabilities stem from our ability to create and use tools - often extremely specialized to a task. Why would an artificial general intelligence be any different?
"don't reinvent the wheel" isn't a law of physics and I think is mostly said by people that never designed anything with wheels.
Bespoke tools can be simpler than general purpose ones
Indeed. And you can make this case about any tooling at all that is model adjacent.
Including by extension all programs, operating systems, or eventually hardware, I suppose.
Yep. That was the entire subtext from the drop in IBM's share price yesterday.
Codex and Claude Code already have the workflow feature which basically does this on demand, so it's getting there.
> Sol Ultra style is the path forward
That’s called brute force so not really.
That’s called brute force so not really.
Except in this case, it isn't yet smart enough. But I agree, building this capability in is coming, and will be really awesome.
It's likely smart enough. It just needs to be told to do it and provided the ability to introspect it. How close could foundational models get to building this harness if explicitly prompted to?
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
Let's maybe say not experienced enough / insufficiently RL'ed then -- 5.6 Sol did not reach for a harness solution like this when it got only 13% or son AGI-3 recently. I agree it's interesting to find the point in the prompting when it could 'tip' and do this. I have no instinct for where that point is, except that it must be somewhere, because I bet the Schema Harness was not hardcoded.
Letting the provider decide for the harness is a terrible idea in my eyes. Outsourcing harnessing is giving up control over the AI and equivalent to abandoning your sovereignty. It is a regression to a pre-enlightenment era.
In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
I've been trying this too: https://www.youtube.com/watch?v=brkP58pZ23w&list=PL8C_UWcLmv...
Cool to see others trying Baba Is You too.
Cool to see others trying Baba Is You too.
I'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.
Heh, probably something like this.
Works fast - tells people how to overthrow the government.
Follows all rules and conventions Google wants - says corporate speak without actually accomplishing anything.
Can actually do complicated things- apt to tell the user to fuck off and do the hard work themselves.
Training models seems more akin to raising a kid than a computer application.
Works fast - tells people how to overthrow the government.
Follows all rules and conventions Google wants - says corporate speak without actually accomplishing anything.
Can actually do complicated things- apt to tell the user to fuck off and do the hard work themselves.
Training models seems more akin to raising a kid than a computer application.
Forget Baba Is You, I want to see an LLM beat Elden Ring
FWIW: "Baba Is You" is 7 years old and heralded as one of the greatest puzzle games of all times, with guides and solutions shared all over the internet. How to beat this game is 100% in the training set.
It was our original assumption. Yet, we went through trajectories and agents did not recall solution. It is with a sharp contrast with task for which agents magically generate solution, e.g. https://openai.com/index/why-we-no-longer-evaluate-swe-bench....
In a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.
In a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.
It turns out knowledge and ability are not the same thing. We should test both.
With this harness or without?
it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it.
it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.
what this harness does is get the model to write a simulator first, it's measuring something entirely different.
it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.
what this harness does is get the model to write a simulator first, it's measuring something entirely different.
> State grounding turns raw observations into objects, variables, and relations that can be tracked. Mechanism discovery finds how that state changes under an action and writes the rule as an executable program
The way I'm reading this isn't that they are writing a game simulator, but rather that they have two things they are evolving - a perceptual model of the game mapping from pixels to objects, and a behavioral model of how each action acts upon these perceptual objects. The behavioral model is written as a program that can be backtested by the game states and actions they have already taken to see if they are correctly predicting the resulting next game state.
The ARC AGI 3 games are non-trivial, and I think it's very impressive to see them doing well using this approach.
I'd agree with their conclusion:
> We read a saturated ARC‑3 as the new beginning: mechanism discovery as a general capability — grounding the causal structure of a world through the agentic loop of action and perception, in environments far richer than a 64×64 grid. This is where we are heading to.
This is the way that an animal learns about it's environment - by observation (and innate biases) to recognize the objects in the environment, and predict their behavior, both autonomous (which AGI ARC 3 doesn't test - the objects in the environment are passive), and in reaction to the animal's behavior. The animal predicts and observes, updating its predictions when it is wrong.
A system that could do this in a messy, dynamic, real-world environment would seem a like genuine step in the direction of animal intelligence, especially if it could ditch the symbolic representations.
The way I'm reading this isn't that they are writing a game simulator, but rather that they have two things they are evolving - a perceptual model of the game mapping from pixels to objects, and a behavioral model of how each action acts upon these perceptual objects. The behavioral model is written as a program that can be backtested by the game states and actions they have already taken to see if they are correctly predicting the resulting next game state.
The ARC AGI 3 games are non-trivial, and I think it's very impressive to see them doing well using this approach.
I'd agree with their conclusion:
> We read a saturated ARC‑3 as the new beginning: mechanism discovery as a general capability — grounding the causal structure of a world through the agentic loop of action and perception, in environments far richer than a 64×64 grid. This is where we are heading to.
This is the way that an animal learns about it's environment - by observation (and innate biases) to recognize the objects in the environment, and predict their behavior, both autonomous (which AGI ARC 3 doesn't test - the objects in the environment are passive), and in reaction to the animal's behavior. The animal predicts and observes, updating its predictions when it is wrong.
A system that could do this in a messy, dynamic, real-world environment would seem a like genuine step in the direction of animal intelligence, especially if it could ditch the symbolic representations.
it's recursion
No - it's not the harness being updated, it's the code representing the action predictions (& perceptual model).
Writing a simulator requires understanding the game well enough to spec the simulator. Acquiring said understanding - within the action limits of the benchmark – seems like the heart of the challenge, so this doesn't strike me as "cheating" at all.
That is a fair assessment, but the "something else" is independently valuable, maybe even more so than model improvements - constructing an architecture for efficient rule determination and execution. In other words, I think the goal here isn't so much to beat Arc-AGI but to develop a generic improvement beyond "Ralph loop", which could dramatically extend frontier capabilities for all kinds of uses.
The simulator the model builds is comparable to the mental model of the game humans create. It is also much more efficient, GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3
> simulator the model builds is comparable to the mental model of the game humans create
>if you have the model simulate them they become trivial
Eh, this is kind of sounds like being a prey animal that develops an almost unbeatable colored camouflage and then the predator develops infrared vision making it useless and the prey saying "no fair, you cheated".
People use algorithmic models all the time on problems that are far too difficult or large for their minds to conceptualize, is this not just an extension of that?
Eh, this is kind of sounds like being a prey animal that develops an almost unbeatable colored camouflage and then the predator develops infrared vision making it useless and the prey saying "no fair, you cheated".
People use algorithmic models all the time on problems that are far too difficult or large for their minds to conceptualize, is this not just an extension of that?
similarly to how they are trivial for a human?
okay, what if this benchmark were just called Arc, and all you knew about it was what questions it asked? it just looks like a bunch of arcade games, which should tell you that it doesn't test AGI at all.
on the flip side, the idea that most tests are bad, even standardized tests, the tests that you scored well on that gave you all your opportunities in life: it cuts to the emotional, grounded core, the absolute foundation, of too many people. in the crowd of hacker news commenters; people who buy anthropic shares at retail; the people who work at tech companies; and their kids, families, etc., who are a bunch of nobodies, there are a lot more incentives to believe "stupid fucking arcade games test AGI" than not.
on the flip side, the idea that most tests are bad, even standardized tests, the tests that you scored well on that gave you all your opportunities in life: it cuts to the emotional, grounded core, the absolute foundation, of too many people. in the crowd of hacker news commenters; people who buy anthropic shares at retail; the people who work at tech companies; and their kids, families, etc., who are a bunch of nobodies, there are a lot more incentives to believe "stupid fucking arcade games test AGI" than not.
This is classic goalpost movement. Arc-AGI-3 was launched this year with roughly 0.5% success for frontier models. being able to 99% it in less than six months sets a new record for Arc-AGI saturation timeline. Speaking of singularity measures. It is definitely a big deal, not least in that Chollet needs to cancel his summer vacation and write Arc-AGI-4 now.
Arc AGI are simple games, the hardness comes from the input being basically adversarial to LLM training. if you use an LLM scaffold that removes the adversarial part you are measuring something else.
the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.
with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.
the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.
with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.
A great deal of mathematics is transforming nonlinear problems into linear ones and solving them with linear techniques. Others are solving non linear problems through stochastic methods. In almost all cases most non trivial math is done by transforming a harder problem into a simpler one.
I get what you mean in terms of testing the model itself to see its improvement in some domain. However if you can transform the domain to be better adapted to the model and achieve the desired results, this is indeed an accomplishment because a whole domain of problems is shown to be practically feasible with this technique without expensive model improvements. Of course the benchmark still exists without the harness, but the harness also exists which allows these problems to be solved.
As noted elsewhere the models themselves were used to build the harness, which means the models can in fact score this scores without intervention but building a harness for themselves adapted to the domain and using it. Is this cheating by the goal posts you’re setting?
There’s a real tension between “I want to solve problems and this technique shows how to solve the problem domain,” and the “I want to measure how something performs unassisted with other techniques.” Fortunately it’s not a mutually exclusive situation. You can do both simultaneously, gain the benefit of the technique to transform the problem into something tractable and keep measuring using the benchmark.
I get what you mean in terms of testing the model itself to see its improvement in some domain. However if you can transform the domain to be better adapted to the model and achieve the desired results, this is indeed an accomplishment because a whole domain of problems is shown to be practically feasible with this technique without expensive model improvements. Of course the benchmark still exists without the harness, but the harness also exists which allows these problems to be solved.
As noted elsewhere the models themselves were used to build the harness, which means the models can in fact score this scores without intervention but building a harness for themselves adapted to the domain and using it. Is this cheating by the goal posts you’re setting?
There’s a real tension between “I want to solve problems and this technique shows how to solve the problem domain,” and the “I want to measure how something performs unassisted with other techniques.” Fortunately it’s not a mutually exclusive situation. You can do both simultaneously, gain the benefit of the technique to transform the problem into something tractable and keep measuring using the benchmark.
To quote the people who make it:
> ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.
This harness does nothing to actually accomplish those goals.
It's a clever trick, sure, but you aren't allowed to use a calculator on your basic algebra tests in school for a reason.
> ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.
This harness does nothing to actually accomplish those goals.
It's a clever trick, sure, but you aren't allowed to use a calculator on your basic algebra tests in school for a reason.
I don't think we got continuous learning here, but we very specifically got interim goal setting and custom world models; the thinking traces demonstrate this round trip of building a world model, mental or coded, then stopping when reality doesn't correlate, then hypothesizing and creating a new model.
And this demonstrates this benchmark does not necessitate achieving those goals to achieve a perfect score. You seem to miss the point that almost all of math, physics, computer science is built on constructing an objective then cheating to attain it, and that demonstrates that there is an equivalency. Maybe the benchmark is flawed, or maybe the goals are not strictly necessary to attain.
For instance, is solving a math proof by enumerating all permutations exhaustively on a computer cheating? Does it matter that it is not a proof by construction? That its not descriptive? Of course not. The proof of the four color theorem is all that’s necessary and sufficient to prove it. Calculator at an algebra exam? Who cares. This isn’t an exam, this is the real world. The fact an AI can use a physics harness to perfectly achieve ARC-AGI-3 without attaining those goals demonstrates the power of the technique and that the goals are not necessary for that class of problems. Then find another benchmark that actually demands the goals be necessary and sufficient to achieve the benchmark goals. But don’t denigrate the fact we have technology today that yesterday was a fantasy.
For instance, is solving a math proof by enumerating all permutations exhaustively on a computer cheating? Does it matter that it is not a proof by construction? That its not descriptive? Of course not. The proof of the four color theorem is all that’s necessary and sufficient to prove it. Calculator at an algebra exam? Who cares. This isn’t an exam, this is the real world. The fact an AI can use a physics harness to perfectly achieve ARC-AGI-3 without attaining those goals demonstrates the power of the technique and that the goals are not necessary for that class of problems. Then find another benchmark that actually demands the goals be necessary and sufficient to achieve the benchmark goals. But don’t denigrate the fact we have technology today that yesterday was a fantasy.
The point of Arc-AGI-3 is to measure model performance. We already know that models can one-shot and iterate on very rudimentary game implementations. And, naturally, once it effectively has a copy of the source code, it can use that to play the game better.
This harness is really moving the goalpost by defeating the entire point of the test. Instead of seeing the strength of a model's world view, its ability to internally derive and intuit rules, and its ability to keep track of game state over time, we're just letting the AI cheat. This is just the LLM equivalent of running a chess engine to the side.
And this harness would not work in a remotely complex game and relies on the fact that Arc-AGI-3 is a focused test that only made the games as complicated as they needed to be for current model performance.
This harness is really moving the goalpost by defeating the entire point of the test. Instead of seeing the strength of a model's world view, its ability to internally derive and intuit rules, and its ability to keep track of game state over time, we're just letting the AI cheat. This is just the LLM equivalent of running a chess engine to the side.
And this harness would not work in a remotely complex game and relies on the fact that Arc-AGI-3 is a focused test that only made the games as complicated as they needed to be for current model performance.
I think this is just too simplistic a take; Arc-AGI-1 was wide open to all models, harnesses, etc, and had quite a lot of innovative structures implemented by hobbyists. At the time, this was seen as a good thing (it was), because we don't know the best system architecture for all sorts of problems right now -> innovation is good.
The games are designed to allow assessment of a system. Knowing better systems to solve the games is a step forward. If any of the frontier labs could have one-shotted -3 in March with a custom harness, they would have done so.
The games are designed to allow assessment of a system. Knowing better systems to solve the games is a step forward. If any of the frontier labs could have one-shotted -3 in March with a custom harness, they would have done so.
Sounds like a distinction between sport and work. How useful is pure model performance if it's known that there are conditions in which even greater performance can be achieved on real tasks? How useful is it to know how fast/far a person can run if they can ride a bicycle or drive a vehicle?
It's funny that the machine intelligences that arose out of the bitter lesson cracked this benchmark by meticulously modeling each individual case with rule based approaches.
When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use. But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?
When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use. But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?
Big jump for sure, but definitely comes with a giant grain of salt lacking open-sourcing the harness itself and measuring performance on the held-out set.
(1) What does it score on the private test set?
(2) Does this approach generalize to, e.g., Atari or NES games, or is it just hard-coding priors about the games into the model (as Chollet specifically warned was a chronic problem in benchmarks in the original Arc-AGI paper)
> Both scores come from a fixed fallback rule: Opus 4.8 and Sol xhigh run first; games scoring below 80 are rerun with Fable 5 and Sol max, respectively, and the higher per-game score is retained.
hmm, this is like pass@n until you get the high watermark? How would this mean anything?
hmm, this is like pass@n until you get the high watermark? How would this mean anything?
Do you throw every problem at your most expensive programmer first? Or do you toss most things to your fleet of juniors and let them knock out the easy work and pass the harder problems to your senior developers. Because that's what real world problems look like and work like in business.
Token costs are about the number one thing discussed by businesses these days.
Token costs are about the number one thing discussed by businesses these days.
How do humans get their results at these tasks? Don't they try until they succeed and claim their best run as their result?
I can't tell based on this article if the authors intend to submit this to Arc for the private / held-out set of games. The final section sorta seems like they won't bother because Arc-AGI-3 is now "saturated"
What does it mean to reach 99% score on Arc-AGI-3? That the agent is able to tackle difficult problems?
It doesn't necessarily mean anything to reach 99% on the public set. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models. ARC-AGI-3 is supposed to measure generalization to unseen games, so the only score that matters is the score on the held out private test set that nobody outside the ARC prize foundation has access to. Also, I believe the private set is significantly harder than the public set.
Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
This is not actually running the Arc-AGI-3 anymore. To summarize TFA:
1. The AI plays the game and records outputs.
2. The AI does TDD using those outputs to create its own copy of the game.
3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
To make an apt comparison... You go to get your chess ELO. You don't know chess at all and you're really bad at it, so you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.
1. The AI plays the game and records outputs.
2. The AI does TDD using those outputs to create its own copy of the game.
3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
To make an apt comparison... You go to get your chess ELO. You don't know chess at all and you're really bad at it, so you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.
>> 2. The AI does TDD using those outputs to create its own copy of the game.-
>> 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
In order to make a copy of the game "the AI" needs to first learn the rules of the game, otherwise the copy is not going to be very faithful and therefore not very useful.
Conversely if the copy is faithful to the game... that's it. The system has learned the rules of the game. As demonstrated by its ability to code a faithful copy of the game. Otherwise, how can it create a copy of the game?
>> 4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
That's precisely what it means to plan with a world model. The copy of the game is a world model and the system is predicting the outcome of its moves in the real game by first trying them out in the copy (i.e. "in its head"). That's exactly how, e.g. PDDL planning works: a planner has a model of an environment given in PDDL and it can try out its actions in that model, and choose an optimal sequence of actions to arrive at a goal state from an initial state, by observing the outcomes of its actions in the model.
It sounds like this "cheating" harness is actually doing something very reasonable and not cheating at all.
Have the ARC people (Chollet and friends) claimed that this result is cheating?
>> 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
In order to make a copy of the game "the AI" needs to first learn the rules of the game, otherwise the copy is not going to be very faithful and therefore not very useful.
Conversely if the copy is faithful to the game... that's it. The system has learned the rules of the game. As demonstrated by its ability to code a faithful copy of the game. Otherwise, how can it create a copy of the game?
>> 4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
That's precisely what it means to plan with a world model. The copy of the game is a world model and the system is predicting the outcome of its moves in the real game by first trying them out in the copy (i.e. "in its head"). That's exactly how, e.g. PDDL planning works: a planner has a model of an environment given in PDDL and it can try out its actions in that model, and choose an optimal sequence of actions to arrive at a goal state from an initial state, by observing the outcomes of its actions in the model.
It sounds like this "cheating" harness is actually doing something very reasonable and not cheating at all.
Have the ARC people (Chollet and friends) claimed that this result is cheating?
> model's ability to intuit game rules naturally, like a human
This line of thinking terrifies me for the future of humanity...
If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention).
Where this becomes really interesting is when the LLM can write novel tools to solve novel problems.
Trying to say that AI has to act like a human to be AGI is something that will end up with us humans going extinct from a digital intelligence we don't understand at all.
This line of thinking terrifies me for the future of humanity...
If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention).
Where this becomes really interesting is when the LLM can write novel tools to solve novel problems.
Trying to say that AI has to act like a human to be AGI is something that will end up with us humans going extinct from a digital intelligence we don't understand at all.
It is not the model now showing intelligence. Tis the dialogue. The medium is the LLM and the harness is then a next level construct.
We need to see private set results, but if this holds then it might represent a breakthrough in other domains as well.
I pretty much predicted this. If such smart models capable of doing math research fail so hard on such simple games the interface is the problem, not the model. Right harness provides a good interface between the problem and the intelligence.
[deleted]
> Schema, the harness we introduce today, reaches 99% on the ARC‑AGI‑3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol.
Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?
---
> When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.
BECs, SVT, Superfluid Quantum Gravity
Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.
(edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati...
> could not detect the medium light was supposed to wave in,
Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.
Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?
---
> When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.
BECs, SVT, Superfluid Quantum Gravity
Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.
(edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati...
> could not detect the medium light was supposed to wave in,
Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.
God, who the fuck are they even writing this slop for? Other machines?
Neat. Maybe even deeply interesting. Absolutely garbage write up.
Neat. Maybe even deeply interesting. Absolutely garbage write up.
I’m pretty excited to see what sort of generalization we come to over the next 12 months on the harness side: if it turns out this can be RLed in as ‘consider if building a world model might help here’ and we get this as another native capacity, that will be interesting. If we get 100 of those problem-solving strategies all included, feels like we will see another hurdle cleared in terms of usefulness.