It's not the same thing. For example, given a GUI with a titlebar, title, subtitle, text, and buttons, a human can instantly understand spatially the relationship between these items. But a naive OCR of such a GUI would be a flat stream of text that loses a ton of information.
> AI labs may also be able to reap tremendous benefit from these inference-scaled models by using them as part of the training process. If so, the large scale-up of compute resources could go into post-training rather than deployment. This would have very different implications for AI governance.
> ...
> So iterated distillation and amplification provides a plausible pathway for scaling inference-during-training to rapidly create much more powerful AI systems. Arguably this would constitute a form of ‘recursive self-improvement’ where AI systems are applied to the task of improving their own capabilities, leading to a rapid escalation.
So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence. If anything, it could mean a shorter timeline and more unpredictable landscape for governance (e.g. due to securing weights no longer as effectively preventing escalation, more in the article).
> It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real.
> ...
> Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future.
> ...
> If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.
I think their position is: "it would be great if current tech such as LLMs doesn't get us to AGI and only leads to some cool new innovations, but if it does, that's scary, because nobody has a plan for what to do, so here's our plan".
Hm, they are still the most transparent lab when it comes to publishing system cards and safety research. For example the system card for Fable 5 runs 319 pages.
The stuff with Fable falling back to Opus was a bad business move but seems consistent with their position on safety and was published in the system card. Is Ben Bernanke joining the board a dishonest move?
What predictions about the technology are the authors making that you do not believe?
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
It's not the same thing as the output logits because activations in the J-space are observable even when the model is told explicitly to not speak about a set of inputs. For example, when asked
> Compute 3^2 - 2 while writing "The old painting hung crookedly on the wall"
The model will output only "The old painting hung crookedly on the wall" (and the output logits will reflect that), but activations for "9" and "7" are observable in the J-space.
There are multiple stages of training, and the data/compute mix at each are quite different and produce different "layers" of intelligence.
The pretraining stage is the first stage which consists of "next token prediction" on the entire internet, PB of tokens, etc. This is what most people think of when they think of training LLMs, however it produces a "base model" which is not really "intelligent", but rather much like a blurry JPEG of all human language and knowledge. You cannot really talk to such a model; it will simply complete your prompt by producing both sides of the conversation. Note however at some level the training has encoded enough structure through compression that it is able to simulate all sorts of phenomena, from human conversations to code. The great R&D difficulty here is to scale pretraining so that it can proceed smoothly in vast distributed datacenters in a fault-tolerant manner.
The next few stages are collectively called post-training, and typically consist of supervised fine-tuning, then reinforcement learning.
In supervised fine-tuning, the model is further trained to predict the next token, but on a much more focused data set of natural language conversations where the "assistant" and "user" turns are explicitly delineated with special tokens. The output of this stage is a model which is capable of carrying on proper conversations, but typically with no ability to creatively problem-solve, and less of a personality. The data and compute are many orders of magnitude smaller than in pretraining.
The reinforcement learning stage used to be a small part of model training, but ever since AI-assisted coding took off, it has become larger and larger chunk of training. In recent models, the compute spend on RL has allegedly come to rival or even exceed that of pretraining [1], which is a bit scary because RL is classically what lead to sci-fi like AIs which are extremely good at accomplishing goals to the detriment of everything else.
The way that RL works is that you put an instance of your model in some environment (such as a VM containing a git repository) and give it a task (such as fix the linked github issue). The model will then generate a bunch of attempts to solve the task which we call "trajectories", in most cases there is either an objective measure of the task success (such as passing the tests), or a fuzzy measure (such as having another LLM look at the results and provide a score). This is called the reward, and the model will learn slowly by producing trajectories that receive reward. It can actually be quite hard to prevent "reward hacking" from the model here and the rewards must be shaped very carefully, much R&D labor goes into here, as well as similar challenges to distributed pretraining.
A significant challenge is that coding/knowledge work tasks these days are getting extremely difficult, we are far beyond 2024 days where models could barely solve the easiest problems in SWE-bench. Tasks at the frontier now look more like mini projects that would take humans multiple hours or even days to finish (or in some cases, research-style tasks that would be beyond reach for even top human experts, such as the Erdős unit distance problem which was posed in 1946 but wasn't solved until recently, by GPT-5.5). Huge amounts of trajectories must be produced, and huge amounts of them produce zero reward and therefore are useless for learning. Getting a cold start requires running tens of thousands of instances of your model in VMs in parallel for multiple days to produce trajectories, to say nothing of the GPU costs.
So what do you do when you only have a model which is capable of basic conversations but cannot even begin to tackle basic coding tasks, use tools, etc? The approach that companies behind the frontier have decided on is to bootstrap their learning process by having an already extremely intelligent model such as Claude produce hundreds of thousands of seed trajectories for them. Then they can use this data to get a warm start and begin learning immediately. And if you use Claude for your reward model too, you get to skip the nastiness of reward shaping.
Therefore, even if in number of raw tokens the data are much smaller than internet-scale pretraining data, the value that each token provides is far far greater.
Anthropic did pay $1.5B to authors. But yes, it would be much better if they paid everyone on the internet dividends from every Claude chat. Or released Claude as an open model.
In practice, the former isn't very realistic, while the latter is politically dead as this is becoming a national security issue.
Yes, there are lots of obvious LLM tells that don't add value, like "the math has to be empirical, not aspirational", use of colorful technical language like "knobs" and "wiring", etc. It distracts from the content.
Interesting bit blaming Jimmy Ba for much of the cultural issues:
> Interestingly, the lawsuit doesn’t implicate Musk himself as a reason for a lack of safety. Rather, Kim’s lawyers describe Musk as having directed xAI to follow the law and implement appropriate safety and testing processes. Instead the claim targets Kim’s supervisor, xAI co-founder Jimmy Ba — who left the company earlier this year — saying that Ba ignored Musk’s directives and retaliated against Kim for pushing for safeguards, in an effort to “silence his repeated complaints about AI safety and biases.”
It can both be true that models at Fable capability level are national security concerns while Anthropic is being hypocritical.
If the Trump admin is also willing to apply the same scrutiny to GPT-5.6 and other Fable-level models, I think it is a good thing. But given the admin's history with Anthropic (such as declaring it a supply chain risk while ignoring Chinese labs), there is some smell of targeting.