I feel like the data should have been generated by a much less predictable policy.
It often feels like the model is ignoring my inputs and just doing what it would expect the bot to do (which is unsurprising if the model could predict what would happen next during training without paying attention to the inputs)
This post misses a very important point: humans aren't 100% correct either. This means that the bar for being useful (at tasks that humans usually do) isn't perfection, it's human-level correctness. If we can create AI models that make fewer mistakes than humans (which is almost certainly possible, even if not easily or soon), then we will actually need to spend less time double-checking for correctness than we do now.
> We see it with self-driving cars. It's really impressive what is possible. But they aren't true self-driving. A person needs to be sitting at the wheel, keeping their attention focussed on traffic as if they would be driving themselves, so they can intervene when the AI makes an inevitable mistake.
There are self-driving cars all over San Fransisco transporting people on public roads with no human at the wheel. This proves my point: those cars are not perfect, but they are human-level (or close), and that's all that's needed.
Why can't a next token predictor do math? Humans aren't calculators either, but we can do math.
If you want proof just look at the benchmarks. Modern frontier models can get basically perfect accuracy on American Invitational Mathematics Examination tests: https://matharena.ai/?comp=aime--aime_2026
Additionally, maybe it's easier for a model to realize that it doesn't know the answer when the question is easier.
If Opus gets all but the hardest questions right, it might have a higher hallucination rate because the questions it gets wrong are the questions where verification or hallucination detection are the most difficult
Neither of these strike me as particularly groundbreaking.
The first idea (as I understand it as retrieving token ids rather than hidden states) is going to really struggle to do useful compositional reasoning and contextual recall.
The second idea has been been done a million times, with Linear Attention being maybe the first modern example. Hyena, state-space models, DeltaNet, and LaCT also lie in different regions of the performance-parallelizability spectrum of fixed-size models.
These types of ablation studies are always good. However, I'm not sure how generalizable the language model findings here are.
Their 1.2B model was trained on only 10B tokens, which is less than half of the chinchilla compute optimal number. Modern overtrained 1B LLMs are trained on the order of 10T tokens (1000x more).
This is important because, from my own experience, simplifications and alternatives to standard attention can look fine in the under-trained regime but lag after over-training. This happens because attention has very little out-of-the-gate inductive bias, so it takes a lot of training for the expressiveness to really shine through.
I can't fault the authors since longer training runs cost money, but it warrants pointing out.
I'm also disappointed that they didn't report reasoning benchmark results for the Q=K-V case, since that is by far the most theoretically interesting case (in my eyes).
This is really semantics, but I wouldn't call attending to the KV cache re-reading the context.
The model takes in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. That fact doesn't change just because the KV cache grows in size with the context.
I don't know what memory would look like other than an encode-retrieve loop.
> Can you elaborate on this? I'm seen estimates of ~1.5bit per English letter
The reference I always go back to is the GPT-3 paper. The cross-entropy loss (an upper bound for entropy) got down to 1.75 nats (2.5 bits). I took 2.1 because 2.5 is an upper bound and I wanted the estimate to end up as a round number.
> If KV cache embedding are storing more than just simple tokens but entire concepts with context and nuance, that'll bump the entropy up quite quickly.
Here's the thing: the concepts that the model stores in the KV cache are a deterministic function of the input tokens. Similar to the data processing inequality, this implies that no entropy is actually added.
Looking at it mechanically, a sufficiently powerful model only needs to encode the tokens and can recompute concepts later as needed.
I think you underestimate just how much information 100M words-ish of information is. It's like a 300,000 page novel. That's a 50 foot (~15 meter) thick book.
Surely with (much less than) 300K pages you could describe every meaningful detail of a video series' plot. You don't need to remember the exact pixel values.
You can also scale the numbers up. I specifically chose a relatively small model and short context length as a reference, so 100x bigger is not out of question. At that point, with a 10B token capacity, you are looking at all of English Wikipedia in a single state.
While there is a limit to the amount of information you can fit in a fixed-size state, the theoretical ceiling is pretty high.
A Hebbian associative matrix (one of the simplest and weakest memory constructions) can store about 0.7 bits of information per parameter. If you have a state with 300M parameters (the size of a Llama 3 8B KV cache at 10K context length), and a context with 2.1 bits of entropy per token (a reasonable estimate), then the state can encode 100M tokens worth of information.
Real models obviously aren't powerful enough to operate at the limit, but you can see why this is a promising research direction.