Founder & CEO - LatentForce | Building LatentGraph - context layer for AI coding agents.| Interested in all things ML | Previously trained deep neural networks for a living and for fun |
I'd say the way to think about it is in terms of the questions you ask being in-distribution or out of distribution w.r.t the model training dataset.
Consider this, if something fundamental has changed in the world after the model was released(ie after the knowledge cut off date), then it would be very difficult for the model to reason about it. One concrete example is the the following: If you ask Opus or any decent coding model to do effort estimation on a coding task, then it would come up with multi week timelines - the models themselves doesn't know that because "they exist", these timelines have now been slashed to a few hours - you can try saying this in the prompt, however, they don't seem to internalise this.
There has been a lot of talk about how continual learning might be "just and engineering challenge" and that we could have agents that continuously learn from experience by just having longer and longer context windows.
What I am trying to argue for in the article is how such a view might be misplaced - just extending the context length and adding more instructions in the context will not get you continual learning - the representational capacity of weights will be the limiting factor.
Just a fun way to think about it. Would love to hear your thoughts.
Nope. Even if context can theoretically encode arbitrary computation under fixed weights, this requires the weights to implement a usable interpreter. Random weights almost surely do not. Training is what constructs that interpreter. Without it context has no meaningful computational semantics.
It's kind of like asking if I make a random circuit with logic gates, does that become a universal computer that can run programs.
I think both the views have their merits. In my mind the hardware vs software analogy for weights vs context holds better because in most modern computing systems, the hardware is fixed and the software changes. What the system can do efficiently, in practice, is a function of both the limitations/capabilities of the hardware and the software their respective capability ceilings.
The brain theory also kind of says the same thing, but it's hard to say what stays fixed vs changes with experience in the brain ig.
I spent the last weekend thinking about continual learning. A lot of people think that we can solve long term memory and learning in LLMs by simply extending the context length to infinity. I analyse a different perspective that challenges this assumption.
We ran this experiment a few weeks ago, but Anthropic’s Mythos report dropped this week and thought this would be relevant to share now.
Surprisingly, we found that for repos already in the training data(pre cut off set), the contamination is at such a high level that even by just giving the model the file name and not file contents, the model is able to tell what is inside that file. Same for file paths. Just given a file name, the model is able to correctly guess the file path - this implies that models already know the structure of these repos and thus understand what to look for and where.
This ability drops sharply for unseen repos(post cutoff set) - raises the question how effectively it will hold for private repos with proprietary scaffoldings and programming patterns. Then the question worth asking is how much of Mythos's capability on well known codebases like Firefox and OpenBSD is genuine reasoning vs parametric familiarity with their structure?
Methodology caveat: modest sample (9-10 repos per group), treat numbers as directional - more experiments in progress....
The 1-2 files per request constraint is interesting. I do the same
thing, but I've started thinking of it as a symptom rather than a
solution. The reason you have to constrain scope is because the
agent doesn't know what it doesn't know — it can't tell you "this
change will break something 4 layers away." So you manually limit
blast radius as a substitute for actual understanding.
Doesn't feel like that's where this should end up?
Personal superintelligence sounds nice until you actually try to use it.
We spent time yesterday arguing through an architecture decision. Today I ask the Agent to help implement it - it knows nothing about any of that. You’re effectively starting over.
Feels like the real problem isn’t intelligence, it’s continuity. And most benchmarks don’t even touch that.