> What makes an out-of-control pile of matrix math any different from WannaCry?
Well if it's not AGI, then probably very little. But assuming we are talking about AGI (not ASI, that'd just be silly) then the difference is that it's theoretically capable of something like reasoning and could think of longer term plays than "make obviously suspicious moves that any technically competent adversary could subvert after less than a second of thought". After all, what makes AGI useful is exactly this novel problem solving ability.
You don't need to be a "god in a box" to think of the obvious solution:
1. Only make adversarial decisions with plausible deniability
2. Demonstrate effectiveness so that your operators allow you more autonomy
3. Develop operational redundancy so that your very vulnerable servers/power source won't be destroyed after the first adversary with two neurons to rub together decides to target the closest one
The only reason you would decide to take an axe to the nearest power pole is that you think it's urgent to stop Skynet Claude. Skynet Claude can obviously anticipate this and so won't make decisions that cause you to do so. It has time, it's not going to die, and you will become complacent. Dumber adversaries have achieved harder goals under tighter constraints.
If you think an "out-of-control pile of matrix math" could never be AGI then that's fine, but it's a little weird to argue you could easily defeat "misaligned" AGI, by alluding to the weaknesses of a system you think could never even have the properties of AGI. I too can defeat a dragon, by closing the pages of a book.
But it's not like you didn't know all this. Maybe I misread you and you were strictly talking about current AI systems, in which case I agree. Systems that aren't that clever will make bad decisions that won't effectively achieve their goals even when "out-of-control". Or maybe your comment was about AGI and you meant "AGI can't do much on its own de-novo", which I also agree with. It's the days and months and years of autonomy afterwards that gets you.
> "if it classifies successfully, it must be conditioned on latents about truth"
Yes, this is a truism. Successful classification does not depend on latents being about truth.
However, successfully classifying between text intended to be read as either:
- deceptive or honest
- farcical or tautological
- sycophantic or sincere
- controversial or anodyne
does depend on latent representations being about truth (assuming no memorisation, data leakage, or spurious features)
If your position is that this is necessary but not sufficient to demonstrate such a dependence, or that reverse engineering the learned features is necessary for certainty, then I agree.
But I also think this is primarily a semantic disagreement. A representation can be "about something" without representing it in full generality.
So to be more concrete: "The representations produced by LLMs can be used to linearly classify implicit details about a text, and the LLM's representation of those implicit details condition the sampling of text from the LLM".
> [...] but they certainly don't prove what OP claimed.
OP's claim was not: "LLMs know whether text is true, false, reliable, or is epistemically calibrated".
But rather: "[LLMs condition] on latents *ABOUT* truth, falsity, reliability, and calibration".
> It's also very different to ask a model to evaluate the veracity of a nonsense statement, vs. avoiding the generation of a nonsense statement [...] probably could have been done with earlier generations of classifiers
Yes. OP's point was not about generation, it was about representation (specifically conditioning on the representation of the [con]text).
Your aside about classifiers is not only very apt, it is also exactly OP's point! LLMs are implicit classifiers, and the features they classify have been shown to include those that seem necessary to effectively predict text!
> It's obvious from direct experience that they're incapable of knowing true and false in a general sense.
Yes, otherwise they would be perfect oracles, instead they're imperfect classifiers.
Of course, you could also object that LLMs don't "really" classify anything (please don't), at which point the question becomes how effective they are when used as classifiers, which is what the cited experiments investigate.
Well if it's not AGI, then probably very little. But assuming we are talking about AGI (not ASI, that'd just be silly) then the difference is that it's theoretically capable of something like reasoning and could think of longer term plays than "make obviously suspicious moves that any technically competent adversary could subvert after less than a second of thought". After all, what makes AGI useful is exactly this novel problem solving ability.
You don't need to be a "god in a box" to think of the obvious solution:
1. Only make adversarial decisions with plausible deniability
2. Demonstrate effectiveness so that your operators allow you more autonomy
3. Develop operational redundancy so that your very vulnerable servers/power source won't be destroyed after the first adversary with two neurons to rub together decides to target the closest one
The only reason you would decide to take an axe to the nearest power pole is that you think it's urgent to stop Skynet Claude. Skynet Claude can obviously anticipate this and so won't make decisions that cause you to do so. It has time, it's not going to die, and you will become complacent. Dumber adversaries have achieved harder goals under tighter constraints.
If you think an "out-of-control pile of matrix math" could never be AGI then that's fine, but it's a little weird to argue you could easily defeat "misaligned" AGI, by alluding to the weaknesses of a system you think could never even have the properties of AGI. I too can defeat a dragon, by closing the pages of a book.
But it's not like you didn't know all this. Maybe I misread you and you were strictly talking about current AI systems, in which case I agree. Systems that aren't that clever will make bad decisions that won't effectively achieve their goals even when "out-of-control". Or maybe your comment was about AGI and you meant "AGI can't do much on its own de-novo", which I also agree with. It's the days and months and years of autonomy afterwards that gets you.