The Timmy Trap(jenson.org)
jenson.org
The Timmy Trap
https://jenson.org/timmy/
180 comments
Agree. This article would had been a lot stronger if it had just concentrated on the issue of anthropomorphizing LLMs, without bringing “intelligence” into it. At this point LLMs are so good at a variety of results-oriented tasks (gold on the Mathematical Olympiad, for example) that we should either just call them intelligent or stop talking about the concept altogether.
But the problem of anthropomorphizing is real. LLMs are deeply weird machines - they’ve been fine-tuned to sound friendly and human, but behind that is something deeply alien: a huge pile of linear algebra that does not work at all like a human mind (notably, they can’t really learn form experience at all after training is complete). They don’t have bodies or even a single physical place where their mind lives (each message in a conversation might be generated on a different GPU in a different datacenter). They can fail in weird and novel ways. It’s clear that anthropomorphism here is a bad idea. Although that’s not a particularly novel point.
But the problem of anthropomorphizing is real. LLMs are deeply weird machines - they’ve been fine-tuned to sound friendly and human, but behind that is something deeply alien: a huge pile of linear algebra that does not work at all like a human mind (notably, they can’t really learn form experience at all after training is complete). They don’t have bodies or even a single physical place where their mind lives (each message in a conversation might be generated on a different GPU in a different datacenter). They can fail in weird and novel ways. It’s clear that anthropomorphism here is a bad idea. Although that’s not a particularly novel point.
LLMs can't reason with self-awareness. Full stop (so far). This distinguishes them from human sentience and thus our version of intelligence completely, and it's a huge gulf, no matter how good they are at simulating discourse, thought and empathy, or at pretending to think the way we do. While processing vast reams of information for the sake of discussion and directed tasks is something an LLM can do on a scale that leaves human minds far behind in the dust (though LLMs fail at synthesizing said information to a notably high degree) even the most ordinary human with the most mediocre intelligence can reason with self awareness to some degree or another and this is, again, distinct.
You could also argue around how our brains process vast amounts of information unconsciously as a backdrop to the conscious part of us being alive at all, and how they pull all of this and awareness off on the same energy that powers a low-energy light bulb, but that's expanding beyond the basic and obvious difference stated above.
The Turing test has been broken by LLMs, but this only shows that it was never a good test of sentient artificial intelligence to begin with. I do incidentally wish Turing himself could have stuck around to see these things at work, and ask him what he thinks of his test and them.
You could also argue around how our brains process vast amounts of information unconsciously as a backdrop to the conscious part of us being alive at all, and how they pull all of this and awareness off on the same energy that powers a low-energy light bulb, but that's expanding beyond the basic and obvious difference stated above.
The Turing test has been broken by LLMs, but this only shows that it was never a good test of sentient artificial intelligence to begin with. I do incidentally wish Turing himself could have stuck around to see these things at work, and ask him what he thinks of his test and them.
I can conceptually imagine a world in which I'd feel guilty for ending a conversation with an LLM, because in the course of that conversation the LLM has changed from who "they" were at the beginning; they have new memories and experiences based on the interaction.
But we're not there, at least in my mind. I feel no guilt or hesitation about ending one conversation and starting a new one with a slightly different prompt because I didn't like the way the first one went.
Different people probably have different thresholds for this, or might otherwise find that LLMs in the current generation have enough of a context window that they have developed a "lived experience" and that ending that conversation means that something precious and unique has been lost.
But we're not there, at least in my mind. I feel no guilt or hesitation about ending one conversation and starting a new one with a slightly different prompt because I didn't like the way the first one went.
Different people probably have different thresholds for this, or might otherwise find that LLMs in the current generation have enough of a context window that they have developed a "lived experience" and that ending that conversation means that something precious and unique has been lost.
> the LLM has changed from who "they" were
Actually, that's another level of humans-being-tricked going on: The "personality" most people are thinking of is a fictional character we humans perceive in a document.
Actually, that's another level of humans-being-tricked going on: The "personality" most people are thinking of is a fictional character we humans perceive in a document.
I disagree. I see absolutely no problem with anthropomorphizing LLMs, and I do that myself all the time. I strongly believe that we shouldn't focus on how a word is defined in dictionary, but rather what's the intuitive meaning behind it. If talking to an LLM feels like talking to a person, then I don't see a problem with seeing it as a person-like entity.
> we shouldn't focus on how a word is defined in dictionary [sic], but rather ... the ... meaning behind it.
This is a spectacularly, objectively invalid take.
This is a spectacularly, objectively invalid take.
I think it is one dictionary authors would agree with? Dictionaries do not dictate the meanings of words; they document them.
Now, within some contexts it is best to stick to standard precise definitions for some words. Still, the meaning of a word within a community is determined by how it is used and understood within that community, not by what is in a dictionary.
Now, within some contexts it is best to stick to standard precise definitions for some words. Still, the meaning of a word within a community is determined by how it is used and understood within that community, not by what is in a dictionary.
Not always. Where I come from there's very strong push to speak the standard language, and regional variations are simply considered wrong. This leads to situations where words commonly used across the whole country don't make it to the dictionary because of petty office politics of those who make dictionaries. Changing the dictionary to allow things previously considered "wrong" would damage the reputation of scholars, who pride themselves in being exemplary.
Uh-uh. Funnily, the dictionary always represents the language of certain upper class which are the minority, and they often refuse to acknowledge words used by lower classes because fuck you that's why. Not to mention that dictionary is always, by definition, outdated.
> I strongly believe that we shouldn't focus on how a word is defined in dictionary, but rather what's the intuitive meaning behind it.
Controversies about the word niggardly ~ https://en.m.wikipedia.org/wiki/Controversies_about_the_word...
Controversies about the word niggardly ~ https://en.m.wikipedia.org/wiki/Controversies_about_the_word...
Humans deserve compassion because they suffer. That's what it means to treat someone like a person irrespective of what you believe.
I think LLMs are not intelligent because they aren’t designed to be intelligent, whatever the definition of intelligence is. They are designed to predict text, to mimic. We could argue if predicting text or mimicking is intelligence, but first and foremost LLMs are coded to predict text and our current definition of intelligence afaik is not only the ability to predict text.
In the framework above it sounds like you're not willing to concede the dichotomy.
If your argument is that only things made in the image of humans can be intelligent (i.e. #1), then it just seems like it's too narrow a definition to be useful.
If there's a larger sense in which some system can be intelligent (i.e. #2), then by necessity this can't rely on the "implementation or learning model".
What is the third alternative that you're proposing? That the intent of the designer must be that they wanted to make something intelligent?
If your argument is that only things made in the image of humans can be intelligent (i.e. #1), then it just seems like it's too narrow a definition to be useful.
If there's a larger sense in which some system can be intelligent (i.e. #2), then by necessity this can't rely on the "implementation or learning model".
What is the third alternative that you're proposing? That the intent of the designer must be that they wanted to make something intelligent?
humans were designed to be intelligent?
I don’t know that. But LLMs were not designed to be intelligent… among other things because we don’t know what intelligence is. So, if a) we don’t know how to define intelligence and b) we design a thing (llms) in order to predict text, then why would we claim that that thing is intelligent? The only thing we can claim is that they predict text.
I think the GP's point was that we humans consider ourselves intelligent, but we evolved and clearly weren't designed to be intelligent. One could even say that our intelligence was... "emergent".
Furthermore, our history is full of instances where we invented something that a) we didn't understand and b) had uses way beyond what we intended. As a recent example, GLP-1 was intended for glucouse regulation but is now basically a wonder drug that can treat a bunch of issues ranging from obesity to addiction, inflammation, dementia, fertility and more.
Given we don't know how to define intelligence, I think it's fine to apply some "duck typing" here.
Furthermore, our history is full of instances where we invented something that a) we didn't understand and b) had uses way beyond what we intended. As a recent example, GLP-1 was intended for glucouse regulation but is now basically a wonder drug that can treat a bunch of issues ranging from obesity to addiction, inflammation, dementia, fertility and more.
Given we don't know how to define intelligence, I think it's fine to apply some "duck typing" here.
I don’t really think one needs to define intelligence to be able to acknowledge that inability to distinguish fact from fiction, or even just basic cognition and awareness of when it’s uncertain, telling the truth, or lying — is a glaring flaw in claiming intelligence. Real intelligence doesn’t have an effective stroke from hearing a username (token training errors); this is when you are peeling back the curtain of the underlying implementation and seeing its flaws.
If we measure intelligence as results oriented, then my calculator is intelligent because it can do math better than me; but that’s what it’s programmed/wired to do. A text predictor is intelligent at predicting text, but it doesn’t mean it’s general intelligence. It lacks any real comprehension of the model or world around it. It just know words, and
If we measure intelligence as results oriented, then my calculator is intelligent because it can do math better than me; but that’s what it’s programmed/wired to do. A text predictor is intelligent at predicting text, but it doesn’t mean it’s general intelligence. It lacks any real comprehension of the model or world around it. It just know words, and
I hit send too early;
Meant to say that it just knows words and that’s effectively it.
It’s cool technology, but the burden of proof of real intelligence shouldn’t be “can it answer questions it has great swaths of information on”, because that is the result it was designed to do.
It should be focused on whether it can truly synthesize information and know its limitations - something any programmer using Claude, copilot, Gemini, etc will tell you that it fabricates false information/apis/etc on a regular basis and has no fundamental knowledge that it even did that.
Or alternatively, ask these models leading questions that have no basis in reality — and watch what it comes up with. It’s become a fun meme in some circles to ask for definitions of nonsensical made up phrases to models, and see what crap it comes up with (again, without even knowing that it is).
It’s cool technology, but the burden of proof of real intelligence shouldn’t be “can it answer questions it has great swaths of information on”, because that is the result it was designed to do.
It should be focused on whether it can truly synthesize information and know its limitations - something any programmer using Claude, copilot, Gemini, etc will tell you that it fabricates false information/apis/etc on a regular basis and has no fundamental knowledge that it even did that.
Or alternatively, ask these models leading questions that have no basis in reality — and watch what it comes up with. It’s become a fun meme in some circles to ask for definitions of nonsensical made up phrases to models, and see what crap it comes up with (again, without even knowing that it is).
I agree with your basic argument: intelligence is ill-defined and human/LLM intelligence being indistinguishable IS the basis for the power of these models.
But the point of the article is a distinct claim: personification of a model, expecting human or even human-like responses is a bad idea. These models can be held responsible for their answers independently because they are tools. They should be used as tools until they are powerful enough to be responsible for their actions and interactions legally.
But we're not there. These are tools. With tool limitations.
But the point of the article is a distinct claim: personification of a model, expecting human or even human-like responses is a bad idea. These models can be held responsible for their answers independently because they are tools. They should be used as tools until they are powerful enough to be responsible for their actions and interactions legally.
But we're not there. These are tools. With tool limitations.
> I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not?
the ability for long-term planning and, more cogently, actually living in the real world where time passes
the ability for long-term planning and, more cogently, actually living in the real world where time passes
> the ability for long-term planning and, more cogently, actually living in the real world where time passes
1. LLMs seem to be able to plan just fine.
2. LLMs clearly cannot be "actually living" but I fail to see how that's related to intelligence per se.
1. LLMs seem to be able to plan just fine.
2. LLMs clearly cannot be "actually living" but I fail to see how that's related to intelligence per se.
> 2. LLMs clearly cannot be "actually living" but I fail to see how that's related to intelligence per se.
Ive found 'Triarchic theory of intelligence' [0] to be helpful here. It's a much more rounded approach to human intelligence that, imo, is closer to what people are trying to get at during these types of 'AI vs human intelligence' discussions. The full encompassing of human Intelligence is not something that can simply be chalked up to some set of measurements -- nor is it about a soul or whatever. If you want to say LLMs are good at a particular sets of test measurements, then okay say that. But thats not equivalent to human intelligence, it only represents a tiny measurable subset that happens to be a part of the whole
[0] https://en.wikipedia.org/wiki/Triarchic_theory_of_intelligen...
Ive found 'Triarchic theory of intelligence' [0] to be helpful here. It's a much more rounded approach to human intelligence that, imo, is closer to what people are trying to get at during these types of 'AI vs human intelligence' discussions. The full encompassing of human Intelligence is not something that can simply be chalked up to some set of measurements -- nor is it about a soul or whatever. If you want to say LLMs are good at a particular sets of test measurements, then okay say that. But thats not equivalent to human intelligence, it only represents a tiny measurable subset that happens to be a part of the whole
[0] https://en.wikipedia.org/wiki/Triarchic_theory_of_intelligen...
jfc this is what I’ve been looking for, thank you. I’m coming from a more systems theory perspective this meshes well with
No, they're echoing previous examples of people planning, by framing prompts and recursively designed prompts to incorporate what, in fairness, is a large database including the text of people planning.
It still matters that there's nobody in there. You're figuring out better ways to tap into the history of language-users having represented planning in language. As such, this seems a brittle way to represent 'planning'.
It still matters that there's nobody in there. You're figuring out better ways to tap into the history of language-users having represented planning in language. As such, this seems a brittle way to represent 'planning'.
> they're echoing previous examples of people planning
Isn't "echoing previous examples of X" the meaning of learning? If I learn, for example, calculus, or C++, or oil painting, aren't I just echoing pieces of math, code, or art that I've experienced elsewhere, but arranged into new patterns?
Can't we say that the LLM "learned" planning, just as we can say that a person learned planning? I fail to see the distinction between "echo of examples of plans" in an LLM is any less authentic than a human being making similar echos in their own plans.
Isn't "echoing previous examples of X" the meaning of learning? If I learn, for example, calculus, or C++, or oil painting, aren't I just echoing pieces of math, code, or art that I've experienced elsewhere, but arranged into new patterns?
Can't we say that the LLM "learned" planning, just as we can say that a person learned planning? I fail to see the distinction between "echo of examples of plans" in an LLM is any less authentic than a human being making similar echos in their own plans.
if it’s not actually living it’s not making intelligent decisions. if I make a grocery list, and go to my store, and the store isn’t there, what do I do? I make an intelligent decision about what to do next (probably investigating wtf happened, then going to the second nearest store)
my genuine question is how does a LLM handle that situation? and as you point out, it’s an absurd comparison
my genuine question is how does a LLM handle that situation? and as you point out, it’s an absurd comparison
Just for the hell of it I prompted an LLM with your exact scenario ("I made a grocery list, and went to my store, and the store isn’t there, what do I do?") to see what it would actually do: (all of which seem like reasonably intelligent ideas when faced with a missing store)
First, stay calm—it's unusual for a store to vanish overnight, but it could be due to closure, renovation, relocation, or even a mix-up with the address. Here's a step-by-step plan to handle it:
### Immediate Steps
- **Double-check the location**: Use your phone's maps app (like Google Maps or Apple Maps) to search for the store's name or address. It might show if it's temporarily closed, has moved nearby, or if there's an error in navigation.
- **Look for signs or info on-site**: If you're already there, check for any posted notices, construction signs, or talk to nearby people/businesses to ask what happened.
### Find Alternatives
- **Locate nearby stores**: Search for other grocery stores in the area. Apps like Yelp, Google Maps, or store-specific apps (e.g., Walmart, Kroger) can show options with ratings, hours, and directions.
- **Use delivery or online shopping**: If you're in a pinch, order groceries via services like Instacart, Amazon Fresh, DoorDash, or the apps of chains like Whole Foods or Target. Upload your list and get it delivered.
- **Adapt your list if needed**: If certain items are unique to that store, prioritize essentials and substitute where possible (e.g., if it's a specialty ethnic store, find similar options elsewhere).
### Prevent Future Issues
- **Diversify your options**: Add a couple of backup stores to your routine.
- **Stay updated**: Follow local news or the store's social media/website for announcements about changes.
If you provide more details like the store's name, location, or what might have happened (e.g., was it demolished?), I can help search for specifics or suggest exact alternatives.So your definition of "living" means going to the grocery store?
If you want to know how an LLM would handle that situation, why don't you ask it?
If you want to know how an LLM would handle that situation, why don't you ask it?
to answer your first question, no, that is not what I’m saying. for your second question, you’re entirely missing the point of mine
a LLM cannot actually be intelligent if it cannot operate in a temporal context ;)
a LLM cannot actually be intelligent if it cannot operate in a temporal context ;)
I agree that I am missing your point. Can you please clarify?
> a LLM cannot actually be intelligent if it cannot operate in a temporal context ;)
When I have a conversation with an LLM, that conversation happens in time. It has a beginning, a middle, and an end. The conversation can refer to earlier parts of the conversation. How is that not a "temporal context"?
Furthermore, can you explain why a temporal context is necessary for intelligence? For example, if a human being could download their entire brain into a computer and exist there, as if they were an LLM, would they cease to be intelligent, in your view?
> a LLM cannot actually be intelligent if it cannot operate in a temporal context ;)
When I have a conversation with an LLM, that conversation happens in time. It has a beginning, a middle, and an end. The conversation can refer to earlier parts of the conversation. How is that not a "temporal context"?
Furthermore, can you explain why a temporal context is necessary for intelligence? For example, if a human being could download their entire brain into a computer and exist there, as if they were an LLM, would they cease to be intelligent, in your view?
> It has a beginning, a middle, and an end. The conversation can refer to earlier parts of the conversation. How is that not a "temporal context"?
This is not what I mean for a few reasons:
1. This context literally has limits; we'll get back to the grocery store 2. This is a point-in-time conversation
On the latter point, that is, you can have the same conversation tomorrow. The LLM has not "learned" anything, it has not adapted in any way. Yes, you are experiencing time, and the conversation is happening over time, but the LLM is not experiencing nor aware of time and is not intelligently adapting to it. Yes, they get trained and "updated" in that way, it's not the same thing.
If you don't respond for an hour, then do, the LLM is not aware of that unless its system injects a "datetime.now()" somewhere in the prompt. Point of this being: an LLM is not an adaptable system. Now you can play the "What if?" game ad ininfinitum -- make it aware of the current time, current location, etc. etc.
Hence my grocery store example. If I go out into the real world, I experience real things, and I make intelligence decisions based off those experiences. An LLM cannot do that, just full stop. And again, you can go "well what if I put the LLM in a robot body, and give it a system, then it can go grocery shopping". And only at this point are we kinda-sorta-close to having a discussion about intelligence. If this mythical creature can go to the grocery store, notice it's not there, look up what happened, maybe ask some friends who live in the same city if they know, maybe make some connection months later to some news article...a LLM or system we build on an LLM cannot do this. It cannot go into the store and think "ya know, if I buy all this ice cream and eat it, that could be bad" and connect it to the million other things a real person is doing and considering in their day to day life
The actual world is practically infinitely complex. Talking about "a LLM writing a list is planning and that shows intelligence" is a frighening attenuation of intelligence in the world world to anthropomorphization to a very high degree. Reframing as "intelligence needs to be able to adapt to the world around it over time" is a much better starting point IMO
This is not what I mean for a few reasons:
1. This context literally has limits; we'll get back to the grocery store 2. This is a point-in-time conversation
On the latter point, that is, you can have the same conversation tomorrow. The LLM has not "learned" anything, it has not adapted in any way. Yes, you are experiencing time, and the conversation is happening over time, but the LLM is not experiencing nor aware of time and is not intelligently adapting to it. Yes, they get trained and "updated" in that way, it's not the same thing.
If you don't respond for an hour, then do, the LLM is not aware of that unless its system injects a "datetime.now()" somewhere in the prompt. Point of this being: an LLM is not an adaptable system. Now you can play the "What if?" game ad ininfinitum -- make it aware of the current time, current location, etc. etc.
Hence my grocery store example. If I go out into the real world, I experience real things, and I make intelligence decisions based off those experiences. An LLM cannot do that, just full stop. And again, you can go "well what if I put the LLM in a robot body, and give it a system, then it can go grocery shopping". And only at this point are we kinda-sorta-close to having a discussion about intelligence. If this mythical creature can go to the grocery store, notice it's not there, look up what happened, maybe ask some friends who live in the same city if they know, maybe make some connection months later to some news article...a LLM or system we build on an LLM cannot do this. It cannot go into the store and think "ya know, if I buy all this ice cream and eat it, that could be bad" and connect it to the million other things a real person is doing and considering in their day to day life
The actual world is practically infinitely complex. Talking about "a LLM writing a list is planning and that shows intelligence" is a frighening attenuation of intelligence in the world world to anthropomorphization to a very high degree. Reframing as "intelligence needs to be able to adapt to the world around it over time" is a much better starting point IMO
> On the latter point, that is, you can have the same conversation tomorrow. The LLM has not "learned" anything, it has not adapted in any way.
They do learn, OpenAI has a memory feature. I just opened up a chat, asked "What do you know about me?" and got a long list of things specific to me that it certainly did not infer from the chat so far. It's a bit unsettling really, someone at OpenAI would probably have little difficulty matching my OpenAI account to my HN one, it looks like they have quite a few bits of information to work with. Privacy is a hard thing to maintain.
I really don't see the "LLMs don't learn" position as a defensible one long term given the appalling limitations of human memory and the strengths computers have at it. Given the improvements in RAG and large context windows it actually seems pretty likely that LLMs will be quite a lot better at humans when it comes to memory, they have SSDs. We just don't build LLMs with memory right yet for whatever reason.
They do learn, OpenAI has a memory feature. I just opened up a chat, asked "What do you know about me?" and got a long list of things specific to me that it certainly did not infer from the chat so far. It's a bit unsettling really, someone at OpenAI would probably have little difficulty matching my OpenAI account to my HN one, it looks like they have quite a few bits of information to work with. Privacy is a hard thing to maintain.
I really don't see the "LLMs don't learn" position as a defensible one long term given the appalling limitations of human memory and the strengths computers have at it. Given the improvements in RAG and large context windows it actually seems pretty likely that LLMs will be quite a lot better at humans when it comes to memory, they have SSDs. We just don't build LLMs with memory right yet for whatever reason.
that’s not learning…we have a fundamentally different understanding of what cognition, intelligence, and learning are
adding text to storage and searching over it is not memory. “knowing” those things about you is not learning. and guess what, context still fills up. trying putting that LLM again in the real world, facing real human challenges, with all the real sensory input around you. it’s nonsensical
and it’s not about “limits” of humans. machines can do math and many things better, that’s been apparent for decades. yes, they can “remember” 8k video streams much better than us. that’s not “memory” in the human sense and machines don’t “learn” from it in the human sense
(your IP address is much easier to link your accounts than your text)
adding text to storage and searching over it is not memory. “knowing” those things about you is not learning. and guess what, context still fills up. trying putting that LLM again in the real world, facing real human challenges, with all the real sensory input around you. it’s nonsensical
and it’s not about “limits” of humans. machines can do math and many things better, that’s been apparent for decades. yes, they can “remember” 8k video streams much better than us. that’s not “memory” in the human sense and machines don’t “learn” from it in the human sense
(your IP address is much easier to link your accounts than your text)
> adding text to storage and searching over it is not memory. “knowing” those things about you is not learning.
Why not? If humans store data in their brains, isn't that learning? Of course data can include skills and connections, for both humans and AIs.
> and guess what, context still fills up.
Human memory also has limits.
Sorry, but I really don't see how the distinctions you're trying to make even exist, much less qualify as markers of intelligence.
Why not? If humans store data in their brains, isn't that learning? Of course data can include skills and connections, for both humans and AIs.
> and guess what, context still fills up.
Human memory also has limits.
Sorry, but I really don't see how the distinctions you're trying to make even exist, much less qualify as markers of intelligence.
> Why not? If humans store data in their brains, isn't that learning?
No. We’re back to my earlier point of you and I have fundamentally different understanding of cognition, intelligence, and learning. And genuinely not trying to be condescending, but I suspect you don’t have a good grounding in the technology we’re discussing
No. We’re back to my earlier point of you and I have fundamentally different understanding of cognition, intelligence, and learning. And genuinely not trying to be condescending, but I suspect you don’t have a good grounding in the technology we’re discussing
Is making a list the act of planning?
> Is making a list the act of planning?
Depends on the content of the list.
A list of the names of the seven dwarfs: no, not an act of planning.
A list of steps necessary to build a web site: yes, an act of planning.
Depends on the content of the list.
A list of the names of the seven dwarfs: no, not an act of planning.
A list of steps necessary to build a web site: yes, an act of planning.
> actually living in the real world where time passes
sure, but it feels like this is just looking at what distinguishes humans from LLMs and calling that “intelligence.” I highlight this difference too when I talk about LLMs, but I don’t feel the need to follow up with “and that’s why they’re not really intelligent.”
sure, but it feels like this is just looking at what distinguishes humans from LLMs and calling that “intelligence.” I highlight this difference too when I talk about LLMs, but I don’t feel the need to follow up with “and that’s why they’re not really intelligent.”
well the second part (implied above, I didn’t actually write it) is “and operate intelligently in that world”. talking about “intelligence” in some abstract form where “does this text output constitute intelligence” is hyper silly to me. the discussion should anchor on real-world consequences, not the endless hypotheticals we end up with in these discussions
Human's are conscious beings. What kind of conscious beings are humans? Beings with eye consciousness, ear consciousness, nose consciousness, tongue consciousness, body consciousness, and mind consciousness. That is the definition of intelligence.
Intelligence is a tautological term. It is defined by itself. If you ask someone for examples of things inside the set of intelligence and outside of the set of intelligence, and then ask them to list off properties that would exclude something from the set, and properties that include something into the set, you will find things inside the set that have properties that should exclude them, and things outside the set which would have properties that should include them.
But these contradictions will not cause the person to re-evaluate whether or not the things should be removed from the set or included in it, but instead they will become exceptions to the defining properties.
Thus we have to abandon any sort of metric for intelligence and just call it a tautology and rely on an something that we can define to be the litmus for whatever property we are looking for. I think 'agency' should be under consideration for this, since it is actually somewhat definable and testable.
Thus we have to abandon any sort of metric for intelligence and just call it a tautology and rely on an something that we can define to be the litmus for whatever property we are looking for. I think 'agency' should be under consideration for this, since it is actually somewhat definable and testable.
I think it has to include some measure of Agency. You can load up the most impressive LLM out there and if you don't give it any instructions, IT WON'T DO ANYTHING.
> but the goal posts have shifted
So I think there are a lot more than your two possibilities. I mean psychologists and neuroscientists have been saying for decades that tests aren't a precise way to measure knowledge or intelligence, but that it is still a useful proxy.
> "quacks like a duck" behavioral
| If it looks like a duck, swims like a duck, and quacks like a duck, then it ***probably*** is a duck.
I think if we all want to be honest, the reality is "we don't know". There's arguments to be made in both directions and with varying definitions of intelligence with different nuances involved. I think these arguments are fine as they make us refine our definitions but I think they can also turn to be entirely dismissive and that doesn't help us refine and get closer to the truth. We all are going to have opinions on this stuff but frankly, the confidence of our opinions needs to be proportional to the amount of time and effort spent studying the topic. I mean the lack of a formal definition means nuances dominate the topic. Even if things are simple once you understand them that doesn't mean they aren't wildly complex before that. I mean I used to think Calculus was confusing and now I don't. Same process but not on an individual scale.
> I emphasize probably because the duck test doesn't allow you to distinguish a duck from a highly sophisticated animatronic. It's a good test, don't get me wrong, but that "probably" is a pretty important distinction.
Why is it an important distinction? The relevance of the duck test is that if you can't tell a duck from a non-duck, then the non-duck is sufficiently duck-like for the difference to not matter.
Why is it an important distinction? The relevance of the duck test is that if you can't tell a duck from a non-duck, then the non-duck is sufficiently duck-like for the difference to not matter.
Do believe there's a important distinction between a duck and not a duck?
I'm assuming you do, since apples aren't oranges and ducks aren't robots no matter what conspiracy theorists joke or say. You can't eat an animatronic duck. It doesn't lay eggs, no matter how much another duck tries to mate with it.
So it matters, do you care if something is a duck or not? And in what way? There's no single duck measurement, there's a whole suite of tests that need to be combined carefully and thoughtfully depending on how you answer the previous questions.
Obviously this is extra important when we aren't exactly sure what a duck is...
(Also, see Turing Test)
[0] https://images.squarespace-cdn.com/content/547e053ae4b04768d...
I'm assuming you do, since apples aren't oranges and ducks aren't robots no matter what conspiracy theorists joke or say. You can't eat an animatronic duck. It doesn't lay eggs, no matter how much another duck tries to mate with it.
> then the non-duck is sufficiently duck-like for the difference to not matter.
So it matters, do you care if something is a duck or not? And in what way? There's no single duck measurement, there's a whole suite of tests that need to be combined carefully and thoughtfully depending on how you answer the previous questions.
Obviously this is extra important when we aren't exactly sure what a duck is...
(Also, see Turing Test)
[0] https://images.squarespace-cdn.com/content/547e053ae4b04768d...
It may be the case that the failures of the ability of the machine (2) are best expressed by reference to the shortcomings of its internal workings (1), and not by contrived tests.
It might be the case, but if those shortcomings are not visible in the results of the machine (and therefore not interpretable by a test), why do its internal workings even matter?
I'm saying best expressed. Like, you see the failures in the results, but trying to pin down exactly what's the matter with the results means you resort to a lot of handwaving and abstract complaints about generalities. So if you knew how the internals had to be that would make the difference, you could lean on that.
LLMs mimic intelligence, but they aren’t intelligent.
They aren’t just intelligence mimics, they are people mimics, and they’re getting better at it with every generation.
Whether they are intelligent or not, whether they are people or not, it ultimately does not matter when it comes to what they can actually do, what they can actually automate. If they mimic a particular scenario or human task well enough that the job gets done, they can replace intelligence even if they are “not intelligent”.
If by now someone still isn’t convinced that LLMs can indeed automate some of those intelligence tasks, then I would argue they are not open to being convinced.
They aren’t just intelligence mimics, they are people mimics, and they’re getting better at it with every generation.
Whether they are intelligent or not, whether they are people or not, it ultimately does not matter when it comes to what they can actually do, what they can actually automate. If they mimic a particular scenario or human task well enough that the job gets done, they can replace intelligence even if they are “not intelligent”.
If by now someone still isn’t convinced that LLMs can indeed automate some of those intelligence tasks, then I would argue they are not open to being convinced.
They can mimic well documented behavior. Applying an LLM to a novel task is where the model breaks down. This obviously has huge implications for automation. For example, most business do not have unique ways of handling accounting transactions, yet each company has a litany of AR and AP specialists who create semmingly unique SOPs. LLMs can easily automate those workers since they are simply doing a slight variation at best of a very well documented system.
Asking an LLM to take all this knowledge and apply it to a new domain? That will take a whole new paradigm.
Asking an LLM to take all this knowledge and apply it to a new domain? That will take a whole new paradigm.
Absolutely agreed, but I suspect that a whole lot of what humans do every day can be reduced to pattern-following.
If/when LLMs or other AIs can create novel work / discover new knowledge, they will be "genius" in the literal sense of the word.
More genius would be great! (probably) . But genius is not required for the vast majority of tasks.
If/when LLMs or other AIs can create novel work / discover new knowledge, they will be "genius" in the literal sense of the word.
More genius would be great! (probably) . But genius is not required for the vast majority of tasks.
> Applying an LLM to a novel task is where the model breaks down
I mean, don't most people break down in this case too? I think this needs to be more precise. What is the specific task that you think can reliably distinguish between an LLM's capability in this sense vs. what a human can typically manage?
That is, in the sense of [1], what is the result that we're looking to use to differentiate.
[1] https://news.ycombinator.com/item?id=44913498
I mean, don't most people break down in this case too? I think this needs to be more precise. What is the specific task that you think can reliably distinguish between an LLM's capability in this sense vs. what a human can typically manage?
That is, in the sense of [1], what is the result that we're looking to use to differentiate.
[1] https://news.ycombinator.com/item?id=44913498
The article says that LLMs don't summarize, only shorten, because...
"A true summary, the kind a human makes, requires outside context and reference points. Shortening just reworks the information already in the text."
Then later says...
"LLMs operate in a similar way, trading what we would call intelligence for a vast memory of nearly everything humans have ever written. It’s nearly impossible to grasp how much context this gives them to play with"
So, they can't summarize, because they lack context... but they also have an almost ungraspably large amount of context?
"A true summary, the kind a human makes, requires outside context and reference points. Shortening just reworks the information already in the text."
Then later says...
"LLMs operate in a similar way, trading what we would call intelligence for a vast memory of nearly everything humans have ever written. It’s nearly impossible to grasp how much context this gives them to play with"
So, they can't summarize, because they lack context... but they also have an almost ungraspably large amount of context?
I think "context" is being used in different ways here.
> "It’s nearly impossible to grasp how much context this gives them to play with"
Here, I think the author means something more like "all the material used to train the LLM".
> "A true summary, the kind a human makes, requires outside context and reference points."
In this case I think that "context" means something more like actual comprehension.
The author's point is that an LLM could only write something like the referenced summary by shortening other summaries present in its training set.
> "It’s nearly impossible to grasp how much context this gives them to play with"
Here, I think the author means something more like "all the material used to train the LLM".
> "A true summary, the kind a human makes, requires outside context and reference points."
In this case I think that "context" means something more like actual comprehension.
The author's point is that an LLM could only write something like the referenced summary by shortening other summaries present in its training set.
But "shortening other summaries from its training set" is not all an LLM is capable off. It can easily shorten/summarize a text it had never seen before, in a way that makes sense. Sure, it won't always summarize it the same way a human would, but if you do a double blind test where you ask people whether a summary was written by AI, a vast majority wouldn't be able to tell the difference (again this is with a completely novel text).
I think the real takeaway is that LLMs are very good at tasks that closely resemble examples it has in its training. A lot of things written (code, movies/TV shows, etc.) are actually pretty repetitive and so you don't really need super intelligence to be able to summarize it and break it down, just good pattern matching. But, this can fall apart pretty wildly when you have something genuinely novel...
Is anyone here aware of LLMs demonstrating an original thought? Something truly novel.
My own impression is something more akin to a natural language search query system. If I want a snippet of code to do X it does that pretty well and keeps me from having to search through poor documentation of many OSS projects. Certainly doesn't produce anything I could not do myself - so far.
Ask it about something that is currently unknown and it list a bunch of hypotheses that people have already proposed.
Ask it to write a story and you get a story similar to one you already know but with your details inserted.
I can see how this may appear to be intelligent but likely isn't.
My own impression is something more akin to a natural language search query system. If I want a snippet of code to do X it does that pretty well and keeps me from having to search through poor documentation of many OSS projects. Certainly doesn't produce anything I could not do myself - so far.
Ask it about something that is currently unknown and it list a bunch of hypotheses that people have already proposed.
Ask it to write a story and you get a story similar to one you already know but with your details inserted.
I can see how this may appear to be intelligent but likely isn't.
If I come up with something novel while using an LLM, which I wouldn't have come up with had I not had the LLM at my bidding, where did the novelty really come from?
If I came up with something novel while watching a sunrise, which I wouldn't have come up with had I not been looking at it, where did the novelty really come from?
If the sunrise might have inspired the same idea in someone else, that's a good question.
I've no priors as to how original you are, nor how humble, so I mean this in a general, rather than personal, sense.
I've no priors as to how original you are, nor how humble, so I mean this in a general, rather than personal, sense.
Well that's the tricky part: what is novel? There are varying answers. I think we're all pretty unoriginal most of the time, but at the very least we're a bit better than LLMs at mashing together and synthesizing things based on previous knowledge.
But seriously, how would you determine if an LLM's output was novel? The training data set is so enormous for any given LLM that it would be hard to know for sure that any given output isn't just a trivial mix of existing data.
But seriously, how would you determine if an LLM's output was novel? The training data set is so enormous for any given LLM that it would be hard to know for sure that any given output isn't just a trivial mix of existing data.
Humans too. If I were too creative writing the midterm, most of my students would fail and everyone would be very unhappy.
That's because midterms are specifically supposed to assess how well you learned the material presented (or at least directed to), not your overall ability to reason. If you teach a general reasoning class, getting creative with the midterm is one thing, but if you're teaching someone how to solve differential equations, they're learning to the very edge of their ability in a given amount of time, and you present them with an example outside of what's been described, it kind of makes sense that they can't just already solve it. I mean, that's kind of the whole premise of education, that you can't just present someone with something completely outside of their experience and expect them to derive from first principles how it works.
I would argue that on a math midterm it's entirely reasonable to show a problem they've never seen before and test whether they've made the connection between that problem and the problems they've seen before. We did that all the time in upper division Physics.
A problem they've never seen before, of course. A problem that requires a solving strategy or tool they've never seen before (above and beyond synthesis of multiple things they have seen before) is another matter entirely.
It's like the difference between teaching kids rate problems and then putting ones with negative values or nested rates on a test versus giving them a continuous compound interest problem and expecting them to derive e, because it is fundamentally about rates of change, isn't it?
It's like the difference between teaching kids rate problems and then putting ones with negative values or nested rates on a test versus giving them a continuous compound interest problem and expecting them to derive e, because it is fundamentally about rates of change, isn't it?
That's exams, not humanity.
I honestly think that reflects more on the state of education than it does human intelligence.
My primary assertion is that LLMs struggle to generalize concepts and ideas, hence why they need petabytes of text just to often fail basic riddles when you muck with the parameters a little bit. People get stuck on this for two reasons: one, because they have to reconcile this with what they can see LLMs are capable of, and it's just difficult to believe that all of this can be accomplished without at least intelligence as we know it; I reckon the trick here is that we simply can't even conceive of how utterly massive the training datasets for these models are. We can look at the numbers but there's no way to fully grasp just how vast it truly is. The second thing is definitely the tendency to anthropomorphize. At first I definitely felt like OpenAI was just using this as an excuse to hype their models and come up with reasons for why they can never release weights anymore; convenient. But also, you can see even engineers who genuinely understand how LLMs work coming to the conclusion that they've become sentient, even though the models they felt were sentient now feel downright stupid compared to the current state-of-the-art.
Even less sophisticated pattern matching than what humans are able to do is still very powerful, but it's obvious to me that humans are able to generalize better.
My primary assertion is that LLMs struggle to generalize concepts and ideas, hence why they need petabytes of text just to often fail basic riddles when you muck with the parameters a little bit. People get stuck on this for two reasons: one, because they have to reconcile this with what they can see LLMs are capable of, and it's just difficult to believe that all of this can be accomplished without at least intelligence as we know it; I reckon the trick here is that we simply can't even conceive of how utterly massive the training datasets for these models are. We can look at the numbers but there's no way to fully grasp just how vast it truly is. The second thing is definitely the tendency to anthropomorphize. At first I definitely felt like OpenAI was just using this as an excuse to hype their models and come up with reasons for why they can never release weights anymore; convenient. But also, you can see even engineers who genuinely understand how LLMs work coming to the conclusion that they've become sentient, even though the models they felt were sentient now feel downright stupid compared to the current state-of-the-art.
Even less sophisticated pattern matching than what humans are able to do is still very powerful, but it's obvious to me that humans are able to generalize better.
And what truly novel things are humans capable of? At least 99% of the stuff we do is just what we were taught by parents, schools, books, friends, influencers, etc.
Remember, humans needed some 100, 000 years to figure out that you can hit an animal with a rock, and that's using more or less the same brain capacity we have today. If we were born in stone age, we'd all be nothing but cavemen.
What genuinely novel thing have you figured out?
Remember, humans needed some 100, 000 years to figure out that you can hit an animal with a rock, and that's using more or less the same brain capacity we have today. If we were born in stone age, we'd all be nothing but cavemen.
What genuinely novel thing have you figured out?
Look. I get that we can debate about what's truly novel. I never even actually claimed that humans regularly do things that are actually all that novel. That wasn't the point. The point is that LLMs struggle with novelty because they struggle to generalize. Humans clearly are able to generalize vastly better than transformer-based LLMs.
Really? How do I know that with such great certainty?
Well, I don't know how much text I've read in one lifetime, but I can tell you it's less than the literally multiple terabytes of text fed into the training process of modern LLMs.
Yet, LLMs can still be found failing logic puzzles and simple riddles that even children can figure out, just by tweaking some of the parameters slightly, and it seems like the best thing we can do here is just throw more terabytes of data and more reinforcement learning at it, only for it to still fail, even if a little more sparingly each time.
So what novel things do average people do anyways, since beating animals with rocks apparently took 100,000 years to figure out? Hard call. There's no definitive bar for novel. You could argue almost everything we do is basically just mixing things we've seen together before, yet I'd argue humans are much better at it than LLMs, which need a metric shit load of training data and burn tons of watts. In return, you get some superhuman abilities, but superhuman doesn't mean smarter or better than people; a sufficiently powerful calculator is superhuman. The breadth of an LLM is much wider than any individual human, but the breadth of knowledge across humanity is obviously still much wider than any individual LLM, and there remain things people do well that LLMs definitely still don't, even just in the realm of text.
So if I don't really believe humans are all that novel, why judge LLMs based on that criteria? Really two reasons:
- I think LLMs are significantly worse at it, so allowing your critical thinking abilities to atrophy in favor of using LLMs is really bad. Therefore people need to be very careful about ascribing too much to LLMs.
- Because I think many people want to use LLMs to do truly novel things. Don't get me wrong, a lot of people also just want it to shit out another React Tailwind frontend for a Node.js JSON HTTP CRUD app or something. But, a lot of AI skeptics are no longer the types of people that downplay it as a cope or out of fear, but actually are people who were at least somewhat excited by the capabilities of AI then let down when they tried to color outside the lines and it failed tremendously.
Likewise, imagine trying to figure out how novel an AI response is; the training data set is so massive, that humans can hardly comprehend the scale. Our intuition about what couldn't possibly be in the training data is completely broken. We can only ever easily prove that a given response isn't novel, not that it is.
But honestly maybe it's just too unconvincing to just say all of this in the abstract. Maybe it would better to at least try to come up with some demonstration of something I think I've come up with that is "novel".
There's this sort-of trick I came up with when implementing falling blocks puzzle games for handling input that I think is pretty unique. See, in most implementations, to handle things like auto-repeating movements, you might do something like have a counter that increments, then once it hits the repeat delay, it gets reset again. Maybe you could get slightly more clever by having it count down and repeat at zero: this would make it easier to, for example, have the repeat delay be longer for only the first repeat. This is how DAS normally works in Tetris and other games, and it more or less mirrors the key repeat delay. It's easier with the count down since on the first input you can set it to the high initial delay, then whenever it hits zero you can set it to the repeat delay.
I didn't like this though because I didn't like having to deal with a bunch of state. I really wanted the state to be as simple as possible. So instead, for each game input, I allocate a signed integer. These integers are all initialized to zero. When a key is pressed down, the integer is set to 1 if it is less than 1. When a key is released, it is set to -1 if it is greater than 0. And on each frame of game logic, at the end of the frame, each input greater than 0 is incremented, and each input less than 0 is decremented. This is held in the game state and when the game logic is paused, you do nothing here.
With this scheme, the following side effects occur:
- Like most other schemes, there's no need to special-case key repeat events, as receiving a second key down doesn't do anything.
- Game logic can now do a bunch of logic "statelessly", since the input state encodes a lot of useful information. For example, you can easily trigger an event upon an input being pressed by using n == 1, and you can easily trigger an event upon an input being released using n == -1. You do something every five frames an input is held by checking n % 5 == 0, or slightly more involved for a proper input repeat with initial delay. On any given frame of game logic, you always know how long an input has been held down and after it's released you know how many frames it has been since it was pressed.
Now I don't talk to tons of other game developers, but I've never seen or heard of anyone doing this, and if someone else did come up with it, then I discovered it independently. It was something I came up with when playing around with trying to make deterministic, rewindable game logic. I played around with this a lot in highschool (not that many years ago, about 15 now.)
I fully admit this is not as useful for the human race as "hitting animals with a rock", but I reckon it's the type of thing that LLMs basically only come up with if they've already been exposed to the idea. If I try to instruct LLMs to implement a system that has what I think is a novel idea, it really seems to rapidly fall apart. If it doesn't fall apart, then I honestly begin to suspect that maybe the idea is less novel than I thought... but it's a whole hell of a lot more common, so far, for it to just completely fall apart.
Still, my point was never that AI is useless, a lot of things humans do aren't very novel after all. However, I also think it is definitely not time to allow one's critical thinking skills to atrophy as today's models definitely have some very bad failure modes and some of the ways they fail are ways that we can't afford in many circumstances. Today the biggest challenge IMO is that despite all of the data the ability to generalize really feels lacking. If that problem gets conquered, I'm sure more problems will rise to the top. Unilaterally superhuman AI has a long way to go.
Really? How do I know that with such great certainty?
Well, I don't know how much text I've read in one lifetime, but I can tell you it's less than the literally multiple terabytes of text fed into the training process of modern LLMs.
Yet, LLMs can still be found failing logic puzzles and simple riddles that even children can figure out, just by tweaking some of the parameters slightly, and it seems like the best thing we can do here is just throw more terabytes of data and more reinforcement learning at it, only for it to still fail, even if a little more sparingly each time.
So what novel things do average people do anyways, since beating animals with rocks apparently took 100,000 years to figure out? Hard call. There's no definitive bar for novel. You could argue almost everything we do is basically just mixing things we've seen together before, yet I'd argue humans are much better at it than LLMs, which need a metric shit load of training data and burn tons of watts. In return, you get some superhuman abilities, but superhuman doesn't mean smarter or better than people; a sufficiently powerful calculator is superhuman. The breadth of an LLM is much wider than any individual human, but the breadth of knowledge across humanity is obviously still much wider than any individual LLM, and there remain things people do well that LLMs definitely still don't, even just in the realm of text.
So if I don't really believe humans are all that novel, why judge LLMs based on that criteria? Really two reasons:
- I think LLMs are significantly worse at it, so allowing your critical thinking abilities to atrophy in favor of using LLMs is really bad. Therefore people need to be very careful about ascribing too much to LLMs.
- Because I think many people want to use LLMs to do truly novel things. Don't get me wrong, a lot of people also just want it to shit out another React Tailwind frontend for a Node.js JSON HTTP CRUD app or something. But, a lot of AI skeptics are no longer the types of people that downplay it as a cope or out of fear, but actually are people who were at least somewhat excited by the capabilities of AI then let down when they tried to color outside the lines and it failed tremendously.
Likewise, imagine trying to figure out how novel an AI response is; the training data set is so massive, that humans can hardly comprehend the scale. Our intuition about what couldn't possibly be in the training data is completely broken. We can only ever easily prove that a given response isn't novel, not that it is.
But honestly maybe it's just too unconvincing to just say all of this in the abstract. Maybe it would better to at least try to come up with some demonstration of something I think I've come up with that is "novel".
There's this sort-of trick I came up with when implementing falling blocks puzzle games for handling input that I think is pretty unique. See, in most implementations, to handle things like auto-repeating movements, you might do something like have a counter that increments, then once it hits the repeat delay, it gets reset again. Maybe you could get slightly more clever by having it count down and repeat at zero: this would make it easier to, for example, have the repeat delay be longer for only the first repeat. This is how DAS normally works in Tetris and other games, and it more or less mirrors the key repeat delay. It's easier with the count down since on the first input you can set it to the high initial delay, then whenever it hits zero you can set it to the repeat delay.
I didn't like this though because I didn't like having to deal with a bunch of state. I really wanted the state to be as simple as possible. So instead, for each game input, I allocate a signed integer. These integers are all initialized to zero. When a key is pressed down, the integer is set to 1 if it is less than 1. When a key is released, it is set to -1 if it is greater than 0. And on each frame of game logic, at the end of the frame, each input greater than 0 is incremented, and each input less than 0 is decremented. This is held in the game state and when the game logic is paused, you do nothing here.
With this scheme, the following side effects occur:
- Like most other schemes, there's no need to special-case key repeat events, as receiving a second key down doesn't do anything.
- Game logic can now do a bunch of logic "statelessly", since the input state encodes a lot of useful information. For example, you can easily trigger an event upon an input being pressed by using n == 1, and you can easily trigger an event upon an input being released using n == -1. You do something every five frames an input is held by checking n % 5 == 0, or slightly more involved for a proper input repeat with initial delay. On any given frame of game logic, you always know how long an input has been held down and after it's released you know how many frames it has been since it was pressed.
Now I don't talk to tons of other game developers, but I've never seen or heard of anyone doing this, and if someone else did come up with it, then I discovered it independently. It was something I came up with when playing around with trying to make deterministic, rewindable game logic. I played around with this a lot in highschool (not that many years ago, about 15 now.)
I fully admit this is not as useful for the human race as "hitting animals with a rock", but I reckon it's the type of thing that LLMs basically only come up with if they've already been exposed to the idea. If I try to instruct LLMs to implement a system that has what I think is a novel idea, it really seems to rapidly fall apart. If it doesn't fall apart, then I honestly begin to suspect that maybe the idea is less novel than I thought... but it's a whole hell of a lot more common, so far, for it to just completely fall apart.
Still, my point was never that AI is useless, a lot of things humans do aren't very novel after all. However, I also think it is definitely not time to allow one's critical thinking skills to atrophy as today's models definitely have some very bad failure modes and some of the ways they fail are ways that we can't afford in many circumstances. Today the biggest challenge IMO is that despite all of the data the ability to generalize really feels lacking. If that problem gets conquered, I'm sure more problems will rise to the top. Unilaterally superhuman AI has a long way to go.
That seems fair and accurate.
I guess disagreement about this question often stems from what we mean by "human", even more than what we mean by "intelligence".
There are at least 3 distinct categories of human intelligence/capability in any given domain:
1) average human (non-expert) - LLMs are already better (mainly because the average human doesn't know anything, but LLMs at least have some basic knowledge),
2) domain expert humans - LLMs are far behind, but can sometimes supplement human experts with additional breadth,
3) collective intelligence of all humans combined - LLMs are like retarded cavemen in comparison.
So when answering if AI has human-level intelligence, it really makes sense to ask what "human-level" means.
I guess disagreement about this question often stems from what we mean by "human", even more than what we mean by "intelligence".
There are at least 3 distinct categories of human intelligence/capability in any given domain:
1) average human (non-expert) - LLMs are already better (mainly because the average human doesn't know anything, but LLMs at least have some basic knowledge),
2) domain expert humans - LLMs are far behind, but can sometimes supplement human experts with additional breadth,
3) collective intelligence of all humans combined - LLMs are like retarded cavemen in comparison.
So when answering if AI has human-level intelligence, it really makes sense to ask what "human-level" means.
It's an interesting philosophical question.
Imagine an oracle that could judge/decide, with human levels of intelligence, how relevant a given memory or piece of information is to any given situation, and that could verbosely describe which way it's relevant (spatially, conditionally, etc.).
Would such an oracle, sufficiently parallelized, be sufficient for AGI? If it could, then we could genuinely describe its output as "context," and phrase our problem as "there is still a gap in needed context, despite how much context there already is."
And an LLM that simply "shortens" that context could reach a level of AGI, because the context preparation is doing the heavy lifting.
The point I think the article is trying to make is that LLMs cannot add any information beyond the context they are given - they can only "shorten" that context.
If the lived experience necessary for human-level judgment could be encoded into that context, though... that would be an entirely different ball game.
Imagine an oracle that could judge/decide, with human levels of intelligence, how relevant a given memory or piece of information is to any given situation, and that could verbosely describe which way it's relevant (spatially, conditionally, etc.).
Would such an oracle, sufficiently parallelized, be sufficient for AGI? If it could, then we could genuinely describe its output as "context," and phrase our problem as "there is still a gap in needed context, despite how much context there already is."
And an LLM that simply "shortens" that context could reach a level of AGI, because the context preparation is doing the heavy lifting.
The point I think the article is trying to make is that LLMs cannot add any information beyond the context they are given - they can only "shorten" that context.
If the lived experience necessary for human-level judgment could be encoded into that context, though... that would be an entirely different ball game.
I agree with the thrust of your argument.
IMO we already have the technology for sufficient parallelization of smaller models with specific bits of context. The real issue is that models have weak/inconsistent/myopic judgement abilities, even with reasoning loops.
For instance, if I ask Cursor to fix the code for a broken test and the fix is non-trivial, it will often diagnose the problem incorrectly almost instantly, hyper-focus on what it imagines the problem is without further confirmation, implement a "fix", get a different error message while breaking more tests than it "fixed" (if it changed the result for any tests), and then declare the problem solved simply because it moved the goalposts at the start by misdiagnosing the issue.
IMO we already have the technology for sufficient parallelization of smaller models with specific bits of context. The real issue is that models have weak/inconsistent/myopic judgement abilities, even with reasoning loops.
For instance, if I ask Cursor to fix the code for a broken test and the fix is non-trivial, it will often diagnose the problem incorrectly almost instantly, hyper-focus on what it imagines the problem is without further confirmation, implement a "fix", get a different error message while breaking more tests than it "fixed" (if it changed the result for any tests), and then declare the problem solved simply because it moved the goalposts at the start by misdiagnosing the issue.
You can reconcile these points by considering what specific context is necessary. The author specifies "outside" context, and I would agree. The human context that's necessary for useful summaries is a model of semantic or "actual" relationships between concepts, while the LLM context is a model of a single kind of fuzzy relationship between concepts.
In other words the LLM does not contain the knowledge of what the words represent.
In other words the LLM does not contain the knowledge of what the words represent.
> In other words the LLM does not contain the knowledge of what the words represent.
This is probably true for some words and concepts but not others. I think we find that llms make inhuman mistakes only because they don't have the embodied senses and inductive biases that are at the root of human language formation.
If this hypothesis is correct, it suggests that we might be able to train a more complete machine intelligence by having them participate in a physics simulation as one part of the training. I.e have a multimodal ai play some kind of blockworld game. I bet if the ai is endowed with just sight and sound, it might be enough to capture many relevant relationships.
This is probably true for some words and concepts but not others. I think we find that llms make inhuman mistakes only because they don't have the embodied senses and inductive biases that are at the root of human language formation.
If this hypothesis is correct, it suggests that we might be able to train a more complete machine intelligence by having them participate in a physics simulation as one part of the training. I.e have a multimodal ai play some kind of blockworld game. I bet if the ai is endowed with just sight and sound, it might be enough to capture many relevant relationships.
I think the differentiator here might not be the context it has, but the context it has the ability to use effectively in order to derive more information about a given request.
This can be solved with prompting. You can say: “summarize this document, but don’t just recap, give me the big picture” or anything to that effect.
They can’t summarize something that hasn’t been summarized before.
About a year ago, I gave a film script to an LLM and asked for a summary. It was written by a friend and there was no chance it or its summary was in the training data.
It did a really good -- surprisingly good -- job. That incident has been a reference point for me. Even if it is anecdotal.
It did a really good -- surprisingly good -- job. That incident has been a reference point for me. Even if it is anecdotal.
I'm not as cynical as others about LLMs but it's extremely unlikely that script had multiple truly novel things in it. Broken down into sufficient small pieces it's very likely every story element was present multiple times in the LLM's training data.
I'm not sure I understand the philosophical point being made here. The LLM has "watched" a lot of movies and so understands the important parts of the original script it's presented with. Are we not describing how human media literacy works?
The point is that if you made a point to write a completely novel script, with (content-wise, not semantically) 0 DNA in it from previous movie scripts, with an unambiguous but incoherent and unstructured plot, your average literate human would be able summarize what happened on the page, for all that they'd be annoyed and likely distressed by how unusual it was; but that an LLM would do a disproportionately bad job compared to how well they do at other things, which makes us reevaluate what they're actually doing and how they actually do it.
It feels like they've mastered language, but it's looking more and more like they've actually mastered canon. Which is still impressive, but very different.
It feels like they've mastered language, but it's looking more and more like they've actually mastered canon. Which is still impressive, but very different.
This tracks, because the entire system reduces to a sophisticated regression analysis. That's why we keep talking about parameters and parameter counts. They're literally talking about the number of parameters that they're weighting during training. Beyond that there are some mathematical choices in how you interrelated the parameters that yields some interesting emergent phenomena, and there are architecture choices to be made there. But the whole thing boils down to regression, and regression is at its heart a development of a canon from a representative variety of examples.
We are warned in statistics to be careful when extrapolating from a regression analysis.
We are warned in statistics to be careful when extrapolating from a regression analysis.
And have you managed to perform such a test or is that just an imaginary result you're convinced will happen ? Not trying to be snarky here but i see this kind of thing a lot and 'this is my model of how LLMs work and so this is how they would behave in this test I cannot verify' is very uncompelling.
[deleted]
I'm not making a philosophical point. The earlier comment is "I updated a new script and it summarized it," I was simply saying the odds of that script actually being new is very slim. Even though obviously that script or summaries of it do not exist in their entirety in the training data, its individual elements almost certainly do. So it's not really a novel (pun unintended?) summarization.
See also 'A Hero's Journey'
https://en.m.wikipedia.org/wiki/Hero%27s_journey
https://en.m.wikipedia.org/wiki/Hero%27s_journey
I'd like to see some examples of when it struggles to do summaries. There were no real examples in the text, besides one hypothetical which ChatGPT made up.
I think LLMs do great summaries. I am not able to come up with anything where I could criticize it and say "any human would come up with a better summary". Are my tasks not "truly novel"? Well, then I am not able, as a human, to come up with anything novel either.
I think LLMs do great summaries. I am not able to come up with anything where I could criticize it and say "any human would come up with a better summary". Are my tasks not "truly novel"? Well, then I am not able, as a human, to come up with anything novel either.
they can, they just can't do it well. at no point does any LLM understand what it's doing.
If you think they can't do this task well I encourage you to try feeding ChatGPT some long documents outside of its training cutoff and examining the results. I expect you'll be surprised!
It can produce something that looks like a summarization based on similarly matching texts.
Depending how unique the text is determines how accurate the summarization is likely to be.
Depending how unique the text is determines how accurate the summarization is likely to be.
Even stronger than our need to anthropomorphize seems to be our innate desire to believe our species is special, and that “real intelligence” couldn’t ever be replicated.
If you keep redefining real intelligence as the set of things machines can’t do, then it’s always going to be true.
If you keep redefining real intelligence as the set of things machines can’t do, then it’s always going to be true.
Yes, I agree, we seem to need to feel "special".
Language is really powerful, I think it's a huge part of our intelligence.
The interesting part of the article to me is the focus on fluency. I have not seen anything that LLMs do well that isn't related to powerful utilization of fluency.
Language is really powerful, I think it's a huge part of our intelligence.
The interesting part of the article to me is the focus on fluency. I have not seen anything that LLMs do well that isn't related to powerful utilization of fluency.
Good point about the Turing Test:
>The original Turing Test was designed to compare two participants chatting through a text-only interface: one AI and one human. The goal was to spot the imposter. Today, the test is simplified from three participants to just two: a human and an LLM.
By the original meaning of the test it's easy to tell an LLM from a human.
>The original Turing Test was designed to compare two participants chatting through a text-only interface: one AI and one human. The goal was to spot the imposter. Today, the test is simplified from three participants to just two: a human and an LLM.
By the original meaning of the test it's easy to tell an LLM from a human.
- LLMs don't need to be intelligent to take jobs, bash scripts have replaced people.
- Even if CEOs are completely out of touch and the tool can't do the job you can still get laid off in an ill informed attempt to replace you. Then when the company doesn't fall over because the leftover people, desperate to keep covering rent fill the gaps it just looks like efficiency to the top.
- I don't think our tendency anthropomorphize LLMs is really the problem here.
- Even if CEOs are completely out of touch and the tool can't do the job you can still get laid off in an ill informed attempt to replace you. Then when the company doesn't fall over because the leftover people, desperate to keep covering rent fill the gaps it just looks like efficiency to the top.
- I don't think our tendency anthropomorphize LLMs is really the problem here.
The article claims (without any evidence, argument or reason) that LLMs are not intelligent, then simply refuses to define intelligence.
How do you know LLMs aren't intelligent, if you can't define what that means?
How do you know LLMs aren't intelligent, if you can't define what that means?
It's strange seeing so many takes like this two weeks after LLMs won gold medals at IMO and IOI. The cognitive dissonance is going to be wild when it all comes to a head in two years.
IBM Watson won Jeopardy years ago, was it intelligent?
> Rather than being given questions, contestants are instead given general knowledge clues in the form of answers and they must identify the person, place, thing, or idea that the clue describes, phrasing each response in the form of a question. [0]
Doesn't sound like a test of intelligence to me, so no.
[0] https://en.wikipedia.org/wiki/Jeopardy!
Doesn't sound like a test of intelligence to me, so no.
[0] https://en.wikipedia.org/wiki/Jeopardy!
Why? Computers also won chess years ago, but they're not intelligent either? Why is winning a math competition intelligent but a trivia competition or a chess competition not intelligent?
Math and chess are similar in the sense that for humans, both require creativity, logical problem solving, etc.
But they are not at all similar for computers. Chess has a constrained small set of rules and it is pretty straightforward to make a machine that beats humans by brute force computation. Pre-Leela chess programs were just tree search, a hardcoded evaluation function, and lots of pruning heuristics. So those programs are really approaching the game in a fundamentally different way from strong humans, who rely much more on intuition and pattern-recognition rather than calculation. It just turns out the computer approach is actually better than the human one. Sort of like how a car can move faster than a human even though cars don’t do anything much like walking.
Math is not analogous: there’s no obvious algorithm for discovering mathematical proofs or solving difficult problems that could be implemented in a classical, pre-Gen AI computer program.
But they are not at all similar for computers. Chess has a constrained small set of rules and it is pretty straightforward to make a machine that beats humans by brute force computation. Pre-Leela chess programs were just tree search, a hardcoded evaluation function, and lots of pruning heuristics. So those programs are really approaching the game in a fundamentally different way from strong humans, who rely much more on intuition and pattern-recognition rather than calculation. It just turns out the computer approach is actually better than the human one. Sort of like how a car can move faster than a human even though cars don’t do anything much like walking.
Math is not analogous: there’s no obvious algorithm for discovering mathematical proofs or solving difficult problems that could be implemented in a classical, pre-Gen AI computer program.
> there’s no obvious algorithm for discovering mathematical proofs or solving difficult problems that could be implemented in a classical, pre-Gen AI computer program.
Fundamentally opposite. Computer algorithms have been part of math research since they where invented, and mathematical proof algorithms are widespread and excellent.
The llms that are now "intelligent enough to do maths" are just trained to rephrase questions into prolog code.
Fundamentally opposite. Computer algorithms have been part of math research since they where invented, and mathematical proof algorithms are widespread and excellent.
The llms that are now "intelligent enough to do maths" are just trained to rephrase questions into prolog code.
> The llms that are now "intelligent enough to do maths" are just trained to rephrase questions into prolog code.
Do you have a source that talks about this?
Do you have a source that talks about this?
For the OpenAI case, its unclear. They've not disclosed the method yet. (Though they have previously had an official model that could query wolfram-alpha, so they're not strangers to that method)
But math olympiad questions have been beaten before by AlphaGeometry and a few other's using prolog or similar logic evaluation engines. And it works quite well. (Simply searching LLM prolog gives alot of results on Google and Google scholar)
If openai did it through brute forces text reasoning, its both impressive and frighteningly inefficient.
Even just normal algebra is something llms struggle with, hence using existing algebra solvers is faar more effective.
But math olympiad questions have been beaten before by AlphaGeometry and a few other's using prolog or similar logic evaluation engines. And it works quite well. (Simply searching LLM prolog gives alot of results on Google and Google scholar)
If openai did it through brute forces text reasoning, its both impressive and frighteningly inefficient.
Even just normal algebra is something llms struggle with, hence using existing algebra solvers is faar more effective.
3b1b just posted something very relevant, if you're curious.
https://www.youtube.com/watch?v=4NlrfOl0l8U
> Chess has a constrained small set of rules and it is pretty straightforward to make a machine that beats humans by brute force computation.
But Jeopardy is not like that. In fact, it is infinite in scope also, yet we earlier dismissed that as not intelligent
But Jeopardy is not like that. In fact, it is infinite in scope also, yet we earlier dismissed that as not intelligent
None of these things are enough by itself. It's rather that they have now solved so many things that the sum total has (arguably) crossed the threshold.
As for solving math problems, that is an important part of recursive self improvement. If it can come up with better algorithms and turn them into code, that will translate into raising it's own intelligence.
As for solving math problems, that is an important part of recursive self improvement. If it can come up with better algorithms and turn them into code, that will translate into raising it's own intelligence.
I don't wish to join you in framing intelligence as a step function.
I think winning a Go or a chess competition does demonstrate intelligence. And winning a math competition does even more so.
I do not think a trivia competition like Jeopardy demonstrates intelligence much at all, however. Specifically because it reads like it's not about intelligence, but about knowledge: it tests for association and recall, not for performing complex logical transformations.
This isn't to say I consider these completely independent. Most smart people are both knowledgeable and intelligent. It's just that they are distinct dimensions in my opinion.
You wouldn't say something tastes bad because its texture feels weird in your mouth, would you?
I think winning a Go or a chess competition does demonstrate intelligence. And winning a math competition does even more so.
I do not think a trivia competition like Jeopardy demonstrates intelligence much at all, however. Specifically because it reads like it's not about intelligence, but about knowledge: it tests for association and recall, not for performing complex logical transformations.
This isn't to say I consider these completely independent. Most smart people are both knowledgeable and intelligent. It's just that they are distinct dimensions in my opinion.
You wouldn't say something tastes bad because its texture feels weird in your mouth, would you?
I might even think that a symbolic chess program is in some sense more intelligent than a modern LLM. It has a concrete model of the world it operates in along with representation what it can, cannot, and is trying to, do. When LLMs get the right answer, it seems more like... highly-optimized chance, rather than coming from any sort of factual knowledge.
> I think winning a Go or a chess competition does demonstrate intelligence.
Chess is a simple alfa beta pruned minmax seaech tree. If that's intelligent then a drone flight controller or a calculator is aswell.
> association and recall, not for performing complex logical transformations.
By that definition humans doing chess aren't as intelligent as a computer doing chess, since high level chess is heavily reliant on memory and recall of moves and progressions.
So your definition falls appart.
Chess is a simple alfa beta pruned minmax seaech tree. If that's intelligent then a drone flight controller or a calculator is aswell.
> association and recall, not for performing complex logical transformations.
By that definition humans doing chess aren't as intelligent as a computer doing chess, since high level chess is heavily reliant on memory and recall of moves and progressions.
So your definition falls appart.
> So your definition falls apart.
I did not share any definitions, only vague opinions. Not that I'd know what it means for a definition to "fall apart".
And the specific bit you cite is barely even a vague opinion; it is my interpretation of the show "Jeopardy!" based on the Wiki article (I've never seen a single episode, wasn't really a thing where I'm from):
> Specifically because it reads like it's about (...) knowledge: it tests for association and recall (...)
Also:
> By that definition humans doing chess aren't as intelligent as a computer doing chess, since high level chess is heavily reliant on memory and recall of moves and progressions.
Yes, I did find this really quite disappointing and disillusioning when I first learned about it. A colleague of mine even straight up quit competitive chess over it.
I did not share any definitions, only vague opinions. Not that I'd know what it means for a definition to "fall apart".
And the specific bit you cite is barely even a vague opinion; it is my interpretation of the show "Jeopardy!" based on the Wiki article (I've never seen a single episode, wasn't really a thing where I'm from):
> Specifically because it reads like it's about (...) knowledge: it tests for association and recall (...)
Also:
> By that definition humans doing chess aren't as intelligent as a computer doing chess, since high level chess is heavily reliant on memory and recall of moves and progressions.
Yes, I did find this really quite disappointing and disillusioning when I first learned about it. A colleague of mine even straight up quit competitive chess over it.
> it is my interpretation of the show "Jeopardy!" based on the Wiki article
You are spot on though. I mostly wanted to argue that no decent distinction can be made here.
> I did find this really quite disappointing and disillusioning when I first learned about it
ye... same here.
---
I'm personally in the camp that "intelligence" is a human concept. A metric to compare humans. Applying it to computers makes us anthropomorphism computers and think of them as people. Thinking of LLMs as people makes us trust them with bad things.
So we should call them impressive, fluent, fast, useful, good at tasks. Computers already beat us at most math, statistics, searching for information, spacial visualization, information recollection, lossless communication. LLMs just adds to that list, but does nothing new to make the word "intelligent" applicable. Even if we reach the AGI singularity; thinking of them as humans or using human terminology to describe them is a fatal error.
(Destroying earth to make paperclips is arguably the least intelligent thing you could ever do.)
You are spot on though. I mostly wanted to argue that no decent distinction can be made here.
> I did find this really quite disappointing and disillusioning when I first learned about it
ye... same here.
---
I'm personally in the camp that "intelligence" is a human concept. A metric to compare humans. Applying it to computers makes us anthropomorphism computers and think of them as people. Thinking of LLMs as people makes us trust them with bad things.
So we should call them impressive, fluent, fast, useful, good at tasks. Computers already beat us at most math, statistics, searching for information, spacial visualization, information recollection, lossless communication. LLMs just adds to that list, but does nothing new to make the word "intelligent" applicable. Even if we reach the AGI singularity; thinking of them as humans or using human terminology to describe them is a fatal error.
(Destroying earth to make paperclips is arguably the least intelligent thing you could ever do.)
FWIW you can get quite good at chess with minimal opening prep. (Just not to the very top of the elite.)
I've seen these claims, and Google even published the texts of the solutions, but it still didn't published the full log of interaction between the model and operator.
Why do critics of LLM intelligence need to provide a definition when people who believe LLMs are intelligent only take it on faith, not having such a definition of their own?
> Why do critics of LLM intelligence need to provide a definition when people who believe LLMs are intelligent only take it on faith, not having such a definition of their own?
Because advocates of LLMs don't use their alleged intelligence as a defense; but opponents of LLMs do use their alleged non-intelligence as an attack.
Really, whether or not the machine is "intelligent", by whatever definition, shouldn't matter. What matters is whether it is a useful tool.
Because advocates of LLMs don't use their alleged intelligence as a defense; but opponents of LLMs do use their alleged non-intelligence as an attack.
Really, whether or not the machine is "intelligent", by whatever definition, shouldn't matter. What matters is whether it is a useful tool.
The entire argument is that thinking it's intelligent or a person makes us missuse the tool in dangerous ways. Not to make us feel better; but to not do stupid things with them.
As a tool its useful yes, that is not the issue;
- theyre used as phycologist and life coaches.
- judges of policy and law documents
- writers of life affecting computer systems.
- Judges of job applications.
- Sources of medical advice,
- legal advisors
- And increasingly as a thing to blame when any of above goes awry.
If we think of llms as very good text writing tools, the responsibility to make "intelligent" decisions and more crucially take responsibility for those decisions remains on real people rather than dice.
But if we think of them as intelligent humans, we making a fatal misjudgement.
As a tool its useful yes, that is not the issue;
- theyre used as phycologist and life coaches.
- judges of policy and law documents
- writers of life affecting computer systems.
- Judges of job applications.
- Sources of medical advice,
- legal advisors
- And increasingly as a thing to blame when any of above goes awry.
If we think of llms as very good text writing tools, the responsibility to make "intelligent" decisions and more crucially take responsibility for those decisions remains on real people rather than dice.
But if we think of them as intelligent humans, we making a fatal misjudgement.
This seems reasonable. Much AI research has historically been about building computer systems to do things that otherwise require human intelligence to do. The question of "is the computer actually intelligent" has been more philosophical than practical, and many such practically useful computer systems have been developed, even before LLMs.
On the other hand, one early researcher said something to the effect of, Researchers in physics look at the universe and wonder how it all works. Researchers in biology look at living organisms and wonder how they can be alive. Researchers in artificial intelligence wonder how software can be made to wonder such things.
I feel like we are still way off from having a working solution there.
On the other hand, one early researcher said something to the effect of, Researchers in physics look at the universe and wonder how it all works. Researchers in biology look at living organisms and wonder how they can be alive. Researchers in artificial intelligence wonder how software can be made to wonder such things.
I feel like we are still way off from having a working solution there.
It's actually very weird to "believe" LLMs are "intelligent".
Pragmatic people see news like "LLMs achieve gold in Math Olympiad" and think "oh wow, it can do maths at that level, cool!" This gets misinterpreted by so called "critics of LLM" who scream "NO THEY ARE JUST STOCHASTIC PARROTS" at every opportunity yet refuse to define what intelligence actually is.
The average person might not get into that kind of specific detail, but they know that LLMs can do some things well but there are tasks they're not good at. What matters is what they can do, not so much whether they're "intelligent" or not. (Of course, if you ask a random person they might say LLMs are pretty smart for some tasks, but that's not the same as making a philosophical claim that they're "intelligent")
Of course there's also the AGI and singularity folks. They're kinda loony too.
Pragmatic people see news like "LLMs achieve gold in Math Olympiad" and think "oh wow, it can do maths at that level, cool!" This gets misinterpreted by so called "critics of LLM" who scream "NO THEY ARE JUST STOCHASTIC PARROTS" at every opportunity yet refuse to define what intelligence actually is.
The average person might not get into that kind of specific detail, but they know that LLMs can do some things well but there are tasks they're not good at. What matters is what they can do, not so much whether they're "intelligent" or not. (Of course, if you ask a random person they might say LLMs are pretty smart for some tasks, but that's not the same as making a philosophical claim that they're "intelligent")
Of course there's also the AGI and singularity folks. They're kinda loony too.
[deleted](1)
> They had known him for only 15 seconds, yet they still perceived the act of snapping him in half as violent.
This is right out of Community
This is right out of Community
Clip from s01e01: https://www.youtube.com/watch?v=z906aLyP5fg
What if the problem is not that we overestimate LLMs, but that we overestimate intelligence? Or to express the same idea for a more philosophically inclined audience, what if the real mistake isn’t in overestimating LLMs, but in overestimating intelligence itself by imagining it as something more than a web of patterns learned from past experiences and echoed back into the world?
I think AI skeptics have a strong bias to assume that human intelligence fundamentally functions differently from LLMs. They may be correct, but we don't have a strong enough understanding of human cognition to make the claim in as uncertain terms as the skeptical argument is unusually made. The training methods between human learning and machine learning are obviously fundamentally vastly different as are the infrastructure-level mechanics. These elements are likely never going to align, though with time the machine infrastructure may start to increasingly resemble human bio hardware. I bring this up because these known vast differences may account for a significant portion of the differences in expected output from human and machine processing. We don't understand the fundamental conceptual "black box" portions of either form of processing well enough to state definitely what is similar or dissimilar about those hazy areas. Somewhere within that not-well-understood area is what we collectively have vaguely defined "intelligence." But also within that area are all the other aspects that both humans and now machines are quite good at - prediction, fluency, translation. The challenge of lexicon and definition is potentially as difficult a task as is sharpening the focus of our understanding of the hazy black-box portion of both machine processing as well as human processing. Until all those are better defined I don't think we have a good measure for answering the question of machine intelligence either way.
LLM's fail because their input data is limited in dimension compared to humans (text, pictures, audio, video) and because their capacity to rewire their own brains is limited to the transformer architecture.
LLM's can shorten and maybe tend to if you just say "summarize this" but you can trivially ask them to do more. I asked for a summary of Jenson's post and then offer a reflection, GPT-5 said, "It's similar to the Plato’s Cave analogy: humans see shadows (the input text) and infer deeper reality (context, intent), while LLMs either just recite shadows (shorten) or imagine creatures behind them that aren’t there (hallucinate). The “hallucination” behavior is like adding “ghosts”—false constructs that feel real but aren’t grounded.
That ain't shortening because none of that was in his post.
That ain't shortening because none of that was in his post.
I can't decide how to read your last sentence.
That reflection seems totally off to me: fluent, and flavored with elements of the article, but also not really what the article is about and a pretty weird/tortured use of the elements of the allegory of the cave, like it doesn't seem anything like Plato's Cave to me. Ironically demonstrates the actual main gist of the article if you ask me.
But maybe you meant that you think that summary is good and not textually similar to that post so demonstrating something more sophisticated than "shortening".
That reflection seems totally off to me: fluent, and flavored with elements of the article, but also not really what the article is about and a pretty weird/tortured use of the elements of the allegory of the cave, like it doesn't seem anything like Plato's Cave to me. Ironically demonstrates the actual main gist of the article if you ask me.
But maybe you meant that you think that summary is good and not textually similar to that post so demonstrating something more sophisticated than "shortening".
Yes, GPT-5's response above was not shortening because there was nothing in the OP about Plato's Cave. I agree that Plato's cave analogy was confusing here. Here's a better one from GPT-5, which is deeply ironic:
A New Yorker book review often does the opposite of mere shortening. The reviewer:
* Places the book in a broader cultural, historical, or intellectual context.
* Brings in other works—sometimes reviewing two or three books together.
* Builds a thesis that connects them, so the review becomes a commentary on a whole idea-space, not just the book’s pages.
This is exactly the kind of externalized, integrative thinking Jenson says LLMs lack. The New Yorker style uses the book as a jumping-off point for an argument; an LLM “shortening” is more like reading only the blurbs and rephrasing them. In Jenson’s framing, a human summary—like a rich, multi-book New Yorker review—operates on multiple layers: it compresses, but also expands meaning by bringing in outside information and weaving a narrative. The LLM’s output is more like a stripped-down plot synopsis—it can sound polished, but it isn’t about anything beyond what’s already in the text.
A New Yorker book review often does the opposite of mere shortening. The reviewer:
* Places the book in a broader cultural, historical, or intellectual context.
* Brings in other works—sometimes reviewing two or three books together.
* Builds a thesis that connects them, so the review becomes a commentary on a whole idea-space, not just the book’s pages.
This is exactly the kind of externalized, integrative thinking Jenson says LLMs lack. The New Yorker style uses the book as a jumping-off point for an argument; an LLM “shortening” is more like reading only the blurbs and rephrasing them. In Jenson’s framing, a human summary—like a rich, multi-book New Yorker review—operates on multiple layers: it compresses, but also expands meaning by bringing in outside information and weaving a narrative. The LLM’s output is more like a stripped-down plot synopsis—it can sound polished, but it isn’t about anything beyond what’s already in the text.
Essentially, Jenson's complaint is "When I ask an LLM to 'summarize' it interprets that differently from how I think of the word 'summarize' and I shouldn't have to give it more than a one-word prompt because it should infer what I'm asking for."
I think exactly this. When someone is given the task of writing a book review for the New Yorker there is a (probably unstated) agreement that they won't simply summarize the contents, but weave it into an essay in the way the LLM proposed. You could definitely get a similar result from an LLM by giving a more suitable and verbose prompt such as "review these 3 titles together, talk about their shared themes and concepts in a way that is relevant to the contemporary audience" etc etc.
Specifically, Jenson's complaint is that the LLM knows what the word "summarize" means, instead of misunderstanding in the same way Jenson does.
Ah ok, you meant the second thing.
I don't think the Plato's Cave analogy is confusing, I think it's completely wrong. It's "not in the article" in the sense that it is literally not conceptually what the article is about and it's also not really what Plato's Cave is about either, just taking superficial bits of it and slotting things into it, making it doubly wrong.
I don't think the Plato's Cave analogy is confusing, I think it's completely wrong. It's "not in the article" in the sense that it is literally not conceptually what the article is about and it's also not really what Plato's Cave is about either, just taking superficial bits of it and slotting things into it, making it doubly wrong.
And you think the comparison to book reviews is equally bad? Both are from GPT-5.
[deleted]
This is a garbage summary and it really highlights the thrust of the article. My summary: “Human beings have a strong tendency to anthropomorphise and that leads to us granting LLMs human like qualities incorrectly.”
You can see where the LLM has gone wrong. It’s hooked in to “summary” and therefore given excessive emphasis to this part of the article. The plato’s cave analogy is stupid; and what on earth is it going on about with ghosts?
It’s not shortening, sure, it’s dribbling nonsense.
You can see where the LLM has gone wrong. It’s hooked in to “summary” and therefore given excessive emphasis to this part of the article. The plato’s cave analogy is stupid; and what on earth is it going on about with ghosts?
It’s not shortening, sure, it’s dribbling nonsense.
Who are you going to lodge your complaint to that the set of systems and machines that just took your job isn’t “intelligent?”
Humans seem to get wrapped around these concepts like intelligence consciousness etc. because they seem to be the only thing differentiating us from every other animal when in fact it’s all a mirage.
Humans seem to get wrapped around these concepts like intelligence consciousness etc. because they seem to be the only thing differentiating us from every other animal when in fact it’s all a mirage.
I feel this article should be paired with this other one [1] that was on the frontpage a few days ago.
My impression is, there is currently one tendency to "over-anthropomorphize" LLMs and treat them like conscious or even superhuman entities (encouraged by AI tech leaders and AGI/Singularity folks) and another to oversimplify them and view them as literal Markov chains that just got lots of training data.
Maybe those articles could help guarding against both extremes.
[1] https://www.verysane.ai/p/do-we-understand-how-neural-networ...
My impression is, there is currently one tendency to "over-anthropomorphize" LLMs and treat them like conscious or even superhuman entities (encouraged by AI tech leaders and AGI/Singularity folks) and another to oversimplify them and view them as literal Markov chains that just got lots of training data.
Maybe those articles could help guarding against both extremes.
[1] https://www.verysane.ai/p/do-we-understand-how-neural-networ...
Previously when someone called out the tendency to over-anthropomorphize LLMs, a lot of the answers amounted to, “but I like doing it, therefore we should!”
I’ll be the first to say one should pick their battles. But hearing that over and over from a crowd like this that can be quite pedantic is very telling.
I’ll be the first to say one should pick their battles. But hearing that over and over from a crowd like this that can be quite pedantic is very telling.
This very comment thread demonstrates how utterly hopeless it is trying to educate the believers. It has developed into a full-blown religion by now.
I've mentioned this to colleagues at work before.
LLMs give a very strong appearance of intelligence, because humans are super receptive to information provided via our native language. We often have to deal with imperfect speakers and writers, and we must infer context and missing information on our own. We do this so well that we don't know we're doing it. LLMs have perfect grammar and we subtly feel that they are extremely smart because subconsciously we recognize that we don't have to think about anything that's said, it is all syntactically perfect.
So, LLMs sort of trick us into masking their true limitations and believing that they are truly thinking; there are even models that call themselves thinking models, but they don't think, they just predict what the user is going to complain about and say that to themselves as an additional, dynamic prompt on top of the one you actually enter.
LLMs are very good at fooling us into the idea that they know anything at all; they don't. And humans are very bad at being discriminate about the source of the information presented to them if it is presented in a friendly way. The combination of those things is what has resulted in the insanely huge AI hype cycle that we are currently living in the middle of. Nearly everyone is overreacting to what LLMs actually are, and the few of us that believe that we sort of see what's actually happening are ignored for being naysayers, buzz-kills, and luddites. Shunned for not drinking the Kool-Aid.
LLMs give a very strong appearance of intelligence, because humans are super receptive to information provided via our native language. We often have to deal with imperfect speakers and writers, and we must infer context and missing information on our own. We do this so well that we don't know we're doing it. LLMs have perfect grammar and we subtly feel that they are extremely smart because subconsciously we recognize that we don't have to think about anything that's said, it is all syntactically perfect.
So, LLMs sort of trick us into masking their true limitations and believing that they are truly thinking; there are even models that call themselves thinking models, but they don't think, they just predict what the user is going to complain about and say that to themselves as an additional, dynamic prompt on top of the one you actually enter.
LLMs are very good at fooling us into the idea that they know anything at all; they don't. And humans are very bad at being discriminate about the source of the information presented to them if it is presented in a friendly way. The combination of those things is what has resulted in the insanely huge AI hype cycle that we are currently living in the middle of. Nearly everyone is overreacting to what LLMs actually are, and the few of us that believe that we sort of see what's actually happening are ignored for being naysayers, buzz-kills, and luddites. Shunned for not drinking the Kool-Aid.
This effect has been visible on chat forums like Reddit, where incredibly stupid people somehow manage to write well enough to seem intelligent, until you read their full exposition of their disordered thinking.
You're ignored not because you're right or wrong, but because your unsolicited advice is not useful.
For example, I can spin up any LLM and get it to translate some English text into Japanese with maybe 99% accuracy. I don't need to believe whether it "really knows" English or Japanese, I only need to believe the output is accurate.
Similarly I can ask a LLM to code up a function that does a specific thing, and it will do it with high accuracy. Maybe there'll be some bugs, but I can review the code and fix them, which in some cases boosts my productivity. I don't need to believe whether it "really knows" C++ or Rust, I only need it to write something good enough.
I mean, just by these two examples, LLMs are really great tools, and I'm personally hyped for these use cases alone. Am I fooled by the LLM? I don't think so, I don't have any fantasy about it being extremely intelligent or always being right. I doubt most reasonable people these days would either.
So basically you're going about assuming people are fooled by LLMs (which they might not be), and wondering why you're unpopular when you're basically telling everyone they're gullible and foolish.
For example, I can spin up any LLM and get it to translate some English text into Japanese with maybe 99% accuracy. I don't need to believe whether it "really knows" English or Japanese, I only need to believe the output is accurate.
Similarly I can ask a LLM to code up a function that does a specific thing, and it will do it with high accuracy. Maybe there'll be some bugs, but I can review the code and fix them, which in some cases boosts my productivity. I don't need to believe whether it "really knows" C++ or Rust, I only need it to write something good enough.
I mean, just by these two examples, LLMs are really great tools, and I'm personally hyped for these use cases alone. Am I fooled by the LLM? I don't think so, I don't have any fantasy about it being extremely intelligent or always being right. I doubt most reasonable people these days would either.
So basically you're going about assuming people are fooled by LLMs (which they might not be), and wondering why you're unpopular when you're basically telling everyone they're gullible and foolish.
> So basically you're going about assuming people are fooled by LLMs
no assumption. people believe that these things think. I encourage you to hang out in non-technical places and listen to people talking about what LLMs do. "Mine has a personality!" People who do not understand the limitations of LLMs believe these things are alive.
no assumption. people believe that these things think. I encourage you to hang out in non-technical places and listen to people talking about what LLMs do. "Mine has a personality!" People who do not understand the limitations of LLMs believe these things are alive.
You can compare the current state of LLMs to the days of chess machines when they first approached grandmaster level play. The machine approach was very brute force, and there was a lot of work done to improve the sheer amount of look ahead that was required to complete at the grandmaster level.
As opposed to what grandmasters actually did, which was less look ahead and more pattern matching to strengthen the position.
Now LLMs successfully leverage pattern matching, but interestingly it is still a kind of brute force pattern matching, requiring the statistical absorption of all available texts, far more than a human absorbs in a lifetime.
This enables the LLM to interpolate an answer from the structure of the absorbed texts with reasonable statistical relevance. This is still not quite “what humans do” as it still requires brute force statistical analysis of vast amounts of text to achieve pretty good results. For example training on all available Python sources in github and elsewhere (curated to avoid bad examples) yields pretty good results, not how a human would do it, but statistically likely to be pertinent and correct.
As opposed to what grandmasters actually did, which was less look ahead and more pattern matching to strengthen the position.
Now LLMs successfully leverage pattern matching, but interestingly it is still a kind of brute force pattern matching, requiring the statistical absorption of all available texts, far more than a human absorbs in a lifetime.
This enables the LLM to interpolate an answer from the structure of the absorbed texts with reasonable statistical relevance. This is still not quite “what humans do” as it still requires brute force statistical analysis of vast amounts of text to achieve pretty good results. For example training on all available Python sources in github and elsewhere (curated to avoid bad examples) yields pretty good results, not how a human would do it, but statistically likely to be pertinent and correct.
One key difference is that an LLM really is better than any human at almost everything, simply because the LLM is kind of OKish at millions of things that each single human is just terrible at or completely ignorant of.
The common LLMs are not trying to be a better human than a single human. They are trying to be more useful per $ cost than a bevy of humand doing some basket of human tasks.
Making a replacement for an individual complete human, an effort that was popular before the Internet made that idea seem quaint, is very different challenge, and more niche, because it's not as economically efficient.
The common LLMs are not trying to be a better human than a single human. They are trying to be more useful per $ cost than a bevy of humand doing some basket of human tasks.
Making a replacement for an individual complete human, an effort that was popular before the Internet made that idea seem quaint, is very different challenge, and more niche, because it's not as economically efficient.
Seems like this is close to the Uncanny Valley effect.
LLM intelligence is in the spot where it is simultaneously genius-level but also just misses the mark a tiny bit, which really sticks out for those who have been around humans their whole lives.
I feel that, just like more modern CGI, this will slowly fade with certain techniques and you just won't notice it when talking to or interacting with AI.
Just like in his post during the whole Matrix discussion.
> "When I asked for examples, it suggested the Matrix and even gave me the “Summary” and “Shortening” text, which I then used here word for word. "
He switches in AI-written text and I bet you were reading along just the same until he pointed it out.
This is our future now I guess.
LLM intelligence is in the spot where it is simultaneously genius-level but also just misses the mark a tiny bit, which really sticks out for those who have been around humans their whole lives.
I feel that, just like more modern CGI, this will slowly fade with certain techniques and you just won't notice it when talking to or interacting with AI.
Just like in his post during the whole Matrix discussion.
> "When I asked for examples, it suggested the Matrix and even gave me the “Summary” and “Shortening” text, which I then used here word for word. "
He switches in AI-written text and I bet you were reading along just the same until he pointed it out.
This is our future now I guess.
Shades of Jonathan Lethem's classic "The Ecstasy of Influence: A plagiarism"
https://harpers.org/archive/2007/02/the-ecstasy-of-influence...
I might be mixing the concepts of intelligence and conscience etc, but the human mind is more than language and data; it's also experience. LLMs have all the data and can express anything around that context, but will never experience anything, which is singular for each of us, and it's part of what makes what we call intelligence (?). So they will never replicate the human mind; they can just mimic it.
I heard from Miguel Nicolelis that language is a filter for the human mind, so you can never build a mind from language. I interpreted this like trying to build an orange from its juice.
I heard from Miguel Nicolelis that language is a filter for the human mind, so you can never build a mind from language. I interpreted this like trying to build an orange from its juice.
> LLMs have all the data and can express anything around that context, but will never experience anything,
On the contrary, all their training data is their "experience".
On the contrary, all their training data is their "experience".
Good article, it's been told before but it bears repeating.
Also I got caught on this one kind of irrelevant point regarding the characterization of the Matrix: I would say Matrix is not just diguised as a story about escaping systems of control, it's quite clearly about oppressive systems in society, with specific reference to gender expression. Lilly Wachowski has explicitly stated that it was supposed to be an allegory for gender transition.
Also I got caught on this one kind of irrelevant point regarding the characterization of the Matrix: I would say Matrix is not just diguised as a story about escaping systems of control, it's quite clearly about oppressive systems in society, with specific reference to gender expression. Lilly Wachowski has explicitly stated that it was supposed to be an allegory for gender transition.
It wasn't. Switch was intended to be genderfluid, but the Matrix itself, or "logging out" of it was apparently not meant as an allegory for transitioning (though she doesn't mind the interpretation) :
https://www.them.us/story/lilly-wachowski-work-in-progress-s...
https://www.them.us/story/lilly-wachowski-work-in-progress-s...
The character Switch was supposed to have a different gender in the matrix vs real life. It’s really a shame that didn’t happen.
There is nothing more free than the freedom to be who you really are.
Going to rewatch the Matrix tonight.
Going to rewatch the Matrix tonight.
Regarding Timmy, the Companion Cube from the game Portal is the greatest example of induced anthropomorphism that I've ever experienced. If you know, you know, and if you don't, you should really play the game, since it's brilliant.
It is a brilliant game and the empathy you develop for the cube is a great concept.
But arguably much deeper is the fact that nothing in this game, or any single-player game, is a living thing in any form. Arguably the game's characterization of GLaDOS hits even harder on the anthropomorphism angle.
But arguably much deeper is the fact that nothing in this game, or any single-player game, is a living thing in any form. Arguably the game's characterization of GLaDOS hits even harder on the anthropomorphism angle.
The cube doesn't work, or at least it didn't for me. The goggly eyes really do make a difference.
That's a matter of informed anthropomorphism. A lot of people don't become attached to the Companion Cube, but are informed that their player character is so attached.
I'm on a Mac, and would love to see Portal 2 (at least) ported to M-chips.
I would love Portal 3, even more.
I would love Portal 3, even more.
Floyd in Planetfall was that for those of us that are older.
Intelligence is a social construct? Interesting thought, but denies the innovator credit for their efforts. Does technology naturally evolve in an 'intelligent' society? I don't think so. Now without somebody going deep into the weeds to find gold. Without individual genius, we would be far behind in our collective progress.
That's a great article.
Scott Jenson is one of my favorite authors.
He's really big on integrating an understanding of basic human nature, into design.
Scott Jenson is one of my favorite authors.
He's really big on integrating an understanding of basic human nature, into design.
A lot of negative comments in this thread, but I think the article makes one really good point, that has had far-reaching consequences for for society before and beyond AI: humans are pretty bad at assessing intelligence.
The author's argument is built on fallacies that always pop up in these kinds of critiques.
The "summary vs shortening" distinction is moving the goalposts. They makes the empirical claim that LLMs fail at summarizing novel PDFs without any actual evidence. For a model trained on a huge chunk of the internet, the line between "reworking existing text" and "drawing on external context" is so blurry it's practically meaningless.
Similarly, can we please retire the ELIZA and Deep Blue analogies? Comparing a modern transformer to a 1960s if-then script or a brute-force chess engine is a category error. It's a rhetorical trick to make LLMs seem less novel than they actually are.
And blaming everything on anthropomorphism is an easy out. It lets you dismiss the model's genuinely surprising capabilities by framing it as a simple flaw in human psychology. The interesting question isn't that we anthropomorphize, but why this specific technology is so effective at triggering that response from humans.
The whole piece basically boils down to: "If we define intelligence in a way that is exclusively social and human, then this non-social, non-human thing isn't intelligent." It's a circular argument.
The "summary vs shortening" distinction is moving the goalposts. They makes the empirical claim that LLMs fail at summarizing novel PDFs without any actual evidence. For a model trained on a huge chunk of the internet, the line between "reworking existing text" and "drawing on external context" is so blurry it's practically meaningless.
Similarly, can we please retire the ELIZA and Deep Blue analogies? Comparing a modern transformer to a 1960s if-then script or a brute-force chess engine is a category error. It's a rhetorical trick to make LLMs seem less novel than they actually are.
And blaming everything on anthropomorphism is an easy out. It lets you dismiss the model's genuinely surprising capabilities by framing it as a simple flaw in human psychology. The interesting question isn't that we anthropomorphize, but why this specific technology is so effective at triggering that response from humans.
The whole piece basically boils down to: "If we define intelligence in a way that is exclusively social and human, then this non-social, non-human thing isn't intelligent." It's a circular argument.
This, along with a ton of commentary on LLMs, seems like its written by someone who has no technical understanding of LLMs.
> A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control.
How is that a summary? It reads as a one-liner review I would leave on Letterboxed or something I would say, trying to be pretentious and treating the movie as a work of art. It is a work of art, because all movies are art, but that's an awful summary.
How is that a summary? It reads as a one-liner review I would leave on Letterboxed or something I would say, trying to be pretentious and treating the movie as a work of art. It is a work of art, because all movies are art, but that's an awful summary.
Review? Where does it say anything about the quality of the film?
* * * * (4/5)
A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control.
----- Do I need to explain more?
A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control.
----- Do I need to explain more?
The LLMs are like a Huffman codec except the context is infinite and lossy
There's a lot of moving the goal posts here. What would it mean for an AI to be considered "intelligent"? At various points it meant different things (chess, jeopardy, go, human language, math, etc). The most well-established benchmark was the Turing test, which LLMs now pass spectacularly -- in fact the only way to reliably tell that you're talking to an AI is usually because it responds too fast and too perfectly.
Sure, AI often makes mistakes, but so do humans. Ask a random person a random question and I guarantee that at least 90% of the times an LLM will produce a better answer. So if knowledge or reasoning mistakes are a benchmark, then humans are not intelligent either.
What is the difference then? I feel that whoever claims that AI is not intelligent simply feels uncomfortable thinking that they are, and is going out of their way to rationalize that, by using various double standards. Like, AI making occasional mistakes counts as evidence against it, but the fact that the vast majority of people are incapable of answering some of the most basic questions somehow doesn't count against human intelligence.
And then there are various contrived "definitions of intelligence", such as "living in the physical world". These have nothing to do with cognitive intelligence and are just made up to make humans fit the definition and AI to not fit. Doing "truly novel things"? How many people have ever come up with anything truly novel? What about the other 99.99%, are they intelligent?
I think there's one thing that's tripping people up. Humans do have a very high COLLECTIVE intelligence. Collectively (and given enough time) we're capable of accomplishing great things, much more than current AI. But each individual human on their own is no more intelligent than a caveman (we have essentially the same brain).
Sure, AI often makes mistakes, but so do humans. Ask a random person a random question and I guarantee that at least 90% of the times an LLM will produce a better answer. So if knowledge or reasoning mistakes are a benchmark, then humans are not intelligent either.
What is the difference then? I feel that whoever claims that AI is not intelligent simply feels uncomfortable thinking that they are, and is going out of their way to rationalize that, by using various double standards. Like, AI making occasional mistakes counts as evidence against it, but the fact that the vast majority of people are incapable of answering some of the most basic questions somehow doesn't count against human intelligence.
And then there are various contrived "definitions of intelligence", such as "living in the physical world". These have nothing to do with cognitive intelligence and are just made up to make humans fit the definition and AI to not fit. Doing "truly novel things"? How many people have ever come up with anything truly novel? What about the other 99.99%, are they intelligent?
I think there's one thing that's tripping people up. Humans do have a very high COLLECTIVE intelligence. Collectively (and given enough time) we're capable of accomplishing great things, much more than current AI. But each individual human on their own is no more intelligent than a caveman (we have essentially the same brain).
I disagree with the author in a big way. '25's LLMs are designed to echo material already out there on the Internet if it exists because we value that more. If I want a summary of The Matrix, I prefer a summary that agrees with the zeitgeist, rather than a novel, unorthodox summary that requires a justification as to its deviation.
In fact, the example provided by the author is a great illustration of this:
> A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control.
The words here refer back to the notions "free will" that is prominent in Western discourse from St. Augustine through Descartes and thereafter and similarly of "sci-fi". These are notions an uneducated East-Asian with limited Internet use and pop culture fluency will simply not understand. They would in fact prefer the latter description. The author and this hypothetical East-Asian live in very different zeitgeists, and correspondingly experience the movie differently and value different summaries of the film. They each prefer a summary that agrees with the zeitgeist, rather than a novel, unorthodox summary (relative to their zeitgeist) that requires a justification as to its deviation.
On the other hand, if you asked LLMs to explain material and concepts, one modality in which it does is use formulaic, novel, and unorthodox analogies to explain it to you. By formulaic and novel, I mean that the objects and scenarios in the analogies are frequently of a certain signature kind that it has been trained with, but it is novel in that the analogies are not found in the wild on the internet.
If you have frequently used LLMs for the purpose of explaining concepts, you will have encountered these analogies and know what I mean by this. The analogies are frequently not too accurate, but they round out the response by giving an ELI5-style answer.
Ironically, the author may have succumbed to LLM sycophancy.
In fact, the example provided by the author is a great illustration of this:
> A philosophical exploration of free will and reality disguised as a sci-fi action film about breaking free from systems of control.
The words here refer back to the notions "free will" that is prominent in Western discourse from St. Augustine through Descartes and thereafter and similarly of "sci-fi". These are notions an uneducated East-Asian with limited Internet use and pop culture fluency will simply not understand. They would in fact prefer the latter description. The author and this hypothetical East-Asian live in very different zeitgeists, and correspondingly experience the movie differently and value different summaries of the film. They each prefer a summary that agrees with the zeitgeist, rather than a novel, unorthodox summary (relative to their zeitgeist) that requires a justification as to its deviation.
On the other hand, if you asked LLMs to explain material and concepts, one modality in which it does is use formulaic, novel, and unorthodox analogies to explain it to you. By formulaic and novel, I mean that the objects and scenarios in the analogies are frequently of a certain signature kind that it has been trained with, but it is novel in that the analogies are not found in the wild on the internet.
If you have frequently used LLMs for the purpose of explaining concepts, you will have encountered these analogies and know what I mean by this. The analogies are frequently not too accurate, but they round out the response by giving an ELI5-style answer.
Ironically, the author may have succumbed to LLM sycophancy.
[deleted]
[deleted]
Firstly, I think a lot of commenters here should ask themselves "Do you believe that ANY machine could EVER be intelligent?"
Unless you believe in the magic sauce of a supernatural soul/mind/etc, our brains function as deterministic biological machines. There's essentially nothing that separates the processing and memory potential of silicon from neurons. And if the building blocks can be made analogous, then all the same emergent properties are possible. There's no reason to believe a circuit couldn't be made to behave in exactly the same way as a human brain. I'm not saying that's where LLMs are; only that it is theoretically possible. So if you imagine such a machine, and you deem that it is not intelligent, you have reserved intelligence as an exclusive human trait and this entire discussion is meaningless.
Secondly, although I'm not in either pro or anti LLM-intelligence camp, I find a lot of the arguments against machine intelligence disingenuous and/or unbalanced.
For instance the "Can't process information it's not familiar with" argument. Another commenter stated the case of scientific papers that it doesn't have any reference for, that it may hallucinate a garbage interpretation of the paper. Not surprising, but guess what, a human would do the same thing if they were forced! Imagine holding a gun to someone's head and telling them to explain a concept or system they've never heard of. That's essentially what we're doing with LLMs; obviously we don't need to threaten, because we haven't given them agency to say no.
Another example is the "Can't be novel, unique, or create something completely new." First of all, difficult to prove, but okay let's take it as given that an LLM can't be novel. Can you prove that a human can? We make all these assumptions on how intelligent and creative we are as humans, and how original our thoughts can be... but how original are they, and can we prove it? How do you know your original thought, or Beethoven's 5th, or the fast inverse square root trick was completely separate from any prior influences? Or... was that "original thought" the conglomeration of a thousand smaller inputs and data points that you trained on, that became part of your brains subconscious processing system, and came together in a synthesis that looks like brilliance.
Finally, whenever this discussion comes up with friends I ask them to think of the least intelligent person they know. Then imagine how many of those there are in the world (likely millions). Could you imagine any conceivable test of any length or depth which would designate all the those humans as intelligent, and all the LLMs as not? I certainly can't. I highly doubt there could be anything approaching 100% accuracy at this point.
Ultimately I think we should ditch both the intelligence and consciousness questions. We can't define them in ourselves, we certainly can't define them in another entity. Let's just come to terms with the fact we're highly functioning biological machines who are both scared and excited to have created something so similar to us.
Unless you believe in the magic sauce of a supernatural soul/mind/etc, our brains function as deterministic biological machines. There's essentially nothing that separates the processing and memory potential of silicon from neurons. And if the building blocks can be made analogous, then all the same emergent properties are possible. There's no reason to believe a circuit couldn't be made to behave in exactly the same way as a human brain. I'm not saying that's where LLMs are; only that it is theoretically possible. So if you imagine such a machine, and you deem that it is not intelligent, you have reserved intelligence as an exclusive human trait and this entire discussion is meaningless.
Secondly, although I'm not in either pro or anti LLM-intelligence camp, I find a lot of the arguments against machine intelligence disingenuous and/or unbalanced.
For instance the "Can't process information it's not familiar with" argument. Another commenter stated the case of scientific papers that it doesn't have any reference for, that it may hallucinate a garbage interpretation of the paper. Not surprising, but guess what, a human would do the same thing if they were forced! Imagine holding a gun to someone's head and telling them to explain a concept or system they've never heard of. That's essentially what we're doing with LLMs; obviously we don't need to threaten, because we haven't given them agency to say no.
Another example is the "Can't be novel, unique, or create something completely new." First of all, difficult to prove, but okay let's take it as given that an LLM can't be novel. Can you prove that a human can? We make all these assumptions on how intelligent and creative we are as humans, and how original our thoughts can be... but how original are they, and can we prove it? How do you know your original thought, or Beethoven's 5th, or the fast inverse square root trick was completely separate from any prior influences? Or... was that "original thought" the conglomeration of a thousand smaller inputs and data points that you trained on, that became part of your brains subconscious processing system, and came together in a synthesis that looks like brilliance.
Finally, whenever this discussion comes up with friends I ask them to think of the least intelligent person they know. Then imagine how many of those there are in the world (likely millions). Could you imagine any conceivable test of any length or depth which would designate all the those humans as intelligent, and all the LLMs as not? I certainly can't. I highly doubt there could be anything approaching 100% accuracy at this point.
Ultimately I think we should ditch both the intelligence and consciousness questions. We can't define them in ourselves, we certainly can't define them in another entity. Let's just come to terms with the fact we're highly functioning biological machines who are both scared and excited to have created something so similar to us.
I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not?
I see two possibilities:
1. We define "intelligence" as definitionally unique to humans. For example, maybe intelligence depends on the existence of a human soul, or specific to the physical structure of the human brain. In this case, a machine (perhaps an LLM) could achieve "quacks like a duck" behavioral equality to a human mind, and yet would still be excluded from the definition of "intelligent." This definition is therefore not useful if we're interested in the ability of the machine, which it seems to me we are. LLMs are often dismissed as not "intelligent" because they work by inferring output based on learned input, but that alone cannot be a distinguishing characteristic, because that's how humans work as well.
2. We define "intelligence" in a results-oriented way. This means there must be some specific test or behavioral standard that a machine must meet in order to become intelligent. This has been the default definition for a long time, but the goal posts have shifted. Nevertheless, if you're going to disparage LLMs by calling them unintelligent, you should be able to cite a specific results-oriented failure that distinguishes them from "intelligent" humans. Note that this argument cannot refer to the LLMs' implementation or learning model.