Two years later, deep learning is still faced with the same basic challenges(garymarcus.substack.com)
garymarcus.substack.com
Two years later, deep learning is still faced with the same basic challenges
https://garymarcus.substack.com/p/two-years-later-deep-learning-is
63 comments
The point Marcus is making is that LLMs are solely next word predictors - intelligence requires far more than that.
LLMs will always have the same basic challenges - their 'kinda copy what a human has written' algorithm has hit its limit.
He is an advocate of symbolic learning - getting that to work is a lot harder, however it's far more likely to result in genuine intelligence.
LLMs will always have the same basic challenges - their 'kinda copy what a human has written' algorithm has hit its limit.
He is an advocate of symbolic learning - getting that to work is a lot harder, however it's far more likely to result in genuine intelligence.
Like Yann LeCun said, Gary Marcus has contributed exactly nothing to the field, he's an influencer that claims to be an expert. Just ignore him.
to wit: his appearance on Ezra Klein was pretty terrible
>It’s a kind of glorified cut and paste. Pastiche is putting together things kind of imitating a style. And in some sense, that’s what it’s doing. It’s imitating particular styles, and it’s cutting and pasting a lot of stuff. It’s a little bit more complicated than that. But to a first approximation, that’s what it’s doing is cutting and pasting things.
This argument is terrible because it proves too much. Either AIs are automata with no connection to the world, or if they are connected to the world its just copy-paste.
And somehow infants aren't just copy paste because uuuuh they have "true understanding" whatever that is.
[1] https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-...
>It’s a kind of glorified cut and paste. Pastiche is putting together things kind of imitating a style. And in some sense, that’s what it’s doing. It’s imitating particular styles, and it’s cutting and pasting a lot of stuff. It’s a little bit more complicated than that. But to a first approximation, that’s what it’s doing is cutting and pasting things.
This argument is terrible because it proves too much. Either AIs are automata with no connection to the world, or if they are connected to the world its just copy-paste.
And somehow infants aren't just copy paste because uuuuh they have "true understanding" whatever that is.
[1] https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-...
I can afford one. But if it tells me lies and picks up the wrong remote I want a refund, as would most customers spending more than $1000 on pretty much anything. And that's the real issue, I think. The marketing has far surpassed the reality of the actual product. People buy helicopters, and jets, and houses larger than an office building. It's not the cost that's at issue. There's room for bullshit in a chat bot. But there's absolutely zero tolerance for bullshit in a physical device roaming around children.
I think the point of the article is that no one has even presented a workable theory for real AI with real comprehension. LLM isn't AI, regardless of the marketing. Until that theory exists, and is implemented, ideas about what AI will someday grow into are ignoring comprehension. That's a required component if you are using the word "intelligence" outside of fiction.
I think the point of the article is that no one has even presented a workable theory for real AI with real comprehension. LLM isn't AI, regardless of the marketing. Until that theory exists, and is implemented, ideas about what AI will someday grow into are ignoring comprehension. That's a required component if you are using the word "intelligence" outside of fiction.
there's absolutely zero tolerance for bullshit in a physical device roaming around children
Why not just put a muzzle on it?
LLM isn't AI, regardless of the marketing
That’s intelism. Different models have different capabilities, you can’t decide which is good enough for you to treat as intelligent. You use terms like intelligence and comprehension, which don’t even have a definition better than “I know it when I see it”.
Not even going to discuss that you want to buy an actually intelligent being for picking up remotes.
Why not just put a muzzle on it?
LLM isn't AI, regardless of the marketing
That’s intelism. Different models have different capabilities, you can’t decide which is good enough for you to treat as intelligent. You use terms like intelligence and comprehension, which don’t even have a definition better than “I know it when I see it”.
Not even going to discuss that you want to buy an actually intelligent being for picking up remotes.
Whoops. You've contradicted yourself.
I didn't advertise a robot picking up a remote for sale as intelligent. I'm advocating that we stop calling it intelligent when it isn't. You're saying it's intelligent.
So either you're ready to offend the previous commenter for their evil advocacy of selling an intelligent robot made to pick up remotes for an advertisement, or you're at odds with your point because you know damn well it's not intelligent.
My guess is the latter.
From the dictionary: the ability to learn or understand or to deal with new or trying situations : REASON also : the skilled use of reason (2) : the ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests)
This robot or ChatGPT, or any LLM theorized to my knowledge does not meet that definition due to the underlying technology not approaching "abstract" or "skilled use of reason."
It's pretty simply not addressing those items. There's no debate. It's a completely different approach to something that has nothing to do with intelligence.
And marketing it as such is frankly a legal issue, in my opinion. I fully expect the lawsuits to continue until marketers are forced to stop using that term entirely.
As an additional note, you've muddied up a lot of concepts here. I'm not sure if you're maybe not a native speaker of English, as that might explain a lot, or if you really accidentally made opposing points sequentially. The latter is a bit funny in a discussion about intelligence. "Intelligent robot" wouldn't refer to one that feels and needs civil rights. The definition is above for your review. It's very much clear, despite your protest, what intelligence means. It is, by definition, not subjective, but objective. And there are absolutely legal definitions with very real legal ramifications.
Also, you should read back through your own comments. You're not a very likable person, and I'm telling you that as a personal favor. You should get some help for this. It's critical for your personal relationships, and it's not very expensive.
I didn't advertise a robot picking up a remote for sale as intelligent. I'm advocating that we stop calling it intelligent when it isn't. You're saying it's intelligent.
So either you're ready to offend the previous commenter for their evil advocacy of selling an intelligent robot made to pick up remotes for an advertisement, or you're at odds with your point because you know damn well it's not intelligent.
My guess is the latter.
From the dictionary: the ability to learn or understand or to deal with new or trying situations : REASON also : the skilled use of reason (2) : the ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests)
This robot or ChatGPT, or any LLM theorized to my knowledge does not meet that definition due to the underlying technology not approaching "abstract" or "skilled use of reason."
It's pretty simply not addressing those items. There's no debate. It's a completely different approach to something that has nothing to do with intelligence.
And marketing it as such is frankly a legal issue, in my opinion. I fully expect the lawsuits to continue until marketers are forced to stop using that term entirely.
As an additional note, you've muddied up a lot of concepts here. I'm not sure if you're maybe not a native speaker of English, as that might explain a lot, or if you really accidentally made opposing points sequentially. The latter is a bit funny in a discussion about intelligence. "Intelligent robot" wouldn't refer to one that feels and needs civil rights. The definition is above for your review. It's very much clear, despite your protest, what intelligence means. It is, by definition, not subjective, but objective. And there are absolutely legal definitions with very real legal ramifications.
Also, you should read back through your own comments. You're not a very likable person, and I'm telling you that as a personal favor. You should get some help for this. It's critical for your personal relationships, and it's not very expensive.
> Why not just put a muzzle on it?
I think he's talking about the Google robot that was linked in the root. "Muzzing" a domestic robot would probably mean putting it in a cage, which would defeat the whole point. That's not a solution.
I think he's talking about the Google robot that was linked in the root. "Muzzing" a domestic robot would probably mean putting it in a cage, which would defeat the whole point. That's not a solution.
I have no idea whom this guy is, and have no thoughts here on his opinions or their falsifiability (happy to agree with you on that aspect if it matters), but just to reply to this particular point here:
> This is well beyond what was considered possible at the time. We're certainly not at 'everything' but shrug. Maybe in ten years.
This feels a lot like what people were saying about self-driving cars being imminent circa... 2015 or so? [1] The skeptical folks rolled their eyes at suggestions that we'd have self driving cars everywhere in a few years, but lo and behold, they were right. Just because we had a massive amount of progress in the years before, that doesn't mean we were on the cusp of achieving the goal. Turns out going from 99.9% accurate to 99.99% accurate (or whatever the numbers were) is harder than all the believers wanted to admit.
This feels like the same thing all over again. Yes, there's been a lot of progress in LLMs. No, that doesn't mean we're anywhere close to deep learning being able to do 'everything' in 10 years, whatever that means.
[1] https://www.theverge.com/24065447/self-driving-car-autonomou...
> This is well beyond what was considered possible at the time. We're certainly not at 'everything' but shrug. Maybe in ten years.
This feels a lot like what people were saying about self-driving cars being imminent circa... 2015 or so? [1] The skeptical folks rolled their eyes at suggestions that we'd have self driving cars everywhere in a few years, but lo and behold, they were right. Just because we had a massive amount of progress in the years before, that doesn't mean we were on the cusp of achieving the goal. Turns out going from 99.9% accurate to 99.99% accurate (or whatever the numbers were) is harder than all the believers wanted to admit.
This feels like the same thing all over again. Yes, there's been a lot of progress in LLMs. No, that doesn't mean we're anywhere close to deep learning being able to do 'everything' in 10 years, whatever that means.
[1] https://www.theverge.com/24065447/self-driving-car-autonomou...
"Turns out going from 99.9% accurate to 99.99% accurate (or whatever the numbers were) is harder than all the believers wanted to admit."
True. But Calfornia's DMV makes Waymo publish their stats, and they get maybe 50% better each year. That's a good rate of gain.
True. But Calfornia's DMV makes Waymo publish their stats, and they get maybe 50% better each year. That's a good rate of gain.
Of course they've been making good gains. We may very well get there someday. It's just nowhere near the timeline many experts were so confident in.
> Waymo [..] get[s] maybe 50% better each year. That's a good rate of gain.
Remind me what we're measuring?
"A seasoned San Francisco cab driver might have avoided the intersection of Jackson Street and Grant Avenue, in the heart of the city's Chinatown on the first day of Chinese New Year. An autonomous Waymo robotaxi, by contrast, drove toward the cross streets on Saturday evening, when crowds were blocking all sides and revelers were lighting fireworks, according to two witnesses. Minutes later, some in the crowd attacked the car and set it on fire."
https://www.reuters.com/business/autos-transportation/san-fr...
Remind me what we're measuring?
"A seasoned San Francisco cab driver might have avoided the intersection of Jackson Street and Grant Avenue, in the heart of the city's Chinatown on the first day of Chinese New Year. An autonomous Waymo robotaxi, by contrast, drove toward the cross streets on Saturday evening, when crowds were blocking all sides and revelers were lighting fireworks, according to two witnesses. Minutes later, some in the crowd attacked the car and set it on fire."
https://www.reuters.com/business/autos-transportation/san-fr...
Would you apply the same line of reasoning to flying? That branch of engineering had a huge amount of detractors too who were confidently right until they weren't. My point is, no one knows how close we are to AGI. Taking the sceptical stance is rewarding because you're right most of the time, except when you aren't in which case you're astronomically wrong.
> Would you apply the same line of reasoning to flying?
No, because I have no idea who predicted what at which point in time.
Or yes, because it took [hundreds? thousands?] of years before someone managed to get it to work. We've only worked on LLMs for so long.
i.e. I have no idea how to answer your question, other than to point out the prediction here was not "never", but rather "not in the next 10 years".
No, because I have no idea who predicted what at which point in time.
Or yes, because it took [hundreds? thousands?] of years before someone managed to get it to work. We've only worked on LLMs for so long.
i.e. I have no idea how to answer your question, other than to point out the prediction here was not "never", but rather "not in the next 10 years".
>The skeptical folks rolled their eyes at suggestions that we'd have self driving cars everywhere in a few years, but lo and behold, they were right.
What? Where?
What? Where?
Lots of places, including here on HN. Here's a random link from Googling right now: https://www.technologyreview.com/2013/10/22/175716/driverles...
> But such projections tend to overlook just how challenging it will be to make a driverless car. [...] It could take decades for the technology to come down in cost, and it might take even longer for it to work safely enough that we trust fully automated vehicles to drive us around.
> But such projections tend to overlook just how challenging it will be to make a driverless car. [...] It could take decades for the technology to come down in cost, and it might take even longer for it to work safely enough that we trust fully automated vehicles to drive us around.
What do mean, "not falsifiable"?
Do we have self-driving cars? LLM attorneys? LLM radiologists?
No, no and no.
And the continuous failure of self-driving cars should worry you.
Do we have self-driving cars? LLM attorneys? LLM radiologists?
No, no and no.
And the continuous failure of self-driving cars should worry you.
The one that jumped out to me was his claiming that computers don’t truly understand language.
At this point I don’t know what that means. I’ll be the first to admit the limitations of statistical language models, but for people like Marcus who have made a name for themselves as skeptics, it’s not in their interests to have falsifiable criticisms.
At this point I don’t know what that means. I’ll be the first to admit the limitations of statistical language models, but for people like Marcus who have made a name for themselves as skeptics, it’s not in their interests to have falsifiable criticisms.
We do have self driving cars.
Tesla says:
>The currently enabled Autopilot, Enhanced Autopilot and Full Self-Driving features require active driver supervision and do not make the vehicle autonomous.
Also, how long will you have self-driving? It is notorious for having dangerous regressions after an update. Those AI systems are black boxes with stochastic behavior.
>The currently enabled Autopilot, Enhanced Autopilot and Full Self-Driving features require active driver supervision and do not make the vehicle autonomous.
Also, how long will you have self-driving? It is notorious for having dangerous regressions after an update. Those AI systems are black boxes with stochastic behavior.
And humans in cars are meat-boxes of stochastic behavior, prone to all kinds of dangerous regressions.
I feel safer, as a cyclist, around Waymos than I do around many human-driven cars. That's even with the cyclist crash last week: it was one of the most common kinds of bicycle-car collisions (bike moving through an intersection hidden behind a truck, hit by a car turning left), and I feel confident that the vehicle responded faster to the situation than a human would have.
I feel safer, as a cyclist, around Waymos than I do around many human-driven cars. That's even with the cyclist crash last week: it was one of the most common kinds of bicycle-car collisions (bike moving through an intersection hidden behind a truck, hit by a car turning left), and I feel confident that the vehicle responded faster to the situation than a human would have.
My partner rides self-driving cars frequently, over a dozen times this year. The technology clearly and manifestly exists.
Glorifying a lane assist that only works on highways by good weather is not self-driving.
From Tesla's own admission:
>The currently enabled Autopilot, Enhanced Autopilot and Full Self-Driving features require active driver supervision and do not make the vehicle autonomous.
From Tesla's own admission:
>The currently enabled Autopilot, Enhanced Autopilot and Full Self-Driving features require active driver supervision and do not make the vehicle autonomous.
Waymo is doing just fine...
After 5.5 billion dollars and 15 years of work... they taxi a few people in a few select cities.
Not what was promised.
https://electrek.co/2015/12/21/tesla-ceo-elon-musk-drops-pre...
A hundred billions have been sunk in other self-driving ventures that flopped without much for show. Wonder what could have been made with that money if the hyperhype didn't sink it there.
Not what was promised.
https://electrek.co/2015/12/21/tesla-ceo-elon-musk-drops-pre...
A hundred billions have been sunk in other self-driving ventures that flopped without much for show. Wonder what could have been made with that money if the hyperhype didn't sink it there.
That was Elon Musk's promise... He sucks, I agree. But he is not the whole story, here.
We do seem to have LLM attorneys, every so often there is a news story where one gets caught because the LLM fabricated a reference.
But if they are getting caught sometimes, that means it is happening undetected in others.
But if they are getting caught sometimes, that means it is happening undetected in others.
"LLM's look poised to upend robotics."
Maybe. Unstructured manipulation is getting better very slowly. But videos such as the Google one and this one [1] are mostly about making the system take instructions in natural language, not doing the actual picking.
Bin-picking of uniform items is a solved problem, and was first working in the 1980s. Bin-picking of random items from a cluttered bin is still a struggle. Amazon has put a fair amount of effort into robotic bin-picking, but it's not yet good enough to deploy.
There's been some good progress on bin-picking recently. Here's Brightpick's system.[2] It has a good vision system which involves LIDAR, and some nice engineering. It's not LLM-based, and it won't do everything, but it can handle most objects that can be picked up by a vacuum picker.
[1] https://www.youtube.com/watch?v=31Xqw_59XZ8
[2] https://www.youtube.com/watch?v=XWlQWgRfh6c
Maybe. Unstructured manipulation is getting better very slowly. But videos such as the Google one and this one [1] are mostly about making the system take instructions in natural language, not doing the actual picking.
Bin-picking of uniform items is a solved problem, and was first working in the 1980s. Bin-picking of random items from a cluttered bin is still a struggle. Amazon has put a fair amount of effort into robotic bin-picking, but it's not yet good enough to deploy.
There's been some good progress on bin-picking recently. Here's Brightpick's system.[2] It has a good vision system which involves LIDAR, and some nice engineering. It's not LLM-based, and it won't do everything, but it can handle most objects that can be picked up by a vacuum picker.
[1] https://www.youtube.com/watch?v=31Xqw_59XZ8
[2] https://www.youtube.com/watch?v=XWlQWgRfh6c
At this point either he is just trolling or writing it to get views from people who already believes this and looking for someone to say it with authority.
[deleted]
Can we have a rate-limiter for submissions from the same domain, proportional to the log(TRANCO rank) of the domain? One or two posts are fine, but I’m a little tired of these hot takes that do little to acknowledge the advances the AI/ML field has made in recent years while berating it for the smallest deficiencies.
You can’t convince me he isn’t responding to the meme of celebrating “hitting the wall day”. I can’t comment on every point he brings up here but I work for a company doing very well reading legal documents. Myth busted.
> “We are still a long way from machines that can genuinely understand human language”. Still true
It's crazy how you can just say things. If by his standards he himself can genuinely understand human language, even given how much of what he says is just smart-sounding hallucinations, then surely Claude 3 well surpasses that threshold.
The secret to grifting is to only highlight your vacuous, unfalsifiable claims. He, like many AI skeptics, is a full-time goalpost mover.
It's crazy how you can just say things. If by his standards he himself can genuinely understand human language, even given how much of what he says is just smart-sounding hallucinations, then surely Claude 3 well surpasses that threshold.
The secret to grifting is to only highlight your vacuous, unfalsifiable claims. He, like many AI skeptics, is a full-time goalpost mover.
I only tried GPT4 but it has a way better command of language than many people I know and I include myself in that. If GPT4 doesn't understand language, then I don't either.
Does MS Word "understand language" because of autocorrect?
Can autocorrect hold a conversation, write code or poetry, answer or pose riddles, summarize text, write novel stories, diagnose medical issues, comprehend diagrams, etc?
ELIZA could do that 50 years ago.
I haven't been really impressed with ChatGPT. It basically seems like a more useful search engine.
I haven't been really impressed with ChatGPT. It basically seems like a more useful search engine.
Are you using GPT 3.5 or GPT-4? And how are you using it? What queries are you giving it?
[deleted]
ELIZA had no ability to translate language, summarize text, generate poetry or anything like that. Extremely idiotic comparison.
Comparing LLMs to ELIZA shows that you understand neither.
Text summarisation routinely hallucinates, code fails outside of boilerplate, and poetry is literally the least coherent form of writing.
And no, AI can't diagnose yet - even expert systems forty years ago did better and it wasn't good enough.
And no, AI can't diagnose yet - even expert systems forty years ago did better and it wasn't good enough.
[deleted]
> Text summarisation routinely hallucinates
LLMs are trained to summarize text with extremely high accuracy. There is no confabulation of text summarized within the context window. Confabulation only occurs with facts recalled from outside of the context window, which is not a text summarization task. Same goes for human memory. Ironically you have hallucinated a "fact" about LLMs.
LLMs are trained to summarize text with extremely high accuracy. There is no confabulation of text summarized within the context window. Confabulation only occurs with facts recalled from outside of the context window, which is not a text summarization task. Same goes for human memory. Ironically you have hallucinated a "fact" about LLMs.
Deep learning is not autocorrect
A sufficiently advanced autocorrect requires understanding language as well as a human. MS Word's autocorrect is not that advanced. I don't understand why people think that this argument is worth bringing up.
Deep learning is still not production ready.
Maybe 90% of the results are great, but the other 10% are not only 'not great' but plain wrong and sometimes even dangerous.
And I think gp posts is also showing that the last bump to make it production ready is still not taken and might even be impossible to take.
Maybe 90% of the results are great, but the other 10% are not only 'not great' but plain wrong and sometimes even dangerous.
And I think gp posts is also showing that the last bump to make it production ready is still not taken and might even be impossible to take.
Last time I checked ChatGPT is very much in production.
From the article: “Deep learning, which is fundamentally a technique for recognizing patterns, is at its best when all we need are rough-ready results, where stakes are low and perfect results optional“
It is somewhat embarrassing how many tasks, from customer service to sportswriting to language translation to article summarizing to concept art for films, can be handled by deep learning at its current level of competence. That says more about the problem spaces and human intelligence that it does about deep learning.
It is somewhat embarrassing how many tasks, from customer service to sportswriting to language translation to article summarizing to concept art for films, can be handled by deep learning at its current level of competence. That says more about the problem spaces and human intelligence that it does about deep learning.
You mean,
- customer service: chatbot invents refund policy that costs the company money and reputation https://www.wired.com/story/air-canada-chatbot-refund-policy...
- sports writing: a private equity group is running Sports Illustrated - which was already in decline - to the ground
- language translation: not much progress, and more risks of hallucinations
- concept art for films: AI is basically doing plagiarism. And so much AI art is flooding the internet that the AI are basically poisoning themselves in the future. And it still can't get the right number of fingers on hands.
- customer service: chatbot invents refund policy that costs the company money and reputation https://www.wired.com/story/air-canada-chatbot-refund-policy...
- sports writing: a private equity group is running Sports Illustrated - which was already in decline - to the ground
- language translation: not much progress, and more risks of hallucinations
- concept art for films: AI is basically doing plagiarism. And so much AI art is flooding the internet that the AI are basically poisoning themselves in the future. And it still can't get the right number of fingers on hands.
I wouldn't really say it's embarassing. If most human tasks required perfection (or at least high levels of correctness at all times), society would probably not have gotten to where we are. Imagine a world where every job was as hard as open-heart surgery! In a sense, it's an achievement of humanity that we're able to recognise when 'good enough' will let us achieve the things we want.
>> From the article: “Deep learning, which is fundamentally a technique for recognizing patterns, is at its best when all we need are rough-ready results, where stakes are low and perfect results optional“
> It is somewhat embarrassing how many tasks, from customer service to sportswriting to language translation to article summarizing to concept art for films, can be handled by deep learning at its current level of competence.
No, it's embarrassing how eagerly leaders will compromise quality to chase automation cost savings. The things you list, especially customer service and sports-writing, cannot "be handled by deep learning at its current level of competence." That doesn't stop people from forcing the results down our throats, though.
> It is somewhat embarrassing how many tasks, from customer service to sportswriting to language translation to article summarizing to concept art for films, can be handled by deep learning at its current level of competence.
No, it's embarrassing how eagerly leaders will compromise quality to chase automation cost savings. The things you list, especially customer service and sports-writing, cannot "be handled by deep learning at its current level of competence." That doesn't stop people from forcing the results down our throats, though.
No personal criticism of this person, but:
"The glass isn't full yet. Sure its filling, but its filling and its not full yet, so obviously no matter how much you fill it, we know that doesn't actually fill it."
..is my breakdown on these non-expert non-specific (mathematically, algorithmically, information theoretically, any relevant expertise at all, ...) cynics.
If he had predicted the progress that was made just in the last two years, it would give some crediblity to his opinions about what wasn't going to improve. But he didn't have that insight. He has no idea what's already cooking in the research kitchen.
But if am going to critque predictions, shouldn't I put myself on the line?
So here is my prediction: when general AI decisively advances past us, a lot of people will feel a strong need to write and read articles about how AI doesn't have a soul. And their creations have no heart. And no progress will ever be able to fix this. Despite a lack of concrete or coherent definitions, this will matter a great deal to them.
"The glass isn't full yet. Sure its filling, but its filling and its not full yet, so obviously no matter how much you fill it, we know that doesn't actually fill it."
..is my breakdown on these non-expert non-specific (mathematically, algorithmically, information theoretically, any relevant expertise at all, ...) cynics.
If he had predicted the progress that was made just in the last two years, it would give some crediblity to his opinions about what wasn't going to improve. But he didn't have that insight. He has no idea what's already cooking in the research kitchen.
But if am going to critque predictions, shouldn't I put myself on the line?
So here is my prediction: when general AI decisively advances past us, a lot of people will feel a strong need to write and read articles about how AI doesn't have a soul. And their creations have no heart. And no progress will ever be able to fix this. Despite a lack of concrete or coherent definitions, this will matter a great deal to them.
"I haven't run into the wall yet. Sure the wall is there, but I am running full speed and I haven't it it yet"
>“Few fields have been more filled with hype and bravado than artificial intelligence. It has flitted from fad to fad decade by decade, always promising the moon, and only occasionally delivering” Still true.
>“We are still a long way from machines that can genuinely understand human language”. Still true, though some have argued there is some superficial understanding.
>“Elon Musk recently said that the new humanoid robot he was hoping to build, Optimus, would someday be bigger than the vehicle industry”. I expressed skepticism. Still early days, but certainly domestic humanoid robots are not in the near term expected to be a big business for anyone.
>The company’s charismatic CEO Sam Altman wrote a triumphant blog posttrumpeting “Moore’s Law for Everything,” claiming that we were just a few years away from “computers that can think,” “read legal documents,” and (echoing IBM Watson) “give medical advice.”Maybe, but maybe not.” Pending/still true. Two years later we don’t have reliable versions of any of that.
Why does anyone listen to a thing this obvious fraud says?
>“We are still a long way from machines that can genuinely understand human language”. Still true, though some have argued there is some superficial understanding.
>“Elon Musk recently said that the new humanoid robot he was hoping to build, Optimus, would someday be bigger than the vehicle industry”. I expressed skepticism. Still early days, but certainly domestic humanoid robots are not in the near term expected to be a big business for anyone.
>The company’s charismatic CEO Sam Altman wrote a triumphant blog posttrumpeting “Moore’s Law for Everything,” claiming that we were just a few years away from “computers that can think,” “read legal documents,” and (echoing IBM Watson) “give medical advice.”Maybe, but maybe not.” Pending/still true. Two years later we don’t have reliable versions of any of that.
Why does anyone listen to a thing this obvious fraud says?
Gary Marcus tries to exit his personal hole of irrelevance. I have been working in the fielf of computational linguistics for 30 years. Back in 2000, I worked with a team of linguists to implement a pretty refined syntactic parser (XIP) based on Shallow Parsing, a symbolic approach. We won with this system a SemEval competition as late as 2016 on Sentiment Analysis. But I never expected LLM to reach this level of competency in my lifetime. Critics are prone to describe the errors these models make, but they seem to forget that 1. It is not a search engine and 2. it is pretty knowledgeable in a huge range of domains, which no humans can equal. I use these models every day to create code or to explain concepts to me. It never ceases to amaze me.
He wrote that deep learning is “hitting a wall” 6 months before ChatGPT was released. Definitely needs to save face somehow.
Furthermore, there has been insane progress over the last few years, which presumably would not have happened if everyone had despaired on reading Marcus' 'hitting a wall' article and dropped everything to work on GOFAI.
But let's widen the lens a little bit:
`In 2015, shortly after Hinton joined Google, The Guardian reported that the company was on the verge of “developing algorithms with the capacity for logic, natural conversation and even flirtation.” In November 2020, Hinton told MIT Technology Review that “deep learning is going to be able to do everything.” I seriously doubt it.`
Well, we've certainly got natural conversation and flirtation, along with some kind of logic - especially if you're OK with accepting python programs as a form of response. This is well beyond what was considered possible at the time. We're certainly not at 'everything' but *shrug*. Maybe in ten years.
`nowhere near the ordinary day-to-day intelligence of Rosey the Robot, a science-fiction housekeeper that could not only interpret a wide variety of human requests but safely act on them in real time.`
It exists, it just isn't anywhere near economical for household tasks... https://www.wired.com/story/google-robot-learned-to-take-ord... LLM's look poised to upend robotics.
Marcus then goes extensively into why everyone should be looking at symbol-manipulation systems. And that's fine... But would be much more convincing if he could actually produce something that works. At some point, one asks, if your idea is so good, why haven't you actually solved any problems?