Trying to make things up to cover for a lack of knowledge is something distinctly different, though. This is a a situation where ChatGPT is able to perfectly describe the mistake it made, describe exactly what it needs to do differently, and then keeps making the same mistake, even with simple tasks. That’s because there’s no greater model that the words are being connected to.
The equivalence would be saying to someone, “put this on the red plate, not the blue one.” And they say sure, then put it on the blue one. You tell them they made a mistake and ask them if they know what it was, and they reply “I put it on the blue plate, not the red one. I should have put it on the red one.” Then you ask them to do it again, and they put it on the blue plate again. You tell them no, you made the same mistake, put it on the blue plate, not the red one. They reply with, “Sorry, I shouldn’t have put it on the blue plate again, now I’m going to put it on the red one,” and then they put it on the blue plate yet again.
Do humans make mistakes? Sure. But that kind of performance in a test wouldn’t be considered a normal mistake, but rather a sign of a serious cognitive impairment.
You can see the difference if you know where to poke. For instance, if you start making spatial abstractions ChatGPT will often make mistakes, you can point it out, they can explain why it's a mistake, but it has no internalized model of what these words mean, so it keeps making the same mistakes (see here for a better idea of what I'm talking about[1]). The fact that you are interacting with it through text means that a lot of the missing abstractions are often hidden.
Partly it's a matter of what to ask Chat GPT. When I try to show people, they usually ask straightforward questions that they can already get a quick answer from in Google, which isn't terrible impressive.
But take this, for example[1]. I asked it to write me a story in a particular genre featuring certain animals, and it did. I asked it to switch genres, which it did well. When I asked for a backstory about how different characters met, it provided a fairly plausible one, as well as songs that would accompany the story if it were a musical, and potential titles for a sequel.
When I asked it to write the beginning of a New York Times article titled "Biden Shocks Nation"[2] I got a fairly convincing news story about Biden deciding not to run for office again. If asked to continue the story and include people who might run, it generates further paragraphs talking about who might run to replace him, starting with Kamala Harris, who it claims is a strong contender.
Is any of this writing amazing? No, but very few writing is. It's amazing how well it's able to generate generic human writing, as well as how easy it is to get it to create what you want with very few prompts.
The equivalence would be saying to someone, “put this on the red plate, not the blue one.” And they say sure, then put it on the blue one. You tell them they made a mistake and ask them if they know what it was, and they reply “I put it on the blue plate, not the red one. I should have put it on the red one.” Then you ask them to do it again, and they put it on the blue plate again. You tell them no, you made the same mistake, put it on the blue plate, not the red one. They reply with, “Sorry, I shouldn’t have put it on the blue plate again, now I’m going to put it on the red one,” and then they put it on the blue plate yet again.
Do humans make mistakes? Sure. But that kind of performance in a test wouldn’t be considered a normal mistake, but rather a sign of a serious cognitive impairment.