This is such a squirrely topic that you presumed that it had goals in this post even though I don't believe you think it does. ;-)
Does it HAVE goals? I don't think there is any evidence that it does. Even if it developed the ability to do all of those things you mentioned, it still wouldn't have any idea what it is doing, let alone why it is doing it, because it's just a language model. That is all it does. And that's why I don't quite understand how people can be so adamant that there is any question about whether it is conscious or not. This whole debate is a trap. It is a brute fact that it is not conscious. It does not have a goal of producing coherent text. That is the goal of the people who wrote it. If we treat the brain as a computer, then GPT lacks the subsystems/algorithms/whatever to do what brains do. It fails at many basic tasks that anything capable of general-purpose reasoning could do. It's often not apparent how spectacularly it can fail because the tasks are so mundane that we don't bother even asking it to do them.
This is a classic example from philosophy of mind. It doesn't completely harmonize with the question you're asking, but it concisely summarizes two opposing views.
What's my view? That we do it all the time as a matter of course. "Why not do it?" isn't the question. The question is how much weight we should give to the result, and whether we should take care to question the results and try something else, too.
The fact that our "native" heuristics fail spectacularly in certain circumstances doesn't imply that they're crappy. They serve us well most of the time. And I say "our" because AFAIK, their efficacy doesn't have much to do with intelligence, though the capacity to question what they tell us does, I suspect.
Does it HAVE goals? I don't think there is any evidence that it does. Even if it developed the ability to do all of those things you mentioned, it still wouldn't have any idea what it is doing, let alone why it is doing it, because it's just a language model. That is all it does. And that's why I don't quite understand how people can be so adamant that there is any question about whether it is conscious or not. This whole debate is a trap. It is a brute fact that it is not conscious. It does not have a goal of producing coherent text. That is the goal of the people who wrote it. If we treat the brain as a computer, then GPT lacks the subsystems/algorithms/whatever to do what brains do. It fails at many basic tasks that anything capable of general-purpose reasoning could do. It's often not apparent how spectacularly it can fail because the tasks are so mundane that we don't bother even asking it to do them.