I read a book, just a few weeks ago regarding this specific topic and my explanaition is directly from that book. We do have specific brain areas for this.
This sudden jerk you can read about it here: https://en.wikipedia.org/wiki/Hypnic_jerk and its not scienctificly clear why and how it works, the best assumption is that its a reflex.
You are not disconnecting your brain from anything.
You generalize this without mentioning that there are LLMs which do not just use random 'dogshit'.
Also what does a normal human do? It looks around how to move one random piece and it uses a very small dictionary / set of basic rules to move it. I do not remember me learning to count every piece and its options by looking up that rulebook. I learned to 'see' how i can move one type of chess piece.
If a LLM uses only these piece moves on a mathematical level, it would do the same thing as i do.
And yes there is also absolutly the option for an LLM to learn some kind of meta game.
There is plenty of AI which learns the rules of games like Alpha Zero.
LLMs might not have the architecture to 'learn', but it also might. If it optimizes all possible moves one chess peace can do (which is not that much to learn) it can easily only 'move' from one game set to another by this type of dictionary.
Yes in theory it could. Depends on how it learns. Does it learn by memorization or by learning the rules. It depends on the architecture and the amount of 'pressure' you put on it to be more efficient or not.
Where did you get the 'brain stem disconnects from your body'? Because thats not how it works in the brain.
We have the part which controls your muscles and we have another part which simulates the movement. Not executing on it has nothing to do with the brain stem disconnecting from the 'body'.
Its the same mechanism as you thinking about a movement but not doing it.
This sudden jerk you can read about it here: https://en.wikipedia.org/wiki/Hypnic_jerk and its not scienctificly clear why and how it works, the best assumption is that its a reflex.
You are not disconnecting your brain from anything.