I like this answer. Been reading “Gödel’s Proof” and for a demonstration where the definition of “tautology” in the main text makes use of the concepts of True and False, there is an appendix explaining that you can arrive at the same result without those concepts, just by treating things as belonging to one class vs another (there is a one to one correspondence with True and False but the meaning is arbitrary)
The model learns an embedding table, where (roughly) each row is used as the model’s internal representation of each token. The numbers in that table are learned. What isn’t learned is the map from token (i.e. combination of characters/byte-pairs) to row-index in the embedding table. That is given by tokenization
These guys [1] used to be called Nerdalize and iirc their first iteration looked a lot like this. I think they have since moved to a model of a few select locations where nodes are closer to each other.