It sees embeddings that is trained to encode semantic meanings.
The way we tokenize is just a design choice. Character level models(e.g. karpathy's nanoGPT) exist and are used for educational purpose. You can train it to count number of 'r' in a word.
The text is converted to embeddings after tokenization. The neural networwk only sees vectors.
Imagine the original question is posed in English but it is translated to Chinese and then the LLM has to answer the original question based on the Chinese translation.
It's a flaw of the tokenization we choose. We can train an LLM using letters instead of tokens as the base units but that would be inefficient.
The possibility seems far beyond gaming(given enough computation resources).
You can feed it with videos of usage of any software or real world footage recorded by a Go Pro mounted on your shoulder(with body motion measured by some sesnors though the action space would be much larger).
Such a "game engine" can potentially be used as a simulation gym environment to train RL agents.
When people say "LLMs are not capable of innovation", what exactly do they consider as innovation? If LLMs are not capable of innovation on their own, what if we augment them with means to interact with the environment so they can obtain new training data?
e.g. The minecraft bot Voyager can explore the game environment and extend its skill library(stored as a vector database), is that considered as innovation? There are also systems like leandojo/alphaproof that discover new proofs and use LLM in non-trivial ways(not just naively predict the next token in one shot). Reinforcement learning algorithms like AlphaGo/AlphaZero use self play and use monte carlo tree search to learn to outperform humans. You can similarly use LLMs to generate actions and estimate state values(check the language agent tree search paper).
Most people use LLMs by prompting them with some additional context(chat history and data retrived from database) but there is nothing that stops us from continuously improving a LLM(either by modifying its weight or augmenting it with external database) by asking it to evaluate the task outcome/error message and feeding it back to the LLM. We can also ask it to just keep on generating new tasks to experiment with the environment/internet to get new knowledge.
You can try STORM(also from Stanford) and see the prompts it generates automatically, it tries to expand on your topic and simulate the collaboration among several domain experts
https://github.com/stanford-oval/storm
I don't get why some people seem to think the only way to use a LLM is for next token prediction or AGI has to be bult using LLM alone.
You want planning, you can do monte carlo tree search and use LLM to evaluate which node to explore next. You want verifiable reasoning, you can ask it to generate code(an approach used by recent AI olympiad winner and many previous papers).
What is even "planning", finding desirable/optimal solutions to some constrained satisfaction problems? Is the llm based minecraft bot voyager not doing some kind of planning?
LLMs have their limitations. Then augment them with external data sources, code interpreters, give it ways to interact with real world/simulation environment.
There's a recently released kungfu movie whose story happened in the Kowloon walled city. The government is planning to relocate the movie set to the original address for exhibition.
There are way more text training data than voice data. It also allows you to use all the benchmarks and tool integrations that have already been developed for LLMs.
Some "agents" like the minecraft bot Voyager(https://github.com/MineDojo/Voyager) have a control loop, they are given a high level task and then they use LLM to decide what actions to take, then evaluate the result and iterate. In some LLM frameworks, a chain/pipeline just uses LLM to process input data(classification, named entitiy extraction, summary, etc).
https://grok.com/share/bGVnYWN5LWNvcHk%3D_572b4955-6265-4210...