> not prediction based on anything statistical but based on one's understanding
It is still statistical. You still want to predict a distribution over expected next token.
However, the function needed to estimate the likely next tokens are not simple max over enumerated next tokens like in a language model. It's more like a transition model in reinforcement learning.
> trying to work the concepts out in my own mind and to anticipate what the authors are going to say before turning each page. I usually fail to guess what the next page holds, but the fact that I've tried and failed makes me more ready to understand why the authors chose the paths that they did
TIL Donald Knuth operates a bit like GPT-3 but for research paper narrative.
It is still statistical. You still want to predict a distribution over expected next token.
However, the function needed to estimate the likely next tokens are not simple max over enumerated next tokens like in a language model. It's more like a transition model in reinforcement learning.