Yes, I also wonder about this! Progress from children books to scientific papers etc.
Could it learn e.g. language structure faster in a pre-training stage?
Also somehow one needs to define a proxy to generalization to compute a loss and do backpropagation.