You literally can do a kind of model PCA, using the Hessian (matrix of second derivatives of the loss function w/r/t the parameters, aka the local curvature of the loss landscape), and diagonalizing. These eigenvectors and eigenvalues (the spectrum of the Hessian) tend to be power-law distributed in just about every deep NN you can think of [1].
That is, there are a few "really important" (highly curved) dimensions in parameter space (the top eigenvectors) which control the model's performance (the loss function). Conversely, there are very many "unimportant"/low curvature dimensions in the model. There was a recent interesting paper that showed that "deleting" these low-curvature dimensions appeared to correspond to removing "memorized" information in LLMs, such that their reasoning performance was left unchanged while their ability to answer questions which require some memorized knowledge was reduced [2].
It appears that sometimes models undergo dramatic transitions from memorization to perfect generalization, which corresponds to the models becoming much more compressible [3].
I'm hopeful that we'll find a way to distill the models down to the most useful core cognitive/reasoning capabilities, and that that core will be far simpler than the current scale of LLMs. But they might need to look stuff up like we do without all that memorized world knowledge!
I don't recall Andrej making "next year!" claims, it was always Elon. I found Andrej's talks from that time to be circumspect and precise in describing their ideas and approach, and not engaging in timeline speculation.
They really should have just marketed the software "as-is" to whatever extent that is allowed by law. I guess they didn't because deployed automobile software is probably not allowed to be considered experimental.
Still, comms that framed it like: "This software purchase upgrades your car with state-of-the-art autonomy capabilities from our AI team, as we approach full self-driving" would have been more honest, still exciting to consumers, and avoided over-promising.
I'm a researcher at Oxford, and I've both taught and studied here and in the US.
The undergraduate teaching here is phenomenal. It's incredibly labor intensive for the staff, but the depth and breadth students are exposed to in their subject is astonishing. It's difficult to imagine how it can be improved.
My favorite study of university rankings comes from faculty hiring markets, which compute implicit rankings by measuring which institutions tend to hire (PhD->faculty) from others. [1] It's not perfect, but at the very least it's a parameter free way to get a sense of how different universities view each other. The parameters in most university rankings are rather arbitrary and game-able.
Some have pointed to things like contextual admissions [2], and more broadly some identity politics capture of the administration for declining standards. While this might be true, in my view Oxford is still far more meritocratic than US institutions on the whole. There are no legacy admissions, and many subjects have difficult tests which better distinguish between applicants who have all done extremely well on national standardised tests (British A Levels are far more difficult than the SAT/ACT/AP exams.)
Lastly, admissions at Oxford are devolved to the individual colleges, of which there are ~40. The faculty at each college directly interview and select the applicants which they will take as students. This devolved system and the friction it creates is surprisingly robust and makes complete ideological capture more difficult.
The most pressing issue for Oxford's long-term viability as a leading institution, in my view, is the funding situation. For one the British economy is in a long, slow decline. Secondly, even though Oxford has money, there are lots of regulations/soft power influence from the British govt to standardise pay across the country, which makes top institutions like Oxford less competitive on the international market for PhD students, postdocs, and faculty in terms of pay.