In my opinion, whether it was actually by Ilya or not is not worthy of debate. Many of them are widely recognized for being good pedagogical resources (e.g. annotated transformer, unreasonable effectiveness of RNNs, understanding LSTM networks), and others are landmark papers which anyone interested in the field would benefit from reading:
- Krizhevsky et al. (2012) introduced AlexNet
- Bahdanau et al. (2014) introduced attention
- He et al. (2015) introduced ResNet
- Vaswani et al. (2017) introduced the Transformer
Other papers are more specialized. Of them, I think Kaplan et al. (2020) by OpenAI is probably most important.
It does not seem that the author cites the source of the control theory map. It was created by Brian Douglas [1], an engineer whose YouTube videos [2] are great for learning core topics.
Also useful is Steve Brunton's channel [3]. He has a freely available book [4] co-authored with Nathan Kutz that ties machine learning and control.
> Maybe it works because the sequences are short and the dimension is high and there's plenty of room for interesting results to fit in the merged key/value space.
In fact, on the second last page of the paper, they discuss this very problem. There is a clear correlation between performance and increasing sequence lengths for the Q-K=V model. While limited to a tight n=3 sample between 512, 1024, 2048 lengths, the degradation decreases from 5.4% to 2.2% as context is increased, suggesting that it is unlikely shorter sequences are the reason K=V performs acceptably.
In addition, replicating Jameson et al. (AIAA 1981-1259) [1], is a worthwhile, more advanced follow up, great if you want to get into serious CFD development eventually.
Don't let the terminology intimidate you. The interesting ideas in quantum computing are far more dependent upon a foundation in linear algebra rather than a foundation in mathematical analysis.
When I started out, I was under the assumption that I had to understand at least the undergraduate real analysis curriculum before I could grasp quantum algorithms. In reality, for the main QC algorithms you see discussed, you don't need to understand completeness; you can just treat a Hilbert space as a finite-dimensional vector space with a complex inner product.
For those unfamiliar with said concepts from linear algebra, there is a playlist [1] often recommended here which discusses them thoroughly.
https://x.com/keshavchan/status/1787861946173186062
In my opinion, whether it was actually by Ilya or not is not worthy of debate. Many of them are widely recognized for being good pedagogical resources (e.g. annotated transformer, unreasonable effectiveness of RNNs, understanding LSTM networks), and others are landmark papers which anyone interested in the field would benefit from reading:
- Krizhevsky et al. (2012) introduced AlexNet
- Bahdanau et al. (2014) introduced attention
- He et al. (2015) introduced ResNet
- Vaswani et al. (2017) introduced the Transformer
Other papers are more specialized. Of them, I think Kaplan et al. (2020) by OpenAI is probably most important.