Ask HN: How do I learn more about LLMs and ML?
17 comments
Aside from Karpathy's videos, I'd say:
Start by any book on traditional neural networks, so you get a decent understanding of what neural networks are, what is an activation function and what the backpropagation is.
From then on, my path was: - numpy user manual, if you don't know it already
- a book about pytorch
- a book about transformers
- a book from Wolfram about LLMs
(forgot the names of books, but you can find plenty on Amazon)
That's as far as reading / theoretical understanding works.
As for understanding in practice (when you've read the theory), I'd say - implement a basic picoGPT/Llama/Mistral model from scratch in python & numpy.
I don't know how good your college is in math, but matrix-vector multiplications, dot products and linear algebra in general is a must.
Start by any book on traditional neural networks, so you get a decent understanding of what neural networks are, what is an activation function and what the backpropagation is.
From then on, my path was: - numpy user manual, if you don't know it already
- a book about pytorch
- a book about transformers
- a book from Wolfram about LLMs
(forgot the names of books, but you can find plenty on Amazon)
That's as far as reading / theoretical understanding works.
As for understanding in practice (when you've read the theory), I'd say - implement a basic picoGPT/Llama/Mistral model from scratch in python & numpy.
I don't know how good your college is in math, but matrix-vector multiplications, dot products and linear algebra in general is a must.
Is this all the math that is required? If I know how to do basic linalg is that about what I need to know to be able to understand how LLMs work? My goal right now is to build stuff on top of LLMs
Let's say one builds a basic LLM in python. What's next?
Aren't we limited by compute and GPU power to do anything further ?
Aren't we limited by compute and GPU power to do anything further ?
That's the algorithm side, you also need training data. Can experiment with different data.
Can‘t recommend the fast.ai course by Jeremy Howard highly enough, it walks you through building your own deep learning stack from scratch. What I really appreciate about it is that it demystifies a lot of jargon to what really are quite simple ideas at their core (e.g. „rectified linear unit“ sounding scary even though its literally only a line with a floor).
The 2022 edition isn’t so much about LLMs as about image generation with stable diffusion, but the underlying techniques are still foundational enough to be generally useful. YMMV, but for me building things from scratch, even if results don‘t reach SOTA, is the single most effective way to learn what‘s really going on.
The 2022 edition isn’t so much about LLMs as about image generation with stable diffusion, but the underlying techniques are still foundational enough to be generally useful. YMMV, but for me building things from scratch, even if results don‘t reach SOTA, is the single most effective way to learn what‘s really going on.
Karpathy’s video “Let's build GPT: from scratch, in code, spelled out” helped me a lot:
https://youtu.be/kCc8FmEb1nY
https://youtu.be/kCc8FmEb1nY
I'm in the exact same situation as the poster rn. Freshman year of college and really interested in LLMs. I'm planning to take ML/Deep learning courses at my school later on, but in the meantime, what would be the best way to learn how to drive the car without exactly knowing how the engine works? What I mean is I want to learn how to fine-tune and build useful apps on top of LLMs
What is your goal ? 1) Know more about how they work on the academic side ? 2) Be able to work in a company that work on LLM ? 3) Be able to work with LLMs 4) agents ? Each of those goal may require different learning "streams"
The book "NLP with transformers", or fast.ai is good for 3). For 1), assuming you do know how they work, I recommend you start reading papers.
I find the discussions around "prompt engineering" to be rather pointless and they are quickly obsolete anyway (newer, more powerful LLMs makes it more and more obvious)
The book "NLP with transformers", or fast.ai is good for 3). For 1), assuming you do know how they work, I recommend you start reading papers.
I find the discussions around "prompt engineering" to be rather pointless and they are quickly obsolete anyway (newer, more powerful LLMs makes it more and more obvious)
Ideally my goal is a mix of 2 and 3. I think agents are quite interesting, but maybe something I’d look at later.
I'm in the exact same situation as you rn. Freshman year of college and really interested in LLMs. Did you decide how you're gonna go about it? Maybe we could share resources
I'm focused on finals at the moment but I'd be more than happy to share what I've been looking at so far (I plan to look into this a lot more in the summer).
When it comes to the theory, 3blue1brown released some really nice videos that solidified a lot of my current understanding, especially with the attention mechanism. I think I am also going to do a mixture of reading papers/watching youtube videos on things that are interesting (ex. Qlora for fine-tuning or diffusion models for image generation) and trying to build out a simple implementation myself and see where that takes me. Maybe I'll start out with the Karpathy nano-gpt videos but try and do it with a different data set.
But for people like myself who lack the math background, data, and compute to be able to train very strong LLMs, I think it is also a good idea to try and build some projects/apps that use a fine-tuned LLM, or just call the OpenAI API.
I'm still a bit lost myself, but in 2 weeks when I'm done with exams, I'm more than happy to keep exchanging resources with you.
When it comes to the theory, 3blue1brown released some really nice videos that solidified a lot of my current understanding, especially with the attention mechanism. I think I am also going to do a mixture of reading papers/watching youtube videos on things that are interesting (ex. Qlora for fine-tuning or diffusion models for image generation) and trying to build out a simple implementation myself and see where that takes me. Maybe I'll start out with the Karpathy nano-gpt videos but try and do it with a different data set.
But for people like myself who lack the math background, data, and compute to be able to train very strong LLMs, I think it is also a good idea to try and build some projects/apps that use a fine-tuned LLM, or just call the OpenAI API.
I'm still a bit lost myself, but in 2 weeks when I'm done with exams, I'm more than happy to keep exchanging resources with you.
Perfect. I have my own finals in a couple weeks haha, planning to look more into this and possibly build a project over the summer. 3B1B's videos sound like a good resource, and the nano-GPT videos look really useful too. I'll share what I come across as well!
Karpathy's "Neural Networks: Zero-to-hero" series has been very useful for me: https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs...
Related question: Does it make sense to buy a RTX 4090 or 3090 Ti for this? Or for finetuning existing open models.
The answer varies wildly. It's a trade-off between what constitutes overall VRAM on your system. For example I have a 3070 Ti and my total VRAM is ~8GB. This is quite insufficient for working with 70b, for example.
So, it depends on the size of the models that you _want_ (or _need_) to work with, the total VRAM on your current system, and other factors.
A 4090, for example, would be massive overkill if you want to play around with just 7b models.
So, it depends on the size of the models that you _want_ (or _need_) to work with, the total VRAM on your current system, and other factors.
A 4090, for example, would be massive overkill if you want to play around with just 7b models.
Build one. Then another. Then another.
Make building them normal not an imaginary exercise.
You don't have to build a good one to learn. Which is good because you probably won't. But no one will care because nobody had to give you permission and nobody is going to give you a grade.
Good luck.
Make building them normal not an imaginary exercise.
You don't have to build a good one to learn. Which is good because you probably won't. But no one will care because nobody had to give you permission and nobody is going to give you a grade.
Good luck.
How should I go about building one?
I am about to finish my freshman year in college and over the past 2 months I have been doing a lot of research on LLMs. I have a good high-level grasp of how these systems work, but I was wondering how I could deepen my knowledge. Should I read papers? What kinds of projects do you think are best to learn how to build these systems?