Mathematical Foundations of Reinforcement Learning(github.com)
github.com
Mathematical Foundations of Reinforcement Learning
https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning
45 comments
During the openai gym era of RL, one of the great selling pts was that RL was very approachable for a new comer as the gym environments were small and tractable that a hobbyist could learn a little bit of RL, try it out on cartpole and see how it'd perform. Are there similarly tractable RL tasks/learning environments with LLMs? From the outside, my impression is that you need some insane GPU access to even start to mess around with these models. Is there something one can do on a normal MacBook air for instance in this LLM x RL domain?
This is entirely doable.
I'm absolutely not versed in RL, but I wanted to understand GRPO, the RL algorithm behind Deepseek's latest model.
I started from a very simple LLM, inspired from Andrej Karpathy's "GPT from scratch" video (https://www.youtube.com/watch?v=kCc8FmEb1nY). Then, I added onto that the GRPO algorithm, which in itself is very simple.
I made a GitHub repo if you want to try it out : https://github.com/Al-th/grpo_experiment
I'm absolutely not versed in RL, but I wanted to understand GRPO, the RL algorithm behind Deepseek's latest model.
I started from a very simple LLM, inspired from Andrej Karpathy's "GPT from scratch" video (https://www.youtube.com/watch?v=kCc8FmEb1nY). Then, I added onto that the GRPO algorithm, which in itself is very simple.
I made a GitHub repo if you want to try it out : https://github.com/Al-th/grpo_experiment
GRPO project is neat. Would you be willing to do a Karpathy-style explainer, breaking down the algorithm from scratch? It’s hard to understand on its own without prior background knowledge.
Find materials on PPO which should be widespread since it is the most popular RL algorithm. GRPO works on the same principles, just makes certain estimates from samples rather than training an auxiliary neural network to make them.
6-lecture series on the Foundations of Deep RL by Pieter Abbeel is also very recommended. gives very good overview and intuition
https://youtu.be/2GwBez0D20A
The best lectures on Reinforcement Learning and related topics are by Dimitris Bertsekas: https://web.mit.edu/dimitrib/www/home.html
Also one by David Silver of Deepmind, AlphaGo fame are good too:
https://www.youtube.com/watch?v=2pWv7GOvuf0
His books tend to be dry and geared towards researchers, in my opinion. He has a new one on RL: https://web.mit.edu/dimitrib/www/RLCOURSECOMPLETE%202ndEDITI...
I'm looking for content (researcher myself) -- mainly on the application side. Should I start with this one? Or anything else?
Very curious about RL for LLMs for example (using data from real use).
Very curious about RL for LLMs for example (using data from real use).
I have not read it but it looks like a comprehensive reference. For a more applied treatment see Foundations of Deep Reinforcement Learning. https://slm-lab.gitbook.io/slm-lab/publications-and-talks/in...
Neither cover LLMs. I don't follow the literature closely so I can only suggest you read papers: https://github.com/WindyLab/LLM-RL-Papers
Neither cover LLMs. I don't follow the literature closely so I can only suggest you read papers: https://github.com/WindyLab/LLM-RL-Papers
Would you mind explicitly indicating whether you have reviewed the submitted materials? And if so, why is it inferior to the material you linked?
Not trying to catch you, genuine interest.
Not trying to catch you, genuine interest.
No. They are outdated and focused on strange things. You wont understand ppo from his textbooks
Which aspects? Foundational textbooks would focus on principles, not necessarily implementations, and don't go "outdated" the same way a snippet does.
Another great resource on RL is Mykel Kochenderfer's suite of textbooks:
https://algorithmsbook.com/
These books are all RL? I’ve got the decision one, I didn’t think the other had anything to do with RL.
He (author) has a strong proclivity for policy-based planning, shall we say.
Highly recommended .. even the main contents diagram is a great visual overview of RL in general, as is the 30 minute intro YT video.
Im expecting to see a lot of hyper growth startups using RL to solve a realworld problem in engineering / logistics / medicine
LLMs currently attract all the hype for good reasons, but Im surprised VCs dont seem to be looking at RL companies specifically.
Im expecting to see a lot of hyper growth startups using RL to solve a realworld problem in engineering / logistics / medicine
LLMs currently attract all the hype for good reasons, but Im surprised VCs dont seem to be looking at RL companies specifically.
RL is definitely really cool but I heavily doubt that we're gonna see 'hyper growth' from RL outside of the context of maybe training reasoning LLMs.
The period from ~2012-2019 of AI research had deepmind (who was the undisputed leader in money and talent) go all in on RL to solve problems and while they did do lots of interesting and useful work, there wasn't anything quite so extraordinary / revolutionary in massively accelerating the field or some sort of crazy breakthrough.
Their over-focus on RL instead of transformers/llms is what allowed OpenAI to surprise everyone and overtake deepmind.
Yes, RL is a useful tool, but outside the context of training LLMs for reasoning there isn't really any breakthrough that makes it more than an interesting tool for certain situations.
The period from ~2012-2019 of AI research had deepmind (who was the undisputed leader in money and talent) go all in on RL to solve problems and while they did do lots of interesting and useful work, there wasn't anything quite so extraordinary / revolutionary in massively accelerating the field or some sort of crazy breakthrough.
Their over-focus on RL instead of transformers/llms is what allowed OpenAI to surprise everyone and overtake deepmind.
Yes, RL is a useful tool, but outside the context of training LLMs for reasoning there isn't really any breakthrough that makes it more than an interesting tool for certain situations.
> Im expecting to see a lot of hyper growth startups using RL to solve a realworld problem in engineering / logistics / medicine
I love when people on hn make market predictions based on how revolutionary they think something is. I guess startup people thank they're also VC people.
FYI Sutton's book came out in 1999; none of this is revolutionary anymore and yet I don't see any "hyper growth". The reason is exactly because while you can train these models to play super Mario, you cannot use them to solve real world problems.
https://www.google.com/books/edition/Reinforcement_Learning/...
I love when people on hn make market predictions based on how revolutionary they think something is. I guess startup people thank they're also VC people.
FYI Sutton's book came out in 1999; none of this is revolutionary anymore and yet I don't see any "hyper growth". The reason is exactly because while you can train these models to play super Mario, you cannot use them to solve real world problems.
https://www.google.com/books/edition/Reinforcement_Learning/...
generally you are right in spirit.
however multi-armed bandit algorithms are highly useful in practice. these are a special case of RL (RL with one state, essentially).
there are even some extensions of applied bandit algorithms to "true RL", e.g. for recommender systems that want to consider history.
this is the place to look for real-world applications of RL.
also RL uses importance-sampling estimators of the gradient. these sometimes show up in other applications though not framed as "RL".
however multi-armed bandit algorithms are highly useful in practice. these are a special case of RL (RL with one state, essentially).
there are even some extensions of applied bandit algorithms to "true RL", e.g. for recommender systems that want to consider history.
this is the place to look for real-world applications of RL.
also RL uses importance-sampling estimators of the gradient. these sometimes show up in other applications though not framed as "RL".
"FYI Maxwell's paper came out in 1865 and now it's 1896 and Marconi's radio, which he invented a whole year ago, still doesn't pick up anything but buzzes and static. The reason is exactly because while you can manipulate the electromagnetic field with current fluctuations, you cannot use it to solve real world problems."
Reinforcement learning is hard to apply to real-world problems, but one cannot deny the success that a company such as OpenAI has.
Sure.. and neural networks came out a very long time ago, but are now arguably approaching usefulness in LLMs.
Perhaps thats because it takes a while for the ideas to get polished/weeded and diffuse into the engineer zeitgeist .. or it could be that compute / GPUs are now powerful enough to run at the scale needed.
re : "RL cannot be used to solve real world problems" .. well, I would argue that these are useful real-world problems :
But I concede I cannot point to a plethora of small startups using RL for these real-world problems .. yet.
This is a prediction, and I could be wrong in many ways - not least that LLMs digest RLs in full and learn to express their logical reasoning, approaching AGI, and use RLs internally, and so subsume and automate the use of RL.
Are VCs better at predicting the future.. I guess that is their job, and they have money on the line... but I think even they would admit they need a large portfolio to capture the unicorns.
VCs probably get a less detailed tech view than founders, but the large number of pitches they review should give them a noisy but wider overview of the whole bleeding edge of innovation.
I think startup founders are in the same future prediction business .. and arguably have more skin in the game.
Predictions would be pretty useless if they weren't somewhat controversial - a prediction we all agree on doesn't say much. Come back and chastize me if we dont see more RL startups in 12 months time !
Perhaps thats because it takes a while for the ideas to get polished/weeded and diffuse into the engineer zeitgeist .. or it could be that compute / GPUs are now powerful enough to run at the scale needed.
re : "RL cannot be used to solve real world problems" .. well, I would argue that these are useful real-world problems :
- predict protein folding structure from DNA sequence
- stabilizing high temperature fusion plasma
- improving weather forecasting efficiency
- improve DeepSeek's recent LLM model
But I concede I cannot point to a plethora of small startups using RL for these real-world problems .. yet.
This is a prediction, and I could be wrong in many ways - not least that LLMs digest RLs in full and learn to express their logical reasoning, approaching AGI, and use RLs internally, and so subsume and automate the use of RL.
Are VCs better at predicting the future.. I guess that is their job, and they have money on the line... but I think even they would admit they need a large portfolio to capture the unicorns.
VCs probably get a less detailed tech view than founders, but the large number of pitches they review should give them a noisy but wider overview of the whole bleeding edge of innovation.
I think startup founders are in the same future prediction business .. and arguably have more skin in the game.
Predictions would be pretty useless if they weren't somewhat controversial - a prediction we all agree on doesn't say much. Come back and chastize me if we dont see more RL startups in 12 months time !
> Come back and chastize me if we dont see more RL startups in 12 months time !
1999 is 26 years ago but ya sure this is the year they finally take off.
> Perhaps thats because it takes a while for the ideas to get polished/weeded and diffuse into the engineer zeitgeist .. or it could be that compute / GPUs are now powerful enough to run at the scale needed.
Or perhaps it could be that you're wrong and they're useless? Nah that couldn't be it.
1999 is 26 years ago but ya sure this is the year they finally take off.
> Perhaps thats because it takes a while for the ideas to get polished/weeded and diffuse into the engineer zeitgeist .. or it could be that compute / GPUs are now powerful enough to run at the scale needed.
Or perhaps it could be that you're wrong and they're useless? Nah that couldn't be it.
And 1967 was 58 years ago, which was when the first deep neural network was trained with stochastic gradient descent. Yet, DNNs didn't take off until the 2010s when the hardware became powerful enough and data became plenty enough to successfully train and utilize them such that they were practical.
I think you GOT caught here. That's why you don't respond to the Nobel prize winning example of RL.
Are we talking about AlphaFold? It did not use RL, right?
I think it does: https://juanraul8.github.io/master-praktikum/
> you cannot use them to solve real world problems
Doesn't waymo and other self-driving systems use reinforcement learning? I thought it was used in robotics as well (i.e., bipedal, quadrupedal movement).
Doesn't waymo and other self-driving systems use reinforcement learning? I thought it was used in robotics as well (i.e., bipedal, quadrupedal movement).
Also worth mentioning Murphy's WIP textbook[0] focused entirely on RL, which is an outgrowth of his excellent ML textbooks.
[0]: https://arxiv.org/abs/2412.05265
[0]: https://arxiv.org/abs/2412.05265
Awesome resource, in case someone is interested I implemented most of suttons book here https://github.com/ivanbelenky/RL
Thanks, looks good
> This book, however, requires the reader to have some knowledge of probability theory and linear algebra.
This is so funny to me, I see it often and I'm always like "yea, right, some knowledge"... these statements always need to be taken with a grain of salt and an understanding that math nerds wrote them. Average programmers with average math skills (like me) beware ;)
This is so funny to me, I see it often and I'm always like "yea, right, some knowledge"... these statements always need to be taken with a grain of salt and an understanding that math nerds wrote them. Average programmers with average math skills (like me) beware ;)
This usually means that average CS or EE university level students should be able to easily follow it even if they have never touched the topic. It's far below the level of math and physics degrees, but still somewhat above what you could expect from an average self taught programmer.
I'm not even self-taught, it's just that when I was studying (CS degree, like 15 years ago) we did have a mandatory linear algebra course, graph theory course, statistics course etc., but we never * actually * used any of that in practice, it was all algo this, big o that, data structures, design patterns, languages, compilers, SQL etc. Now that I'm thinking about it pretty much the only course we had to use some linear algebra in was the 3d rendering one. ...
And then you work on .net/java/sql/server crap for a decade and you forget even the little math you used to know :D
And then you work on .net/java/sql/server crap for a decade and you forget even the little math you used to know :D
I don't know how to go from understanding this material to having a job in the field. Just stuck as a SWE for now.
Step 1: Build something cool with it.
- Do you understand the material?
- Can you utilize your understanding to build successful models/algorithms?
A way I like to gauge someone's understandings of things is by getting them to explain the limitations. This is often less explicitly stated in learning and a deeper understanding is acquired through experience and most importantly, reflection on that experience. This is often an underutilized tactic but it is very effective. If you can't do this, then the good news is that starting now will only accelerate your understanding :)
Just a random thought:
Understanding the limitations is a complicated thing in tech. You can finnangle most systems into doing mostly anything, as inefficient as that may prove to be.
The question then becomes up to what point is it "a reasonably better than most others" solution. And that's a question of an understanding of a field, not a space in the field.
Understanding the limitations is a complicated thing in tech. You can finnangle most systems into doing mostly anything, as inefficient as that may prove to be.
The question then becomes up to what point is it "a reasonably better than most others" solution. And that's a question of an understanding of a field, not a space in the field.
> is a complicated thing in tech
> You can finnangle most systems into doing mostly anything
> an understanding of a field, not a space in the field.
[deleted]
Thank you. This is great. I also appreciated the linked code for MinRL (https://github.com/10-OASIS-01/minrl).
Having done research in RL, a big problem with incremental research was to reproduce comparative works, and to validate my own contributions. A simple library like this, with built-in tools for visualization and a gridworld sandbox where I can validate just by observation, is very helpful!
Having done research in RL, a big problem with incremental research was to reproduce comparative works, and to validate my own contributions. A simple library like this, with built-in tools for visualization and a gridworld sandbox where I can validate just by observation, is very helpful!
And if you want to understand the theory of Skinner's Verbal Behavior check out
https://bfskinner.org/wp-content/uploads/2020/11/978_0_99645...
https://bfskinner.org/wp-content/uploads/2020/11/978_0_99645...
Amazing resource. Highly recommended for both content and approachability.