Scaffolding to Superhuman: How Curriculum Learning Solved 2048 and Tetris(kywch.github.io)
kywch.github.io
Scaffolding to Superhuman: How Curriculum Learning Solved 2048 and Tetris
https://kywch.github.io/blog/2025/12/curriculum-learning-2048-tetris/
34 comments
> To learn, agents must experience high-value states, which are hard (or impossible) for untrained agents to reach. The endgame-only envs were the final piece to crack 65k. The endgame requires tens of thousands of correct moves where a single mistake ends the game, but to practice, agents must first get there.
This seems really similar to the motivations around masked language modeling. By providing increasingly-masked targets over time, a smooth difficulty curve can be established. Randomly masking X% of the tokens/bytes is trivial to implement. MLM can take a small corpus and turn it into an astronomically large one.
This seems really similar to the motivations around masked language modeling. By providing increasingly-masked targets over time, a smooth difficulty curve can be established. Randomly masking X% of the tokens/bytes is trivial to implement. MLM can take a small corpus and turn it into an astronomically large one.
This is less about masked modelling and more about reverse-curriculum.
e.g. DeepCubeA 2019 (!) paper to solve Rubik cube.
Start with solved state and teach the network successively harder states. This is so "obvious" and "unhelpful in real domains" that perhaps they havent heard of this paper.
e.g. DeepCubeA 2019 (!) paper to solve Rubik cube.
Start with solved state and teach the network successively harder states. This is so "obvious" and "unhelpful in real domains" that perhaps they havent heard of this paper.
perhaps I'm missing something. Why not start the learning at a later state?
If the goal is to achieve end-to-end learning that would be cheating.
If you sat down to solve a problem you’ve never seen before you wouldn’t even know what a valid “later state” looking like.
If you sat down to solve a problem you’ve never seen before you wouldn’t even know what a valid “later state” looking like.
Why is it cheating? We literally teach sports this way? Often times you teach sports by learning in scaled down scenarios. I see no reason this should be different.
If the goal is to learn how to solve a Rubik's Cube when you've never seen a Rubik's Cube before, you have no idea what "halfway solved" even looks like.
This is precisely how RL worked for learning Atari games: you don't start with the game halfway solved and then claim the AI solved the end-to-end problem on its own.
The goal in these scenarios is for the machine to solve the problem with no prior information.
This is precisely how RL worked for learning Atari games: you don't start with the game halfway solved and then claim the AI solved the end-to-end problem on its own.
The goal in these scenarios is for the machine to solve the problem with no prior information.
This isn't accurate, though? Halfway solved, for most teachings, is to have the first layer solved.
Indeed, this is a key to teaching people to know how to advance. Do not focus on a side, but learn to advance a layer.
Indeed, this is a key to teaching people to know how to advance. Do not focus on a side, but learn to advance a layer.
That's effectively what you get in either case. With MLM, on the first learning iteration you might only mask exactly one token per sequence. This is equivalent to starting learning at a later state. The direction of the curriculum flows toward more and more of these being masked over time, which is equivalent to starting from earlier and earlier states. Eventually, you mask 100% of the sequence and you are starting from zero.
You can watch these agents play live, and you can also intervene
* 2048: https://kywch.github.io/games/2048.html
* Tetris: https://kywch.github.io/games/tetris.html
I've always found curriculum learning incredibly hard to tune and calibrate reliably (even more so than many other RL approaches!).
Reward scales and horizon lengths may vary across tasks with different difficulty, effectively exploring policy space (keeping multimodal strategy distributions for exploration before overfitting on small problems), and catastrophic forgetting when mixing curriculum levels or when introducing them too late.
Does any reader/or the author have good heuristics for these? Or is it still so problem dependent that hyper parameter search for finding something that works in spite of these challenges is still the go to?
Reward scales and horizon lengths may vary across tasks with different difficulty, effectively exploring policy space (keeping multimodal strategy distributions for exploration before overfitting on small problems), and catastrophic forgetting when mixing curriculum levels or when introducing them too late.
Does any reader/or the author have good heuristics for these? Or is it still so problem dependent that hyper parameter search for finding something that works in spite of these challenges is still the go to?
I think Go-Explore (https://arxiv.org/abs/1901.10995) is promising. It'll provide automatic scaffolding and prevent catastrophic forgetting.
If one can frame the problem into a competition, then self-play has been shown to work repeatedly.
If one can frame the problem into a competition, then self-play has been shown to work repeatedly.
Star the puffer
https://github.com/PufferAI/PufferLib
Curriculum learning helped me out a lot in this project too https://www.robw.fyi/2025/12/28/solve-hi-q-with-alphazero-an...
Is there value in using deep RL for problems that seem more suited to planning-based approaches?
Unless I am mistaken, this would be the first heuristic-free model trained to play tetris, which is pretty incredible, since mastering tetris from just raw game state has never been close to solved, till now(?)
Pufferlib already had a pretty good model before: https://puffer.ai/ocean.html?env=tetris
I wonder if he tried NNUE
NNUE is for deep searches, as far as I understand this just says what move to do based on the state?
I'm gonna go out on a limb and say that this is LLM written slop that is badly edited by a human. Factually correct but the awful writing remains.
What I like about this writeup is that it quietly demolishes the idea that you need DeepMind-scale resources to get “superhuman” RL. The headline result is less about 2048 and Tetris and more about treating the data pipeline as the main product: careful observation design, reward shaping, and then a curriculum that drops the agent straight into high-value endgame states so it ever sees them in the first place. Once your env runs at millions of steps per second on a single 4090, the bottleneck is human iteration on those choices, not FLOPs.
The happy Tetris bug is also a neat example of how “bad” inputs can act like curriculum or data augmentation. Corrupted observations forced the policy to be robust to chaos early, which then paid off when the game actually got hard. That feels very similar to tricks in other domains where we deliberately randomize or mask parts of the input. It makes me wonder how many surprisingly strong RL systems in the wild are really powered by accidental curricula that nobody has fully noticed or formalized yet.
The happy Tetris bug is also a neat example of how “bad” inputs can act like curriculum or data augmentation. Corrupted observations forced the policy to be robust to chaos early, which then paid off when the game actually got hard. That feels very similar to tricks in other domains where we deliberately randomize or mask parts of the input. It makes me wonder how many surprisingly strong RL systems in the wild are really powered by accidental curricula that nobody has fully noticed or formalized yet.
You never needed DeepMind scale resources to get superhuman performance on a small subset of narrow tasks. Deep Blue scale resources are often enough.
The interesting tasks, however, tend to take a lot more effort.
The interesting tasks, however, tend to take a lot more effort.
Those are not hard tasks ...
Great, add "curriculum" to the list of words that will spark my interest in human learning, only for it to be about garbage AI. I want HN with a hard rule against AI posts.
Are we really dismissing the entire field of AI just because LLMs are overhyped?
LLMs show the problems of energy economy in this form of computing. It costs way too much in resources and power for minimal and generally worthless results. 2048 is a game with a several known algorithm for winning. Tetris is an obscenely simple game that unassisted humans could reliably take to the kill screen 20 years ago.
Does any of this used energy benefit any other problem?
Also using "Superhuman" in the title is absurd given this paltry outcome.
Does any of this used energy benefit any other problem?
Also using "Superhuman" in the title is absurd given this paltry outcome.
Believe it or not, you can visit more than 1 website. How about a guideline to put (AI) like we do with (video). I'm just sick of having to click to figure out if it's about humans or computers. They've hijacked every single word related to the most fascinating thing in the entire universe just to generate ad revenue and VC funding.
The famous Hacker News website is about computers. It is also about ad revenue and VC funding. It was originally named Startup News, and its patron and author is the multibillionaire founder of a well-known "startup accelerator" called "Y Combinator."
> Believe it or not, you can visit more than 1 website.
> Believe it or not, you can visit more than 1 website.
Why garbage ai? I thought it was a very interesting post, personally.
> HN with a hard rule against AI posts.
Greasemonkey / Tampermonkey / User Scripts with
Array.from( document.querySelectorAll(".submission>.title") ).filter( e => e.innerText.includes("AI") ).map( e => e.parentElement.style.opacity = .1)
Edit: WTH... how am I getting downvoted for suggesting an actual optional solution? Please clarify.
Greasemonkey / Tampermonkey / User Scripts with
Array.from( document.querySelectorAll(".submission>.title") ).filter( e => e.innerText.includes("AI") ).map( e => e.parentElement.style.opacity = .1)
Edit: WTH... how am I getting downvoted for suggesting an actual optional solution? Please clarify.
What you get is an iterator over the dataset that samples based on how far you are in the training.
0: https://github.com/omarkamali/curriculus