Yes exactly, and also there is auto-research by Karpathy, Fable-5 on parameter golf etc which point to the direction of highly-assisted / autonomous AI training as the future. Wild.
I am also curious! The inner-RL-loop models are only trained once, then disgarded. But the outer-RL-loop model is trained on the same tasks over and over again. I imagine it would overfit after many more steps, but perhaps with a larger set of diverse tasks, the model would simply improve.
Yes I do agree with this. I believe we are shifting from "make the model good" (prompt/context engineering, etc) to "define good for the model" (success criteria/rubrics). Over time I believe this will become increasingly obvious (as long as model capabilities continue to increase).
I chose the key technical decision and direction (such as the system architecture, the tasks to train on, the stack of Tinker, Prime-RL & Runpod - all of which I know well) etc.
The problems it would do well on are training small agentic (multi-turn, tool use) task based models using the prime-rl stack, which are close to the distribution trained upon. It would likely not transfer to other training frameworks such as SLIME, ART or ROLL, it would also likely not transfer well to RL for complex agents such as coding agents etc.
It is limited due to its scale. As a single person, the resources required to train this on a more diverse dataset, with more complex tasks on a larger variety of models, is outside my abilities! I believe there are many avenues to explore to improve performance for this to be genuinely valuable. For now, this just a proof of concept to show the possible.
I would like to think I have a good understanding of RL, evaluations, and agentic systems after a few years of working on these areas. However, I will always have gaps. I use Fable to help accelerate me, and fill those gaps at the same time, from which I can learn from too.
Yes I'd heard the labs (Anthropic mostly) speaking about LLMs training LLMs, so I wanted to make things a little more concrete and test it out myself! Essentially you are correct though, my framework allows the agent access to compute, but also the agent itself is being trained to become better at training models with that compute.
I RL-trained an agent whose job is to write RL training jobs for smaller models, and open-sourced the whole thing.
The setup is two nested RL loops:
- Outer loop: the trainer agent (Qwen3.6-35B-A3B, LoRA) is handed a task spec ("teach a small model to do X"). It works in a sandboxed workspace with file tools and writes a complete prime-rl training job: a verifiers environment + rubric, a dataset, and a hyperparameter config. Submitting triggers a validation probe with capped retries.
- Inner loop: each validated job is dispatched to a warm pool of up to 16 Runpod GPU pods, where prime-rl GRPO-trains a small Qwen (0.6B or 1.7B). The checkpoint is scored pre/post on a hidden eval the agent never sees.
- The inner model's improvement flows back up as the outer loop's reward (plus a validation-efficiency term and a small train-speed tie-breaker). The outer loop is tinker-cookbook's importance-sampling GRPO, run async off-policy so one slow episode doesn't stall a batch.
Results, over 54 outer-loop steps (~1,750 real GPU training jobs):
- Episode reward went from ~0.0 to a ~0.63 peak.
- Learning came in two distinct rungs: first "stop failing validation and dying on GPUs", then "make better models". GRPO took the steepest gradient first — the entire early gain was process reliability, and only once that saturated did the hidden-eval scores of the trained models start climbing.
- It transferred to a held-out task family that the agent never trained on: mean reward 0.399 untrained → 0.545 at step 34, easing to 0.49 by step 54 (n=10 per arm, so noisy — a rise then a plateau/dip).
- The agent learned to stop picking the weaker 0.6B base model (1.7B share of its jobs: 42% → 95%) and started actually using the hyperparameter surface (21% → ~78% of episodes).
Cost: the headline arc was ~$1.3k all-in (~$810 Runpod, ~$465 Tinker). Each inner training job cost ~$0.13–0.30 — a benchmark matrix over GPU × base-model picked cheap pairs (mostly A40s in practice, since the cost-winner was rarely in stock).
Two honesty notes: the outer loop runs through Tinker's managed API rather than local GPUs — the inner loop is all open-source stack on rented pods. And ~$1.3k is the headline arc, not the project; the pilots and blind alleys that got me there cost a few hundred more, and every one of them is written up in the retros in the repo, including the failures.
I did this because I think agents that improve other AI systems are going to be a big part of the next few years, and I wanted to know what it actually takes to get the reward moving. Turns out: way more debugging of the process than the policy, and it's all more accessible than it looks.
Happy to answer questions about the reward design, the GPU orchestration, or the things that didn't work.
My RL trained multi-agent-coding model Orca-Agent-v0.1-14B reached a 167% higher relative score than its base model on Stanford's TerminalBench. I've open sourced everything.
*What I did:*
- I trained a 14B orchestrator model to better coordinate explorer & coder subagents (subagents are tool calls for orchestrator)
- Scaled to 32x H100s that were pushed to their limits across 4 bare-metal nodes
- Scaled to 256 Docker environments rolling out simultaneously, automatically distributed across the cluster
*Key results:*
- Qwen3-14B jumped from *7% → 18.25%* on TerminalBench after training
- Model now within striking distance of Qwen3-Coder-480B (19.7%)
- Training was stable with smooth entropy decrease and healthy gradient norms
*Training approach:*
Reward design and biggest learning: Kept it simple - *just unit tests*. Every "smart" reward signal I tried to craft led to policy collapse
Curriculum learning:
- Stage-1: Tasks where base model succeeded 1-2/3 times (41 tasks)
- Stage-2: Tasks where Stage-1 model succeeded 1-4/5 times
Dataset: Used synthetically generated RL environments and unit tests
*More details:*
I have added lots more details in the repo linked to this submission, including training code, model weights, datasets.
Huge thanks to:
- Tara for providing the compute
- Prime Intellect team for building prime-rl and dealing with my endless questions
- Alex Dimakis for the conversation that sparked training the orchestrator model
Thanks for reading!
Dan
(Evaluated on the excellent TerminalBench benchmark by Stanford & Laude Institute)
Hitting a million brick walls with multi-turn RL training isn't fun, so I thought I would try something new to climb Stanford's leaderboard for now! So this weekend I was just tinkering with multi-agent systems and... somehow ended up beating Claude Code on Stanford's TerminalBench leaderboard (#12)! Genuinely didn't expect this - started as a fun experiment and ended up with something that works surprisingly well.
*What I did:*
- Built a multi-agent AI system with three specialised agents:
- Orchestrator: The brain - never touches code, just delegates and coordinates
- Explorer agents: Read & run only investigators that gather intel
- Coder agents: The ones who actually implement stuff
- Created a "Context Store" which can be thought of as persistent memory that lets agents share their discoveries.
- Tested on TerminalBench with both Claude Sonnet-4 and Qwen3-Coder-480B.
*Key results:*
- Orchestrator + Sonnet-4: 36.0% success rate (#12 on leaderboard, ahead of Claude Code!)
- Sonnet-4 consumed 93.2M tokens vs Qwen's 14.7M tokens to compete all tasks!
- The orchestrator's explicit task delegation + intelligent context sharing between subagents seems to be the secret sauce
*(Kind of) Technical details:*
- The orchestrator can't read/write code directly - this forces proper delegation patterns and strategic planning
- Each agent gets precise instructions about what "knowledge artifacts" to return, these artifacts are then stored, and can be provided to future subagents upon launch.
Exactly my first thought when I realised the cost!
Currently LoRA is not supported by rLLM (The team told me they aim to support in next release), but it is certainly possible to port to verl directly or another RL framework for sure. I just did not have the time to port again (already done 2x as other RL frameworks had issues)