Easily 5-10x or even more in certain special cases (when it'd take me a lot of upfront effort to get context on some problem domain). And it can do all the "P2"s that I'd realistically never get to. There was a day where I landed 7 small-to-medium-size pull requests before lunch.
There are also cases where it fails to do what I wanted, and then I just stop trying after a few iterations. But I've learned what to expect it to do well in and I am mostly calibrated now.
The biggest difference is that I can have agents working on 3-4 parallel tasks at any given point.
Not to downplay the issue you raise but I haven't noticed this.
Every iteration I make on the prompts only make the request more specified and narrow and it's always gotten me closer to my desired goal for the PR. (But I do just ditch the worse attempts at each iteration cycle)
Is it possible that reasoning models combined with the actual interaction with the real codebase makes this "prompt fragility" issue you speak of less common?
I work at OpenAI (not on Codex) and have used it successfully for multiple projects so far. Here's my flow:
- Always run more than one rollout of the same prompt -- they will turn out different
- Look through the parallel implementations, see which is best (even if it's not good enough), then figure out what changes to your prompt would have helped nudge towards the better solution.
- In addition, add new modifications to the prompt to resolve the parts that the model didn't do correctly.
- Repeat loop until the code is good enough.
If you do this and also split your work into smaller parallelizable chunks, you can find yourself spending a few hours only looping between prompt tuning and code review with massive projects implemented in a short period of time.
I've used this for "API munging" but also pretty deep Triton kernel code and it's been massive.
For those who are interested in getting involved with online math circles (as parents or potential instructors), check out https://theglobalmathcircle.org (Jeremy, the author of this post graduated from our training program)
This isn't accurate. The bottleneck in very-large-scale-training BY FAR is communication between devices. If you have a million CPUs, the communication cost will be significantly higher than a thousand A100s (perhaps in the order of 100x or even more). So this is only possible to replicate with very dense and high compute chips with extremely fast interconnect.
OP here. I help run the Global Math Circle. Happy to answer any questions.
Here's a short blurb about our approach and history:
---
The GMC approach
Our approach is to treat math education as an accessible mystery. We combine a small group of children and an experienced guide. They go deep for ~8 weeks seeking to understand the insight behind the mystery. It’s an awesome experience where once a week the children share ideas and debate with each other what the next question should be, discovering deep mathematics along the way. The guide contributes minimal input because the discussion is led by the students, helping them learn leadership skills along the way. We never turn anyone away for lack of funds. All of our prices for circles and leader training are sliding scale and we regularly accept kids who can’t pay at all.
---
Our History
The Global Math Circle (originally named “The Math Circle”), the earliest math circle in the US, was founded in Boston, MA in 1994 by Robert and Ellen Kaplan[1], authors of such books as Out of the Labyrinth[2] and The Art of the Infinite[3]. The intention, in Bob's words, "Small groups meet with their leader, each seeing and talking with all; and as always, the leader no more than posing a deep and exciting question (an accessible mystery) and the students probing it collegially together. They come up with examples, counter-examples, insights and proofs: intense fun."
In 2015, the GMC moved online and has been running online circles and leader training institutes since. A group of enthusiasts have been helping Bob and Ellen, and with Bob's passing in 2022, continue to operate the GMC on the same principles that guided the organization from its inception. In addition to our main presence across the United States, GMC actively collaborates with groups around the world to spread our approach, from Senegal[4] to Brazil[5] and from Kenya[6] to Columbia[7].