For some reason, perhaps some sort of Freudian self-defense mechanism, we tend to downplay how impressive solving never seen problems that require deep understanding of the concepts at play requires.
Look for final exams of advanced courses in CS or math. It will be clarifying how close (or plainly, harder) the questions from the study are. And so how impressive the capabilities these models are achieving...
The math works for codebases that survive past year one but not so much for less mature ones, no?
I shipped a small Stripe storefront with Claude Code over the weekend. Three pages, four integrations, one database. At that size you can read every file before merging, and that's basically the lever. Shore's argument really bites when you can't.
The thing missing from his model, I think, is project shape. Bounded greenfield has a maintenance ceiling because the code itself does. A long-lived monolith with the agent extending it is where the math gets ugly.
Look for final exams of advanced courses in CS or math. It will be clarifying how close (or plainly, harder) the questions from the study are. And so how impressive the capabilities these models are achieving...