The flow you describe in that comment is rather simple in my opinion and with the right harness even Sonnet would drive most of that.
I judge by the ability to bugfix complex codebases and the direction it takes in architecture. In my opinion, that's a tad more complex (and easier to objectively measure) than orchestrating tickets, no matter how complex.
Step 1: don't trust benchmarks you don't understand - they might measure irrelevant things
Step 2: test it on things you know Opus failed
My day-to-day take, for the coding I do (not security related): incremental, modest improvement, if any. Not worth the 2x cost. I've calmly continued to use Opus, happy that it seems like it got an allowance upgrade.
I'm more interested why you think my understanding is flawed honestly. I thought I distilled it decently well in two sentences. The bottom line is, in this hyperdimensional space you can find relationships that are not easily distinguished by human minds, but the corpus is still fixed, a llm can't truly know anything beyond its training data.
The strongest argument for this is structural: what LLMs are.
In a brutal simplistic way: each token is represented in a high dimensional vector. LLMs operate on them. They are the true, underlying meaning of the token for the LLM. Think of it as 1000+ ways to think of that word/token. Those meanings are baked in at training time. So, LLMs might be able to cross-reference them and solve a class of problems that flew under our radar, but can't come up with revolutionary theories that were never in the training set.
Of course, they will help winning a Nobel in the years to come, no doubt, but can't speak mathematics we can't understand (beyond simple obfuscation) and won't discover anything substantial on their own.
AI (in this form) will never be able to solve things we truly cannot solve yet. It might catch things that we didn't project properly or brute force things no human can , but it will never unify general relativity with quantum mechanics. It's amazing at finding hidden truths in large datasets, but won't win a Nobel unassisted.
Agreed. As someone who was always curious but had difficulties learning math the way it's taught at the university, AI teaching me the way no professor ever could is a blessing. I fail to see the point of the memo besides: we got here first and we decide what math is because we can. I'm really optimistic about AI and the value it brings in education. Gatekeepers will complain, but ultimately, will either adapt or be left behind.
We're releasing this next week.