Seems like the definition of success could skew things in a couple of ways. An engineer might use Claude to evaluate an approach and conclude it's not worth pursuing for technical reasons. There'd then be no commit and the session likely doesn't count as verified success even though that was the right call.
And is tests passing a weaker signal than it looks? If I add new functionality without adding tests, existing tests can all pass but it does't tell you whether my new code actually works.
Not quibbling but to be fair that report shows problems in Europe too, not the same speed of change, and its a different situation, but if you care about democracy its not great.
I've started using LLMs to help with card creation, and reviewing historic cards. Its been increasingly successful, I still end up with cards tailored to me, and the card quality has gone up (as I've asked the LLM to look for other views/sources and to do associate research to produce high quality cards).
Having ability to bake some of that into the tool in a configurable way would be ideal, and I hope thats sort of path they go down.
I realise in meantime plugins are an option, but I've found the quality of plugins very mixed.
"I implemented a formula for Jeffrey Emanuel’s “Rule of Five”, which is the observation that if you make an LLM review something five times, with different focus areas each time though, it generates superior outcomes and artifacts. So you can take any workflow, cook it with the Rule of Five, and it will make each step get reviewed 4 times (the implementation counts as the first review)."
And I guess more generally, there is a level of non-determinism in there anyway.
Yeah and its not a big focus of the posts which is interesting. I'd have thought he'd spend a lot more time talking about the workflow he's using, the specs/feature definitions he's writing, and so on.
"Agent 1, refactor that method to be more efficient. Agent 5, tighten up the graphics on level 3!"
I'm not sure its even that, his description of his role in this is:
"You are a Product Manager, and Gas Town is an Idea Compiler. You just make up features, design them, file the implementation plans, and then sling the work around to your polecats and crew. Opus 4.5 can handle any reasonably sized task, so your job is to make tasks for it. That’s it."
And he says he isn't reviewing the code, he lets agents review each others code from look of it. I am interested to see the specs/feature definitions he's giving them, that seems to be one interesting part of his flow.
I don't know, your talking about an incredibly talented engineer saying:
"In the past week, just prompting, and inspecting the code to provide guidance from time to time, in a few hours I did the following four tasks, in hours instead of weeks"
Its up to you to decide how to behave, but I can't see any reasons to completely dismiss this. It ends with good guidance what to do if you can't replicate though.
I would assume there are open source solutions to the problem it could have trained on, if so it would be interesting yo see how they had influenced what Claude produced here.
And is tests passing a weaker signal than it looks? If I add new functionality without adding tests, existing tests can all pass but it does't tell you whether my new code actually works.
To be fair this is accepted near the end.