Not including their best model in a max subscription would otherwise be truly a good reason for once to consider going back to openai for me. I'll at least try it.
I mean these were all solved before I assume so 100% not the same human ofc but models are expected to be good at a variety of code bases while human can specialize in one and learn. I think it's fair to compare to an individual that is used to working on a product.
I mean if you have an idea or project that convinces you its worth to found a company, a thousand bucks is not a great hinderance. If the 1000 bucks are a hinderance you probably shouldnt found a company. It's a decent filter if you are serious about it.
This mostly reads as a comparison between Opus 4.7 and 4.1 it would be more interesting if they reran the experiment against a team of humans with 4.7 and see how much the humans still improve the results today.
yes I agree with this, more granular going back, letting me interrupt where it went off the rails, or even editing file reads myself etc would be lovely. Ingesting parts of other conversations would also be cool!
None of his math really checks out. Building a piece of software is or at least was orders of magnitudes more expensive than maintaining it. But how much money it can make is potentially unbounded (until it gets replaced).
So investing e.g. 10 million this year to build a product that produces maybe 2 million ARR will have armortized after 5 years if you can reduce engineering spend to zero. You can also use the same crew to build another product instead and repeat that process over and over again. That's why an engineering team is an asset.
It's also a gamble, if you invest 10 million this year and the product doesn't produce any revenue you lost the bet. You can decide to either bet again or lay everyone off.
It is incredibly hard or maybe even impossible to predict if a product or feature will be successful in driving revenue. So all his math is kinda pointless.
Doesn't look as bad as I expected tbh.
Sure some stuff could be better but I've seen much shittier vibe coded projects (including my own). I'd be more interested in their workflows and testing pipeline though. They ship pretty often but Boris still says he has 10+ PRs a day. I would be really curious what triggers a release, since it doesn't seem like every PR is released. I'm also curious how large their PRs really are.
There is a big difference between:
> Build plugins
and:
> Add 3px padding in line 5
if you claim "No code is written by humans anymore"
Yes and even now if you tell the LLM any private information inside the sandbox it can now leak that if it gets misdirected/prompt injected.
So there isn't really a way to avoid this trade-off you can either have a useless agent with no info and no access. Or a useful agent that then is incredibly risky to use as it might go rogue any moment.
Sure you can slightly choose where on the scale you want to be but any usefulness inherently means it's also risky if you run LLMs async without supervision.
The only absolutely safe way to give access and info to an agent is with manual approvals for anything it does. Which gives you review fatigue in minutes.
FWIW I reported your post to the mods because it reads completely AI generated to me. My judgement was that it might have been slightly edited but is largely verbatim LLM output.
Some tells that you might wanna look at in your writing, if you truly did write it yourself without Any LLM input are these contrarian/pivoting statements. Your post is full of these and it is imo the most classic LLM writing tell atm. These are mostly variants of the 'Its not X but Y" theme:
- "Not whether they've adopted every tool, but whether they're curious"
- "I still drive the intuition. The agents just execute at a speed I never could alone."
- "The model doesn't save you from bad decisions. It just helps you make them faster."
- "That foundation isn't decoration. It's the reason the AI is useful to me in the first place."
- "That's not prompting. That's engineering"
It is also telling that the reader basically cant take a breather most of the sentences try to emphasize harder than the last one. There is no fluff thought, no getting side tracked. It reads unnatural, humans do not think like this usually.
When did they stop putting competitor models on the comparison table btw?
And yeh I mean the benchmark improvements are meh. Context Window and lack of real memory is still an issue.
Socials:
- github.com/Askir
- linkedin.com/in/jaschabeste
Opinions are my own