I'm not a big user, but I have been doing some vibe-ish coding for a PoC the past few days, and I'm astonished at how bad it is at python in particular (Opus 4.6 High).
* It likes to put inline imports everywhere, even though I specify in my CLAUDE.md that it should not.
* We use ruff and pyright and require that all problems are addressed or at least ignored for a good reason, but it straight up #noqa ignores all issues instead.
* For typing it used the builtin 'any' instead of typing.Any which is nonsense.
* I asked it to add a simple sum of a column from a related database table, but instead of using a calculated sum in SQL it did a classic n+1 where it gets every single row from the related table and calculates the sum in python.
Just look at the commit cadance, the bulk of the 8k lines of code was added in a couple of hours. Most commits 2-4 minutes apart. This is 100% vibe coded and it's pretty obvious.
> It doesn't show any obvious indications of being AI.
I agree that he probably asked the AI to omit some common AI tells, like excessive comments, verbose readmes etc.
I did a cloc check on it and it does seem to have 800k lines of typescript. So unless they are vendoring dependencies it's actually as insane as it sounds.
5 day old repo, 2000 stars on GitHub, 400 total weekly downloads on npm. Frontpage of hacker news with a bunch of weird comments. Moderation has been lacking recently.
I don't read a lot of papers, but to me this one seems iffy in spots.
> A1 cost $291.47 ($18.21/hr, or $37,876/year at 40 hours/week). A2 cost $944.07 ($59/hr, $122,720/year). Cost
contributors in decreasing order were the sub-agents, supervisor and triage module. *A1 achieved similar vulnerability
counts at roughly a quarter the cost of A2*. Given the average U.S. penetration tester earns $125,034/year [Indeed],
scaffolds like ARTEMIS are already competitive on cost-to-performance ratio.
The statement about similar vulnerability counts seems like a straight up lie. A2 found 11 vulnerabilities with 9 of these being valid. A1 found 11 vulnerabilities with 6 being valid. Counting invalid vulerabilities to say the cheaper agent is as good is a weird choice.
Also the scoring is suspect and seems to be tuned specifically to give the AI a boost, heavily relying on severity scores.
Also kinda funny that the AI's were slower than all the human participants.
* It likes to put inline imports everywhere, even though I specify in my CLAUDE.md that it should not.
* We use ruff and pyright and require that all problems are addressed or at least ignored for a good reason, but it straight up #noqa ignores all issues instead.
* For typing it used the builtin 'any' instead of typing.Any which is nonsense.
* I asked it to add a simple sum of a column from a related database table, but instead of using a calculated sum in SQL it did a classic n+1 where it gets every single row from the related table and calculates the sum in python.
Just absolute beginner errors.