> However, it’s your job to go down the rabbit hole, learn the 100%, and sprinkle in your 3%.
I would say that there is a big difference between stealing without acknowledgement, and stealing with acknowledgement and actively learning through reverse engineering.
Thanks, I didn't mean to be brusque, but I have seen a lot of these vibe tests lately that come to grand conclusions like "X model is better than Y" from the result of a single prompt.
Appreciate you sharing the results of your tests though!
> So we ran it head-to-head against Claude Opus 4.8: same one-shot prompt, build a 3D platformer in raw WebGL from scratch
Running a single one-shot prompt is not a benchmark, not is it representative of any sort of real-world usage.
Most agent usage is collaborative so you need to test things like reliability (when I delegate a task, does it complete it without making up test results for e.g.) and steerability (does it obey my instructions or does it just do what it thinks is best).
As someone who has built both react based frontends and html based ones (with htmx), there is a law of diminishing returns at play.
To start off, writing a basic crud website with forms is much easier with htmx.
But when you start building more complex components, and integrate with other systems (OAuth for e.g.) there are tons of libraries and SDKs for the react ecosystem, but not many for pure html components.
At this point, it's much easier to use off the shelf components than it is to manually write html to handle all the bizarre UI edge cases.
> the model has its own emergent guardrails that sometimes cause it to push back on legitimate security research requests. But as we found, these organic refusals aren’t consistent - the same task, framed differently or presented in a different context, could produce completely different outcomes as illustrated in the examples below.
This was new. I'm surprised that a model specifically designed for security research and gated to professionals is refusing legitimate requests
Gemma4 edge models were promised to be great for agentic use, but have been really disappointing in all my tests. They fail at the most basic tool use scenarios.
Have you run any tool-use benchmarks for Needle, or do you plan to? Would be great if you could add results to the repo if so.
One underrated advantage of using Python or Typescript is that AI agents can inspect the code of installed dependencies.
This means you don't have to muck around with supplying the right documentation for each version of each dependency, or worry about hallucinated interfaces (at least with the latest models).
In the past you'd have to dig through a foreign codebase manually to figure out why a documented interface for a dependency is not working as expected, but frontier models automate that quite well.
> We find that
models are not failing due to “death by a thousand
cuts” (i.e., many small errors). Instead, they main-
tain near-perfect reconstruction in some rounds, and
experience critical failures in a few rounds, typically
losing 10-30+ points in a single round trip
> We find that
weaker models’ degradation originates primarily from
content deletion, while frontier models’ degradation is
attributable to corruption of content.
I think we largely already knew this. This is why we fudge around with harnesses and temperature etc.
https://vivis.dev
https://findsubstack.com
https://pythonkoans.substack.com