Ex-Pythagora (YC W24), built & sold A Web Whiteboard (acq'd by Miro) and two earlier startups. Runs GoodCode.io (dev agency). Currently doing AI-related consulting.
Clearly much better than the Terra version. I'd say its on par with Fable, and the observed differencies are more due to random luck and open-ended prompt, rather than model capability. (Edit: after some more testing, perhaps not on par - somewhere between Opus and Fable, is a better description).
Fable did better pathfinding and has more terrain variety, visually the map looks better, especially soft edges of the fog of war. And the enemies.
Sol took more care with tiny ux details, added help, and more building varieties.
> Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).
It even has enemies! (I'm not too mad about it not following my instructions because it can be fun to play :) And I generated that from Claude Code on my phone.
Well, it is a silly test, not a scientific benchmark.
However, I would say it is a measure (not the measure). If you look at the entries, there's a lot of variation - definitely not something they memorized outright.
And the test itself is deceptively simple. You need to do canvas rendering, there's pathfinding, command queueing, terrain generation, etc. There are some subtle click handler bugs (various LLMs often stumble on those). And I ask the model to do it all in one file, further increasing the complexity of the task.
And the result is something that you can instantly evaluate. And if the result is any good, even play! So yeah, I think it's a fair test.
I'm sure it'll get saturated at some point. Actually I started with Minesweeper and switched to RTS last December, because Minesweeper was being saturated. I'm expecting (hoping?) the RTS test will last until the end of this year...
Comparing this to other models, I find it similar to GPT-5.5 and a bit behind Sonnet 5. You can see how other models fared here: https://senko.net/vibecode-bench/ (you can also fetch the prompt and the the 5.6 Terra resulting code on from that page).
I don't have access to Sol yet (on a Plus sub, which should get it according to what I've read), so can't do the more interesting test. I'll update the above page as soon as I get access - hopefully soon.
> feed 2bln a year into it and make a compelling almost SOTA model
...and the reason is, if you give a bunch of people €2b a year and tell them "go try and make something", they'll make a ton of paperwork covering their asses and very little actual output.
This is irrespective if those people are European ("european google killer"), American ("cost plus" old US aerospace companies) or Chinese (which is why they do it a little different).
If there are no incentives to really try really hard, they won't do it.
In many high-tech cases in Europe, the formula for "let's subsidise the hell out of research and hope a commercially-viable business comes out" has a really poor track record.
Your second option - and possibly the best bet - is to find an existing company that already showed they're capable, and shower them with money, which is what French are doing with Mistral.
LinkedIn in particular is quite aggressively blocking any automated attempts to read or navigate through it.
I post quite a lot there and wanted to have a copy of my posts on my blog[0] to preserve them. For a few months I was able to use a headless browser + claude code, then LI wised up and started logging it out, so I had to use a regular Chrome, log in manually and then tell the LLM to take over and slowly go through my feed.
If you're accessing sites which are not actively blocking bots, or - gasp - have an API, it's much better.
Good catch. I haven't looked deeply into it. This is with Vulkan backend on Linux which I understand should be roughly comparable to CUDA? Gfx is rtx 3060(ti?).
I should play a bit more with llama.cpp options and see what bappened there. Thanks!
The result is decent, but it had a few bizzare/trivial syntax errors I had to fix manually: it would do an extra closing bracket or paren a few times, and wanted to separate function definitions with comma. Not sure what that was about, but otherwise the output run just fine.
So, with those qualifiers, I think it's a decent local coding model. It roughly compares with GPT-4.1 (!!), released 14 months ago, on the output: https://senko.net/vibecode-bench/2025/minesweeper-gpt-4.1.ht... (actually I'd call it better, but those syntax errors...)
I ran the quantized version (4-bit GGUF) on my consumer-grade card with 12G of VRAM and got 5t/s for output. Not for interactive use for coding, but fairly capable model.
To me, it's fascinating how much progress we got in over a year. GPT-4.1 was considered an extremely capable coding model. Now we got something with 12B of params performing roughly the same (in this specific benchmark, disclaimers, etc).
Yeah, it's weird, nobody's saying "we should make all the data centres use closed loop cooling even if it's more expensive for them!", but a lot of voices are yelling "AI uses water!", referring to the same thing.
I mean, email and Hacker News and Netflix use water, too.
I do believe there's going to be a lot of left-behinds, as a sort-of digital rust belt. Even though, as an industry, we've always been in the business of automating and replacing ourselves, the shift will hit too quick.
I lay none of the blame on AI the technology, and all the blame on AI as a mindset and excuse.
Layoffs are not due to AI, but it's a convenient excuse: "more productive, don't need people, we're firing on all cylinders and yeah, firing 20% of the workforce while we're at it". Everything else being equal, the "more productive so we earn 20%" counterfactual makes more sense - but of course, not everything else is equal.
Treadmill speed will increase, no doubt. We haven't lowered working hours from 40 to 20 when computers 2x'd us all, we for sure won't lower them now.
We'll manage the nondeterministic imperfections, but boy, will there be bumps on the road.
What I fear most: AI will give us all more power. This includes profitmaxxing no-holds-barred corpos, from preseed startups hustling 996-style to big multinacionals. Even now, with locked-down devices and subscriptions for everything and owning things replaced with "owning a limited nontransferrable revokable end-user license", it's not good. AI is going to multiply that.
> And things like AC and clothes dryers are taken for granted.
Not sure where you get your impression of Europe, but if you feel amenities like these are not standard, it’s a few decades out of date.
North Europeans traditionally didn’t need AC, but everywhere where it gets hot - which is everywhere now - they got them installed. Very few buildings with integrated HVAC systems for the entire buildings tho, mostly independent units.
European here. Yes, houses are smaller, apartments can be comparatively tiny. Street parking can be a challenge.
However: I got stores, cinemas, cafes, restaurants within walking distance. My kids can roam around in the neighbourhood without someone calling social service on me. I can walk anywhere in the city at any hour day or night without someone robbing me. I can cheaply purchase free range eggs and organic vegetables. Tap water is fine, actually excellent. 30hour commute is considered too long. Coast is mere 3h away, people come from all over the planet to enjoy it, I spend 5 weeks a year there, just chillin and enjoying life. I get fast, cheap internet, and order groceries, do my taxes and doctors appointment online.
Tell me again how I’m suffering without poorly insulated detached houses, HSA, spam calls, an SUV to drive myself to the bakery, school shooting drills, healthcare bills, homeless people rejected by society, and that circus you have for a government right now?
Interests: AI, startups, open source.
Twitter: @senkorasic
Bluesky: @senko.net
Work: https://senkorasic.com
Personal: https://senko.net & https://blog.senko.net
GitHub: http://github.com/senko
Mail: [email protected]