Not quite. These cost-per-task benchmarks report the cost of the task after the model gives its initial answer. The total cost is irrelevant, and isn't factored into the model's decisions - a run of the full benchmark for something like Fable might cost $10k.
What I'm looking for is the inverse. I want to give the model a budget of $100, and see how much it can accomplish with that $100. For smaller models, this means they can do more than just choose thinking amount, they can do something like a /loop to keep iterating on a problem until they get it right.
Can something like Deepseek V4 Flash get more answers correct than Fable, when given equal budgets?
Think of it as answering this question: How much intelligence can you get out of a model given a budget of $100? A cost-per-task dash correlates, but it doesn't give you an answer to that question.
I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests?
Give us something that measures a combination of efficiency and intelligence.
I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have budget to self-test.
Cloudflare is obviously more trustworthy/robust here, but if name of the url matters to you, my site non.io [1] allows for named uploads, ie https://html.non.io/solara [2]
Somewhat useful if you want a url that isn't a hash / is more self descriptive.
This feels like a regulation whose effectiveness will expire in the next couple of years (as driverless cars become the norm), but which will set a precedent that this is the norm. This with the EU chat control coming up really set a tone.
These "pull the loop over" kinda knots are delightfully simple, and shockingly secure. The Palomar fishing knot is somewhat similar, and is one of my go-to ones when I need to tie a hook quickly: https://www.youtube.com/watch?v=uWw_f7CQQLg
A non-trivial amount of their ARR is still from Valve-made games. Counterstrike still nets a bit over 1b per year from just case unboxings, and Dota is in the hundreds of millions. I wouldn't call 8-10% a margin of error.
What's fascinating to me are the Valve comparables here.
<500 employees vs 18k at Xbox
17B ARR vs 20B ARR
At the end of the day there are two strong differences here. Valve has always been lead by people who were game devs, and have always conveyed a message that the gaming experience matters most. Xbox was led by Phil Spencer, who at least was known as being an avid gamer, but in his tenure pushed for things like xbox game pass to drive continual revenue and windows integrations that affected performance of games. Now it's being led by an industry outsider.
It boils down to trust in the end, and willingness to place profit over brand. If you look at the responses to this in r/xbox or other communities, it's overwhelmingly a stance of zero surprise. Xbox has always placed the business first, and this is the natural end of that mission - you get a bloated org with a platform that people don't end up trusting.
I do think resetting is the correct thing to do; there's no reason for Xbox to have 10k+ employees. Still it's another black mark against the brand. Also look at the framing of this message - it's about how their structure has affected the business. In this entire 47 sentence post, there is a single sentence that talks about the affect on the players:
> That complexity has slowed decisions, blurred accountability, and made it harder to deliver for players.
It says a lot when the players are the secondary consideration.
If you want the core loop of Factorio but with a fresh spin, I highly recommend Dyson Sphere Program. It's my personal fav of the factory genre for the pure scale of it. As a tip, there's full multiplayer support from the mod community that works great.
I used cursor to port it to rust as well, and it got the conversion time down to 20s. Still likely not worth it as even with the rust port it's a 400x increase in processing time (that scales exponentially) for a ~2% decrease in size.
IMO the most interesting thing about this is Kimi K2.6, an extremely capable model, can be relatively easily post-trained to allow pen tests.
This in its own right proves that the defenses of Fable and others are temporary blocks, and AI based hacking is going to be effectively available to all parties regardless of stop gaps, as long as open models exist.
I think what I find fascinating about this is it's a native app with no web version... and they still decided to write it in html/js. This is after Microsoft's commitment to rebuild things in WinUI.
Don't get me wrong, I totally understand the barrier of friction that native presents compared to html/js, but that barrier has lowered so much with the advent of agentic development. It just feels like things weren't thought out.
What is the current state of ZFS? I know it had some licensing issues traditionally, despite it being a delight to use every time I've tried it. Is it back?
RE longer prompts, yes. Generally speaking I expand most prompts to be around a full page of text as it is already, so adding more detail in just refines that expanded prompt. That's more for a single screen though. It sounds like what you're asking about is something like a design.md for an entire brand / docs for a design system.
For that, I have a different approach, which is to extract your design system from screenshots. After which you can just select the brand you want when generating. There's sample images in the sibling comment in this thread.
Also it might be worth noting my pedigree here - I ran the design systems features over at Figma for around 5 years, but quit to build out diffui. The project is heavily oriented towards being able to replicate brands consistently, since the target audience I'm going after is enterprise design teams who are having trouble with existing tools capturing their brand look/feel.
From there you just select the brand at generation time. I've found you don't need a design.md or a npm package - simple screenshots are plenty good enough. Here's a prompt for "a landing page for a new satellite connectivity" feature I generated in reddit/netflix/slack's brands: https://image.non.io/b5e23f19-5041-4f87-9b97-0af39986d1b0.we...
I've found that starting using diffusion to render your creation, then using a LLM to build from the image creates much less of a slop feel than just starting out with a LLM. You wouldn't tell a construction crew to just build you a house without an architectural plan, so why tell a LLM what visual result you want without a visual guide?
my thing is diffui.ai if you want to check it out. It's basically a harness for diffusion models to generate UI, as well as agent integration for handoff.
Working on diffui.ai - diffusion for UI design.
Formerly Figma, Atlassian, and Microsoft.
AMA about design tokens, webcomponents, and design systems!
+1 808 366 1708 [email protected]