I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.
It is interesting though!
Perhaps in some way this means we should decide which eval set aligns best with our taste?
Back to the blog post. This is an excellent write up of an excellent technical achievement.
I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.
I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.
No. It makes you faster. You can prove it by looking at the lines of code.
Source: Gary Tan, who can write more code than Jeff Dean and John Carmack.
If you want to go really fast, you put a Ralph Wiggum dungeon. It's where you orchestrate a team of Ralph Wiggum loops together using subagents to win the economic Darwin Award which companies are handing out for who can burn the most tokens.
To hide the severity of the price increase, the plan is to move everyone right one model.
Haiku = essentially phased out
Sonnet = the Haiku use cases
Opus = the new Sonnet class
Fable = the new Opus class
If I am right, the other "5.0" models will be conspicuously absent, possibly even for a couple of months. (If Opus 5 follows soon and is even modestly better than 4.8 then I was wrong.)
Perhaps "potentially billions" is justifiable? In that it's a "change the scenario entirely" level of change.
Certainly if you compare it to another likely scenario where Vercel buys them and fast forward 2 years, it's plausible that a huge number of projects went one way or the other because of what the AIs defaulted to.
The reason this is worth it to CloudFlare is it will cause AI to recommend them more.
The agents already reach for Vite. When they reach for Vite it's very logical they will default to CloudFlare after. (Much like they will guide users to setup Vercel for NextJS).
This could be a $20m acquisition which will generate $billions from the increase in the agent equivalent of SEO.
But it's also possible they haven't spent much of that money.
The investors don't need to be happy. They just need to be made whole (assuming they have a minority control).
It could literally be that only $2m ever got spent and that's been paid back.
It could also be that when literally nobody said they would pay for Vite+ the investors and team in general lost confidence and were actually very happy just to get their money back and pivot into this acquisition.
So much respect for Black Magic. They are absolutely World Class and their business model is extremely generous.
Having said that, for all the AI features, the big one would be setting key frames etc. with an agent, driving the general editing workflow with text,etc. I realize this is non trivial but it's certainly viable for a team of this calibre.
I think if BM added a paid for agent which helped execute their traditional video editing tools (even if it "only" supported a subset) then that's a subscription a lot of people would be willing to pay for, especially as their core tool is so generous.
Anthropic capitalized upon a brief window of being more code-focused, which turned into enterprize contracts.
Then on renewal rug-pulled those same enterprises - going from your seat includes all the usage a user would reasonably need, to being you pay for the seat + all tokens at API pricing. (Which they raised by how many times in a year? I don't know the actual number.)
Revenue spikes like crazy through basically hostage taking made possible by Sonnet 3.5 era sentiment + enterprise purchasing lag.
Parlay the revenue spike into the valuation.
Crazy. Those same enterprises will get sticker shock and leave. Absurd short-term thinking.
OpenAI is the better company (transparency, open sourcing things, how they handle things in general e.g. OpenClaw, how they compete, etc.) and they have the vastly better brand, the better consumer presence, and (for me and many others) they have the better coding app + models.
Anthropic doing deeply customer hostile stuff - again and again - to produce a short term revenue spike does NOT make for a long-term sustainable business.
For such a young business to have such a long history of bait-and-switch is absolutely crazy. (Raising prices repeatedly, lowering rate-limits repeatedly, changing the terms, banning calls which contain "OpenClaw", turning on their IDE partners, turning on their enterprise partners.)
AFAICT anyone who's ever shown faith in Anthropic has been immediately exploited by them to some degree. They will quickly get the reputation of being "the Oracle of AI companies".
They note that Mythos "found a way to inject code into a config file that would run with elevated privileges, and designed the exploit to delete itself after running".
This is more impressive than what the benchmark was supposed to be measuring. The Kobiachi Maru.
Sometimes outcomes and achievements and work product are useful beyond just... stack ranking yourself against your peers. Seems so odd to me that this is your mentality unless you're earlier in your career.
I was reminded of the factorio blog. That game's such a huge optimization challenge even by today's standards and I believe works with the design.
One interesting thing I remember is if you have a long conveyor belt of 10,000 copper coils, you can basically simplify it to just be only the entry and exit tile are actually active. All the others don't actually have to move because nothing changes... As long as the belts are fully or uniformly saturated. So you avoid mechanics which would stop that.
I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.
It is interesting though!
Perhaps in some way this means we should decide which eval set aligns best with our taste?
Back to the blog post. This is an excellent write up of an excellent technical achievement.
I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.
I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.