Arena can definitely be benchmaxxed a bit, if you try. The distribution of prompts there is very different than usage by regular coders. E.g., lots of requests for one-shot games from scratch. So if you fine-tuned your model to be great at making fun one-shot games from underspecified prompts, your coding model might look better than it is (on general tasks, at least).
I work at OpenAI, and am happy to say we don't try to juice our scores here, as doing so would be counterproductive and make Arena a worse signal for everyone.
> Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.
We agree models should be compared on a fair basis. Unfortunately, adaptive was the only publicly available number. Anthropic doesn't generally let us run their models for evals, so we rely on whatever Anthropic or third parties have published. In this case, the Agents' Last Exam leaderboard has Fable Adaptive, but not Fable Max.
Would have loved to publish a full curve for Fable if anyone makes the data available.
Although we do bias toward publishing evals where we're ahead, we have historically been unafraid to publish evals where we're behind (e.g., GDPval). The point is give people useful information to decide what's best, not to trick people.
Edit: Now I see there's a second entry with xhigh effort. Not sure if that was added or recently or we skipped it.
As usual, even though GPT-5.6 is releasing today, the rollout in ChatGPT and Codex will be gradual over many hours so that we can make sure service remains stable for everyone (same as our previous launches). We usually start with Pro/Enterprise accounts and then work our way down to Plus. We know it's slightly annoying to have to wait a random amount of time, but we do it this way to keep service maximally stable.
The timescale is typically hours not minutes, so if you don't see it now, I'd try again later today.
We mention it will be a gradual rollout over the next 24 hours in the Availability section at the bottom of the blog but I admit it's pretty buried.
Pointing out problems (e.g., hidden tests that assume narrow implementation details) is much easier than fixing them (e.g., creating tests that work for any possible choice of implementation).
- everyone will get access Thursday (barring banned countries / individuals)
Historically, some companies and individuals have gotten alpha access before public launches, to give feedback and adapt their products to the new models. With GPT-5.6, some folks had early alpha access, but this was paused while the model was being evaluated and approved. Now, alpha access will be enabled for partners in the next two days before our wider launch.
> Input and output tokens from each step are carried over, while reasoning tokens are discarded.
Keeping reasoning tokens around is better for caching and for remembering past insights, so you might reasonably wonder why we designed it this way. The main benefit of dropping reasoning tokens is that you can fit a lot more work inside the model's context window before you're forced into a slow and lossy compaction step. This was a larger consideration with our earlier reasoning models that had shorter context windows (~200k), longer thinking times (up to ~100k per message), and poor compaction. However, now that we've shipped longer context windows, we've trained our models think much more efficiently, and we've made compaction way better than it used to be, the balance of factors is changing. Tune in Thursday!
Unfortunately we're not in a position where we can promise an exact date, but we expect it to take weeks (not days or months). It's the best coding model we've ever trained and we're bummed we can't release it to everyone yet. When we do launch, we'll share a lot more evals and testimonials and demos that help show what it's good/bad at. Personally hoping that both GPT-5.6 Sol and Fable 5 get broadly released soon so that everyone (myself included) can try them head to head.
Yeah, we'll share a lot more details and evals when we can release GPT-5.6 widely. We focused on cyber (and bio) here to help explain why it's being held back for now. We would have loved to launch it to everyone - it's the best coding model I've ever used - and we plan to do so as soon as we can ('coming weeks').
Makes sense, thanks. I suppose error bars are tricky if trying to handle problem-to-problem variance, rubric-to-rubric variance, and run-to-run variance all at once.
Very cool! So glad to see people building and sharing evals that are better than SWE bench.
I'm curious - any particular reason you didn't put error bars on the graphs? Seems like it could be helpful when there are only 50 unique problems in the diamond set.
For what it's worth, I work at OpenAI and I can guarantee you that we don't switch to heavily quantized models or otherwise nerf them when we're under high load. It's true that the product experience can change over time - we're frequently tweaking ChatGPT & Codex with the intention of making them better - but we don't pull any nefarious time-of-day shenanigans or similar. You should get what you pay for.
To clarify the title, Daybreak is not a new AI model or a new product. It's a rebranding of OpenAI for Cyber, which is an umbrella over multiple things that OpenAI is doing with companies.
Whether a problem is "good" or "bad" is not always objective or simple.
For example, you can have problems that are underspecified, with hardcoded tests for a particular solution (out of multiple possible solutions). If your solution works fine but used a different function name than the one hardcoded in the tests, you can unfairly score 0.
When an eval has underspecified problems like these, you can still score 100% if you remember the original solution from your training data or if you just have taste similar to the original human authors. And both of these qualities - good memory and good taste - are great, but they'll be rewarded unfairly relative to a model that still did exactly what it was asked but in a different way than the hardcoded tests expected.
@sandersted