All of the benchmarks are pretty terrible when you look under the hood.
For context, I've been iterating on a "supervisor" to replace a lot of the rigamarole spent when working with Codex/Claude Code, and recently ran this agent against Terminal Bench 2.1
At first I was excited, because my spec-driven supervisor outperformed vanilla codex on a bunch of tasks, however as I looked deeper, I found a ton of issues with the tasks themselves.
The main takeaway is that the instructions are often ambiguous while the test cases are overly specific.
A few examples:
- For `configure-git-webserver` the task includes language like "so that I can run" which blurs the line between what the agent should deliver vs. what should be removed. This causes an overthinking agent to configure the server, and then remove the exact files that the verifier checks, because if the user were to run the same commands, they would conflict.
- For `make-mips-interpreter` the task includes the language "I will check that you booted doom correctly" which causes the agent to retain the generated file `/tmp/frame.bmp` because the supervisor expects the user to check that _it_ booted Doom correctly, not that Doom boots correctly in an isolated way. The verifier then fails to start Doom, because it exits when an existing `/tmp/frame.bmp` exists, not checking to see that it's created from the boot[0].
- For `mcmc-sampling-stan` the supervisor agent often reached the right value, but produced a domain-specific numeric output in scientific notation, rather than a simple decimal form. The verifier fails because it parses the result incorrectly[1].
These are just a few of the inconsistencies I've found, which leads me to believe that Terminal Bench 2.1 is already saturated, and the results from GPT-5.6 and Mythos are basically at the top of the expected threshold (88.8% and 88% respectively).
The biggest issue, as I can tell, is that most benchmarks are "one-shot" and rarely test the model+harness on long iteration tasks, which is the primary way most users use these tools in practice.
Don't appreciate the slander, but I'll respond anyhow.
Contrary to your predisposition, we're actually quite peeved that we might be seeing results from 5.6 instead of 5.5, as it's muddying our own internal data.
We've run the tasks on this benchmark hundreds of times for our own internal harness. It got magically better yesterday. Last week we were seeing worse performance (sub-80%).
I agree that benchmarks don't mean much for real world use, and I'm a bit disappointed at the lack of variety in the published benchmarks so far.
With that said, 88.8% is higher than Mythos, and the highest I've seen from vanilla Codex. If 5.6 is any better than 5.5, you'd think they would avoid publishing just one coding-related benchmark with a score that equals their previous model.
> I'm not sure why a higher scores on a few tests [..]
It's not just higher scores, the API is no longer flagging tests for cybersecurity warnings that it's been flagging for weeks.
If you used GPT-5.5 over the last 24 hours or so, you may have already had access to 5.6.
I've been running some tests on a harness we're building, and suddenly saw a jump in a few points yesterday. I reran the vanilla codex benchmark and saw an ~88% score on Terminal Bench 2.1 from GPT-5.5 on vanilla Codex.
The biggest indicator, beyond the score, was that 3 tests which frequently hit "safety" blockers with 5.5 started succeeding last night without warning.
As an American with mostly Western European ancestors (according to a popular DNA testing site), I've always considered Romans as some distant/tangentially related group.
It was surprising to find out that I have "ancient" DNA matches with a couple of Roman and Etruscan individuals.
I've been quite impressed with DeepSeek v4 Flash running via antirez's ds4[0].
It feels like a GPT-4 class model in terms of "stored knowledge" but is better at long-horizon tool calling than any of the GPT-4 class models.
Running on a 128GB MBP M4 Max, I'm getting ~24 t/s on generation and ~200 t/s on prefill. I was expecting it to feel slow, and it certainly does when e.g. generating code, but it's surprisingly useful as a "machine orchestrator" for simple tasks.
For non-agentic usecases, it's a decent enough model to converse with, and has the benefit of being entirely self-contained/private.
> the conversation that generates the code is becoming the true source of our software
This is close, but not quite spot on. I've found that I'll test more ideas _with code_ using agentic tools, then before, leading to an excess of conversation history that is no longer representative of the final outcome.
A simple example I encountered recently was dealing with performance issues on an iOS application (I haven't written mobile code since before Swift..). If you viewed the chat, you'd dive down a diverging path of rabbit holes, few of which were relevant to the final outcome[0].
To solve this in my own work, I've started relying on "context hierarchy" - which is essentially live documentation that lives next to the source files (using markdown).
This approach avoids comments being removed erroneously, and helps codify the intent behind the code and how it relates to the overall architecture. As an added bonus, it also forces the LLM to edit _two_ things instead of just one (which might actually be the biggest benefit).
My workflow is currently maintained via some repo level scripts and AGENTS.md prompts, but I've tried to pull it out into a skill for others to use[1].
Candidly, I'm not sure the skill is the best approach yet, as the agent can sometimes get too focused on the "skill" as a separate tool rather than a core part of the workflow. I'm currently exploring other options here (repo bootstrap, side-loaded subagents, hooks, etc.)
[0]For more context, I was using a 3rd party library and trying to make it performant during a streaming operation, by removing the SwiftUI view layer (LazyVStack) and implementing a custom rendering path with UIViewController. The final solution ended up as a custom implementation of the 3rd party library, and moving back to LazyVStack.
This is neat! I love that your Step 15 shows an accurate version of the 3d helix, rather than the highly-viral "vortex" animation from a few years back[0]
It'd be awesome to scale this up to the Milk Way, and beyond, watching everything move in relation to larger time scales.
It's interesting that we're seeing these gains when it seems Mythos/Fable is "just" a scaled up version of their existing architecture[0].
When GPT 4.5 launched, the gains compared to the model size didn't seem that great, leading some to believe that the only progress we'd see would come from RL.
This model certainly has quite a "substantial amount of post-training and fine-tuning", but it's also based on a new pretrain[1][3], which given the cost, indicate that it is in fact quite a bit larger than Opus 4.X.
[0] One of the early testers mentioned: "As far as I can tell from talking to people internally at Anthropic, there's nothing special about architecturally"[2]
[3] There were rumors going around when Mythos was first announced that it was the first 10T parameter model, but I can't find a verifiable source for that number.
It’s interesting to me how similar attempting to understand LLMs is to neuroscience.
“When we turn this bit off, this other thing happens… if we change these weights the Eiffel Tower is now in Rome”
We’re basically just probing around and trying to reverse engineer an emergent system.
To your point, this system may be quite different from model to model (human to human) although some similarities likely occur.
The comment I was responding to tried to belittle the OP’s understanding of transformers, by mentioning that running an LLM at scale is much harder than the simple white board diagram.
My point was simply that we don’t know why they work, and all the extra optimizations isn’t the “thing” that makes it emergent.
Simply scaling the “GPT” is good enough to see it, so the OP’s awe should stand.
(On a side note, what other architectures can we scale to find similar emergent behavior?)
So this isn't quantum computing (in the qubit sense), but instead a different computer architecture (demonstrated on an FPGA) that's based on Fowler–Nordheim (FN) quantum tunneling (a real physical effect, used in flash memory, but simulated here).
From the paper:
> The FN-dynamics may be realized either by a physical FN-tunneling device or via a digital emulation of the FN-tunneling dynamical systems. In this work, we employ the digital emulation to achieve the precision required for simulated annealing in the low-temperature regime.
With a "real" (read: analog) FN device, you potentially get large speed ups and even larger cost/energy savings, because the physics is essentially working for "free" -- that's the quantum part.
What's unclear is how scalable the autoencoder architecture would be with analog FN devices today.