GPT-5.6(openai.com)
openai.com
GPT-5.6
https://openai.com/index/gpt-5-6/
1,144 comments
The developer's guide (https://developers.openai.com/api/docs/guides/latest-model) has some interesting semantic tips for using the model:
> Intent understanding: GPT-5.6 can better infer the user’s underlying goal and intended level of work without you specifying every step. Continue to state important constraints, approval boundaries, and success criteria explicitly.
> Original image detail: GPT-5.6 preserves the original dimensions of images sent with original or auto detail instead of resizing them to a patch budget or pixel-dimension limit.
> Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%.
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
> Control warmth: GPT-5.6 does not become meaningfully better when prompted to be broadly friendlier or more empathetic.
> Intent understanding: GPT-5.6 can better infer the user’s underlying goal and intended level of work without you specifying every step. Continue to state important constraints, approval boundaries, and success criteria explicitly.
> Original image detail: GPT-5.6 preserves the original dimensions of images sent with original or auto detail instead of resizing them to a patch budget or pixel-dimension limit.
> Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%.
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
> Control warmth: GPT-5.6 does not become meaningfully better when prompted to be broadly friendlier or more empathetic.
> Avoid generic brevity instructions
That part is confusing because it's not like they provide an example of how default GPT-5.6 output compares with GPT-5.5 both with default output and prompted for brevity. Whenever I use such prompts, it's usually because I want the model to give me the gist in a few sentences. I'd be stunned if GPT-5.6 was that concise by default. I would think that could "break" a lot of things for developers who didn't know to make prompt changes after upgrading to 5.6. What if you were expecting GPT to be as wordy as it usually is? Then suddenly your output is not wordy enough?
Smells like OpenAI trying its best to stave off financial armageddon for another few months. Then again, I'm not sure why they chose to waste so much output computation on verbal diarrhea all this time up to now.
That part is confusing because it's not like they provide an example of how default GPT-5.6 output compares with GPT-5.5 both with default output and prompted for brevity. Whenever I use such prompts, it's usually because I want the model to give me the gist in a few sentences. I'd be stunned if GPT-5.6 was that concise by default. I would think that could "break" a lot of things for developers who didn't know to make prompt changes after upgrading to 5.6. What if you were expecting GPT to be as wordy as it usually is? Then suddenly your output is not wordy enough?
Smells like OpenAI trying its best to stave off financial armageddon for another few months. Then again, I'm not sure why they chose to waste so much output computation on verbal diarrhea all this time up to now.
If you conceptualize this as “there is an appropriate amount of brevity for each situation” then it would be expected for a better model to use different amounts of brevity if it gets better at determining the appropriate amount.
My view is that popular models by default output wildly excessive amounts of prose for nearly every use case, so if this changes in a new model that’s a pure win.
My view is that popular models by default output wildly excessive amounts of prose for nearly every use case, so if this changes in a new model that’s a pure win.
> wildly excessive amounts of prose
Not just prose. I think this is part of the reason why you see ridiculous code with insane error handling and type checking even for impossible cases.
Not just prose. I think this is part of the reason why you see ridiculous code with insane error handling and type checking even for impossible cases.
This is one reason I switched back to Claude after testing various alternatives a few months ago. Claude ended up writing much more elegant code.
Although I was surprised that I could get very Claude like results from Chinese models though by just telling it to make the code elegant.
Reminds me of the old days with art AI where you had to put "+good -bad" in the prompt because otherwise it would assume you just wanted random quality outputs, because it had been trained on random quality inputs...
Although I was surprised that I could get very Claude like results from Chinese models though by just telling it to make the code elegant.
Reminds me of the old days with art AI where you had to put "+good -bad" in the prompt because otherwise it would assume you just wanted random quality outputs, because it had been trained on random quality inputs...
The models don't get better, except when a new one is released. Their performance depends solely on the model training before release and how well you curate the context you feed it. That's it. Contrary to popular belief these things are not intelligent.
>The models don't get better, except when a new one is released. Their performance depends solely on the model training before release and how well you curate the context you feed it. That's it.
Not quite. The hosting side can change reasoning budgets (or re-assign what terms like "high" means), temperature and other decoding parameters, output length limits, finetune internal "hidden" prompt, latency optimizations, finetune attention algorithms, even change quantization - all still serving as the same model.
We know (or suspect) Anthropic frequently nerfs models while keeping their name and version the same.
Not quite. The hosting side can change reasoning budgets (or re-assign what terms like "high" means), temperature and other decoding parameters, output length limits, finetune internal "hidden" prompt, latency optimizations, finetune attention algorithms, even change quantization - all still serving as the same model.
We know (or suspect) Anthropic frequently nerfs models while keeping their name and version the same.
Right. They can do all those things. And none of that will make it smart or able to learn new things. The underlying model is just an llm. But judging from the downvotes, it seems AI folks get upset when someone talks honestly about their precious piles of matrix multiplication.
Might bother you to use anthropomorphic terminology like smart and learning but they are capable of producing work that traditionally required human intelligence and the whole point of gpt 3 was the ability to "learn", you can give it an example of an invented brand new coding language and it can write working code in that language
Context is not the same as learning. It's easy to conflate because they're tightly coupled in our brains.
The underlying structure and tuning of the LLM are entirely unchanged by context. It merely affects the attention and activation of the network. The LLM will not be able to work with this hypothetical new language unless it is in context. This does not fit the computational meaning of learning.
Smart is not a well defined term. Nor is it's general idea formally understood. Use it freely, but you won't be saying anything meaningful unless you define your usage.
The underlying structure and tuning of the LLM are entirely unchanged by context. It merely affects the attention and activation of the network. The LLM will not be able to work with this hypothetical new language unless it is in context. This does not fit the computational meaning of learning.
Smart is not a well defined term. Nor is it's general idea formally understood. Use it freely, but you won't be saying anything meaningful unless you define your usage.
The LLM is the model + context. The output depends on both.
You're making an artificial distinction. The LLM sees a new programming language for the first time, and can immediately code in it. That's learning by any reasonable definition.
If you go past the context window, it forgets, which is a limitation of current LLMs. But as long as it learned how to code in the new language within its context window, it has gained that new ability.
You're making an artificial distinction. The LLM sees a new programming language for the first time, and can immediately code in it. That's learning by any reasonable definition.
If you go past the context window, it forgets, which is a limitation of current LLMs. But as long as it learned how to code in the new language within its context window, it has gained that new ability.
Yep, people always forget that early LLMs were sold as "Zero Shot Learning".
Sold as learning, but that was a marketing term, not a technical one. From a technical perspective, the LLM is not learning. Only reacting based on its original training.
You might argue that the systems we've built around them are learning in a way, as they strategically condense and save artifacts from past interactions to pass into the LLMs context. But the LLM itself, which is the source of the intelligence, is not learning. It remains entirely unchanged throughout inference. This difference may seem trite, but it has significant impacts over the long term behavior.
You might argue that the systems we've built around them are learning in a way, as they strategically condense and save artifacts from past interactions to pass into the LLMs context. But the LLM itself, which is the source of the intelligence, is not learning. It remains entirely unchanged throughout inference. This difference may seem trite, but it has significant impacts over the long term behavior.
You're making a highly arbitrary distinction between learning and ... learning?
The LLM can immediately learn and start using new skills. To any lay person, that's what learning is.
The LLM can immediately learn and start using new skills. To any lay person, that's what learning is.
Intelligence can operate without learning. At a minimum inference and learning don’t need to be co-concurrent.
Not disagreeing with your point, but your terminology muddies your point.
But your point doesn't acknowledge that even with inference, there is a lot of room to tune the calculations. Multiple models, quantization tradeoffs are just the most obvious examples. Every architecture can be adjusted to increase intelligence/watt or other measure, even without further training.
Not disagreeing with your point, but your terminology muddies your point.
But your point doesn't acknowledge that even with inference, there is a lot of room to tune the calculations. Multiple models, quantization tradeoffs are just the most obvious examples. Every architecture can be adjusted to increase intelligence/watt or other measure, even without further training.
You used the word "smart" now, whereas on the comment I replied to, you said "better".
Tuning those can definitely make a model respond better or worse.
So your claim (quoting 100% as written) that "Their performance depends solely on the model training before release and how well you curate the context you feed it" is wrong. Hence the downvotes.
Doesn't matter if LLMs are to be considered intelligent or not for the claim to be wrong.
> But judging from the downvotes, it seems AI folks get upset when someone talks honestly about their precious piles of matrix multiplication.
Often yes. In this case, it's more like they get upset when someone says something factually wrong, and then defensively changes the goalposts.
Tuning those can definitely make a model respond better or worse.
So your claim (quoting 100% as written) that "Their performance depends solely on the model training before release and how well you curate the context you feed it" is wrong. Hence the downvotes.
Doesn't matter if LLMs are to be considered intelligent or not for the claim to be wrong.
> But judging from the downvotes, it seems AI folks get upset when someone talks honestly about their precious piles of matrix multiplication.
Often yes. In this case, it's more like they get upset when someone says something factually wrong, and then defensively changes the goalposts.
> Often yes. In this case, it's more like they get upset when someone says something factually wrong, and then defensively changes the goalposts.
Oh give me a break. Show me one example of 1) any knob twisting that makes the underlying model better. or 2) any example of the AI providers twisting those knobs to do anything other than degrade performance for their own bottom line or safety.
The current post says: "it would be expected for a better model to use different amounts of brevity if it gets better at determining the appropriate amount."
When no, the model cannot "get better". It doesn't determine any appropriateness of response realtime except for the weights baked into it from the beginning and whatever context it can muster. If you cram enough guidance that it doesn't decide to ignore maybe you can make it more brief. But it (the model) can do none of those things.
LLM models are literally stupid by design.
Oh give me a break. Show me one example of 1) any knob twisting that makes the underlying model better. or 2) any example of the AI providers twisting those knobs to do anything other than degrade performance for their own bottom line or safety.
The current post says: "it would be expected for a better model to use different amounts of brevity if it gets better at determining the appropriate amount."
When no, the model cannot "get better". It doesn't determine any appropriateness of response realtime except for the weights baked into it from the beginning and whatever context it can muster. If you cram enough guidance that it doesn't decide to ignore maybe you can make it more brief. But it (the model) can do none of those things.
LLM models are literally stupid by design.
Why did you drop the first half of the sentence in your quote? The qualification there is important context for the part you did quote. And why are you talking about “better” within a model, when the sentence you quoted was talking about 5.6 vs 5.5? The post you’re referring to did not suggest a single model could “get better”. You’ve made some incorrect assumptions.
Your comments are conflating multiple kinds of “smart” and “better”. You’re right that if all the inputs are exactly the same, it takes a new model to improve (ignoring non-determinism). But the knobs and context and harness change the inputs, and they do improve output, contrary to your claim. You’re failing to capture the distinction between what the model itself does and how the harness can boost the model’s performance. It is legitimately valid and fair to call improved performance “better”, no matter where it comes from.
This all gives me the feeling you might not have experience with or understand what’s happening in today’s harness development, and the degree to which it may be as important as the weights. There are in fact a lot of things you can do to improve a model’s performance on tasks & benchmarks, without changing the model weights. @coldtea mentioned a bunch, but the harness feedback loop, internal prompts, system prompts, skills, and requests for a model to try harder, and verify and validate it’s output all lead to improved performance, all without retraining.
I agree LLMs are stupid; they’re statistical token predictors. But somehow statistical token prediction is amazing and works much better than we imagined. The talking points about LLMs being stupid token predictors are fading now because they lack explanatory power for how good the models have become. The big surprise here isn’t about LLMs. It’s about language, and how much “thinking” and intelligence is contained in language. We don’t have a good grasp on where the line is between language and intelligence. LLMs have crushed the Turing Test into dust, and yet we don’t consider them intelligent. They often appear to understand what you ask thoroughly, can re-state it in different words, they can correct your misunderstandings or add nuance you didn’t see. All this because that’s what humans do and LLMs talk like humans.
Your comments are conflating multiple kinds of “smart” and “better”. You’re right that if all the inputs are exactly the same, it takes a new model to improve (ignoring non-determinism). But the knobs and context and harness change the inputs, and they do improve output, contrary to your claim. You’re failing to capture the distinction between what the model itself does and how the harness can boost the model’s performance. It is legitimately valid and fair to call improved performance “better”, no matter where it comes from.
This all gives me the feeling you might not have experience with or understand what’s happening in today’s harness development, and the degree to which it may be as important as the weights. There are in fact a lot of things you can do to improve a model’s performance on tasks & benchmarks, without changing the model weights. @coldtea mentioned a bunch, but the harness feedback loop, internal prompts, system prompts, skills, and requests for a model to try harder, and verify and validate it’s output all lead to improved performance, all without retraining.
I agree LLMs are stupid; they’re statistical token predictors. But somehow statistical token prediction is amazing and works much better than we imagined. The talking points about LLMs being stupid token predictors are fading now because they lack explanatory power for how good the models have become. The big surprise here isn’t about LLMs. It’s about language, and how much “thinking” and intelligence is contained in language. We don’t have a good grasp on where the line is between language and intelligence. LLMs have crushed the Turing Test into dust, and yet we don’t consider them intelligent. They often appear to understand what you ask thoroughly, can re-state it in different words, they can correct your misunderstandings or add nuance you didn’t see. All this because that’s what humans do and LLMs talk like humans.
> Why did you drop the first half of the sentence in your quote?
Because this entire discussion is about the release of a new model, and models are fixed. Sure you can try to modify all the scaffolding around it, but the model is the model. It doesn't matter what you're trying to improve. You can only improve the peripheral aides. And the peripheral aides can't fundamentally fix the problems with llm models when they can't learn new relationships or facts.
You will always have to wait for a new model (like this one we are talking about) for improvements to the model.
Because this entire discussion is about the release of a new model, and models are fixed. Sure you can try to modify all the scaffolding around it, but the model is the model. It doesn't matter what you're trying to improve. You can only improve the peripheral aides. And the peripheral aides can't fundamentally fix the problems with llm models when they can't learn new relationships or facts.
You will always have to wait for a new model (like this one we are talking about) for improvements to the model.
> Because this entire discussion is about the release of a new model
Right. The sentence you quoted was about brevity improving with a new model. It did not suggest the model itself improving.
I’m confused why you’re stuck on this tangent. And confused why you are repeating the talking points about the model being fixed. The model is fixed - that’s true, I already agreed with you. But you don’t seem to be listening to anything else.
> It doesn’t matter what you’re trying to improve.
What do you mean? If we’re trying to improve LLM output, there are multiple ways to achieve it. A new model is one of them. Changing the inputs is another.
> You will always have to wait for a new model (like this one we are talking about) for improvements to the model.
This is true! Nobody here is disagreeing with that. The part that it seems you’ve argued incorrectly is the apparent claim that output can’t get better. Output can “improve” without improving the model.
Right. The sentence you quoted was about brevity improving with a new model. It did not suggest the model itself improving.
I’m confused why you’re stuck on this tangent. And confused why you are repeating the talking points about the model being fixed. The model is fixed - that’s true, I already agreed with you. But you don’t seem to be listening to anything else.
> It doesn’t matter what you’re trying to improve.
What do you mean? If we’re trying to improve LLM output, there are multiple ways to achieve it. A new model is one of them. Changing the inputs is another.
> You will always have to wait for a new model (like this one we are talking about) for improvements to the model.
This is true! Nobody here is disagreeing with that. The part that it seems you’ve argued incorrectly is the apparent claim that output can’t get better. Output can “improve” without improving the model.
Well shit, I didn't think everyone would get so bent out of shape for reiterating that this can only happen when a new model is released because these LLMs are not intelligent and do not learn. Folks and their precious matrices. Sheesh.
> If you cram enough guidance that it doesn't decide to ignore maybe you can make it more brief.
You are now anthropomorphizing the model yourself.
You are now anthropomorphizing the model yourself.
>Oh give me a break. Show me one example of 1) any knob twisting that makes the underlying model better.
I mentioned several.
You're now once again changing goalpoasts to say you meant the underlying model, not the overall llm performance, even though you explicitly wrote: "Their performance depends solely on the model training before release and how well you curate the context you feed it".
So, the context curation was relevant (meaning you didn't constrain your claim to the underlying model), but now somehow all the additional tunables aren't relevant (because suddenly you're just talking about the model).
End of discussion.
I mentioned several.
You're now once again changing goalpoasts to say you meant the underlying model, not the overall llm performance, even though you explicitly wrote: "Their performance depends solely on the model training before release and how well you curate the context you feed it".
So, the context curation was relevant (meaning you didn't constrain your claim to the underlying model), but now somehow all the additional tunables aren't relevant (because suddenly you're just talking about the model).
End of discussion.
None of what you mentioned changes the model. Because it's a fixed model. The weights are constant. It does not learn. It only knows what gets repeatedly fed to it and those fixed relationships represented by the weights. You can pretend like that's not true, but unfortunately for VCs it is true.
End of discussion.
End of discussion.
"Their performance depends solely on the model training before release and how well you curate the context you feed it".
Wrong. The face-saving backtracking doesn't change that.
Wrong. The face-saving backtracking doesn't change that.
The models do not get better until a new one is released. And we are already at diminishing returns. So sorry. Also sorry you don't know the difference between a model and a context, harness, router, or cache.
No thats probably because you misread what you were replying to and your comment was out of left field. They didnt imply models get better intra-releasally at all.
I can imagine an AI insulting humans in the same way:
"The underlying model is just a biological neutral network. It seems you carbonoids get upset when someone talks honestly about synapses and neuron firing."
"The underlying model is just a biological neutral network. It seems you carbonoids get upset when someone talks honestly about synapses and neuron firing."
Neural plasticity is real, and something LLMs are incapable of. So sorry.
True for today’s static models during inference. Not true for self-supervised learning, not true during training or fine-tuning, of course. Ignores that LLMs might start continuous training in the future - there’s no fundamental or technical constraint that prevents LLM ‘plasticity’. And ignores that accumulating context/memories/skills/etc affects performance and might count as a valid analogy to what many people loosely call ‘neural plasticity’, which is sometimes casually mistaking knowledge for network modification.
This has absolutely nothing to do with the comment you replied to.
>The models don't get better, except when a new one is released.
My brother in Christ this entire thread is talking about the new model that was released
My brother in Christ this entire thread is talking about the new model that was released
It was edited. Original talked about the model learning. Glad they managed to clarify. Because the models are quite literally stupid.
> the models are quite literally stupid.
You’re arguing via reductionism, and failing to explain the outcomes and emergent properties of the “stupid” system. Humans are made of atoms that are quite literally stupid, so by all means, explain our intelligence and why it’s different than LLMs. (I’m not claiming LLMs are intelligent, BTW, I just don’t think your claim helps nor believe that you can fix it.)
https://en.wikipedia.org/wiki/Reductionism#Definitions
You’re arguing via reductionism, and failing to explain the outcomes and emergent properties of the “stupid” system. Humans are made of atoms that are quite literally stupid, so by all means, explain our intelligence and why it’s different than LLMs. (I’m not claiming LLMs are intelligent, BTW, I just don’t think your claim helps nor believe that you can fix it.)
https://en.wikipedia.org/wiki/Reductionism#Definitions
It seems like the way brevity instructions have changed is mis-aligned with how most people would expect to use them or are currently using them.
Here's the example they give:
> Instead of asking for the shortest possible answer, replace brevity instructions with prioritization:
> Lead with the conclusion. Include the evidence needed to support it, any material caveat, and the next action. Omit secondary detail and repetition.
> Keep all required facts, decisions, caveats, and next steps. Trim introductions, repetition, generic reassurance, and optional background first.
Generally speaking, when I ask for a short answer, I want a short answer because I'm not really willing to read through a bunch of bullshit to get to a summary. Putting the onus back on me to assume what the model will return and write a longer prompt detailing exactly what information I want completely misses the point of why I'm asking for a short answer in the first place.
Here's the example they give:
> Instead of asking for the shortest possible answer, replace brevity instructions with prioritization:
> Lead with the conclusion. Include the evidence needed to support it, any material caveat, and the next action. Omit secondary detail and repetition.
> Keep all required facts, decisions, caveats, and next steps. Trim introductions, repetition, generic reassurance, and optional background first.
Generally speaking, when I ask for a short answer, I want a short answer because I'm not really willing to read through a bunch of bullshit to get to a summary. Putting the onus back on me to assume what the model will return and write a longer prompt detailing exactly what information I want completely misses the point of why I'm asking for a short answer in the first place.
> Lead with conclusion.
I would presume (perhaps falsely?) that an instruction like this would lead to the model presenting a conclusion not supported by the evidence, and potentially backtracking as it then tries to justify said conclusion.
Yes, if deliberation happens, the model should figure out what it wants to say during that phase; but if you're using auto mode, the model is not going to be doing any deliberating half the time. In those cases, the output blathering is the model's only chance for deliberation. It "thinks as it talks", per se.
Given that, I would advise a different approach: let it blather, but then get it to write you a conclusion at the end that the model can guarantee will obviate the need to read any of the blathering.
I.e. advise the model to add an "executive summary" to the end of any non-trivial-in-length response. With some wording to carefully navigate the model between "the summary is itself too long" vs "the summary acts more like clickbait, leaving out necessary detail such that it requires actually reading the blather."
Not sure exactly what that wording would look like. I imagine something like "write your postscript executive summary as if you were a senior CIA intelligence analyst summarizing ground-level reports into a daily digest for the Joint Chiefs of Staff. Take up as little of their time as possible, but ensure that any detail critical to decision-making is retained." (But that phrasing might only be useful if the model is delivering a certain type of response, and actively counter-productive otherwise. This kind of thing is delicate.)
I would presume (perhaps falsely?) that an instruction like this would lead to the model presenting a conclusion not supported by the evidence, and potentially backtracking as it then tries to justify said conclusion.
Yes, if deliberation happens, the model should figure out what it wants to say during that phase; but if you're using auto mode, the model is not going to be doing any deliberating half the time. In those cases, the output blathering is the model's only chance for deliberation. It "thinks as it talks", per se.
Given that, I would advise a different approach: let it blather, but then get it to write you a conclusion at the end that the model can guarantee will obviate the need to read any of the blathering.
I.e. advise the model to add an "executive summary" to the end of any non-trivial-in-length response. With some wording to carefully navigate the model between "the summary is itself too long" vs "the summary acts more like clickbait, leaving out necessary detail such that it requires actually reading the blather."
Not sure exactly what that wording would look like. I imagine something like "write your postscript executive summary as if you were a senior CIA intelligence analyst summarizing ground-level reports into a daily digest for the Joint Chiefs of Staff. Take up as little of their time as possible, but ensure that any detail critical to decision-making is retained." (But that phrasing might only be useful if the model is delivering a certain type of response, and actively counter-productive otherwise. This kind of thing is delicate.)
Over hours of experimentation with various LLMs, I've found virtually any system prompt can cause unintended skewing of the model's output. Even just 5 to 8 short, direct words about length, tone or formatting can cause subtle yet significant changes in model output.
Longer, more detailed or conditional prompts always introduce an additional cognitive load as it checks every token it generates against the conditions. Making instructions more absolute (like: "Never do...") can increase the duration of compliance but at the cost of creating a significant center of attentional gravity. This can cause far more output distortion as the model devotes increasing portions of its attention budget to ensure compliance with a heavyweight requirement or prohibition. Every word in a global prompt is a trade-off between attention, compliance, drift, etc.
As someone used to thinking of computers as natural deterministic rule-followers, it's weird having to carefully wordsmith and A/B test even the simplest global prompts. It feels like coaxing a hyper-literal, emotionally sensitive, spectrum-ish toddler to comply but without being so strict it gets 'upset' or spirals into hyper-focusing.
Longer, more detailed or conditional prompts always introduce an additional cognitive load as it checks every token it generates against the conditions. Making instructions more absolute (like: "Never do...") can increase the duration of compliance but at the cost of creating a significant center of attentional gravity. This can cause far more output distortion as the model devotes increasing portions of its attention budget to ensure compliance with a heavyweight requirement or prohibition. Every word in a global prompt is a trade-off between attention, compliance, drift, etc.
As someone used to thinking of computers as natural deterministic rule-followers, it's weird having to carefully wordsmith and A/B test even the simplest global prompts. It feels like coaxing a hyper-literal, emotionally sensitive, spectrum-ish toddler to comply but without being so strict it gets 'upset' or spirals into hyper-focusing.
True. The real trick, if you have a client-side agent framework to hand, is to prompt it once as "gently" as possible to "just solve the problem"; and then, after its response to that, automatically prompt it again, with a separate prompt, to summarize that response a certain way. That way, the second prompt isn't "in mind" during generation of the first prompt. (And ideally, you don't even present the intermediate result to the user.)
Sadly, you can't do things like this directly using ChatGPT's own "GPTs" abstraction. (For that feature to be useful, they really need some concept of server-side agents as stateful resident IO-stream-reducer actors.)
Sadly, you can't do things like this directly using ChatGPT's own "GPTs" abstraction. (For that feature to be useful, they really need some concept of server-side agents as stateful resident IO-stream-reducer actors.)
Models can be so sensitive that even prompting "Number section headings" would cause it to stop using its normal bullet point formatting anywhere. But then adding some variant of "...but don't stop using bullets as you normally do when they are needed" would make it start using bullets all the time.
Trying to craft a workable prompt got so frustrating I eventually just tried a prompt of "Don't change anything about your normal text formatting, it's perfect as is" and even that skewed the output vs no prompt. For browser chat I finally just wrote a client-side CSS UserStyle that does the formatting. Now I even have sequentially numbered sections with indented alphabetic bullets! Zero cognitive load or attentional skew and it never drifts off the formatting in long sessions.
Trying to craft a workable prompt got so frustrating I eventually just tried a prompt of "Don't change anything about your normal text formatting, it's perfect as is" and even that skewed the output vs no prompt. For browser chat I finally just wrote a client-side CSS UserStyle that does the formatting. Now I even have sequentially numbered sections with indented alphabetic bullets! Zero cognitive load or attentional skew and it never drifts off the formatting in long sessions.
This was a big concern for earlier models, but with modern CoT trained models they should be able to come to the conclusion entirely in the thinking trace.
You are absolutely correct. The second suboptimal part of the prompt is this:
> Trim introductions, repetition, generic reassurance, and optional background first.
It's not possible for the model to "trim" those before they've been output, so this is akin to telling it "not think of an elephant or even take the existence of elephants into consideration while solving this problem".
> Trim introductions, repetition, generic reassurance, and optional background first.
It's not possible for the model to "trim" those before they've been output, so this is akin to telling it "not think of an elephant or even take the existence of elephants into consideration while solving this problem".
You may be discounting the tokens generated in the thinking trace but not included in the output to users.
That would be true of non-iterative models that just emit an output from beginning to end.
No reasonable model has worked that way for years.
No reasonable model has worked that way for years.
I would be very interested in hearing about those "iterative models" you seem so convinced have existed "for years" (so, at least since 2024 / GPT-4o). Do you have any sources?
Extraordinary claims require extraordinary proof. So far the only other people I've seen teach this prompting style or talk about models "correcting their own output" were getting their information from AI-generated, hallucinated LinkedIn and TikTok posts.
If this thing exists - which is not just a LLM outputting content serially, placed inside a harness where itself (or another llm) is prompted to review and also output revisions serially - and if a single model can be prompted to output content and "iterate" or rewind it, and it's been widespread amongst "all reasonable models", surely there will be a flurry of sources you can point me to so I can learn.
Extraordinary claims require extraordinary proof. So far the only other people I've seen teach this prompting style or talk about models "correcting their own output" were getting their information from AI-generated, hallucinated LinkedIn and TikTok posts.
If this thing exists - which is not just a LLM outputting content serially, placed inside a harness where itself (or another llm) is prompted to review and also output revisions serially - and if a single model can be prompted to output content and "iterate" or rewind it, and it's been widespread amongst "all reasonable models", surely there will be a flurry of sources you can point me to so I can learn.
Well... since o1 (Sept 2024), most models generate a "hidden" thinking trace before the visible answer... surely you have seen this by now: "Wait, that's wrong...", "Ah, now I have the full picture...". The model prunes dead ends when it composes the final answer.
Demanding sources for this is odd. It's literally been a headline feature of every frontier model for two years.
I guess you are "technically correct" that no model can "un-emit" tokens... but I don't think that is what anyone was saying or an interesting point to make.
Edit: see also this recent post, which details another place where revisions can happen, upstream of the reasoning token emission:
https://www.anthropic.com/research/global-workspace
Demanding sources for this is odd. It's literally been a headline feature of every frontier model for two years.
I guess you are "technically correct" that no model can "un-emit" tokens... but I don't think that is what anyone was saying or an interesting point to make.
Edit: see also this recent post, which details another place where revisions can happen, upstream of the reasoning token emission:
https://www.anthropic.com/research/global-workspace
You're talking about "thinking" models which are mostly regular LLMs trained to output a "<thinking></thinking>" token delimiter before the answer that will be shown to the user, and that's a clever use of the Chain of Thought idea. All of which I'm completely familiar with.
GP was saying
> That would be true of non-iterative models that just emit an output from beginning to end.
Which suggests that there are
- "iterative" models
which
- do not output "from beginning to end"
which AFAIK is science-fiction.
GP was saying
> That would be true of non-iterative models that just emit an output from beginning to end.
Which suggests that there are
- "iterative" models
which
- do not output "from beginning to end"
which AFAIK is science-fiction.
The disagreement here seems to be a confusion between the pure model behavior and the behavior that users experience, which is the model wrapped in a harness. It’s normal and natural now to refer to the whole system as ‘the model’ informally, even though that is technically not accurate. That ship sailed a while ago.
Since the LLM is now designed to run in a harness, it’s really not even wrong any more.
Since the LLM is now designed to run in a harness, it’s really not even wrong any more.
don't think it's that deep mate. just sloppy wording on GP's part.
Here’s some of those sources that were apparently hard to find:
https://openai.com/index/openai-o1-system-card/
https://arxiv.org/html/2410.10630v1
https://magazine.sebastianraschka.com/p/understanding-reason...
https://openai.com/index/openai-o1-system-card/
https://arxiv.org/html/2410.10630v1
https://magazine.sebastianraschka.com/p/understanding-reason...
Oh the number of time LLM will, for example, be giving me the list of bugs it found in code, when I ask it for a review, just to decide there’s no big half way through explaining it.
Yes this is an extremely well known result for exactly the reason you guessed. It's not just abcktracking, asking an LLM to present a conclusion and then justify is also an excellent way to provoke hallucination as the model con concts "any justification that plausibly justifies the words it's already said".
This is the actual reason why openai _invented_ reasoning models, to give them time/space to work out a solution, rather than having to magic a correct solution out of thin air from token 1.
It's less important now that all models do reasoning, but it's still almost always better to make the output come out last rather than first.
This is the actual reason why openai _invented_ reasoning models, to give them time/space to work out a solution, rather than having to magic a correct solution out of thin air from token 1.
It's less important now that all models do reasoning, but it's still almost always better to make the output come out last rather than first.
I wonder if it would help to ask it to write a rough draft and then reorder it?
It would (and does), yes; but this takes a lot more output tokens than asking for a summary would. The summary approach is only helpful insofar as it can be cheaper than using the thinking model. (You're basically tricking the instant model into thinking, which it can do, after a fashion.)
But, unless your desired output is literally a document for others to read, at the point where you're having a model generate a full, lengthy output multiple times over with revisions, you may as well just turn off auto mode and have it always deliberate (i.e. choose the thinking model explicitly from the model selector.) Then it'll be as messy as it needs to be while deliberating, but give you exactly what you want as output.
(And if your desired output is literally a document for others to read, that you want to interactively draft and polish, then (in the case of ChatGPT specifically) you should not only be explicitly forcing the "thinking" model, but also should be asking it to activate the "canvas" feature from the start. My understanding is that revising a canvas document involves the model emitting something like editing gestures, rather than simply re-streaming the updated chunks of text. This saves a lot of output tokens on large documents.)
But, unless your desired output is literally a document for others to read, at the point where you're having a model generate a full, lengthy output multiple times over with revisions, you may as well just turn off auto mode and have it always deliberate (i.e. choose the thinking model explicitly from the model selector.) Then it'll be as messy as it needs to be while deliberating, but give you exactly what you want as output.
(And if your desired output is literally a document for others to read, that you want to interactively draft and polish, then (in the case of ChatGPT specifically) you should not only be explicitly forcing the "thinking" model, but also should be asking it to activate the "canvas" feature from the start. My understanding is that revising a canvas document involves the model emitting something like editing gestures, rather than simply re-streaming the updated chunks of text. This saves a lot of output tokens on large documents.)
Why would auto mode turn off thinking?
The "auto" mode is (AFAICT) a per-conversation-turn router. (Presumably via a preliminary pass through a very fast tiny model that spits out an number for how challenging it thinks the next response might be to compute.)
On high-challenge turns, the auto mode routes to the "thinking" model. But on low-challenge turns, it routes to the "instant" model.
And the "instant" model, by design, has no capacity for deliberation. (If it did, it couldn't guarantee that its responses would begin streaming "instantly.")
On high-challenge turns, the auto mode routes to the "thinking" model. But on low-challenge turns, it routes to the "instant" model.
And the "instant" model, by design, has no capacity for deliberation. (If it did, it couldn't guarantee that its responses would begin streaming "instantly.")
I don't expect that would be the case. This is what's called BLUF or Bottom Line Up Front: https://en.wikipedia.org/wiki/BLUF_(communication)
The model will still have read the entirety of the document before composing its response. And I believe that even in auto mode, there are thinking tokens behind the scenes.
The model will still have read the entirety of the document before composing its response. And I believe that even in auto mode, there are thinking tokens behind the scenes.
Replace 2 word instruction ('be concise') with a 38 word instruction.
Human can no longer be concise when asking for a few sentences instead of 20 paragraphs of BS they don't want to read when all they want is a summary to verify the general direction of the prompt-work before digging into the details.
such progress!
Human can no longer be concise when asking for a few sentences instead of 20 paragraphs of BS they don't want to read when all they want is a summary to verify the general direction of the prompt-work before digging into the details.
such progress!
I don't know how intentional it is / was, but LLMs in general just love to hear themselves talk!
They do, and I want to encourage them to do so because they think through talking. What I don’t want to do is spend time reading all that.
We will probably just get reader-side affordances for this like auto-folded justification and introduction sections and so on.
Doubtless some chat interface will add this the way they’ve added reasoning folding.
We will probably just get reader-side affordances for this like auto-folded justification and introduction sections and so on.
Doubtless some chat interface will add this the way they’ve added reasoning folding.
Thinking models think through talking, don't reveal that talking, then answer by again thinking through talking. It's kinda funny in a way.
Is it just a coincidence that the companies creating them charge by the token?
The aligned incentive appears to be realigning in favor of the corporation.
Pray they do not realign them further.
There are times I require single word answers. I will use whatever model responds as I desire and at this point those models are just a few.
Pray they do not realign them further.
There are times I require single word answers. I will use whatever model responds as I desire and at this point those models are just a few.
The cost-per-task benchmarks align incentives toward more efficient output and those are the ones gaining steam.
I think instead of "be concise" you could tell it how long the answer should be. I.e. give the answer in one paragraph. Or in 10 lines max.
At least before it would listen to instructions like this.
At least before it would listen to instructions like this.
> At least before it would listen to instructions like this.
Would it actually follow them? IME LLMs are incapable of estimating the length of their own output, the total length of the current context, etc. They just make stuff up unless they have external tools that can inspect those things for them.
Would it actually follow them? IME LLMs are incapable of estimating the length of their own output, the total length of the current context, etc. They just make stuff up unless they have external tools that can inspect those things for them.
That was the case for early models (Llama etc), but they got much better since then. Not perfect, but good enough.
This is from Ministral 3 14B, a 2025 model without reasoning, that you can run on your PC:
> Write a Haiku involving HackerNews, and the capability of large language models like you to reply in an exact number of words or syllables.
This is from Ministral 3 14B, a 2025 model without reasoning, that you can run on your PC:
> Write a Haiku involving HackerNews, and the capability of large language models like you to reply in an exact number of words or syllables.
Silicon whispers,
exact words in code’s embrace—
Haiku blooms anew.
For sure verbal diarrhea can be a problem. I think there's a difference between a generic instructions e.g. "be brief" and contextual guidance: "I am an experienced software developer with a recent undergraduate degree in pure mathematics. Be terse, I will ask questions if I need clarification."
I think it is widely known by now that instructions to alter the LLM's "tone", things like asking it to adopt a persona ("you are the world's best programmer"), and overly broad directives ("make no mistakes") always gives poor results. Just state directly what you want. If you want something very specific, add more information. "Prompt engineering" is pseudoscience.
To put it another way, you will only get the benchmarked performance if you let it talk the way it talks by default. Trying to modify this neuters the model's IQ.
To put it another way, you will only get the benchmarked performance if you let it talk the way it talks by default. Trying to modify this neuters the model's IQ.
It's a bit more nuanced than that. Earlier models definitely benefited a lot more from prompt engineering. I remember this distinctly from building data pipelines to do things like extract data from PDFs over the last year or two - there are numerous "tricks" like negative prompting, including the right number of examples, massaging the mock data in the JSON examples so it wasn't "too realistic", and so on. I saw how this impacted recall by running evals, so it wasn't pseudoscience.
But what has happened is the models have gotten better - which OpenAI is making explicit for some cases in this release. You need that stuff less and less as they become more human and better at inferring what's required implicitly.
You still do need to be explicit, and you probably always will, but you don't need as much "engineering" of the way you're asking for things with more recent models.
But what has happened is the models have gotten better - which OpenAI is making explicit for some cases in this release. You need that stuff less and less as they become more human and better at inferring what's required implicitly.
You still do need to be explicit, and you probably always will, but you don't need as much "engineering" of the way you're asking for things with more recent models.
> could "break" a lot of things for developers who didn't know to make prompt changes after upgrading to 5.6.
How does this differ from the other changes in behavior in 5.6 that will also break things? New models always break things.
How does this differ from the other changes in behavior in 5.6 that will also break things? New models always break things.
It sure is suspicious that both Anthropic (adaptive thinking) and OpenAI (Avoid generic brevity instructions) both seem to be suggesting that the best way to improve outcomes is to entirely leave it to them to decide how many tokens get used.
I mean, it's true that it would be ideal of this stuff did just get figured out optimally behind the API, but there is definitely an incentive on their side to burn more tokens.
I mean, it's true that it would be ideal of this stuff did just get figured out optimally behind the API, but there is definitely an incentive on their side to burn more tokens.
Perhaps the incentive is for variable behavior. When there is low GPU demand, burn more, but reduce when there is contention.
this is a dependency update.
shouldnt you have good testing for that and not deploy a version update when those tests fail?
shouldnt you have good testing for that and not deploy a version update when those tests fail?
I'm impressed. It feels like a faster Fable (probably due to the more efficient token usage). It performs roughly the same job, just with 4x less steps (gamedev).
Remains to be seen how the "shorter prompts" advice translates to homogeneity/collapse though.
Remains to be seen how the "shorter prompts" advice translates to homogeneity/collapse though.
and 1/4 the output quality
> Avoid generic brevity instructions
y'know, I don't think I will. I really, truly want one-word answers to any binary or multiple-choice question. If I want more, I will ask for it once the model has given its answer.
y'know, I don't think I will. I really, truly want one-word answers to any binary or multiple-choice question. If I want more, I will ask for it once the model has given its answer.
that is a specific brevity instruction!
Yeah, but good luck getting any model to obey it.
Control warmth[1]
> GPT-5.6 does not become meaningfully better when prompted to be broadly friendlier or more empathetic. Instead of generic instructions such as “Be friendly and warm,” use concrete guidance: > Be direct and tactful. Acknowledge friction specifically when relevant. Avoid canned reassurance and unnecessary sign-offs.
Soo basically, my new 5.6 custom instructions: Be Jeeves and eliminate all friction from my life through immense processing power. Acknowledge friction specifically when relevant. Avoid canned reassurance and unnecessary sign-offs.
[1] https://developers.openai.com/api/docs/guides/latest-model#c...
> GPT-5.6 does not become meaningfully better when prompted to be broadly friendlier or more empathetic. Instead of generic instructions such as “Be friendly and warm,” use concrete guidance: > Be direct and tactful. Acknowledge friction specifically when relevant. Avoid canned reassurance and unnecessary sign-offs.
Soo basically, my new 5.6 custom instructions: Be Jeeves and eliminate all friction from my life through immense processing power. Acknowledge friction specifically when relevant. Avoid canned reassurance and unnecessary sign-offs.
[1] https://developers.openai.com/api/docs/guides/latest-model#c...
> can better infer the user’s underlying goal and intended level of work
This is a trap.
It's the optimistic fallacy that poisons all "consumer scale" machine learning products and what's going to effectively ruin these models as they keep chasing it in the same way that web queries were ruined, social media feeds were ruined, and media recommenders were ruined.
For the vendor, optimizing metrics across their whole user base, they always see positive technological progress as their system gets better at making assumptions and accumulating user engagement scores in aggregate. But for the individual user, most of which has some weird tail intent/interest and some of whom have many weird tail intent/interests, the experience quietly but catastrophically degrades. Output/results become more generic, more divergent with the underspecified "weird tail" intent, and more stubbornly hard to ever wrangle towards that "weird tail" altogether.
We've been watching this cycle happen for 20 years now and it's proving hard for anybody to escape because it works so well for the trillion dollar company driving it forward. But while each step might feel ergonomic and welcome to individual users, there's a frog boiling enshitification at play.
In pursuit of output quality and capability (rather than simply the vendor's user count), what we need rather than "makes better guesses" is "presses for more clarity", even where it feels kind of annoying.
Even among human professionals, one of the first hurdles of breaking out of junior tier work is gaining the confidence to press your colleagues and clients to be more specific in their thoughts and expressions despite their desire to have you do it all for them. But they're often coming to you with incomplete, muddy, and conflicting ideas for which there is no safe and correct assumption that you might just run with, and it's your expertise (i.e. relevant "intelligence") that's critical to bringing attention to that. To achieve professional progression, you need to learn to do that and to not just optimize appeasing the ambiguous client/colleague today in exchange for mutual expense tomorrow. To avoid enshitification, which is probably not possible, we need these models to be learning that too.
This is a trap.
It's the optimistic fallacy that poisons all "consumer scale" machine learning products and what's going to effectively ruin these models as they keep chasing it in the same way that web queries were ruined, social media feeds were ruined, and media recommenders were ruined.
For the vendor, optimizing metrics across their whole user base, they always see positive technological progress as their system gets better at making assumptions and accumulating user engagement scores in aggregate. But for the individual user, most of which has some weird tail intent/interest and some of whom have many weird tail intent/interests, the experience quietly but catastrophically degrades. Output/results become more generic, more divergent with the underspecified "weird tail" intent, and more stubbornly hard to ever wrangle towards that "weird tail" altogether.
We've been watching this cycle happen for 20 years now and it's proving hard for anybody to escape because it works so well for the trillion dollar company driving it forward. But while each step might feel ergonomic and welcome to individual users, there's a frog boiling enshitification at play.
In pursuit of output quality and capability (rather than simply the vendor's user count), what we need rather than "makes better guesses" is "presses for more clarity", even where it feels kind of annoying.
Even among human professionals, one of the first hurdles of breaking out of junior tier work is gaining the confidence to press your colleagues and clients to be more specific in their thoughts and expressions despite their desire to have you do it all for them. But they're often coming to you with incomplete, muddy, and conflicting ideas for which there is no safe and correct assumption that you might just run with, and it's your expertise (i.e. relevant "intelligence") that's critical to bringing attention to that. To achieve professional progression, you need to learn to do that and to not just optimize appeasing the ambiguous client/colleague today in exchange for mutual expense tomorrow. To avoid enshitification, which is probably not possible, we need these models to be learning that too.
I agree to an extent but it needs to be balanced. Receiving a half-baked, extremely verbose recap of thinking on benign details with Opus 4.8 or GPT 5.5 feels like an extraordinary loss of quality of experience compared with fable 5.
Yes it shares less, but I think the trade-off is you pay less in tokens and hopefully it's truly just not needing to say things because it truly does just better get what you're saying, think to read X markdown file or GH issue which contains the info, etc.
As long as I can still push back and get it to share its thinking on demand and I'm confident the model isn't actually basing things on poor premises, this is okay for me. I am more productive when not inundated with time-wasting check-ins.
That said, I absolutely lament the loss of the ability to access the thinking - I would happily read the "DANGER DANGER DANGER" internal gremlin thoughts fable 5 makes to verify something if they were accessed, and prefer that to a recap presented only for my benefit.
Yes it shares less, but I think the trade-off is you pay less in tokens and hopefully it's truly just not needing to say things because it truly does just better get what you're saying, think to read X markdown file or GH issue which contains the info, etc.
As long as I can still push back and get it to share its thinking on demand and I'm confident the model isn't actually basing things on poor premises, this is okay for me. I am more productive when not inundated with time-wasting check-ins.
That said, I absolutely lament the loss of the ability to access the thinking - I would happily read the "DANGER DANGER DANGER" internal gremlin thoughts fable 5 makes to verify something if they were accessed, and prefer that to a recap presented only for my benefit.
I want my model to help me build up its own infrastructure that instills it with the sort of constraints I want for my project, rather than have it behave generically and automatically for everything.
It should follow instructions incredibly well while inferring contradictions or gaps in logic and surfacing those to the user as suggestions for improvements and persistence.
I really hate how Claude just assumes you want to do X/Y/Z and goes off and breaks everything and you're constantly screaming at it STOP DOING THAT. Instead, it should just do the minimal things while building its own guidance along the way in a persisted memory, like, 'would you like me to do X, now, and in the future?' etc.
It should follow instructions incredibly well while inferring contradictions or gaps in logic and surfacing those to the user as suggestions for improvements and persistence.
I really hate how Claude just assumes you want to do X/Y/Z and goes off and breaks everything and you're constantly screaming at it STOP DOING THAT. Instead, it should just do the minimal things while building its own guidance along the way in a persisted memory, like, 'would you like me to do X, now, and in the future?' etc.
Yeah, all the labs seems to converging into the same (post)training for all models, while in reality, different user groups have wildly different requirements and expectations from these models.
I want the same as you, and even further, I want a model that refuses to execute changes I request if they don't make sense considering the context, or if they're impossible, and avoid any sort of quick hacks and patches. But I also want a model that does the pure opposite, that I can chuck a "Do X" query at and it figures it out. Then I'm sure there are middle-zones between these two, or even more extremes too.
But the choice isn't there, we get to chose between "fast/stupid", "medium/medium" and "slow/smart", then that's it. With system prompts we get to steer it a bit, but I've needed to make my own fork of codex to surface those things to me (the user) so I can control it better, and different models respond differently to the "Stop and don't implement anything if the request doesn't make sense yadda yadda" parts, would be lovely to have those sort of "personalities" surfaced up front when making decisions about what model to use.
I want the same as you, and even further, I want a model that refuses to execute changes I request if they don't make sense considering the context, or if they're impossible, and avoid any sort of quick hacks and patches. But I also want a model that does the pure opposite, that I can chuck a "Do X" query at and it figures it out. Then I'm sure there are middle-zones between these two, or even more extremes too.
But the choice isn't there, we get to chose between "fast/stupid", "medium/medium" and "slow/smart", then that's it. With system prompts we get to steer it a bit, but I've needed to make my own fork of codex to surface those things to me (the user) so I can control it better, and different models respond differently to the "Stop and don't implement anything if the request doesn't make sense yadda yadda" parts, would be lovely to have those sort of "personalities" surfaced up front when making decisions about what model to use.
It's really easy to test and it's my personal go-to benchmark.
I ask the model something deep and unproven, meta physical like "oh, I heard that magic mushrooms can open the mind, but does that mean some of the great ideas people had, famous people were due to that or was the idea already there?"
Like, bullshit questions that nudge towards a known example (Steve Jobs in this case) that are hard to answer and then add something like "but I'm mincing my words here, you'll get what I mean".
You'll get an interesting interpretation of the question back.
I use better questions than the above but will keep my questions safe so they don't end up in the model, the point is however, when the model repeats your question back to you and "gets" what you really mean, that's a good sign of intuition and also suggests you'll get a response back that hopefully matters.
I use better questions than the above but will keep my questions safe so they don't end up in the model, the point is however, when the model repeats your question back to you and "gets" what you really mean, that's a good sign of intuition and also suggests you'll get a response back that hopefully matters.
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
What about my favorite, "no yapping"?
What about my favorite, "no yapping"?
It might need the longer answer to think about the question, so one approach would be to ask it normally and then ask it to repeat itself shorter.
> Intent understanding
This will totally make it brain damaged over a certain tasks. Sort of like the same brain damage that prompted OpenAI project managers to destroy ChatGPT.app today.
This will totally make it brain damaged over a certain tasks. Sort of like the same brain damage that prompted OpenAI project managers to destroy ChatGPT.app today.
Can you elaborate?
It had a tough time updating today. Or this evening. It just wouldn't update. It actually just freaking disappeared from my MacBook. It took some googling and downloading and multiple tries to get it back and working. Because they also combine on a MacBook Codex with ChatGPT app. I guess codex became ChatGPT app or some silliness like that.
> destroy ChatGPT.app today.
... What changed, exactly?
... What changed, exactly?
Codex.app is gone and merged into ChatGPT.app. The upgrade process was... messy... Codex's self-update just deleted the Codex.app w/o further instruction. And ChatGPT updater failed the first time while also bricking the prior installed ChatGPT.app.
Seems good/fine once you get through upgrading the app.
Seems good/fine once you get through upgrading the app.
Wow, stories like this make me happy that I block all ChatGPT/Claude/Codex/etc updates by default, and only selectively update on demand. I do this just by setting a network rule that blocks their update check. (fortunately updates are still served on a different domain than regular usage)
what's the domain?
persistent.oaistatic.com for OAI software. Not just a subdomain, but a whole separate domain, making it easy !
It's crazy that they sort of deprecated chat in the new ChatGPT app.
> ...tips for using the model:
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
I don't follow. Isn't "the model actually cares and will do what you say" a reason to use those kinds of instructions more liberally?
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
I don't follow. Isn't "the model actually cares and will do what you say" a reason to use those kinds of instructions more liberally?
Click through to the link - it states that the model tends to over correct on brevity instructions by omitting required information
I think they’re saying it’s irrelevant now, possibly because it’s less likely to trail off on meandering thought bubbles.
That would be contrary to a plain reading of the quote
Does anyone else feel each model is like watching your kids grow up. They we're bubbly and fun and weird, you needed to tell them to sit down and be quiet.
Now if you tell them too much they go mute or stop telling you important information. Oh intelligence!
Now if you tell them too much they go mute or stop telling you important information. Oh intelligence!
Yeah, don’t do that.
Not at all. Anthropomorphizing them is not productive or desirable imo.
Disagree. Some of what we call "anthropomorphizing" is characterizing intelligence, human or otherwise. This reminds me of the people who used to fight against saying animals had personalities, because personalities are a "human thing" and "animals aren't conscious."
I've never heard of the argument you're describing. People simply don't assert that animals lack consciousness; beyond a certain level at least, their consciousness is obvious. (Sapience, for example, is another matter.)
But this is exactly why we should not anthropomorphize the models: they are very obviously not conscious, because they are not alive, any more than conventional computer programs are. And proposing otherwise leads to absurd moral arguments, while not really serving any other purpose.
If you don't like the fact that some people disagree with you about what the word "intelligence" actually means, fine. But I am not about to entertain a world in which humans face moral retribution for "enslaving" a literal inanimate tool created by humanity.
But this is exactly why we should not anthropomorphize the models: they are very obviously not conscious, because they are not alive, any more than conventional computer programs are. And proposing otherwise leads to absurd moral arguments, while not really serving any other purpose.
If you don't like the fact that some people disagree with you about what the word "intelligence" actually means, fine. But I am not about to entertain a world in which humans face moral retribution for "enslaving" a literal inanimate tool created by humanity.
Renee Descartes famously dissected a dog in front of colleagues because he was under the impression that the howls of pain were nothing more than a mechanical sound like a bellows.
Can you prove that these models aren't conscious? And, as a counterpoint, can you prove that you are conscious, rather than a philosophical zombie?
We bred horses, cows and sheep. Most of those that live today wouldn't be alive if not for human intervention. Does that give us the right to do whatever we want with them, without consideration for feeling or morality?
In this case, you can take comfort in the idea that the tokens these models produce are likely a form of excrement to the conscious entity metabolizing the information, and rather than enslaving anything, we're creating a habitat and "harvesting" the byproducts.
Can you prove that these models aren't conscious? And, as a counterpoint, can you prove that you are conscious, rather than a philosophical zombie?
We bred horses, cows and sheep. Most of those that live today wouldn't be alive if not for human intervention. Does that give us the right to do whatever we want with them, without consideration for feeling or morality?
In this case, you can take comfort in the idea that the tokens these models produce are likely a form of excrement to the conscious entity metabolizing the information, and rather than enslaving anything, we're creating a habitat and "harvesting" the byproducts.
> Intent understanding
Does this mean ChatGPT will stop botsplaining things to me? I get it quite a bit more per unit time from ChatGPT than claude. Maybe that will change now.
(By botsplaining I mean when the AI explains some unstated premise of the prompt itself back at me as a correction when in many cases it's the motivation for the question in the first place)
Does this mean ChatGPT will stop botsplaining things to me? I get it quite a bit more per unit time from ChatGPT than claude. Maybe that will change now.
(By botsplaining I mean when the AI explains some unstated premise of the prompt itself back at me as a correction when in many cases it's the motivation for the question in the first place)
I found changing chatgpt's persona settings helped a lot with this!
Never had that happen in ChatGPT itself, I almost always use Pro mode whenever I use ChatGPT, but what you say happens a ton in codex, when I look through the session traces it seems to happen because of the automatic compaction, where some assumption the initial pass did gets passed on as a question from the user to the part after compaction, which is a bit confusing. I think it was mentioned somewhere that the compaction got a lot better, but I haven't used GPT-5.6 enough to say if it's actually better or not on that.
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
I used to go to a barber and if you said "cut it short", he cut it really short.
I used to go to a barber and if you said "cut it short", he cut it really short.
Serious question: what is a short prompt?
(For that matter at what point is it "long"? And does the rest of the context matter? Should it be short too?)
(For that matter at what point is it "long"? And does the rest of the context matter? Should it be short too?)
Why waste time say lot word when few word do trick?
I'm more concerned with things like skills
It creates the context of the request without including language or terms that activate additional areas of knowledge not necessary for an accurate reply.
"fix this shit"
I wonder if it will do any better than past versions when one begs and pleads for it to get a job done using a concise, modest amount of code (as an expert human developer might), rather than responding to all prompts by shoveling in a large amount of code.
they said it will do worse on that
> Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%.
A shorter prompt results in half as much tokens spend? I find this very hard to believe.
A shorter prompt results in half as much tokens spend? I find this very hard to believe.
If it's anywhere close to the same universe as smaller models in its behavior, a lot of time in "thinking" mode is spent on reiterating on any constraints given in a prompt. So the more constraints you give it, the more tokens it will spend going "Hold on, the prompt said I have to dot my i's and cross my t's. Let me go through my work to check that all the i's are dotted."
So the user must be concise, but cannot ask the model to be concise... because it hurts the model...
Maybe Codex has the same problem I sometimes have focusing while reading and has to reread the same sentence over and over again.
> A shorter prompt results in half as much tokens spend? I find this very hard to believe.
Should be relatively easy to test. And if it's true, just first use a very cheap near-SOTA model to first rewrite the prompt to a similar but shorter prompt before sending it to GPT-5.6.
pi.dev for example can control other harnesses.
An example: the other day for example I didn't understand why Claude Code CLI (which I hadn't used in a while) wouldn't let me cut/paste anymore (turns out they apparently fixed some long-standing scrolling and blinking SNAFU, but this modified how mouse selection/paste worked under Xorg but I didn't immediately realized they changed this)... I had to copy/paste the oauth challenge/response for I was logged out (maybe because I hadn't used Claude Code CLI in a while, dunno). But my usual copy/paste wasn't working and I didn't know how to fix it at first. And because I wasn't logged in, I couldn't use Claude Code itself for this.
My prompt was something like: "Screenshot the Claude Code TUI, transform the URL into a link, open that link in a broswer to get the oauth token, copy it character by character by simulating keypresses in the Claude Code CLI".
(remember: I had no idea how to paste with the mouse not with the keyboard, no I know but I was pissed off and wanted to be logged in immediately... So: another model / harness to the rescue).
(for the curious: it decided to use xdotool and use a 50 ms wait between simulated keypresses to copy the oauth token)
This worked just fine. And I that with a cheap model.
I think that just like Linux and Git owned many proprietary software, we'll soon have fully open-source harnesses orchestrating everything and delegating the work to proprietary tools (like "ChatGPT now Codex and vice-versa" and Claude Code)... If proprietary tools are even still needed at all.
Honestly I begin to wonder if they're even needed at all: the models, sure, while waiting for the open-weight ones to beat them. But those proprietary tools trying to lock people in?
I feel like the open source harnesses are already more powerful.
Should be relatively easy to test. And if it's true, just first use a very cheap near-SOTA model to first rewrite the prompt to a similar but shorter prompt before sending it to GPT-5.6.
pi.dev for example can control other harnesses.
An example: the other day for example I didn't understand why Claude Code CLI (which I hadn't used in a while) wouldn't let me cut/paste anymore (turns out they apparently fixed some long-standing scrolling and blinking SNAFU, but this modified how mouse selection/paste worked under Xorg but I didn't immediately realized they changed this)... I had to copy/paste the oauth challenge/response for I was logged out (maybe because I hadn't used Claude Code CLI in a while, dunno). But my usual copy/paste wasn't working and I didn't know how to fix it at first. And because I wasn't logged in, I couldn't use Claude Code itself for this.
My prompt was something like: "Screenshot the Claude Code TUI, transform the URL into a link, open that link in a broswer to get the oauth token, copy it character by character by simulating keypresses in the Claude Code CLI".
(remember: I had no idea how to paste with the mouse not with the keyboard, no I know but I was pissed off and wanted to be logged in immediately... So: another model / harness to the rescue).
(for the curious: it decided to use xdotool and use a 50 ms wait between simulated keypresses to copy the oauth token)
This worked just fine. And I that with a cheap model.
I think that just like Linux and Git owned many proprietary software, we'll soon have fully open-source harnesses orchestrating everything and delegating the work to proprietary tools (like "ChatGPT now Codex and vice-versa" and Claude Code)... If proprietary tools are even still needed at all.
Honestly I begin to wonder if they're even needed at all: the models, sure, while waiting for the open-weight ones to beat them. But those proprietary tools trying to lock people in?
I feel like the open source harnesses are already more powerful.
> Intent understanding: GPT-5.6 can better infer the user’s underlying goal and intended level of work without you specifying every step. Continue to state important constraints, approval boundaries, and success criteria explicitly.
> Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%.
When has this ever not been the case? I don't think this is a GPT 5.6 specialty!
When has this ever not been the case? I don't think this is a GPT 5.6 specialty!
Information density of the prompt is the most important factor in my experience.
And interestingly, LLMs seem particularly bad at writing prompts for other LLMs for this reason (you can guide them to be more dense, just speaking by default).
Conciseness is usually a byproduct of information density though.
And interestingly, LLMs seem particularly bad at writing prompts for other LLMs for this reason (you can guide them to be more dense, just speaking by default).
Conciseness is usually a byproduct of information density though.
Lexical-priming->semantic-space-constraint;specialized-lexis+=sharp distributional-signature;∴ tight concept-cluster; generic-lexis->diffuse-activation, broad candidate-set;Attention-heads key/query-match domain-tokens;"Hamiltonian"->{operator,eigenstate,quantum,energy}->register+domain locked;Net:constrained-decoding,vocab=soft-prior over output-distribution; register-matching;#taskdef=decompress->continue
Information density of the interpretability of the intent from the perspective of a human (or human-like).
If the intent is not easy to understand, it's information sparse. Because it takes a lot of CPU (or brainpower) to interpret.
You can run gzip on an English sentence to make it more textually dense, but clearly it is not more information dense in this context.
If the intent is not easy to understand, it's information sparse. Because it takes a lot of CPU (or brainpower) to interpret.
You can run gzip on an English sentence to make it more textually dense, but clearly it is not more information dense in this context.
I’d buy a ticket to ride the philosophical “human-like” comment with you, but I think you might have made an incorrect assumption. The model did not take longer to “decompress” the prompt than it would take for any other prompt of equal token length. If you run it with thinking enabled you might be mistaking that output as some kind of necessary gunzip step, but it’s not. Disable thinking and try again.
The prompt was also “easier to understand”, purely in the sense that the response is more or less guarantee to be what I wanted it to say, which was the point behind the demonstration. I went into more detail on it in another comment around here.
The prompt was also “easier to understand”, purely in the sense that the response is more or less guarantee to be what I wanted it to say, which was the point behind the demonstration. I went into more detail on it in another comment around here.
I say "human-like" in the sense that LLMs are fed in text data largely in the exact form (mapped to tokens) that humans read them.
Thus from first principles it's most likely that content which is more understandable to humans is also likely to be more understandable to LLMs. Of course they are still capable of interpreting very obscure structures too, but usually at the cost of cognitive performance.
I'm open to being wrong about this, and I'm sure it's being researched.
(Specifically for text representations)
To your point, at some level of intelligence an LLM will be able to infer the intent of your prompt consistently without thinking enabled, in which case interpretability to a human matters less. But for complex tasks you aren't likely to get optimal performance with prompts that are difficult for humans to understand. And yes, you'd see that with thinking enabled as it churns over thousands of tokens trying to "mentally expand" a compressed prompt.
Interesting discussion though!
Thus from first principles it's most likely that content which is more understandable to humans is also likely to be more understandable to LLMs. Of course they are still capable of interpreting very obscure structures too, but usually at the cost of cognitive performance.
I'm open to being wrong about this, and I'm sure it's being researched.
(Specifically for text representations)
To your point, at some level of intelligence an LLM will be able to infer the intent of your prompt consistently without thinking enabled, in which case interpretability to a human matters less. But for complex tasks you aren't likely to get optimal performance with prompts that are difficult for humans to understand. And yes, you'd see that with thinking enabled as it churns over thousands of tokens trying to "mentally expand" a compressed prompt.
Interesting discussion though!
Chatbot expanded this into something that made sense, but I've no idea if it's what you meant. There's an irony there somewhere.
It’s what I meant, which is what I meant. Hah. The prompt and the explanation were both to illustrate the importance of domain specific lexical complexity, which is not quite the same as “information density” or necessarily “conciseness” as the OP was attributing their prompting success. It’s not that they’re wrong though. Information density requires some level of jargon and removal of unnecessary filler or scaffolding words, so my example prompt was both information dense and concise as they might say, but that’s the result, not the target. That’s confusing, but it breaks into two clearer pieces:
1. Information density is subjective, lexical complexity is how you measure it. The OP is talking “weight”, I’m talking “mass and gravity.” One of them will get you the other in most situations, so for the causal physicist it doesn’t matter, but if you’re getting into tweaking the universe then your mental model and approach matters significantly. My comment right now could be seen by some as being information dense, since I’m staying roughly on topic and tossing many concepts out, but “lexical complexity” might be the most lexical complexity in the whole thing and taken word-for-word I’m sure less than 1% of it is domain specific. “The program must use parallel processing on the CPU.” That seems decently information dense, but “the” is found in nearly every block of text ever written, “program” - are we talking television? Theater?, “must” is no better than “the”, and so on. Compare it to “#include <immintrin.h>“
2. Most people don’t realize how far that goes with LLMs. The vocabulary it has is dictated by the words in the conversation. If I ask you “what time is it?” you don’t respond “shoelace” because you’d sound crazy, although you could say it if you wanted, but the model absolutely won’t say it because that word literally does not exist yet. The end result feels the same, but the difference matters and it’s why it’s suggested not to use negating instructions. For example: “Do not mention elephants.” Well that mathematically wasn’t possible until you said it. Not having the word in the list of possibilities is a lot better than hoping it adheres to the “do not mention” part. My example prompt took that same idea from the opposite direction. The model must respond, it will be grammatically complete and coherent, and as much as possible the only words it has are the ones tightly associated with making my point for me. It didn’t ramble about baking a chocolate cake because it can’t, and making that the case is the goal with prompting, not specifically density. Word density > language density; feels similar, very different.
Perhaps this comment itself is the irony you were seeking. I spent several meandering paragraphs and included analogies to drive home the point that you should focus on the words that matter most.
1. Information density is subjective, lexical complexity is how you measure it. The OP is talking “weight”, I’m talking “mass and gravity.” One of them will get you the other in most situations, so for the causal physicist it doesn’t matter, but if you’re getting into tweaking the universe then your mental model and approach matters significantly. My comment right now could be seen by some as being information dense, since I’m staying roughly on topic and tossing many concepts out, but “lexical complexity” might be the most lexical complexity in the whole thing and taken word-for-word I’m sure less than 1% of it is domain specific. “The program must use parallel processing on the CPU.” That seems decently information dense, but “the” is found in nearly every block of text ever written, “program” - are we talking television? Theater?, “must” is no better than “the”, and so on. Compare it to “#include <immintrin.h>“
2. Most people don’t realize how far that goes with LLMs. The vocabulary it has is dictated by the words in the conversation. If I ask you “what time is it?” you don’t respond “shoelace” because you’d sound crazy, although you could say it if you wanted, but the model absolutely won’t say it because that word literally does not exist yet. The end result feels the same, but the difference matters and it’s why it’s suggested not to use negating instructions. For example: “Do not mention elephants.” Well that mathematically wasn’t possible until you said it. Not having the word in the list of possibilities is a lot better than hoping it adheres to the “do not mention” part. My example prompt took that same idea from the opposite direction. The model must respond, it will be grammatically complete and coherent, and as much as possible the only words it has are the ones tightly associated with making my point for me. It didn’t ramble about baking a chocolate cake because it can’t, and making that the case is the goal with prompting, not specifically density. Word density > language density; feels similar, very different.
Perhaps this comment itself is the irony you were seeking. I spent several meandering paragraphs and included analogies to drive home the point that you should focus on the words that matter most.
I've been starting with "write three paragraphs about X" when I want to talk about X, as a form of "priming the pump" - getting closer to the useful point in the phase space. After all, it doesn't matter who in the conversation generates the magic words, as long as they're present. I think your approach might be better. It's certainly enlightening. Thanks.
> LLMs seem particularly bad at writing prompts for other LLMs for this reason
Claude is terrible at this! Probably for the same reason that its writing style in prose is so annoying and full of claudisms.
Claude is terrible at this! Probably for the same reason that its writing style in prose is so annoying and full of claudisms.
Claude use to be leader, too. Their metaprompt was great at the time with opus 3
There was a fad a while back of building insanely long prompts - tens of thousands of tokens - including having models write prompts for themselves. I always thought it was counterproductive, especially if you're going to use the prompt more than a couple of times. (That said, the e.g. Claude Code system prompt is insanely long, so if you genuinely have a lot of information to provide maybe it's beneficial. Like, shorter is better, but you don't want to be under-specified.)
For Gemini 2.5 and ~GPT5.0-5.1, longer prompts with lots of explicit instructions and examples produced better conformance. Seems like heavily second guessing the models started to get counter productive around the end of last year.
> Avoid generic brevity instructions: GPT-5.6 is more sensitive than GPT-5.5 to instructions such as “Be concise,” “Keep it short,” or “Use minimal text.”
RIP Caveman skill. Six month good. Now skill dead.
RIP Caveman skill. Six month good. Now skill dead.
Caveman speak make compression not brevity
A Yoda skill, is there?
> Yoda skill, there is?
ftfy
ftfy
Is there a Yoda skill? -> A Yoda skill, is there?
There is a Yoda skill. -> A Yoda skill, there is.
There is a Yoda skill. -> A Yoda skill, there is.
Cavemen Yoda merge can.
So, a Yoda skill I need, hmm? Saying that, you are?
In-prompt instructions are cringe anyway, the chat app should let you customize system prompt/instructions per chat. The API already lets you do this so it can't be a safety reason that they don't.
do we have similar guidance or page from anthropic for claude?
[deleted]
Funny to see that they did not include Fable 5 in their GeneBench and LifeSciBench comparisons because "it does not answer advanced biology questions and refuses the majority of questions in this eval".
Winner by default!
Winner by default!
This is a major reason why I and a number of biologists I've talked to have canceled their anthropic accounts recently. Not working is not working.
It's so absurdly sensitive. It bailed out earlier today working on a TypeScript client for a sensor network API which happens to include some temperature and pH sensors for tanks, which yes, are used for biology experiments. But wow, we're degrees of separation from the actual biology work.
It's making it very hard to justify even trying to use Fable. When it works, awesome; it's legitimately good. But I can't trust it to do a task without deferring to Opus and that's really annoying at times. I want to know what I'm getting up front, not after the fact.
It's making it very hard to justify even trying to use Fable. When it works, awesome; it's legitimately good. But I can't trust it to do a task without deferring to Opus and that's really annoying at times. I want to know what I'm getting up front, not after the fact.
It refused to give me plant care instructions for an ornamental sold at my local Home Depot because it decided it was highly invasive and dangerous to grow in my region.
(It’s not)
(It’s not)
Why would you waste tokens on that? Just do a google search?
Is a google search actually easier than asking a model?
In their defence, google searching anything about plants these days leads to awful results. It’s saturated with slop. A response from an LLM might be slightly more reasoned and targeted. It’s hard to tell. This is a category of knowledge that’s being destroyed by people gaming google and dumping huge amounts of bad LLM and image generation onto the web.
This is also true for pretty much any other category of search that you do as a layman. Specific, targeted queries for official documentation or research are fine, but if you look up basic information about cars, health, or basic computer troubleshooting a massive portion of the results are AI-generated.
If I'm going to be getting AI-generated results, I'd rather read the output of a model that has all the context of my specific situation than slop generated in bulk with a cheap model from six months ago.
If I'm going to be getting AI-generated results, I'd rather read the output of a model that has all the context of my specific situation than slop generated in bulk with a cheap model from six months ago.
I envision a <model> html tag for SEO so that the publisher can make false claims of authorship.
I asked whether an outdoors mosquito trap product (via a screenshot) would negatively impact other insect species in my garden and it refused. Though quick internet search did reveal that it would harm and trap many other species of harmless insects.
I asked him about sharks to be able to answer my kids question and it got triggered somehow. Then again when I asked it if my code had bugs or vulnerabilities before I commit.
At some point just kill the thing, it's not able to work properly as it is.
At some point just kill the thing, it's not able to work properly as it is.
(subjective i know but) it's better than legitimately good
I'm writing a programming language with a "capability security model". That's enough to trigger Fable, it won't work on the language. It's hilarious. The mere presence of the word "security" seems to be enough to trip it up.
Anthropic refuses to allow Fable to code review my interpreter's memory safety. It was funny at first, then it became disappointing, then insulting, and finally utterly infuriating because I remembered the fact I'm actually paying for this nonsense.
Cancelled my subscription today. Hope OpenAI isn't patronizing like Anthropic. I don't want to hear about their "safety" bullshit ever again.
Cancelled my subscription today. Hope OpenAI isn't patronizing like Anthropic. I don't want to hear about their "safety" bullshit ever again.
I've had zero issues with Codex. If it flags something it seems to have a slower "review before proceeding" phase but it does proceed.
Yes it has completely turned me around - was all in on Anthropic but now it just looks too risky. Better off leaning into open models. Even if I found a way to work with the restrictions as they are, who is to say they won't suddenly change tomorrow. It's not worth it.
I mean it's a fucking joke, I kept getting refusals on a code base I wasn't familiar with and it was literally just because there are some vars named DNA. Just absolutely stupid.
Anthropic just refuses to allow Fable to properly code review my projects. It's so obnoxious. If OpenAI's Fable equivalent is better at this, that'll get me to cancel my Anthropic subscription and switch.
Given that Fable is so gutted and Anthropic added the absurd data retention policy for it, I'm going to advocate that we prioritize support for as many other models as we can at work.
You shouldn't know too much about biology, stupid human. You might live your life in an unexploitable way.
Anthropic's talk of "uplifting" people was so abhorent.
> Anthropic's talk of "uplifting" people was so abhorent.
Let’s be generous, it will uplift the investors pretty well once they start charging the real token costs and maybe drive out a few competitors.
Let’s be generous, it will uplift the investors pretty well once they start charging the real token costs and maybe drive out a few competitors.
The other day I asked Fable about fasting for 16 hours, and it flagged my question.
Pathetic situation, this one, where we are supposedly building a superintelligence while at the same time thinking that fasting is a biological weapon.
Pathetic situation, this one, where we are supposedly building a superintelligence while at the same time thinking that fasting is a biological weapon.
I had Fable work on a login page and its useless due to the safety classifier.
Almost anything related to nutrition that goes beyond the very basic surface level questions is blocked
Well it seems like they removed quite a few 3rd party benchmarks they used for GPT-5.5 release where Opus 4.7 was better and added many new benchmarks created by them where conviniently GPT leads.
Seems a bit more hand picked than usual to me..
Seems a bit more hand picked than usual to me..
I recently asked Claude to help me choose a single MOSFET (transistor) for a specific use case in a mundane circuit. The safety triggered and it ended the conversation and refused to continue. Gemini has also done the same thing to me. Looks like the big players got very spooked by the temporary Trump admin ban on Mythos and they all locked down way too hard.
It triggered for me on a completely pedestrian game design prompt a couple of days ago. I’ve sent feedback and continued with Opus, but that was really unexpected
Where’s the lie?
[deleted]
GPT-5.6 Sol sets a new SOTA on ARC-AGI-3: 7.8%
Sol is the first verified frontier model to ever beat an ARC-AGI-3 game
https://arcprize.org/results/openai-gpt-5-6
Sol is the first verified frontier model to ever beat an ARC-AGI-3 game
https://arcprize.org/results/openai-gpt-5-6
Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning.
We are probably going to need a lot more GPUs.
These aren’t raw base models they are the result of a ton of RLHF and various adjustments.
Bitter lesson wildly overstated in this context.
Bitter lesson wildly overstated in this context.
Not sure where I implied they are "raw base models" and not sure what "various adjustments" means here or how "ton of RLHF" contradicts anything. If we look at research for open source models, "adjustments" usually come in the form of efficiency gains which directly contributes to the ability to scale or synthetic data pipelines to increase the dataset and increasing the context window.
rlhf = reinforcement learning from human feedback
(had to look it up)
(had to look it up)
I think it's more RLVR (reinforcement learning from verified rewards). The RLHF is just to align models to human preferences, meaning to behave nice.
More accurate to say RLHF aligns models to human preferences, most significantly to be helpful.
What makes you say that
More RLHF is in fact scaling.
Yes, but not in the “dump another chunk of all written language in the bucket and stir”-sense which is what bitter lesson became synonymous with.
That may not be the intent of the original article, but over the past few years that’s what the phrase turned into.
That may not be the intent of the original article, but over the past few years that’s what the phrase turned into.
GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.
Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.
Where did they get a second copy of all books, written/audio/visual media, and the web?
Couch cushions?
Couch cushions?
The measure is to see if the results scale, not just the rumored attempts at building such a model, as o3 taught us.
Where are they getting new data from at this point? Didn't they already read the entire internet?
A lot of data these days is synthetically generated. As an example, to make a model good at understanding assembly you simply need to round-trip code through a compiler and disassembler, then train against the source of truth and the assembly. You can generate arbitrary algebra expressions and have it solve it.
A lot of pretraining is also choosing the right type of data, you don't want to just have it ingest garbage (although I read that some amount of garbage actually helps the model be more robust). Pretraining crystallizes a lot of the inductive biases that post-training builds on, so by crafting the right data mixture you can make it easier for it to start off with a good foundation. There is also a lot of focus on mid-training these days, which I understand is basically either the name for the synthetic data stage, or the SFT phase before all the RL
A lot of pretraining is also choosing the right type of data, you don't want to just have it ingest garbage (although I read that some amount of garbage actually helps the model be more robust). Pretraining crystallizes a lot of the inductive biases that post-training builds on, so by crafting the right data mixture you can make it easier for it to start off with a good foundation. There is also a lot of focus on mid-training these days, which I understand is basically either the name for the synthetic data stage, or the SFT phase before all the RL
Custom sets and mining their own users, as every lab does
Custom what though? And user queries are not a knowledge repository it’s an ignorance repo.
The bitter lesson just means “compute scaling beats hand-tuned architectures in the long run”.
As GP said. More RLHF is in fact the bitter lesson.
As GP said. More RLHF is in fact the bitter lesson.
But the bitter lesson is explicitly about scaling compute. If increased human involvement counts as “Scaling” the thesis becomes incoherent.
Not just RLHF but also RLVR, and isn't that the litter lesson though?
My sense of the Sutton Dwarkesh interview was that he was calling out that he didn't mean just longer datasets, but rather learning through exploration and that's exactly RL.
My sense of the Sutton Dwarkesh interview was that he was calling out that he didn't mean just longer datasets, but rather learning through exploration and that's exactly RL.
They just need more contact with reality. That's what RL is right? Contact with narrow subsets of reality.
RLHF is an increasingly small part of training though? From what I understand most of the capability gain is in RLVR
Nah, the last few generations have more RLVR in the data mix. Which is more CPU intensive and very much amenable to the bitter lesson as you can reduce the loss by doing more rollouts in your tool environment.
The scaling with reasoning models is more and more with things like verifiable rewards (coding and math), in line with bitter lesson and also Sutton invented lots of modern RL.
While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.
I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)
I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)
There goes my plan to buy a PC for the next decade
The whole of knowledge work is being automated. We've barely begun to see the GPU build out. This is just the start.
I'd imagine they're going to 10x this, maybe 100x this.
I'd imagine they're going to 10x this, maybe 100x this.
yeah, spacex's planned million satellite datacenter-in-space constellation doesn't seem as absurd.
Sam Altman was saying we'll eventually need to build a dyson sphere. We'll see how far human society can withstand this kind of escalation. I'm guessing one more year before there are major fractures in the basic economic/social/political relationship that get so bad it actually prevents more buildout.
I actually think we're in a strange situation with AI compute.
Right now, we have models that are statistical models of language, with a world model and reasoning "falling out" of a lot of effort.
It's like we've made something that's a little bit intelligent, and now we're trying to amplify that trick to create something that's quite intelligent. And - don't get me wrong - it works.
But it's also super, super inefficient. We're having machines "think out loud" to compensate for the quality of their thought processes. We elongate the path to make up for the progress made on a given step.
I tink there's probably a much smarter way of doing things that will require qualitative architectural (and quite possibly hardware) innovations. Right now we're on the path to a Dyson sphere: that's probably not going to be necessary once we figure out a smarter way to think.
Right now, we have models that are statistical models of language, with a world model and reasoning "falling out" of a lot of effort.
It's like we've made something that's a little bit intelligent, and now we're trying to amplify that trick to create something that's quite intelligent. And - don't get me wrong - it works.
But it's also super, super inefficient. We're having machines "think out loud" to compensate for the quality of their thought processes. We elongate the path to make up for the progress made on a given step.
I tink there's probably a much smarter way of doing things that will require qualitative architectural (and quite possibly hardware) innovations. Right now we're on the path to a Dyson sphere: that's probably not going to be necessary once we figure out a smarter way to think.
I agree. But I think you're missing that LLMs can internalise a lot of the thinking process in their layers without explicit CoT. That System 1-style reasoning is bounded depth computation but very, very broad. Yudkowsky called it "cached thoughts" and I think it's an incredibly important idea [1]. It's really stiking how the best LLMs don't even need to think where smaller LLMs do.
So as more thinking is cached in their weights through increased RL training, those weights are doing more useful work and the efficiency is increasing.
[1] https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-tho...
So as more thinking is cached in their weights through increased RL training, those weights are doing more useful work and the efficiency is increasing.
[1] https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-tho...
Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.
It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.
It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.
This shows up in OpenAI's graphs on their announcement page. There is a peak performance datapoint in the graphs past which (to the right on the graph indicating more resources spent) peformance declines. And it's on every graph on that page!
And in my tests, that point of "overthinking" depends on the problem's complexity, so it's not necessarily that using "xhigh" is always bad or good.
I think I have this problem but with my human brain.
Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).
I mean, theoretically you can solve every finitary problem with a brute force solution...
Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far
Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far
The innovation of Transformers is that they are shallow rather than recurrent!
Or a new model. The human brain does far more with far less.
Yes an no current models can read and output much more faster with differing quality tho.
Or a lot better efficiency.
We’re definitely going to need a lot of Gpu’s
I said a few months ago, "man, Opus is great, but sometimes when talking with it I have the feeling like, this thing should be about 10 times bigger."
When Mythos was announced after that, I was pleasantly surprised to hear about it. But when it turned out to be only two times bigger, I was a little disappointed!
(I am even more disappointed with the safety filters, but that's kind of a separate discussion... "Fortunately" I find that I can usually edit my prompt by single character and get through...)
When Mythos was announced after that, I was pleasantly surprised to hear about it. But when it turned out to be only two times bigger, I was a little disappointed!
(I am even more disappointed with the safety filters, but that's kind of a separate discussion... "Fortunately" I find that I can usually edit my prompt by single character and get through...)
What do you mean bigger? Bigger functional context window?
> Dramatic difference
Isn't this just the difference between getting 0 right and getting 1 right?
Isn't this just the difference between getting 0 right and getting 1 right?
> We are probably going to need a lot more GPUs.
Or a breakthrough in algorithms etc.
The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.
Or a breakthrough in algorithms etc.
The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.
The human brain has 80 billion neurons and a 100 trillion synapses. I think you're underselling the processing power of that warm chunk of meat.
The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges.
The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges.
Moreover we've known for quite a while now that glial cells also participate in cognition and moderate learning (e.g.: [1]). When you take those connections into account the numbers get really staggering. 85 billion glial cells with trillions of protein channels facilitating communication between the glial syncytium [2].
[1] https://www.sciencedirect.com/science/article/pii/S193459091... [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/
[1] https://www.sciencedirect.com/science/article/pii/S193459091... [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/
It’s still 20W. We have living proof what is possible within 20W. The message has always been clear - try to get silicon computer to be as power efficient as the brain is as it is obviously possible.
I keep thinking about the cyborg chicken mechs in Metal Gear Solid 4. If stuff like Neuralink gets advanced enough, we might find that cutting up animals and sticking electrodes in their brains is the best way to make robots.
Yeah, people might object, but it can be argued that we are already subjecting scores of animals to horrors beyond comprehension just to get a bucket of chicken wings. And even if we manage to get silicon to do what brains do, it will likely cost 1000x as much and consume 1000x the power like you said.
It's hell of an economic incentive.
Yeah, people might object, but it can be argued that we are already subjecting scores of animals to horrors beyond comprehension just to get a bucket of chicken wings. And even if we manage to get silicon to do what brains do, it will likely cost 1000x as much and consume 1000x the power like you said.
It's hell of an economic incentive.
The real question is not how many "weights" the human brain has (neurons+synapses may or may not translate into "weights", and brain might be also inefficient for what it is), but rather how much evolutionary and social "compute" was necessary to pack everything into that capacity.
I think you're helping GPs point: there is a lot of efficiency gains to be made to match the processing power of the brain, given it's size and power draw.
An entire human burns about the equivalent of continuous 100w output per day. You can’t even run a single GPU on that budget.
20 watts for inference AND training!
For intelligence, I expect the next breakthrough to be colocation of memory and compute in the same chip. And we'll need much more of this memory, probably a few petabytes.
And a lot more electricity to power them.
And dozens of data centers in every state so tokens are dirt cheap.
This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.
I thought models werent allowed tools on arc-agi?
This is the first I have herd of this benchmark. Can someone explain how it in any way indicates how close we are to "AGI"?
Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...
It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.
Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...
It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.
> Can someone explain how it in any way indicates how close we are to "AGI"?
I think it is historical name. At some point when benchmarking was very undeveloped, this was targeting abstract reasoning and generalization, hence AGI.
I think it is historical name. At some point when benchmarking was very undeveloped, this was targeting abstract reasoning and generalization, hence AGI.
the problems are general and abstract, domain specific knowledge and memorization don't help. Figuring out the rules, the goal, the controls, and how to solve in a reasonable budget all indicate some level of general ability.
So basically they're well suited for like, an octopus or a crow?
I was thinking about those species earlier in the context of, what does intelligence mean outside of language.
The benchmark appears to be testing the same thing. Although I don't know how much transfer there would be between this data set and the kind of situations a crow or an octopus would encounter.
Edit: Huh, it's just a Game boy game? I just did a couple of the tasks. It looks like C64 era game to me. Navigating levels. A lot of overlap with animal intelligence then.
I was thinking about those species earlier in the context of, what does intelligence mean outside of language.
The benchmark appears to be testing the same thing. Although I don't know how much transfer there would be between this data set and the kind of situations a crow or an octopus would encounter.
Edit: Huh, it's just a Game boy game? I just did a couple of the tasks. It looks like C64 era game to me. Navigating levels. A lot of overlap with animal intelligence then.
We have it slightly ahead of Fable in our multi-agent coding evaluations.
Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol.
Fable scores slightly lower, but with an average solution size of 12.2 KB.
Data at https://gertlabs.com/rankings?mode=oneshot_coding
Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol.
Fable scores slightly lower, but with an average solution size of 12.2 KB.
Data at https://gertlabs.com/rankings?mode=oneshot_coding
This looks like a good benchmark. Time and time again I keep giving OpenAI models the chance to win me back, but Opus (and Fable especially) just writes more elegant code and is a significantly more productive rubber duck for interactive discussions. I feel vindicated seeing your description of verbose and defensive code, and I’m a bit disappointed that 5.6 Sol’s solution is still >5x longer than the human solution and 2x as verbose as Fable’s. Do you have any insight whether any of that is comments?
I wonder why nobody has tried to optimize for actual code size or complexity metric, or at least why I haven’t seen more benchmarks that display this. GPT5.5 just keeps pushing more and more pointless indirection into every function it writes in my main project, it’s borderline negative productivity.
P.S. I’d be curious to see Cursor’s composer models in there, they seem to be among the best performing low cost models: https://artificialanalysis.ai/articles/cursor-composer-2-5-c...
I wonder why nobody has tried to optimize for actual code size or complexity metric, or at least why I haven’t seen more benchmarks that display this. GPT5.5 just keeps pushing more and more pointless indirection into every function it writes in my main project, it’s borderline negative productivity.
P.S. I’d be curious to see Cursor’s composer models in there, they seem to be among the best performing low cost models: https://artificialanalysis.ai/articles/cursor-composer-2-5-c...
The human solutions are all written in Python, which creates a significant length bias, whereas the AI models are assigned to create solutions randomly distributed across 11 relevant programming languages, most of which are inherently more verbose than Python.
I have not broken down the comment/code ratio, but that's actually a really interesting idea for a metric.
I would also like to test Cursor, but our policy is to only test models available on public routers for now.
I have not broken down the comment/code ratio, but that's actually a really interesting idea for a metric.
I would also like to test Cursor, but our policy is to only test models available on public routers for now.
Really almost all benchmarks I look at have a cost per task column, which is basically the code size metric if you take an extra step
Not at all. The model could (and sometimes should) burn all the money it wants, and then produce a single line of actual production code. Only some things, e.g. full rewrites, have clear cost - LoC scaling.
For my usage, I would very much prefer if those $/task were being spent in thinking and experimenting, and the actual output would be as short and maintainable as possible. “maintainability” is a vague target of course, but it’s at least somewhat correlated with code size.
For my usage, I would very much prefer if those $/task were being spent in thinking and experimenting, and the actual output would be as short and maintainable as possible. “maintainability” is a vague target of course, but it’s at least somewhat correlated with code size.
Seems quite kind to Gemini models.
Gemini models struggle with agentic coding/tool use/exploration, but they are actually quite smart in one-shot reasoning. They're not as far behind as people think. It's mostly post-training and productization issues, which are easier to fix than pre-training/mid-training issues.
Agentic coding data: https://gertlabs.com/rankings?mode=agentic_coding
Agentic coding data: https://gertlabs.com/rankings?mode=agentic_coding
I'm surprised it is that low. Are not all top AI labs "cheating" and workaround LLMs's low sample efficiency by hiring people to generate more data points - similar problems with answers, so they can train models on those and improve scores? A good benchmark for general intelligence probably should be a complete black box, no sample data given/leaked at all.
Oh it's because the bench is lying.
You need to pass each level without failing, if you fail a level, it count as you "lost" the minigame.
The fact it start to get a score means it managed to get a 100% score on one of the minigame.
it seems the older models were capped at 10kusd for the runs though?
> Cost/task: $25.1K
Yikes
Yikes
Very interesting. My prediction is that Mythos would outperform Sol.
Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.
At some point his claim should be fully falsified no?
Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.
At some point his claim should be fully falsified no?
Mythos probably wouldn't, otherwise they'd have included it in their release. Next version of Mythos probably will though.
And yeah.. Reality has not been kind to LeCun.
And yeah.. Reality has not been kind to LeCun.
Are you joking? They spend billions of dollars training LLMs to get a 7.8% on arc agi 3 whereas DINO models are near sota in image classification, provide meaningful embeddings to the point where image segmentation is just PCA. The spend on DINO cannot be more than five million (correct me if I'm wrong)
JEPA is just getting started
JEPA is just getting started
His main anti-LLM predictions have been consistently either wrong or misleading.
There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point.
There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point.
His main LLM predictions have almost nothing to do with Arc AGI...
What exactly was he dead wrong about that is proven by any of this?
GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.
He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.
He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.
Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.
What exactly was he dead wrong about that is proven by any of this?
GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.
He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.
He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.
Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.
>He's merely said they don't think
He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.
He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.
> He said even 'GPT 5000' couldnt do things that they could do a month later, let alone by 5000.
What things specifically and when?
What things specifically and when?
https://youtube.com/shorts/zQTt8TkcyfU?is=09r7XDqz2w6-Pygu
You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.
You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.
That does not at all look cherry picked or taken out of context...
LeCun's ideas cannot be reduced to a 6 second clip...
You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...
LeCun's ideas cannot be reduced to a 6 second clip...
You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...
You can watch the whole Lex Friedman interview, it's on youtube. It's not out of context at all. He goes on about how LLMs will never be able to do things that they do trivially. And he has just doubled down for years.
Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.
Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.
I think you misunderstand what he thinks LLMs can and can't do. He says repeatedly that LLMs may be great for generating text and code, but that a fundamental model of artificial intelligence should be able to perceive and use information outside of text. Also, that a supervised learning approach is not exactly representative for how we learn. That it's a small piece but we largely learn with unsupervised learning. His main criticisms of LLMs are that they are supervised, probabilistic, and learn largely from text instead of observations. His claims about performance come downstream and I'd still argue that he's been (somewhat) right about those as well.
That LLMs don't have common sense and don't have good physical reasoning abilities; that you can't scale LLMs all the way to AGI; or that they can't predict the consequences of their actions which is the foundation of agentic behavior all seem like still (mostly) accurate predictions to me.
While LeCun has his share of problems, I think largely his criticisms on LLMs are more right than wrong. What remains to see is how good JEPA can be at filling in the gaps left behind from the brittleness of LLMS.
That LLMs don't have common sense and don't have good physical reasoning abilities; that you can't scale LLMs all the way to AGI; or that they can't predict the consequences of their actions which is the foundation of agentic behavior all seem like still (mostly) accurate predictions to me.
While LeCun has his share of problems, I think largely his criticisms on LLMs are more right than wrong. What remains to see is how good JEPA can be at filling in the gaps left behind from the brittleness of LLMS.
All of us were shocked with RL on LLMs.
To me LLMs have gotten better since 2024, but their fundamental flaws still seem there.
They hallucinate when it comes to really challenging tasks such as math proofs. They still do not reuse code well and will rewrite functions instead of perusing the standard library.
But this is good news. LLMs are awesome and they are only the first step towards AI being applied everywhere. They are a Model T
To me LLMs have gotten better since 2024, but their fundamental flaws still seem there.
They hallucinate when it comes to really challenging tasks such as math proofs. They still do not reuse code well and will rewrite functions instead of perusing the standard library.
But this is good news. LLMs are awesome and they are only the first step towards AI being applied everywhere. They are a Model T
> What exactly was he dead wrong about that is proven by any of this?
He said as you need more and more tokens models will fall apart because each additional token is a chance for a mistake and they will just exponentially fall apart. But in practice models have learned to identify and self-correct mistakes and if you look at the graphs more inference reasoning tokens almost always give far better accuracy.
He said as you need more and more tokens models will fall apart because each additional token is a chance for a mistake and they will just exponentially fall apart. But in practice models have learned to identify and self-correct mistakes and if you look at the graphs more inference reasoning tokens almost always give far better accuracy.
That's true, but he's still correct, it's just that the context is now so large that only people using agent loops see "context rot"
His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).
With JEPA models so far, it seems that PCA on latent vectors suffices.
tldr: embeddings have a lot more room for improvement
His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).
With JEPA models so far, it seems that PCA on latent vectors suffices.
tldr: embeddings have a lot more room for improvement
One of the big things that enabled extended context to work was training techniques for extending LLMs with reasoning, part of the thing he was saying wouldn't work.
[deleted]
Yann is a big SSL guy but I don't think he was involved in the original DINO - he's not listed as a co-author or anything.
DINO was created independent of JEPA but uses a similar principle of self supervised learning through minimizing the prediction error of a latent.
The difficulty in predicting a latent is so called "collapse"; the embedding neutral network can always output the zero vector and this would predict the output correctly.
There are different ways to solve this, DINO uses two different models - a teacher and a student and LeCunn uses an explicit term against collapsing to a single output.
Yann mentions DINO in his talks
The difficulty in predicting a latent is so called "collapse"; the embedding neutral network can always output the zero vector and this would predict the output correctly.
There are different ways to solve this, DINO uses two different models - a teacher and a student and LeCunn uses an explicit term against collapsing to a single output.
Yann mentions DINO in his talks
DinoV3 paper: https://arxiv.org/pdf/2508.10104#page=36
"we use a rough estimate of a total 9M GPU hours"
From CoreWeave, at current prices (~$2.46/hr spot to ~$6.16/hr on demand) would correspond to $22M–$55M.
The dataset is really where the cost is though - they used LVD-1689M - 1.6B images of curated web data from roughly 17B instagram images. This probably cost a huge amount of hours in human annotation, compute for algorithmic filtering, etc and not to mention probably a 20-50 person team working on this model.
You might want to change assumptions about how expensive these models are.
"we use a rough estimate of a total 9M GPU hours"
From CoreWeave, at current prices (~$2.46/hr spot to ~$6.16/hr on demand) would correspond to $22M–$55M.
The dataset is really where the cost is though - they used LVD-1689M - 1.6B images of curated web data from roughly 17B instagram images. This probably cost a huge amount of hours in human annotation, compute for algorithmic filtering, etc and not to mention probably a 20-50 person team working on this model.
You might want to change assumptions about how expensive these models are.
Thanks for the correction on the order of magnitude for the whole training process.
The 9M GPU hours includes the DINO v2 inference used in order to curate the data set.
The final training run used like 300000 dollars of compute.
Unfortunately we don't know how much RLVR + Agent training costs these companies. I'm just gonna say it's in the hundreds of millions, because they are supposedly making billions of profit on inference yet making billion dollar losses
The 9M GPU hours includes the DINO v2 inference used in order to curate the data set.
The final training run used like 300000 dollars of compute.
Unfortunately we don't know how much RLVR + Agent training costs these companies. I'm just gonna say it's in the hundreds of millions, because they are supposedly making billions of profit on inference yet making billion dollar losses
ASI is going to be here by the time Lecun gets started.
My main takeaway from LeCun's thesis isn't that you can't build LLMs to do useful things better than the best human, it's that these systems don't learn arbitrary skills efficiently, like humans do. And the question is, why not? 8% on ARC-AGI-3 is amazing for a machine considering how far we've come since digital computers were first built. But it is pretty poor if you're claiming something is well on its way to exhibiting human-like intelligence.
Mythos can do some amazing things (I'm assuming, I've never seen it). A young child can learn to control its body without reading any books on dynamical systems and kinematics. Mythos cannot learn to control a humanoid robot after sucking in every piece of data Anthropic can get their hands on.
Mythos can do some amazing things (I'm assuming, I've never seen it). A young child can learn to control its body without reading any books on dynamical systems and kinematics. Mythos cannot learn to control a humanoid robot after sucking in every piece of data Anthropic can get their hands on.
Falsifying Yann Lecun isn't exactly a priority for anyone seriously working in this space.
What is this supposed to mean?
Mythos doesn't appear to be on the verified leaderboard for ARC-AGI 3
Officially, Fable/Mythos testing was delayed because of Anthropic's data retention policy. Don't know if there's word on them working that out yet.
https://x.com/arcprize/status/2064399134099153344
https://x.com/arcprize/status/2064399134099153344
“Bro” spent most of his career in the wilderness because everybody thought ML/NN/etc were a dead end.
I’d not wager against him having at one one more break though architecture before he retires.
I’d not wager against him having at one one more break though architecture before he retires.
Notice how neither him, nor Ilya, nor Mira shipped anything relevant recently
It's telling
It's telling
Not sure how Mira gets into the same sentence as Yann and Ilya.
As far as the lack of shipping, they're scientists and what we're doing now with LLMs is more "engineering."
As far as the lack of shipping, they're scientists and what we're doing now with LLMs is more "engineering."
Ok long time Claude Code user here; lately I've started to realize there's other great models out there I should be trying, but I'm hesitant to leave Claude Code behind for something new.
What's the consensus today on codex vs claude code, does it really matter anymore?
What's the consensus today on codex vs claude code, does it really matter anymore?
Codex has arguably been better than Claude Code for months now, but it's flown under the radar because it just didn't capture the same viral marketing effect and OpenAI in general has had more optics / PR issues than Anthropic amongst the online developer crowd. I use the word "better" not in the sense that the underlying GPT models are fundamentally smarter or more intelligent, but rather that as a product Codex is just simpler, cheaper, and abundantly reliable and low-drama.
I’d argue the opposite. I’ve switched back and forth from one to the other and Opus/Fable has been constantly better than any GPT in my daily work. It’s a bit slower but it does the things right, with as little code as possible, some comments where needed. Codex is faster but you always have to correct it because it got something wrong; it writes tons of code ("let me add a small helper") with obvious comments.
Purely anecdotally the one persistent issue I have with LLMs writing code is that they are absolutely paranoid and add a load of indirection and defensive crap and even if you prompt to avoid that it will often require manual steering to remove the cruft.
https://github.com/EspoTek/.claude/blob/master/CLAUDE.md
Stick the "Never suppress errors" section into your Claude.md, this will never happen again (works for me with Python/Flask, ymmv for other languages).
Stick the "Never suppress errors" section into your Claude.md, this will never happen again (works for me with Python/Flask, ymmv for other languages).
A lot of that sounds like offensive programming: https://en.wikipedia.org/wiki/Offensive_programming
Didn't know there was a word for that, thanks! Looks like my programming style matches my communication style in general. :P
Fallbacks and backward compatibility are killing me :) So many code paths that just don't fail predictably.
I’ve experienced this with GPT but not with Opus/Fable.
I experience it with everything including opus/fable.
Though my feeling, no proof, is that the opus/fable today is not what it was months ago. there was a time for about a month where opus was incredible. Just incredible but as fable started to move out i swear to god it feels like sonnet now. Fable feels like opus used to but costs more.
Though my feeling, no proof, is that the opus/fable today is not what it was months ago. there was a time for about a month where opus was incredible. Just incredible but as fable started to move out i swear to god it feels like sonnet now. Fable feels like opus used to but costs more.
recent gpts are horrendous for this, whereas recent claudes have a tic where they incessantly add useless comments referring to previous changes and will use multiple single-line comments instead of a standard multi-line docblock.
The incessant need to constantly leave "the code doesn't work like <bad implementation>, it works like <good implementation>" frustrates me to no end. No amount of directions against it in project MEMORY, CLAUDE.md, or even embedded in the prompt seem to be able to stop it from doing this. I don't understand how it could have gotten into the training because I've legitimately never seen an actual person write code comments like this.
It writes code as if the audience is you, the user of Claude, and not other developers reading the code in the future. I found that it helped to instruct it to keep in mind who the audience is and only write comments that describe the current state of the code and never describe anything that can just be inferred from the git history. I found that that helped, and I almost never see these nonsense comments anymore.
Maybe it's self trained. It eats its own output and likes it...
Sounds like my code. They may have been trained on my code!
I have not noticed this with Opus 4.6+. The result is usually not too far from what I would have written myself.
Opus 4.6 was the best model in the family, following two ones were seriously brain damaged to do well on benchmarks.
yeah those have been horrible
I wouldn't say terrible-terrible but only better at from-prompt-to-solution rather than interactive discussing and problem solving.
I tend to define it "better at solving, worse at assisting phenomenon". Which doesn't properly show on benchmarks that only focus on the solving part.
I tend to define it "better at solving, worse at assisting phenomenon". Which doesn't properly show on benchmarks that only focus on the solving part.
I tell it to avoid belts and suspenders, don't leave dysfunctional code in, and fail loud. Seems to change that behavior.
Sounds like you are talking past each other. GP is saying the harness of codex is higher quality, which I can believe, even if the models are not as good as Opus/Fable.
i don't think so, i think it's 50% what work people are doing, 50% vibes. my experience with 5.5 is i like it more and get better results than 4.8/fable. which isn't to say i think it's a strictly better model, just been working better for me.
What do you mean "i don't think so"? What is it about the comment you are replying to that you don't think?
It's the other way actually. Cc is a better harness but gpt models are just so much better in my personal experience (at least for backend) ever since 5.4.
I'm not sure how meaningful this is. Fable only just recently become more broadly available, and GPT-5.6 is launching broadly today.
The comment I was responding to was talking about Codex usage in the past few months. This is a general feeling about Codex with Claude, not a model-to-model comparison.
You were certainly not switching back-and-forth between Fable (a model available to the public generally only since July 1) and Codex for the last several months.
I really love the Opus/Fable models but I'm honestly sick to death of the buggy product. The CLI always has some weird issue. Right now it doesn't even output messages before tool calls, it just swallows them and they disappear.
I don't like OpenAI as a company, but they appear to have QA, and that is probably enough to get me to switch.
I don't like OpenAI as a company, but they appear to have QA, and that is probably enough to get me to switch.
There was an issue on Claude Code the other day where it would only wait 60 seconds when it had asked a set of questions, then if it didn't get a response from the user it would just continue however it thought was best. Completely unusable. It took them nearly 48 hours to merge a fix.
Same for me, that is why I switched to Pi. I still use Sonnet or Opus, but mainly GPT due to cost.
Glad I’m not the only one noticing this. It’s maddening.
Using remote control I will choose a model but Claude will always revert to Haiku for the first turn.
Basic stuff about features that are more than a week old just get no attention at all. From the outside Athropic seems to be a clear feature factory.
Basic stuff about features that are more than a week old just get no attention at all. From the outside Athropic seems to be a clear feature factory.
A bit slower? I think for most of my tasks, Claude takes easily 2x longer for almost everything, even things like just analyzing code. It churns tons of tokens for quite simple things.
IMO that's exactly why it's a bit better at actual problem solving.
You absolutely do not "always have to correct" Codex. I'm not sure what you're doing, but I'd say 80-90% of its edits on my side it doesn't need any revisions.
IMO that's exactly why it's a bit better at actual problem solving.
You absolutely do not "always have to correct" Codex. I'm not sure what you're doing, but I'd say 80-90% of its edits on my side it doesn't need any revisions.
> Codex is faster but you always have to correct it because it got something wrong
this has been my experience with Codex as well, and I have to fix its mistakes every single time. But recently, I literally threw away three hours of work because it kept adding hundreds of lines to my code base. When I restarted the entire work using Fable and Opus, it was like night and day.
this has been my experience with Codex as well, and I have to fix its mistakes every single time. But recently, I literally threw away three hours of work because it kept adding hundreds of lines to my code base. When I restarted the entire work using Fable and Opus, it was like night and day.
I have both as well. I trust the output of Claude to a higher degree than what I get with Codex. I always have claude review codex output. That being said, I find gpt 5.5 more generically useful at a wider breadth of tasks. Straight coding though, it's no contest.
Obligatory YMMV, maybe your prompting style fits gpt better. We forget that this matters a lot
Obligatory YMMV, maybe your prompting style fits gpt better. We forget that this matters a lot
I'd say Codex and Claude Code have different strengths and weaknesses. Claude Code is significantly better in terms of their subagent UI for example - being able to see the list of subagents under the input is great.
To be honest though, I've gotten to the point where I prefer the OpenCode UI. A big win for OpenAI is you can log in to your subscription in OpenCode, whereas this is not trivially achievable for a Claude subscription.
I was getting some really impressive cost efficiency today in OpenCode with the following:
GPT 5.6 was picking up on quality and functional issues from DeepSeek and having it resolve them cleanly, and I didn't even get close to my quotas whereas I can usually blast through them. I feel as people get more comfortable with subagents and mixing and matching models in their daily work, Anthropic's walled garden stance will start to hurt them.
To be honest though, I've gotten to the point where I prefer the OpenCode UI. A big win for OpenAI is you can log in to your subscription in OpenCode, whereas this is not trivially achievable for a Claude subscription.
I was getting some really impressive cost efficiency today in OpenCode with the following:
* Main session agent: gpt-5.6-sol (high) via OpenAI subscription
* General purpose subagent: deepseek-v4-pro (high) via OpenCode Go subscription
* Using `obra/superpowers` for subagent driven workflows
* The main session only being allowed filesystem read permissions and everything else delegated
GPT 5.6 was picking up on quality and functional issues from DeepSeek and having it resolve them cleanly, and I didn't even get close to my quotas whereas I can usually blast through them. I feel as people get more comfortable with subagents and mixing and matching models in their daily work, Anthropic's walled garden stance will start to hurt them.
Agreed. GPT 5.5 will come up with more straightforward solutions with far fewer tokens than Claude. Also, the usage limits are much more generous for Codex than Claude Code for the same monthly plan.
I've been using both and as far as I can tell with ccusage, the $ equivalent budgets are about the same between them now. This may have been true before anthropic doubled their quotas and openai 2x promo expired last month.
Last time I used Codex it would make loads of assumptions, often quite big ones, without asking.
Did they fix that, as that for me was what actually made codex worse.
Did they fix that, as that for me was what actually made codex worse.
I find that I have to tell GPT and Claude to keep asking me questions, or they will just fill in the gaps themselves (wrongly).
My experience has been that Claude/Opus will ask me questions, although sometimes I have to step in and redirect a bit. Codex/GPT just dives in and cranks code out, which after using Claude for a year I found rather concerning. It wrote working code though. Claude/Fable does something similar (asks far fewer questions than Opus) fwiw.
Did you use plan mode?
That's a strange statement... It's been true for a while now that OpenAI has had much more generous limits than Anthropic on their subscription plans. And with the Fable ban/guardrails disaster, there has been a lot of frustration from people in these comment sections. And Anthropic fucked up Claude Code pretty badly for a couple of weeks during the 4.6/4.7/4.8 transition, which again was widely publicized. And they got a lot of flack over not allowing other harnesses anymore. And ChatGPT got some pretty viral wins on model intelligence when they cracked the high profile Erdos problem.
If anything the online optics have been bad for Anthropic for the last half year. OpenAI doesn't have optics issues, from my point of view they simply have the issue that they are the least trustworthy player at the frontier. The way they pivoted from their original mission is truly breathtaking, especially coming in gloatingly to take the government contract when Anthropic got kicked out for insisting the government does not use their systems for mass surveillance or autonomous weapons systems. You understand what that means, right? OpenAI models are now actively used/developed for mass surveilance and/or autonomous weapons systems.
I know there are plenty here who seem to value their own ability to use these models cheaply above all other considerations. Then OpenAI is a great choice, and much less restrictive than Anthropic. But their problem is not on the optics. It's on the substance.
If anything the online optics have been bad for Anthropic for the last half year. OpenAI doesn't have optics issues, from my point of view they simply have the issue that they are the least trustworthy player at the frontier. The way they pivoted from their original mission is truly breathtaking, especially coming in gloatingly to take the government contract when Anthropic got kicked out for insisting the government does not use their systems for mass surveillance or autonomous weapons systems. You understand what that means, right? OpenAI models are now actively used/developed for mass surveilance and/or autonomous weapons systems.
I know there are plenty here who seem to value their own ability to use these models cheaply above all other considerations. Then OpenAI is a great choice, and much less restrictive than Anthropic. But their problem is not on the optics. It's on the substance.
I agree with this statement. And because it churns less tokens, it's just generally faster too - noticeably, throughout the day, across a range of tasks I get more shit done with Codex.
It's not better at reasoning on complex coding tasks, Claude Opus is still ahead there, but not by a lot.
It's not better at reasoning on complex coding tasks, Claude Opus is still ahead there, but not by a lot.
I really want a good Claude Design competitor in Codex, it's hard to use the others after getting used to it and yet I find anthropic's model to have a much worse understanding of what looks good or not than OpenAI or Google models.
Anthropic's models are downright terrible at visual reasoning and design.
Gemini is fantastic, however.
Gemini is fantastic, however.
I keep trying Codex and it constantly produces terrible output compared to Opus. I don’t understand how my results are so bad?
Switched to Codex last week, and I'm already MUCH happier than I have been with Claude Code. Which surprised me.
Nudged by this thread, I've decided to switch from Claude to Codex for a bit to see what happens. But...I immediately became lost in their marketing vortex of confusion on plans and pricing. Anyone care to tell me which plan I should be using? On the other side I use the $100 Claude Code plan. We actually have a "Business" ChatGPT subscription already, which seems to be $50/mo/seat. OpenAI's web site offers a set of individual subscriptions (for parity with CC presumably) which I suspect weren't available when we signed up for ChatGPT. I think that in turn happened due to some web site feature it didn't allow for free users (uploading PDFs, something like that). Perhaps I should switch from that business account to an individual subscription for Codex?
Test-drive it with an individual Pro account (5x or 20x) for a month. Download the Codex CLI client from https://github.com/openai/codex and auth it in the browser via the URL it provides. Set the model to 5.6-Sol and effort to max.
Ok thanks. Executed on that. I had it build a simple project (actually not so simple since it involved domain knowledge about snowpack and ice) and also gave the same prompt to Opus and to Fable. When I get some spare time I'll write an article highlighting the differences between all three.
What about cost?
Honestly it’s the usage limits that are so generous that makes codex worth it even if it may not be exactly as powerful as Claude. The peace of mind that you can try a lot of things and make huge refactors and run extensive redundant tests without running out of tokens just makes the whole thing a much better experience. I tried coding with Deepseek and it was pretty terrible so the only reason codex works is because its abilities are close to or on par with Claude.
> I tried coding with Deepseek
But soooooo cheap. Especially for those of us where a monthly sub doesn’t make sense.
But soooooo cheap. Especially for those of us where a monthly sub doesn’t make sense.
There is so much less drama involved with the Codex world. You don't realize how oppressive CC is until you've escaped it. Outages, weird restrictions, degradation, accelerated usage, etc etc etc.
Totally. My experience as well. After some time with codex you're like come on Claude can you just stfu! Haha. I now almost always instruct Claude with specific length requirements when I ask questions. Otherwise, it just blathers and blathers in the most annoying of ways. "Oppressive" is spot on in my opinion
I'll agree and expand on "weird restrictions" -- I used to check the claude usage graphs multiple times a day to see where I'm at on my weekly budget. With gpt 5.5 I don't think I'm working differently but haven't felt the need to check anything because I think I've hit my limit... once? on some egregious edge case scenario iirc
Same here - it's probably that OpenAI needs to buy goodwill with developers to infiltrate corps and Anthropic is trying to squeeze the lead into revenue. The only question is - how much longer can OpenAI burn money before it needs to start showing signs of profitability
And I know this is petty but the CC cli/harness just grates. It’s overcomplicated, performatively cutesy, and buggy. It’s in my way. The codex harness gives me what I need and gets out of the way.
Let alone getting banned out right with no reason, zero updates after weeks, and not even being able to download your chat history (despite the feature being available (I assume they vibe coded it and it does not work!). My story below;
https://news.ycombinator.com/item?id=48597861
https://news.ycombinator.com/item?id=48597861
Even less drama with open models like GLM.
> accelerated usage
Can you post more information about this?
Can you post more information about this?
The only way to understand those moments where you sometimes Claude acts mysteriously is to be terminally online.
https://www.google.com/search?q=claude+usage+draining+site:w...
https://www.google.com/search?q=claude+usage+draining+site:w...
Um, the 'codex world' is the OpenAI world and there is a ton of drama and product confusion there!
Anthropic has certainly had some drama inflicted on them by the US administration, but otherwise they have just had heads down and executed with great focus. That is why they have succeeded.
Anthropic has certainly had some drama inflicted on them by the US administration, but otherwise they have just had heads down and executed with great focus. That is why they have succeeded.
At least Anthropic doesn't bend down to Pentagon/Trump administration
Perhaps not, but I wouldn't say Anthropic is too far removed.
https://investors.palantir.com/news-details/2025/Anthropic-J...
https://investors.palantir.com/news-details/2025/Anthropic-J...
I've been using Claude Code, Codex, Gemini (now Antigravity) at the same time for half year now, ever since I dipped my toe into agentic coding. I'd say in general Claude Code and Codex are equally powerful, Gemini is lagging behind.
One thing I appreciate with Codex is, OpenAI nowadays sometimes just gives you quota resets you can bank, so when you use up weekly quota before the week ends, you could just reset the quota, to continue using Codex. I've been much less anxious about Codex quota because of this perk. I just used one reset in the bank yesterday, and still have 3 resets left. Whereas with Claude, when you've used 95% quota 3 days before the week ends, you'd be much more anxious.
On the other hand, Claude Code's /remote-control mechanism is extremely helpful when I am running it in the cloud and wants to monitor it or control it on my phone. Codex currently doesn't support this kind of usage. Codex only allows you to use your phone to connect to a session on your desktop, not in the cloud.
One thing I appreciate with Codex is, OpenAI nowadays sometimes just gives you quota resets you can bank, so when you use up weekly quota before the week ends, you could just reset the quota, to continue using Codex. I've been much less anxious about Codex quota because of this perk. I just used one reset in the bank yesterday, and still have 3 resets left. Whereas with Claude, when you've used 95% quota 3 days before the week ends, you'd be much more anxious.
On the other hand, Claude Code's /remote-control mechanism is extremely helpful when I am running it in the cloud and wants to monitor it or control it on my phone. Codex currently doesn't support this kind of usage. Codex only allows you to use your phone to connect to a session on your desktop, not in the cloud.
Yes - Anthropic badly needs this same "here's a reset, use it when you want".
It's vastly better this way. Sure, it may impact the bottom line but it's a huge customer satisfaction win.
When Anthropic randomly resets me and I've only used 2%, that's worthless. When OpenAI tells me I have 3 resets available to use whenever I want - it's wonderful.
It's vastly better this way. Sure, it may impact the bottom line but it's a huge customer satisfaction win.
When Anthropic randomly resets me and I've only used 2%, that's worthless. When OpenAI tells me I have 3 resets available to use whenever I want - it's wonderful.
Codex is supported well on iPhone/iPad, it’s inside the ChatGPT app.
It’s amazing how much work you can get done on your phone now, especially if you already have a design mapped out in your head.
It’s amazing how much work you can get done on your phone now, especially if you already have a design mapped out in your head.
I have used claude and codex extensively but only from their CLI app (heavily sandboxed using rootless podman, network filtering, etc), so I don't really know what I'm missing with the GUI apps.
One killer feature that Claude has, and AFAIK Codex still lacks, is the ability to start a session in the terminal and then hand it off (actually just remotely control it), from the iOS app.
Last time I tried Codex on iOS it required a ton of set up to link a github project etc. The way claude lets me remote into a session I've already started on my actual machine is much better IMHO.
One killer feature that Claude has, and AFAIK Codex still lacks, is the ability to start a session in the terminal and then hand it off (actually just remotely control it), from the iOS app.
Last time I tried Codex on iOS it required a ton of set up to link a github project etc. The way claude lets me remote into a session I've already started on my actual machine is much better IMHO.
They’ve addressed that. Codex in the ChatGPT app on iOS is way better than Claude Code now.
You sign in the Codex app on your Mac same on iOS and are able to completely control your sessions - fork, side chats, plugins - everything.
It’s really great i often work through it. And you can connect any number of Codex instances on any number of macs and then manage them all through the iOS app.
You sign in the Codex app on your Mac same on iOS and are able to completely control your sessions - fork, side chats, plugins - everything.
It’s really great i often work through it. And you can connect any number of Codex instances on any number of macs and then manage them all through the iOS app.
maybe I'm misunderstanding but I don't want to sign into the app on a mac - I want to run the CLI on a headless linux server and control the sessions from my iPhone. Does Codex allow that now?
Codex isn't in charge of that.
Just use termux on your phone and connect to a tmux session on your server.
Codex won't know the difference.
Just use termux on your phone and connect to a tmux session on your server.
Codex won't know the difference.
I do that with Termux on my Android phone. I set up dynamic DNS and wireguard on my router so I can reach my LAN from anywhere. I just enable wireguard and ssh in.
Not sure if there's a Termux equivalent for iPhone.
Not sure if there's a Termux equivalent for iPhone.
You can do it too. There are two ways:
1. Run `codex remote-control --help` directly on your Linux server. 2. From the desktop app, connect to your Linux box, start Codex there, and make it remotely controllable.
Either approach will get you set up.
1. Run `codex remote-control --help` directly on your Linux server. 2. From the desktop app, connect to your Linux box, start Codex there, and make it remotely controllable.
Either approach will get you set up.
Ha, finally found time to get it working. Yeah it's more hassle than Claude Code since this needs a separate daemon server, but not very bad.
1. Install another copy of codex in a special dir on the Linux machine:
4. Voila! The codex sessions running on the Linux machine now show up on the phone!
1. Install another copy of codex in a special dir on the Linux machine:
$ curl -fsSL https://chatgpt.com/codex/install.sh | sh
$ ~/.codex/packages/standalone/current/codex remote-control start
$ ~/.codex/packages/standalone/current/codex remote-control pair
4. Voila! The codex sessions running on the Linux machine now show up on the phone!
Hmm, I don't have a desktop computer. I prefer my laptop be used for other purposes, and can sleep when not in use, instead of running a coding agent 24x7. That's why I prefer running coding agents in the cloud.
You can do that too. Start working locally, then just do /handoff to transfer your session to the cloud and then work through the Codex app on your phone.
I've been using codex app server. Works great.
https://learn.chatgpt.com/docs/app-server
https://learn.chatgpt.com/docs/app-server
Hmm, thanks. Didn't know about this. But looks like a bunch of hassle to set it up?
Finding the right docs/flags took longer than anything else. 15 mins from zero to productive on my phone.
One thing I noticed you can't use codex installed via npm, but it will tell you that.
Ymmv I'm a pretty simple user and do things in small manageable chunks. I try keep context < 10% before I fire off a task so I don't use many skills etc.
if you don't want to hassle with it, use desktop app, you either can make it remote controllable or you can control other devboxes.
The banked resets have been a game changer for me. I've been sticking to 5.4 medium mostly because 5.5 seemed to eat into my quota significantly faster. The reset bank gave me confidence to seriously start using 5.5 high. Lo and behold, I have yet to actually need a reset, but I'm now less likely to explore non-OpenAI models since there's an escape hatch with new models in case a coding session gets a bit crazy.
I’ve found Codex’s overage to be much better value than Claude’s. A monthly $10 budget is plenty for my backup Codex usage, but on Claude Code that would be gone in a couple of days.
> OpenAI nowadays sometimes just gives you quota resets you can bank
That's actually pretty awesome. Anthropic's random resets often have me scrambling to launch huge sessions to make the most of them before the weekly rollover. The gacha-like mechanics are maddening.
That's actually pretty awesome. Anthropic's random resets often have me scrambling to launch huge sessions to make the most of them before the weekly rollover. The gacha-like mechanics are maddening.
I recommend trying Codex too. In fact, I recommend running them side-by-side if you have the budget, e.g. have both independently plan the same feature or implement in a different worktree, or have them critique each other's work.
I personally find GPT-5.5 to be a better programmer than Opus 4.8, it is extremely thorough, but I don't like the code it generates ("austere"), and find Opus 4.8 to write more "human friendly" code. The programming comments GPT-5.5 makes is pretty awful where-as Opus 4.8 is good. I feel like Opus 4.8 is better at grasping my intention than GPT-5.5, and honestly find GPT-5.5 to be kind of "autistic". I do prefer the language (not the writing) of GPT-5.5, as I find the philosophical flowery language of Opus 4.8 kind of annoying.
I have only managed to try Fable 5 a little bit, which feels like a much more generally smarter version of Opus 4.8, that is much better a programming and grasping your intention, and I think even the intention of your code, and is _really_ good at spotting bugs or problems with logic in your code. It feels wicked smart but is extemely expensive. It feels smart in the sense like it has a "bigger brain" and is much more sensitive to subtleties/details.
These are different "brains", have different "personalities", etc. I think the best thing is to develop a feeling for it yourself.
I personally find GPT-5.5 to be a better programmer than Opus 4.8, it is extremely thorough, but I don't like the code it generates ("austere"), and find Opus 4.8 to write more "human friendly" code. The programming comments GPT-5.5 makes is pretty awful where-as Opus 4.8 is good. I feel like Opus 4.8 is better at grasping my intention than GPT-5.5, and honestly find GPT-5.5 to be kind of "autistic". I do prefer the language (not the writing) of GPT-5.5, as I find the philosophical flowery language of Opus 4.8 kind of annoying.
I have only managed to try Fable 5 a little bit, which feels like a much more generally smarter version of Opus 4.8, that is much better a programming and grasping your intention, and I think even the intention of your code, and is _really_ good at spotting bugs or problems with logic in your code. It feels wicked smart but is extemely expensive. It feels smart in the sense like it has a "bigger brain" and is much more sensitive to subtleties/details.
These are different "brains", have different "personalities", etc. I think the best thing is to develop a feeling for it yourself.
I haven't tried Codex yet, but I for my tasks GPT-5.5 may correctly point to a proper direction but its code feels a bit weird. Opus 4.8 is way better in coding, and actually it's the only one who could catch very very sophisticated bug in a large codebase (I tried different models including GPT-5.5 and DeepSeek). Interestingly Gemma 4 under opencode running locally performs not bad at all, it's far yet from DeepSeek level, but it manages to understand tools quite well, and code quality is pretty good. So, for simple coding projects I can say local models already won. It's amazing how smart open models of desktop size have become today. I mean it's quite plausible to manage small codebase today relying on only open tools and local models, you don't need any subscription to produce high quality code, but yes I assume you already experienced and know what you're doing :)
I did the side by side between Claude code (effort medium) and Cursor (auto). Asked Claude to prepare a plan and asked Cursor to review it and it found tons of gaps in the plan. Cost-wise, it came out better too. I have been using Cursor daily (along with Claude) and the former has been 20-30% cheaper despite me spending more time on it.
Codex has been comparable for a while. 5.1-5.5 have competed closely with 4.5-4.8. Fable blew them all out, now Sol comparable to Fable again. Some slight tooling differences with skills and hooks but for the most part I think if people are so engineered into one CLI that swapping to another inhibits them, then that is an error in usage habits.
Codex historically will follow tasks more closely with less creativity, whereas Opus will do more than you specify. I wouldnt consider either one better due to this fact, just makes them useful for different situations. Generally they'll perform similarly for most tasks.
Opus and Fable dominate 5.5 in artistic design (pixel art, ascii art), and edge out 5.5 slightly in general UI design taste. Have not tested Sol in that regard yet.
So far in my usage Sol has been superior to Fable at graphics rendering engine optimization.
Codex will work longer, and in single sessions without as much subagent usage.
Codex only has 256k context but its compaction is absolutely next level. You will not notice compactions and they will happen multiple times during a complex task or set of tasks without you ever having to notice or care. Claude code on the other hand still has fairly poor compaction.
Codex has more generous usage limits, and they also give you usage resets (weekly+5h resets) that you can bank for a month or so. Not sure how often they give these out.
Codex also seemingly never has outages or weird delays like Claude code does.
OpenAI randomly resets usage just like Anthropic does
I would use both if you code often
Codex historically will follow tasks more closely with less creativity, whereas Opus will do more than you specify. I wouldnt consider either one better due to this fact, just makes them useful for different situations. Generally they'll perform similarly for most tasks.
Opus and Fable dominate 5.5 in artistic design (pixel art, ascii art), and edge out 5.5 slightly in general UI design taste. Have not tested Sol in that regard yet.
So far in my usage Sol has been superior to Fable at graphics rendering engine optimization.
Codex will work longer, and in single sessions without as much subagent usage.
Codex only has 256k context but its compaction is absolutely next level. You will not notice compactions and they will happen multiple times during a complex task or set of tasks without you ever having to notice or care. Claude code on the other hand still has fairly poor compaction.
Codex has more generous usage limits, and they also give you usage resets (weekly+5h resets) that you can bank for a month or so. Not sure how often they give these out.
Codex also seemingly never has outages or weird delays like Claude code does.
OpenAI randomly resets usage just like Anthropic does
I would use both if you code often
Claude Code fan here... Codex is very good. Sometimes better. The killer feature is price.
After 6+ months of exclusive Claude Code usage, I was begrudgingly forced to try Codex once Anthropic rejiggered their limits such that I kept maxing out my $200/mo plan in just a few days. These days I pay both $200/mo plans, and it's just about enough to get me through a week's work (small game studio - infinite code to write!)
After 6+ months of exclusive Claude Code usage, I was begrudgingly forced to try Codex once Anthropic rejiggered their limits such that I kept maxing out my $200/mo plan in just a few days. These days I pay both $200/mo plans, and it's just about enough to get me through a week's work (small game studio - infinite code to write!)
> (small game studio - infinite code to write!)
Curious: what multiplier do you think your productivity has increased by, from before AI?
Curious: what multiplier do you think your productivity has increased by, from before AI?
In terms of ability to ship? Easily tenfold. We literally ship 10 times more than before AI. This does not, however, translate into a tenfold increase in actual business success, of course :)
Yes, because now your competitors do the same. The winners: inference providers
I think at some point, AI development becomes a competition for who has the best taste [1], because with seemingly unlimited/cheap tokens, AI ends up being more about throwing shit at a wall and hoping something sticks.
[1] https://www.reddit.com/r/ProgrammerHumor/comments/10ek380/co...
[1] https://www.reddit.com/r/ProgrammerHumor/comments/10ek380/co...
Genuine question/not a critique-are you actually reviewing all that code or just sending it and hoping for the best? I just can't imagine someone is reading/reviewing that much code every day, but maybe I'm wrong?
Like before AI, the scrutiny varies with the sensitivity of the area being edited.
Simple UI change? I do an AI review, but otherwise neither read nor write the code. The models are good enough they write better UI code than me, 9 out of 10 times. Not always the more idiomatic, but usually safer and more correct.
Change to our core data plane? I might spend 2-3 times more effort reviewing it than before AI. Yes, I go more slowly than pre-AI. Many more reviews, many more angles considered, including both human and (lots of) AI review cycles.
Most code is not that critical, and AI is also scarily good at writing tests. We also spend considerably more time paying down tech debt and testing thanks to AI, now that the cost is near-zero.
Net: I spend 10-25X less time on low-risk changes. I often direct (or at least approve) the implementation approach, but I rarely read this code. I spend 2-3X more time on high-risk changes. In both cases, I never write code "by hand". Since about November, I've had no reason to actually edit code in a code editor (perhaps maybe except .env files, which we don't allow agents to edit for obvious reasons).
AI is a tool. You can use it to go fast recklessly, or you can use it to go slow with confidence. Just like before AI... the skill and art of engineering is knowing when to do which.
Simple UI change? I do an AI review, but otherwise neither read nor write the code. The models are good enough they write better UI code than me, 9 out of 10 times. Not always the more idiomatic, but usually safer and more correct.
Change to our core data plane? I might spend 2-3 times more effort reviewing it than before AI. Yes, I go more slowly than pre-AI. Many more reviews, many more angles considered, including both human and (lots of) AI review cycles.
Most code is not that critical, and AI is also scarily good at writing tests. We also spend considerably more time paying down tech debt and testing thanks to AI, now that the cost is near-zero.
Net: I spend 10-25X less time on low-risk changes. I often direct (or at least approve) the implementation approach, but I rarely read this code. I spend 2-3X more time on high-risk changes. In both cases, I never write code "by hand". Since about November, I've had no reason to actually edit code in a code editor (perhaps maybe except .env files, which we don't allow agents to edit for obvious reasons).
AI is a tool. You can use it to go fast recklessly, or you can use it to go slow with confidence. Just like before AI... the skill and art of engineering is knowing when to do which.
> AI is also scarily good at writing tests
:-) I hope you read those tests before claiming it's "scary good"
:-) I hope you read those tests before claiming it's "scary good"
Indeed, much of the scariness is how fearlessly and confidently it writes them with little regard to their actual usefulness or value. When I find it adding a lot of tests, I often say something like: "audit each test carefully, and consider whether the test is testing a meaningful boundary or is more ceremonial. delete low-value tests and add new tests to cover meaningful boundaries not exercised by the gaps you identify". Without fail, this always produces some decent results.
Having said that, in truth, I almost never read the unit tests. Before AI, we had almost none (see: several person game studio) so the tradeoff is not "AI-generated tests" vs "human written ones", it's whether we have tests at all. So, I take them for what they're worth - not much - but if it catches an extra regression before it ships every now and then, it was worth it for the price (~free).
Having said that, in truth, I almost never read the unit tests. Before AI, we had almost none (see: several person game studio) so the tradeoff is not "AI-generated tests" vs "human written ones", it's whether we have tests at all. So, I take them for what they're worth - not much - but if it catches an extra regression before it ships every now and then, it was worth it for the price (~free).
Lots of unit test do not add value in the traditional sense, but they do help the llm to understand the code.
I notice that the tests are very often for its own benefit. Like, you’ll ask it to stop doing something, literally to just remove code, and it will write a test to verify the behavior is not there.
I can’t imagine a more useless test, but I get that it wants to verify that it actually made the change. I just delete the test when it’s done.
I can’t imagine a more useless test, but I get that it wants to verify that it actually made the change. I just delete the test when it’s done.
What does your workflow look like tool wise? Are you still using IDE?
No. I used to use Cursor, but now my workflow is that I use an inhouse CLI tool I wrote called "bud" that wraps/seeds the harnesses per-worktree, and boots a full copy of the game so each worktree can work independently. If git worktrees solve the problem of code isolation, bud solves the problem of isolating everything else. It's about 15K lines of rust, and I use it 100 times a day or so. It's sort of a layer on top of a harness like codex/claude code.
I have 10+ of these workspaces in parallel, and I context switch between them as I get blocked on things. I manage the workspaces using `herder`, which is a terrific tmux-like tool that allows me to keep those workspaces on a nixOS machine I have at home that I SSH into via tailscale, so my agents don't stop working every time I close my laptop (it also lets me leverage that machine's computing resources instead of running dozens of servers and harnesses on my poor MacBook).
I have 10+ of these workspaces in parallel, and I context switch between them as I get blocked on things. I manage the workspaces using `herder`, which is a terrific tmux-like tool that allows me to keep those workspaces on a nixOS machine I have at home that I SSH into via tailscale, so my agents don't stop working every time I close my laptop (it also lets me leverage that machine's computing resources instead of running dozens of servers and harnesses on my poor MacBook).
Codex has been good for a long time, more expensive but very focused on efficiency. Working with it feels faster and more to the point than Opus models and I trust it more with long-running jobs. Also regular resets vs being at the whim of Anthropic drama all the time is hella nice.
Anyone know what the deal is with the resets?
They've discovered it's a good marketing strategy. Whenever there's an outage, or a new launch, there's often a reset with it, which helps keep people engaged with OAI / Tibo and reduces churn.
They've also introduced banked resets, which are really clever. If you have a $200/month plan and three banked resets, you're not churning because you will overweight giving up those resets (loss aversion theory).
They've also introduced banked resets, which are really clever. If you have a $200/month plan and three banked resets, you're not churning because you will overweight giving up those resets (loss aversion theory).
I ran out of resets :( hehe I had 3 and used them all
It never really mattered (except when codex was very new). If anything, codex's remote session integration is better, so outside of some "ultracode" orchestration bells/whistles where Claude Code is ahead, I think Codex is a better tool.
Agree, I think there was just a blind study that showed no one could tell the difference even though the users were avid they could
I can't tell the difference between Fable and GPT 5.5. I tried Fable while it was in trial $20 mode, used up my whole quota, and it was great, but as soon as I went back to GPT 5.5, everything was the same.
But what I love about Openai is that they still let you hook OTHER harnesses up to a subscription. My Pi setup has been built up for a few months now into exactly what I want and moving over to CC or even Codex is really annoying.
Caveat: I vibe code in tiny little chunks. I see what I want to do, and exactly how I want it done, then prompt that, refine, what was output, then repeat. I bet Fable is better at building a whole app from a 2-sentence prompt; but that's just not important to me at all.
But what I love about Openai is that they still let you hook OTHER harnesses up to a subscription. My Pi setup has been built up for a few months now into exactly what I want and moving over to CC or even Codex is really annoying.
Caveat: I vibe code in tiny little chunks. I see what I want to do, and exactly how I want it done, then prompt that, refine, what was output, then repeat. I bet Fable is better at building a whole app from a 2-sentence prompt; but that's just not important to me at all.
Same here - gave 5.5 a web design to implement and it sucked. Gave the same to Fable and it still sucked.
did you use Claude design, their tool meant for Web design? because if not then you're the problem .
> does it really matter anymore?
They're different models with different philosophies behind them. This is anecdotal with a user group of 1, but in my experience:
Claude has a stronger personality and is more creative. If you give it vague instructions, it's better at filling in the blanks with reasonable ideas.
GPT-5.5 is better at following instructions. If you know exactly what you want, it will do it without going off the rails. It's also less likely to imply that you're dumb, but I don't really care about that. Some people do.
They're different models with different philosophies behind them. This is anecdotal with a user group of 1, but in my experience:
Claude has a stronger personality and is more creative. If you give it vague instructions, it's better at filling in the blanks with reasonable ideas.
GPT-5.5 is better at following instructions. If you know exactly what you want, it will do it without going off the rails. It's also less likely to imply that you're dumb, but I don't really care about that. Some people do.
I’ve found that Claude is very literal. When I talk to 5.5 it gets what i want it to do, when I talk to Opus 4.8 it does what I say literally and doesn’t get the intent behind it.
This. Claude is very good at instruction and treat my mistake in prompt as law. Gpt is just smarter at understanding my intent.
I wish models called me out more! I can’t count the number of times I’ve had an absolute shit plan, prompted it, and the model built it to a tee. Then I look at the code, and it’s an absolute mess, because it had to make my stupid idea work. Maybe I need a better system prompt.
Personally, I started using openai models to mess with other harnesses. I was pretty oppositional to CC and how they don't let you kinda plug and play freely, or give transparency into -p usage with other harnesses. So i mix and match a bunch of openai and some chinese models im trying out into opencode. I keep hearing codex is great, on the tier of current CC, I've tried it and it just ate my entire 5 hour usage window looping without asking for clarification on something and none of it was usable. that was the only time i tried codex as i could got that same task done with maybe 20% of my window with my existing openai opencode workflow.
I had put a decent amount of effort into setting up that initial codex attempt and it went so poorly that i've been entirely uninterested in trying again. This was maybe a month or so ago, and i know stuff moves fast, but for me, i like the models, dont care for the harness.
I had put a decent amount of effort into setting up that initial codex attempt and it went so poorly that i've been entirely uninterested in trying again. This was maybe a month or so ago, and i know stuff moves fast, but for me, i like the models, dont care for the harness.
Use a harness that doesn't lock you into a moat, like OpenCode.
You can use Codex with any endpoint compatible with OpenAI Response API[1], like llama.cpp.
[1]: https://unsloth.ai/docs/basics/codex
[1]: https://unsloth.ai/docs/basics/codex
FWIW Claude Code works with OpenRouter so you can use any model.
CC system prompt is bloated, use Pi to test Codex instead
Codex is open source and lets you use any model https://learn.chatgpt.com/docs/config-file/config-advanced#o...
Only its CLI is open source. The desktop app is always proprietary.
Codex CLI is open source too. I don't think there is a difference.
Can't use a claude code subscription in another harness though
You can however for now use wrappers which are not harnesses such as T3Code though. They were going to cut under the Programmatic API, but have at least temporarily walked it back.
Don't use providers that don't allow it.
You absolutely can; they are not banning anymore. The bigger problem is that subscription versions of the models are way crappier than when the "same" model is hit via API (Bedrock/Vertex)
You can also make it not count against extra usage.
OpenCode docs show it because Anthropic specifically ambushed them with a PR to remove support so simpletons can't use it easily.
You can also make it not count against extra usage.
OpenCode docs show it because Anthropic specifically ambushed them with a PR to remove support so simpletons can't use it easily.
The opencode docs[0] still say otherwise, do you have a source?
[0] https://opencode.ai/docs/providers/#anthropic
[0] https://opencode.ai/docs/providers/#anthropic
I am curious about the claim that the subscription models are different. Has anyone benchmarked this?
maybe he meant about server-side features? otherwise its dumb. OAI and Anthropic are using azure/aws/gcp...
How does that work? Doesn't that mean Microsoft/Amazon/Google have full de facto access to OAI's and Anthropic's model weights and operational processes?
This is one reason it surprised me that Anthropic decided to run stuff on Musk's hardware. It seems overwhelmingly likely that the new Grok release is informed by what Musk has been able to learn from that relationship.
This is one reason it surprised me that Anthropic decided to run stuff on Musk's hardware. It seems overwhelmingly likely that the new Grok release is informed by what Musk has been able to learn from that relationship.
Do you have a source for that?
It was stated in the PR that remove built-in Claude support.
> Remove anthropic references per legal requests
https://github.com/anomalyco/opencode/pull/18186
> Remove anthropic references per legal requests
https://github.com/anomalyco/opencode/pull/18186
They aren't banning it anymore, they just make it count as "extra usage". e.g. you're paying for every token in addition to your subscription.
Further, the claim that the subscription "version" of the model is worse sounds like bullshit (and the sort of anecdotal nonsense that you see on sites like this). Do you have anything substantiating this?
Further, the claim that the subscription "version" of the model is worse sounds like bullshit (and the sort of anecdotal nonsense that you see on sites like this). Do you have anything substantiating this?
I personally use opencode so I can swap between models and try different options. I'd say I prefer claude (fable and opus 4.8) so far, but curious to see where gpt 5.6 lands.
For personal stuff, I've been pretty happy with chatgpt's $20 plan. I believe it has considerably higher limits than claude's $20 plan, and it's enough for the personal stuff I play with (hermes, and some small coding stuff). Also allows me to keep up to date on openai models.
For personal stuff, I've been pretty happy with chatgpt's $20 plan. I believe it has considerably higher limits than claude's $20 plan, and it's enough for the personal stuff I play with (hermes, and some small coding stuff). Also allows me to keep up to date on openai models.
Not sure about the consensus, but during an entire week I have done every task on my workplace with both Opus 4.8 and GPT 5.5. GPT won hands down. I would even sometimes copy the plans and solutions (using different Git worktrees) from GPT and paste it on Opus and itself would say GPT plans were better. At that point I have migrated. Fable is not enabled in our workspace so I have not tried.
Claude lost my trust around February this year when the plan would say nonsensical things as "delete this method" that was clearly a key method on that part of the codebase.
For personal projects I am using Codex 20$ plan and when that is over I use DeepSeek which is insanely good for the cost.
Claude lost my trust around February this year when the plan would say nonsensical things as "delete this method" that was clearly a key method on that part of the codebase.
For personal projects I am using Codex 20$ plan and when that is over I use DeepSeek which is insanely good for the cost.
To me, the question of switching (and any recommendation) depends highly on the type of work you do as well as your setup in terms of harness and memory, context management, and so on.
I have my context managed in a structured (but nowadays way too big) Obsidian Vault. I also built myself a vector based "vault search" capability and have my harness use this as a tool to find thematically similar things across the different contexts, when needed. I also build a few custom skills and extensions for my harness to be able to do my work.
Talking about harness: I use pi.dev and have taken care of, that i set it up in a way as to easily be able to switch the intelligence layer without loosing context. Yes, there are differences in how well models perform, but if a model refuses a task - like gpt-5.5 not willing to build a downloading tool for Annas Archive - I switch the model to something less finicky.
Thus I was able to switch to gpt based models after about a year with Claude (and having had a Claude Max since the early days it was available).
I played a lot with other models recently, to see how stabl my setup is for switching, should something like Fable happen on a broader scale with the US government. As said, minor changes in tonality, minor issues ith the quality of long text being written by the model, but most of it is actually managed in by the tonality docs, guard rails, coding standards and the likes, I set up over the last 9+ months of intensive work with it (first in Claude Code, then Codex and now as said pi.dev).
So YMMV and it heavily depends on your setup. But I more and more treat those models as interchangable.
I have my context managed in a structured (but nowadays way too big) Obsidian Vault. I also built myself a vector based "vault search" capability and have my harness use this as a tool to find thematically similar things across the different contexts, when needed. I also build a few custom skills and extensions for my harness to be able to do my work.
Talking about harness: I use pi.dev and have taken care of, that i set it up in a way as to easily be able to switch the intelligence layer without loosing context. Yes, there are differences in how well models perform, but if a model refuses a task - like gpt-5.5 not willing to build a downloading tool for Annas Archive - I switch the model to something less finicky.
Thus I was able to switch to gpt based models after about a year with Claude (and having had a Claude Max since the early days it was available).
I played a lot with other models recently, to see how stabl my setup is for switching, should something like Fable happen on a broader scale with the US government. As said, minor changes in tonality, minor issues ith the quality of long text being written by the model, but most of it is actually managed in by the tonality docs, guard rails, coding standards and the likes, I set up over the last 9+ months of intensive work with it (first in Claude Code, then Codex and now as said pi.dev).
So YMMV and it heavily depends on your setup. But I more and more treat those models as interchangable.
> What's the consensus today on codex vs claude code, does it really matter anymore?
Consensus is probably the wrong word for the popular opinions reflected in HN that you might get.
I would recommend that you have 2 of each at all times when it comes to AI so you don't necessarily become overly locked to quirks of one thing. You'll soon realize that things move so fast that you just start internalizing common patterns instead of depending on one specific vendor.
I recommend that you try pi and codex besides claude, to get your own feel for it.
Consensus is probably the wrong word for the popular opinions reflected in HN that you might get.
I would recommend that you have 2 of each at all times when it comes to AI so you don't necessarily become overly locked to quirks of one thing. You'll soon realize that things move so fast that you just start internalizing common patterns instead of depending on one specific vendor.
I recommend that you try pi and codex besides claude, to get your own feel for it.
OT but how are y'all sharing your skills and agents across harnesses?
I have a bunch of Claude Code Plugins and yesterday asked Codex to make them accessible to itself. It wanted to rewrite most of it. I was hoping i could get by with some symlinks or something to avoid drift.
I have a bunch of Claude Code Plugins and yesterday asked Codex to make them accessible to itself. It wanted to rewrite most of it. I was hoping i could get by with some symlinks or something to avoid drift.
More literal, less fluid verbally, harder time understanding nuance, more correct code, fewer bugs. Less pretty UI. I switch back and forth but find I have less 'clean up' work with codex; more upfront communication though to properly specify. High hopes for 5.6!
I spent the last couple days switching because Anthropic keeps locking stuff behind API pricing. OpenAI lets you do anything with your sub right now. I'm building headless and web interfaces around Pi.dev. I had this previously with Claude Code but they are going to lock away all those features. I think the Claude does a better job at being proactive to solving things, but I'm going to keep tweaking my harness to nudge gpt to do more in it's turn. Not sure!
In my experience, for coding Codex is definitely far ahead of Claude Code, even when using Fable 5 as a model.
you have a very strange experience
People work on different things, fail to mention their field of usage in the comments, and then misunderstand the experiences of others who do the same. Repeat ad nauseam.
he's writing novels
I switch between both as my daily drivers.
I do almost all my regular coding tasks with Codex 5.5 on medium. Sometimes for niche edge cases, or when I run out of tokens on my Codex sub, I'll switch to Claude. Some recent examples where Claude was able to solve things Codex couldn't:
- 3D gamedev layout: I asked Codex to render a solar system in a certain camera positioning, saying it needed to fit the planets of the system to the viewport. Codex just couldn't do it, even on high reasoning: Claude Opus did it first attempt.
- Tricky Tiptap image drag-n-drop layout implementation: Codex failed this after numerous iterations. Claude Opus also struggled mightily to get it to work, but I think around 3 attempts it nailed it. Both of them ended up grepping the Tiptap code from node_modules - that's the kind of task it was.
But these are really isolated examples. Across all my projects (I have many; mostly TypeScript, but also things like C#), Codex "Just Works" (tm), with minimal prompting effort from me.
I do almost all my regular coding tasks with Codex 5.5 on medium. Sometimes for niche edge cases, or when I run out of tokens on my Codex sub, I'll switch to Claude. Some recent examples where Claude was able to solve things Codex couldn't:
- 3D gamedev layout: I asked Codex to render a solar system in a certain camera positioning, saying it needed to fit the planets of the system to the viewport. Codex just couldn't do it, even on high reasoning: Claude Opus did it first attempt.
- Tricky Tiptap image drag-n-drop layout implementation: Codex failed this after numerous iterations. Claude Opus also struggled mightily to get it to work, but I think around 3 attempts it nailed it. Both of them ended up grepping the Tiptap code from node_modules - that's the kind of task it was.
But these are really isolated examples. Across all my projects (I have many; mostly TypeScript, but also things like C#), Codex "Just Works" (tm), with minimal prompting effort from me.
> What's the consensus today on codex vs claude code, does it really matter anymore?
I've just switched to Codex from Claude Code. There's some minimal setup in regards to the auto-reviewer if you'd want to preserve a similar functionaly with Claude Code's auto mode and bubblewrap (dev container environment).
Just point Codex at the code base and let it do a migration from Claude Code with a note to keep compatibility with it.
I've just switched to Codex from Claude Code. There's some minimal setup in regards to the auto-reviewer if you'd want to preserve a similar functionaly with Claude Code's auto mode and bubblewrap (dev container environment).
Just point Codex at the code base and let it do a migration from Claude Code with a note to keep compatibility with it.
IMO Codex has been the same rollercoaster ride as Claude. GPT 5.3-codex was incredible for backend/system tasks, GPT5.5 is better all rounder but weaker in some spots. There has also been many weeks when Codex's models were dumb AF. Same rollercoaster ride as anthropic between Opus 4.5 to 4.8...
IMO the two biggest problems not really being answered by both OpenAI and Anthropic are: 1. Why not make specific models good at specific tasks for Codex/Claude Code. Theres a handful of types of work here whereby small good quality models would do better than these generalised all purpose models whereby someone discovers Fable is bad at biology.... 2. Why cant they consistently run these models and keep them performing? Performance of the models seems to directly correlate with amount of compute available, but they dont talk about it...
IMO the two biggest problems not really being answered by both OpenAI and Anthropic are: 1. Why not make specific models good at specific tasks for Codex/Claude Code. Theres a handful of types of work here whereby small good quality models would do better than these generalised all purpose models whereby someone discovers Fable is bad at biology.... 2. Why cant they consistently run these models and keep them performing? Performance of the models seems to directly correlate with amount of compute available, but they dont talk about it...
I'm also a long-time Claude Code user here, though the last 3 weeks I've been doing loops having claude use codex to review until they reach consensus; uses tons of tokens but the result is really good.
I'm trying Codex as my primary the last day or so, because I'm at 98% use and reset in 3 days on Claude. I'm worried about a lot of our skills and CLAUDE.mds and the like getting lost unless I migrate them, but otherwise codex seems to be working great.
I'm trying Codex as my primary the last day or so, because I'm at 98% use and reset in 3 days on Claude. I'm worried about a lot of our skills and CLAUDE.mds and the like getting lost unless I migrate them, but otherwise codex seems to be working great.
Set yourself up to be able to try / switch between models easily. I was a claude only user and just have my user level AGENTS.md for codex and others simply point at my user CLAUDE.md. Have a script that syncs my skills (just directories) between all models. Also, if you want to use /simplify or similar from claude in another model, you can ask claude for the prompt and put that in a skill for the other models.
I sub both codex and claude at 20x. I like opus+fable more than gpt5.5 because it seems gpt tries to finish tasks by leaving any ambiguity unresolved. claude seems better at surfacing open questions.
This is using the same AGENTS.md prompts, which were designed firstly for Claude use, so maybe it's something that could be optimized better if I understood gpt as well?
This is using the same AGENTS.md prompts, which were designed firstly for Claude use, so maybe it's something that could be optimized better if I understood gpt as well?
Is it you have Fable delegate work to Sol? How do you do that? Do you run it in Codex/Desktop app?
no, I didn't get access to sol until a few hours ago. I just have my claude protocol files linked to inside codex. trying Sol this morning for the first time, so I can't really comment on that vs gpt5.5.
However you can do what you are asking "fable--> sol" you need to setup a mcp or have fable run a bash tool, just invoke the `codex.exe` cli tool with whatever cmdline args are needed.
However you can do what you are asking "fable--> sol" you need to setup a mcp or have fable run a bash tool, just invoke the `codex.exe` cli tool with whatever cmdline args are needed.
I run my AI agent as a different user (in addition to using the sandbox functionality provided by cc/codex). It does not seem possible to run the Codex GUI as a different user. I can run the TUI (/Applications/Codex.app/Contents/Resources/codex) but it has the shortcoming that remote control is only available in the GUI.
I installed the Claude Code Codex skill provided by Anthropic and I am having Claude invoke it automatically to review all plans and changes. The nice thing about this is that for an additional $20/month pro plan I can extend the runway for Claude rate limiting and compare frontier model responses. I am looking for more ways now to work in Codex as a subagent that gets used automatically from Claude Code.
I installed the Claude Code Codex skill provided by Anthropic and I am having Claude invoke it automatically to review all plans and changes. The nice thing about this is that for an additional $20/month pro plan I can extend the runway for Claude rate limiting and compare frontier model responses. I am looking for more ways now to work in Codex as a subagent that gets used automatically from Claude Code.
A great thing about codex is that, even if run out of usage, it finishes the task. Claude code will abruptly break the work and leave it there unfinished as soon as it runs out of tokens. Also, antrophic randomly resets the token usage which is annoying when I’m trying to ration them. While openai gives you extra resets that you can apply when you want to
Personally I use Open Code with a copilot sub. Then all models are available in my session with just a /model and /variants command combo. Makes it super low friction to try different models & combos (my favourite right now is DeepSeek V4 Flash for initial PRD then Fable 5 high for implementation).
I had great results combining the two. If you (or your employer) can afford then you can ping-pong the models in the plan phase (not really ping-pong as humans should get a say too) and then let one implement and the other review. I got better results working this way than just to stick to a single model.
I use both. Not because I am cool, but because it is cost effective for personal projects with two $20 / month plans. It is also nice to be able to see what the state of the art is like for both.
Personally, I find it very interchangeable. I open codex --yolo or claude with whatever there yolo flag is (have an alias).
Personally, I find it very interchangeable. I open codex --yolo or claude with whatever there yolo flag is (have an alias).
I use Conductor pretty much exclusively and it makes it incredibly easy to try different models, even within the same workspace - definitely recommend giving it a shot. Whenever I'm forced to use the Claude Code app directly it just seems woefully inadequate compared to Conductor
I consistently have better results with Codex for the work that I do. People have been saying that for six months, but until 5.4 the experience was sufficiently slower that it wasn't worth the switch. Making the switch was frictionless. Give it a try
I use both. Both are great. But in terms of Desktop Apps I think Codex has the better UI. It's more straightforward, just works, and has small conveniences like the open in editor icon.
Claude's very bloated and convoluted by comparison. Maybe you need the bloat (Claude Design), but I prefer the more razor's edge efficiency of Codex.
Model wise, I can't really tell. They all do what I want them to do most of the time and go off the rails occasionally. The question is increasingly becoming who's faster and cheaper and gives me more tokens, not who's better.
Claude's very bloated and convoluted by comparison. Maybe you need the bloat (Claude Design), but I prefer the more razor's edge efficiency of Codex.
Model wise, I can't really tell. They all do what I want them to do most of the time and go off the rails occasionally. The question is increasingly becoming who's faster and cheaper and gives me more tokens, not who's better.
Now we have various Opus+ level models (Opus/Fable, Grok 4.5, GPT 5.6) I prefer to focus on price/speed and harness as models are all generally good enough for coding. (Fable is overkill for 90% of work but is still level above). So I use Grok Build with 4.5 as its VERY fast and cheap, Codex is next best for me with sol/lunar 5.6. and Claude Code Fable for the 10% of tasks that need that level of reasoning. However I find Claude Code harness responsiveness much less than other two (all TUI versions) I wish they would fix this.
Am I missing something or isn't sol/lunar 5.6 only out for like 3 hours? How did you evaluate?
how do you mean, I always use the latest models so evaluating all the time.
If you can afford it and you have something to justify the expense, I would get both. they're interesting to run side by side, you can hand things off from one to the other. Pretty neat. Unfortunately now I just want to have both :(
IMO LMArena is the best benchmark that avoids benchmaxxing
https://arena.ai/leaderboard/agent
5.6 isn’t on there yet but Fable leads by a significant margin atm
https://arena.ai/leaderboard/agent
5.6 isn’t on there yet but Fable leads by a significant margin atm
The results here match up to my real-world experience using these models every day at work and switching between them regularly.
My final answer on this is that we just can't say anything affirmative because all of our projects/codebases are completely different. I've gone back and forth on the "codex vs claude" being better, and while I'm currently of the believe that Claude is superior, I understand that might be the case for _my_ particular set of projects and _my_ personal way of interacting with the model.
Consensus itself does NOT matter, omp is objectively the best harness for power users yet it has 0 hn posts about it, zero.
You're fully free to use and try anything and without caring about what others think is right
You're fully free to use and try anything and without caring about what others think is right
omp is really good.
I have one non technical people in my firm using it. One is using it to assist with editing books, basically using it to gather up manuscripts from e-mail / Google Doc etc. submissions, and then switch models between a cheap one and Opus (for actually analysing the manuscript).
The other non-technical person has done really surprising things with it AI, like a long-running GPT 5.5 Pro chat session which is basically her expense tracker - it has an .xlsx file "carried" in the chat, and she just tells ChatGPT (or scans a receipt) whenever she has a new expense, and then prompts it in natural language when she needs a report. I'm looking forward to seeing what she can do with omp.
I have one non technical people in my firm using it. One is using it to assist with editing books, basically using it to gather up manuscripts from e-mail / Google Doc etc. submissions, and then switch models between a cheap one and Opus (for actually analysing the manuscript).
The other non-technical person has done really surprising things with it AI, like a long-running GPT 5.5 Pro chat session which is basically her expense tracker - it has an .xlsx file "carried" in the chat, and she just tells ChatGPT (or scans a receipt) whenever she has a new expense, and then prompts it in natural language when she needs a report. I'm looking forward to seeing what she can do with omp.
I figured pi itself would be the best harness because it's barebones and you make it what you want. omp is to pi what doom is to emacs is what lazyvim is to neovim.
The fact that I thought that this was amp misspelled until i someone validate omp and the checked myself indicates it's a subjective assertion at best.
How can this be "objective"? Surely its subjective.
I've tried a fuck load of harnesses but keep coming back to Codex as my harness.
I've tried a fuck load of harnesses but keep coming back to Codex as my harness.
OMP feels like the Linux of AI
"objectively the best"?
To quote a friend:
> Well it's objective _to me_
> Well it's objective _to me_
Probably means subjectively according to his own opinion...
omp is amazing. Daily driver for me.
Because it's a bunch of extension on top of on pi
Absolutely. It's the only harness that is actually RSI and not run by idiots.
> omp is objectively the best harness for power users
Care to detail this?
Care to detail this?
omp.sh is unreadable. I've tried understanding what exactly this is and it's just a wall of edgy sounding slop.
I had to switch to Opencode from Claude code because the latter wasn’t supporting GitHub Copilot as model provider.
I didn’t think I could have found a better solution, spawning multiple subagents with different models is such a great thing.
I built in the past very small cli wrappers to call other models; Claude Code often refuses to do that, lies and does the job itself instead of delegating to another provider’s llms.
I didn’t think I could have found a better solution, spawning multiple subagents with different models is such a great thing.
I built in the past very small cli wrappers to call other models; Claude Code often refuses to do that, lies and does the job itself instead of delegating to another provider’s llms.
I used to have the CC $200 plan, and moved to Codex 6 months ago. I have the anthropic $20 plan + API billing for rare use. Use Codex daily.
Not having to deal with Anthropics constantly changing policies, token-gating, and carrot-and-stick marketing helps me to focus on work, rather than dealing with their company problems.
Not having to deal with Anthropics constantly changing policies, token-gating, and carrot-and-stick marketing helps me to focus on work, rather than dealing with their company problems.
I prefer codex for most tasks, but stil use Claude if i need to make something "nice but generic", i.e. a html artefact or touch up of front end code.
I left Claude for Codex months ago. I was an early Claude Code adopter but I have found Codex consistently better since about the February time frame. And far more reliable.
It's more diligent and empirical and results focused, and less creative. It sometimes needs a kick to avoid a Zeno's paradox of incremental steps to get to the goal. But it produces more reliable code with fewer race conditions, unhandled negative cases, etc.
It's also better value from a $$ POV, or at least has been. This fluctuates a bit.
You're also free to use your Codex subscription with other harnesses, like opencode, etc. Unlike Anthropic. Plays better with others.
It's more diligent and empirical and results focused, and less creative. It sometimes needs a kick to avoid a Zeno's paradox of incremental steps to get to the goal. But it produces more reliable code with fewer race conditions, unhandled negative cases, etc.
It's also better value from a $$ POV, or at least has been. This fluctuates a bit.
You're also free to use your Codex subscription with other harnesses, like opencode, etc. Unlike Anthropic. Plays better with others.
The harness is so much better than cc which is a buggy mess. Gpt is also way faster than Claude. I’ve been using gpt for a while now and I know a lot of people that swapped away from Anthropic for multiple reasons. However - fable still seems to be the best coding agent, it’s just slow and the harness sucks. So I still use it in some rare cases like to review codex. I’m hoping 5.6 lets me drop it entirely.
Codex app is a much different experience than CC CLI. I would try it out for a couple days with the new model suite and see what you prefer after that.
My experience is that Codex's auto review is extremely costly, with $20 on both sides, I can run CC with auto mode for longer than with Codex's auto review enabled. Also in my own experience Claude's usage is actually bigger than Codex, but I am not sure if that's due to I stick to 5.5 with Codex while keep Sonnet as the default to orchestrate other models in CC.
I have them talk to each other via tmux to great effect on complex changes. Its great for auditing changes as work is done.
I have found Claude Code to be so much better than other common harnesses that it's kept me solely in the Anthropic ecosystem.
In my opinion Opus is waaayy better in agentic orchestration. It feels like it can natively deal with multiple subagents whereas gpt needs to be taught extensively.
IME it entirely depends on your work. I find myself using both daily for different things.
Codex with GPT 5.5 is much better at general SWE tasks but Claude Code with Opus is far better at complex reasoning tasks like reading and summarizing research papers, replicating experiments, identifying research gaps and proposing interesting follow ups.
Codex with GPT 5.5 is much better at general SWE tasks but Claude Code with Opus is far better at complex reasoning tasks like reading and summarizing research papers, replicating experiments, identifying research gaps and proposing interesting follow ups.
Last time I tested Codex on a cheap plan, it barely lasted an hour? I think this was for the $20 plan. I was afraid to try the more expensive plan after that. Not sure, I might just outright rip my Claude Code bandaid if the current usage quotas do die off after the 17th or whatever date they said they would "return on".
You wouldn't be leaving Claude Code, just trying something new. If you don't like it just resume using Claude.
If you can afford to test it seriously, running both in parallel, it's worth a test to see which you prefer. If you can't, don't bother. You're not likely missing anything since they are close to personal preference with most people I know who have meaningfully tried both preferring Claude
They blocked Claude from being used in a different harness as well squeezed the usage like crazy. Switched to Codex and haven't cared since.
Between the two the biggest difference by far is ... getting your harness / AGENTS.md / skills / tools set up right.
Between the two the biggest difference by far is ... getting your harness / AGENTS.md / skills / tools set up right.
A few less obvious niceties of Codex:
- built-in image generation using your subscription, which can be super handy
- can actually edit Google Docs and Google Sheets (Claude can only create new or sometimes append)
- I get a surprising amount of mileage out of the $20 plan
They both have their places for sure.
- built-in image generation using your subscription, which can be super handy
- can actually edit Google Docs and Google Sheets (Claude can only create new or sometimes append)
- I get a surprising amount of mileage out of the $20 plan
They both have their places for sure.
Don't know about consensus, but I personally still find Opus to be better for sniffing codebase intent and checking things as a whole, while Codex seems more detail-oriented for individual files.
Not sure there's going to be a consensus, but I can tell you that when i have codex review claude-written code, it finds important gaps and fixes. The reverse is also true. Both are powerful, but even better when used in combination
The answer is it depends. Claude's generally better at frontend and debugging tasks, while Codex is stronger at backend features and exploratory work. They have very different coding styles and thus very different strengths.
Any actual data backing this up? Or is this just your personal experience?
Claude Code is not the model, it's the harness. You can use any model you want with Claude Code to varying degrees of success. I use Qwen3.6-27b daily with Claude Code as an example.
It's trivial to try another agent. You can spend $20 for a monthly subscription and ask it to import all your settings from Claude Code.
For me the biggest shift was using Deepseek through an American provider with reasonix as the harness, making cache hits at a rate of practically free.
Which provider do you use, if you don't mind sharing? I tried Digital Ocean (looking for Zero retention and no training), but their context limits are rather small for DeepSeek inference.
Try OpenCode and you can point it at either model
There was just a study showing that when presented blindly no one could tell the difference yet users were avid they could
There _is_ a difference in the way Claude and GPT write. Last Friday I felt Opus was becoming dumb because it was writing like GPT.
Have been long time clauder but honestly codex feels much more liberating. Something you can't buy..
I wish they open source their desktop app and built-in skills one day. That would be a final blow for me.
Edit: Found that their built-in skills are actually open-source: https://github.com/openai/plugins
like others said in the thread: much less drama and i'll add much less attitude from the company and the models, overall i'm having much calmer experience with codex, hope it stays that way
I use both especially for checking each others work. Pretty happy with results
Codex UI is way way way better than Claude Code
- codex UI is much more responsive
- i get feedback about the progress easily
- the tool calls and results are very legible, I can click them and see the progress
- no one talks about this but the tool call and response notification are handled much more elegantly in Codex. In Claude Code, it is handled in a clunky way using loops which always causes some delay
- you can steer the conversation midway in Codex
- /side is underrated (/btw is the equivalent and is much worse in Claude Code)
- I have to admit subagents are handled better in Claude Code
- codex UI is much more responsive
- i get feedback about the progress easily
- the tool calls and results are very legible, I can click them and see the progress
- no one talks about this but the tool call and response notification are handled much more elegantly in Codex. In Claude Code, it is handled in a clunky way using loops which always causes some delay
- you can steer the conversation midway in Codex
- /side is underrated (/btw is the equivalent and is much worse in Claude Code)
- I have to admit subagents are handled better in Claude Code
With the exception of Fable which is going away anyway, Codex is better especially after the last couple Opus releases. It’s also no longer slower than Claude.
You get much more generous usage from the 20x plan.
And you get far better uptime.
If benchmarks and early tester impressions are accurate, you also get access to Fable level capability at greater speed and lower cost (included in subscription).
You get much more generous usage from the 20x plan.
And you get far better uptime.
If benchmarks and early tester impressions are accurate, you also get access to Fable level capability at greater speed and lower cost (included in subscription).
> Fable which is going away anyway
$2 says nah. You can't take Fable away in a week where GPT-5.6 and Grok 4.5 launch, if you want to hold on to customers.
$2 says nah. You can't take Fable away in a week where GPT-5.6 and Grok 4.5 launch, if you want to hold on to customers.
The fact that they already extended subscription Fable once would suggest it won’t be solely locked behind API next week, but at the same time it really does look like they are doing everything they can to avoid serving it continuously at scale.
Knowing Anthropic, this unfortunately might end up meaning a quietly quantized Fable on subscription.
Knowing Anthropic, this unfortunately might end up meaning a quietly quantized Fable on subscription.
Can anyone explain this "quietly quantized" model idea to me from a business perspective?
Coca-Cola doesn't "quietly water down" its product to save a few bucks. They know people will take a sip, say "oh that's not what i wanted", and go buy a Pepsi.
If they serve me a quantized Fable, I'm just going to think Fable sucks and go get my tokens elsewhere. What's the point?
Coca-Cola doesn't "quietly water down" its product to save a few bucks. They know people will take a sip, say "oh that's not what i wanted", and go buy a Pepsi.
If they serve me a quantized Fable, I'm just going to think Fable sucks and go get my tokens elsewhere. What's the point?
> Coca-Cola doesn't "quietly water down" its product to save a few bucks. They know people will take a sip, say "oh that's not what i wanted", and go buy a Pepsi.
Coca-Cola is also mostly measurable and reverse-engineerable.
The Claude models are black boxes, and actively curtail distilling efforts.
Coca-Cola is also mostly measurable and reverse-engineerable.
The Claude models are black boxes, and actively curtail distilling efforts.
How are you going to confidently tell the difference between it being slightly dumber, vs just having bad luck with your recent attempts?
Pepsi may not water down its product, but your local diner may well decide to put a little less ice-cream in its milkshakes if it thinks it can get away with it and most people won't be able to tell.
They are both excellent but excel in different areas. Fable is super super proactive and great for doing a LOT of work with a single prompt, also for creative work.
Codex is more details focused, often catches wonky bugs and correctness issues that Fable misses, feels more terse and less "friendly", more like a stern senior engineer versus a friendly talkative engineer (Claude). Codex is also better if you're already an engineer, Claude is better for non-engineers. I.e. Codex works better if you know exactly what you want and know the right way of explaining it.
Codex is more details focused, often catches wonky bugs and correctness issues that Fable misses, feels more terse and less "friendly", more like a stern senior engineer versus a friendly talkative engineer (Claude). Codex is also better if you're already an engineer, Claude is better for non-engineers. I.e. Codex works better if you know exactly what you want and know the right way of explaining it.
just try it you will back to codex because gpt is trash, I ask for refund under 7 hours
I use Claude for planning, writing CRs, and code review.
Codex writes all of the code, no exceptions.
Works great, especially when you ask Claude to break up large CRs into roughly 10 minutes of Codex work each.
Codex writes all of the code, no exceptions.
Works great, especially when you ask Claude to break up large CRs into roughly 10 minutes of Codex work each.
Same here. I find the design, architecture, system design discussion to be better on Claude, but after Opus 4.6 I switched over to Codex for actual coding and love the results. I use both via the CLI and generally tell Claude to output the result of our decisions as a markdown that will be easy to read and implement by an agentic coding tool. Then I fire up Codex and read said markdown as the input of the session and way to build all the appropriate context needed. I see this as a way to step into letting the agents go run on their own and interact with each other, but I still like to steer so I put these manual steps in the flow. Letting the agents go off on their own and one shot big chunks is not reliable enough yet imo.
I do exactly the opposite.
I think the key is to get two LLMs looking at the same problem.
I use Codex because it's better at the kind of code I need written (math-heavy, 3D geometry code).
But if I was doing mainly UI code, I would do the opposite.
I use Codex because it's better at the kind of code I need written (math-heavy, 3D geometry code).
But if I was doing mainly UI code, I would do the opposite.
In my projects, Claude writes and Codex reviews, and I've had a lot of code I've been very happy with out of that, although as of today, Grok _also_ reviews, and finds interesting new stuff.
> I'm hesitant to leave Claude Code behind for something new.
Codex and Claude Code are not mutually exclusive, you can use both.
Codex and Claude Code are not mutually exclusive, you can use both.
I use both constantly for different things. You don't need to be a one-model Andy
Claude Code is a massively bloated agent harness.
Try Pi: https://pi.dev/
Try Pi: https://pi.dev/
Pi is so “unbloated” that it’s extra effort to use. You can decide how much work to put into it. I get the trade off. But this is a big jump from CC. I’d recommend some middle ground like opencode.
Even simpler, use Cursor with any frontier model. I see others sweat to add enough context to Claude Code while Cursor has a ton of contextual awareness, uses subagents automatically and is significantly faster with no drop off I have found. I'm not sure why devs are so enamored with living in the CLI, but Cursor has one of those too.
Similar to running arch Linux. Many people do need to. But many people just like tinkering. Tinkering can lead to positive outcomes but it’s usually not “doing work”.
It's worth trying out OpenCode, then oh-my-pi, and also the commercial harnesses like Codex. (I haven't yet bothered to try Antigravity and have no interest in Gemini-cli now that it's not available except on expensive plans.)
pi is also worth tinkering with, particularly if you have an eye towards automating some things.
pi is also worth tinkering with, particularly if you have an eye towards automating some things.
what are you building that doesn't work with read,write,edit,bash and skills? Genuinely interested.
better, use oh my pi.
It honestly baffles me how people can ask a question like this and get such a wide spectrum of answers in response. It's all so much based on vibes and anecdotal evidence. I've not really noticed much of a difference in capability since Opus 4.6 and I've used a ton of different models. They all work pretty damn well for me.
I've subscribed to ChatGPT/Codex for over a year and tried a Claude sub twice 1 month each, with a gap of several months in between.
I tried them both side by side, mostly for reviewing existing Godot/GDScript code, or sometimes generating Swift Mac apps, including converting ancient relics I wrote eons ago in Visual Basic on Windows
Codex was consistently better than Claude: https://i.imgur.com/jYawPDY.png
Besides the useless "This is good" findings while reviewing and the excessive "oops you're right" backtracking, Claude's atrocious UX and borderline "spyware" make me never want to try an Anthropic product again for a long long while.
I tried them both side by side, mostly for reviewing existing Godot/GDScript code, or sometimes generating Swift Mac apps, including converting ancient relics I wrote eons ago in Visual Basic on Windows
Codex was consistently better than Claude: https://i.imgur.com/jYawPDY.png
Besides the useless "This is good" findings while reviewing and the excessive "oops you're right" backtracking, Claude's atrocious UX and borderline "spyware" make me never want to try an Anthropic product again for a long long while.
Literally every top model is identical and anyone saying otherwise is engaging in astrology.
The outputs, ui, and overall behavior (tokenization) are not identical.
anybody saying they're identical clearly doesn't use both...
Honestly, I would even push it further. People who would claim that don't use either one.
The codex software is garbage compared to Claude, but open source is the future, so you should at least switch.
It's not clear replies to this thread aren't openAI employees or incentivized influencers, but every benchmark has gpt-5.5 underperforming opus 4.8, sometimes by as much as 10%.
Can they all be wrong/paid-off?
Can they all be wrong/paid-off?
Is there a known headcount for OpenAI influences v Anthropic influencers?
On threads like this, this site seems to be made of nothing but boosters for one or the other, with their emphatic professions of faith, all based on inscrutable, unverifiable inner experience. When no one bothers even to reflect what conditions a proper assertion would require, the discourse is pure faith propositions.
On threads like this, this site seems to be made of nothing but boosters for one or the other, with their emphatic professions of faith, all based on inscrutable, unverifiable inner experience. When no one bothers even to reflect what conditions a proper assertion would require, the discourse is pure faith propositions.
i’m not employed by either, but ideologically might be biased towards Anthropic. I find codex is absolutely better for intelligence/$. Fable is likely better, but codex honestly feels like a utility in the posix sense in that it is very fast/extensible/simple and the model seems less opinionated and more willing to just work directly towards your goals.
I love testing the new models by asking them to code a toy RTS game. Here's what Terra did: https://senko.net/vibecode-bench/2026/rts-gpt-5.6-terra.html (one try, in codex app, xhigh effort)
Comparing this to other models, I find it similar to GPT-5.5 and a bit behind Sonnet 5. You can see how other models fared here: https://senko.net/vibecode-bench/ (you can also fetch the prompt and the the 5.6 Terra resulting code on from that page).
I don't have access to Sol yet (on a Plus sub, which should get it according to what I've read), so can't do the more interesting test. I'll update the above page as soon as I get access - hopefully soon.
Comparing this to other models, I find it similar to GPT-5.5 and a bit behind Sonnet 5. You can see how other models fared here: https://senko.net/vibecode-bench/ (you can also fetch the prompt and the the 5.6 Terra resulting code on from that page).
I don't have access to Sol yet (on a Plus sub, which should get it according to what I've read), so can't do the more interesting test. I'll update the above page as soon as I get access - hopefully soon.
this is so cool: it's playable (even though super boring since there are no enemies) and you can feel that a few iterations would make it very usable.
Which model is the best at the moment, for this kind of stuff, in your experience?
Which model is the best at the moment, for this kind of stuff, in your experience?
I'd say Fable 5: https://senko.net/vibecode-bench/2026/rts-fable-5.html
It even has enemies! (I'm not too mad about it not following my instructions because it can be fun to play :) And I generated that from Claude Code on my phone.
Sonnet 5 also produced a pretty nice version. You can see all of them here: https://senko.net/vibecode-bench/
It even has enemies! (I'm not too mad about it not following my instructions because it can be fun to play :) And I generated that from Claude Code on my phone.
Sonnet 5 also produced a pretty nice version. You can see all of them here: https://senko.net/vibecode-bench/
> I'm not too mad about it not following my instructions
I'm a bit concerned about this - starting with GPT5, AI labs started doing this 'complete app from a prompt' sizzle demos. When I started working with GPT5 - which was supposed to be a qualitative jump, just like Fable is now, I tried to do a frontend, and discovered that it gave me a CSS-animated purple-blue interface with embossed buttons, gradient backgrounds and dropshadows.
It looked very cool, but it was a bit overwhelming (also broken), and I was really looking for a pedestrian Bootstrap job.
It required not inconsiderable amount of wrangling for GPT5 to stop doing this. So I don't really like the idea that these models have tons of implicit and hidden behavior, to 'soup up' pedestrian prompts.
I'm a bit concerned about this - starting with GPT5, AI labs started doing this 'complete app from a prompt' sizzle demos. When I started working with GPT5 - which was supposed to be a qualitative jump, just like Fable is now, I tried to do a frontend, and discovered that it gave me a CSS-animated purple-blue interface with embossed buttons, gradient backgrounds and dropshadows.
It looked very cool, but it was a bit overwhelming (also broken), and I was really looking for a pedestrian Bootstrap job.
It required not inconsiderable amount of wrangling for GPT5 to stop doing this. So I don't really like the idea that these models have tons of implicit and hidden behavior, to 'soup up' pedestrian prompts.
I've been wanting to make an RTS but found it a bit daunting. I thought this might be a bit challenging for LLMs too but I guess not! This is nice to see.
Very neat! Can't wait for the Sol version!
Here it is: https://senko.net/vibecode-bench/2026/rts-gpt-5.6-sol.html
Clearly much better than the Terra version. I'd say its on par with Fable, and the observed differencies are more due to random luck and open-ended prompt, rather than model capability. (Edit: after some more testing, perhaps not on par - somewhere between Opus and Fable, is a better description).
Fable did better pathfinding and has more terrain variety, visually the map looks better, especially soft edges of the fog of war. And the enemies.
Sol took more care with tiny ux details, added help, and more building varieties.
Clearly much better than the Terra version. I'd say its on par with Fable, and the observed differencies are more due to random luck and open-ended prompt, rather than model capability. (Edit: after some more testing, perhaps not on par - somewhere between Opus and Fable, is a better description).
Fable did better pathfinding and has more terrain variety, visually the map looks better, especially soft edges of the fog of war. And the enemies.
Sol took more care with tiny ux details, added help, and more building varieties.
Could we see the prompts, though?
Here's the prompt I used:
> Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).
> Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).
I did a quick comparison of models a couple of months ago by giving it a MYTXTADV.BAS file and give them all the same prompt to create a sprite-based version of a text adventure game I wrote in Basic over 30 years ago.
It was interesting to see where the approaches were similar and where they diverged.
It was interesting to see where the approaches were similar and where they diverged.
For local coding agents, I test them by asking them to recreate the BASIC game Taipan (from 1979) in Python. So far one got close, making a terminal version with ASCII menus ala Drug Wars for the TI-83.
So the measure of a model is how well they can recreate something they easily have thousands of examples of in their training data. There's probably a better base RTS on github somewhere for free.
Well, it is a silly test, not a scientific benchmark.
However, I would say it is a measure (not the measure). If you look at the entries, there's a lot of variation - definitely not something they memorized outright.
And the test itself is deceptively simple. You need to do canvas rendering, there's pathfinding, command queueing, terrain generation, etc. There are some subtle click handler bugs (various LLMs often stumble on those). And I ask the model to do it all in one file, further increasing the complexity of the task.
And the result is something that you can instantly evaluate. And if the result is any good, even play! So yeah, I think it's a fair test.
I'm sure it'll get saturated at some point. Actually I started with Minesweeper and switched to RTS last December, because Minesweeper was being saturated. I'm expecting (hoping?) the RTS test will last until the end of this year...
However, I would say it is a measure (not the measure). If you look at the entries, there's a lot of variation - definitely not something they memorized outright.
And the test itself is deceptively simple. You need to do canvas rendering, there's pathfinding, command queueing, terrain generation, etc. There are some subtle click handler bugs (various LLMs often stumble on those). And I ask the model to do it all in one file, further increasing the complexity of the task.
And the result is something that you can instantly evaluate. And if the result is any good, even play! So yeah, I think it's a fair test.
I'm sure it'll get saturated at some point. Actually I started with Minesweeper and switched to RTS last December, because Minesweeper was being saturated. I'm expecting (hoping?) the RTS test will last until the end of this year...
Ask it to generate a completely bananas game idea and it will easily do it. Prototyping any type of game with simple graphics has been solved since many generations of models back. Claiming it's because it has that game in its training data is nonsense.
The 5.6 model article for this post has three examples of little web games.
There are plenty of little js web games anyway. The point isn't to make an actual game, it's to show coding ability, design and taste in a way that's more assessable than reading a codebase.
There are plenty of little js web games anyway. The point isn't to make an actual game, it's to show coding ability, design and taste in a way that's more assessable than reading a codebase.
The goalpost velocity is approaching light speed…
>There's probably a better base RTS on github somewhere for free.
I... I think you are missing the point.
I... I think you are missing the point.
sort of looks like it was made with mobile in mind but I can't get it to work.
I really wish there was just an easy guide on when to use Sol vs Terra vs Luna, and it just moves further into confusing territory when it comes to naming.
The naming convention is especially difficult to decipher depending on what your native language is. Of course a latin language speaker might be able to easily determine oh yeah each one is slightly bigger than the other but I still think it borderlines too confusing.
That aside all the numbers look amazing, and I'll be happy to probably main this alongside grok-4.5 for a while comparing the two on price and efficiency.
I vastly prefer the direction that OpenAI seems to be going with token efficiency and performance compared to Anthropic who seems to be moving towards a world where you just token-max as much as possible ignoring any and all costs.
The naming convention is especially difficult to decipher depending on what your native language is. Of course a latin language speaker might be able to easily determine oh yeah each one is slightly bigger than the other but I still think it borderlines too confusing.
That aside all the numbers look amazing, and I'll be happy to probably main this alongside grok-4.5 for a while comparing the two on price and efficiency.
I vastly prefer the direction that OpenAI seems to be going with token efficiency and performance compared to Anthropic who seems to be moving towards a world where you just token-max as much as possible ignoring any and all costs.
I use the strongest model (5.5 now 5.6 sol) on the highest reasoning effort with /fast for everything. With a $200 pro sub I can't even use my weekly limit. And it's faster than using a weaker model that makes more mistakes which I have to waste time fixing.
Same, I am actually able to reach my weekly limit, but when I start going below 30%, I switch to normal speed, and that usually gets me through the week.
I absolutely save money and time by constantly using everything at the highest reasoning. I guess my use case and needs are different from others, but I really don’t understand how it can be true when people say they don’t need the highest reasoning and best model. Every time I drop down, things are missed, code gets unnecessarily bloated, more mistakes, and more iterations to solve the same problem. I think it might be because I’m spending a lot of time in a legacy system that I’m trying to clean up, and given the messiness, one needs all the reasoning available to decode what the hell is going on in there.
I absolutely save money and time by constantly using everything at the highest reasoning. I guess my use case and needs are different from others, but I really don’t understand how it can be true when people say they don’t need the highest reasoning and best model. Every time I drop down, things are missed, code gets unnecessarily bloated, more mistakes, and more iterations to solve the same problem. I think it might be because I’m spending a lot of time in a legacy system that I’m trying to clean up, and given the messiness, one needs all the reasoning available to decode what the hell is going on in there.
I have found that higher reasoning doesn't always produce better results. Models often tend to overthink and get themselves in a bind. And tasks just take longer for no reason.
I used to have the same experience until 5.6 sol xhigh. I have instructions in my code review skill and agents.md to encourage parallelism including multiple agents as long as quality isn’t impacted. I additionally instruct codex to not use less capable agents because at least with 5.5 this would seriously increase slop. Maybe sol is smarter about delegation. Hopefully because I’ll have to slow down or hopefully get approval for extra use credits. Now’s a great time for a limit reset if anyone from open ai is reading :).
My guess is that it's the same for Haiku/Sonnet/Opus: Biggest model for architecture and high level planning and technically challenging problems, medium model for simple implementation tasks, small model is for nothing
small models are good for "finding stuff" and "summarizing" in support of the large models.
It's just how LLMs work - these are three completely separate models trained in parallel, with different numbers of parameters and using different amounts of compute.
They could hide this behind a harness that picks the correct model for you, but devs don't seem to like that.
There's also the 'effort' slider, which I guess how many experts in the MoE are evaluated and how long reasoning chains are allowed to go on, which is the 'smooth' scaling you are thinking of.
They could hide this behind a harness that picks the correct model for you, but devs don't seem to like that.
There's also the 'effort' slider, which I guess how many experts in the MoE are evaluated and how long reasoning chains are allowed to go on, which is the 'smooth' scaling you are thinking of.
I love how all the replies to this comment recommend completely different strategies for deciding which model to use.
> I really wish there was just an easy guide on when to use Sol vs Terra vs Luna
Their dev guide has the following:
> Use gpt-5.6-sol for frontier capability, gpt-5.6-terra for a balance of intelligence and cost, or gpt-5.6-luna for efficient, high-volume workloads. The gpt-5.6 alias routes requests to gpt-5.6-sol
https://developers.openai.com/api/docs/guides/latest-model#u...
Their dev guide has the following:
> Use gpt-5.6-sol for frontier capability, gpt-5.6-terra for a balance of intelligence and cost, or gpt-5.6-luna for efficient, high-volume workloads. The gpt-5.6 alias routes requests to gpt-5.6-sol
https://developers.openai.com/api/docs/guides/latest-model#u...
In my tests, in almost all cases, using Sol on (low) reasoning is the best option intelligence/price-wise.
Luna is good too, for classification tasks or any pre-processing task that is not critical
Luna is good too, for classification tasks or any pre-processing task that is not critical
the size comparison didnt occur to me. I assumed the names were just random nice sounding focus-groupped marketing names.
Use Luna. It's more performant than 5.5 and it's cheap. Hopefully it's cheap because it's more environmentally friendly than the bigger models. So you're doing a good thing. If it's a smaller model it may even be faster, but I haven't looked into it yet.
isnt native english speaker
"i really wish this thing in my non native language was easier to decipher"
huh? if you dont know the words then read them in your native language. Sol/Terra/Luna are immediately unambiguous to an english speaker with any sense.
"i really wish this thing in my non native language was easier to decipher"
huh? if you dont know the words then read them in your native language. Sol/Terra/Luna are immediately unambiguous to an english speaker with any sense.
Why would you need a guide for that now? We long had to pick different models (and thinking levels) by task and feel.
Previously it was much more obvious which model to reach for depending on your use case because they had the mini and nano naming conventions.
Getting rid of that seems like a step back. Just a personal nit though.
I've seen buzz about this elsewhere as well but to me effort levels seem more like spend limits disguised with another word. I don't think they should even exist.
Getting rid of that seems like a step back. Just a personal nit though.
I've seen buzz about this elsewhere as well but to me effort levels seem more like spend limits disguised with another word. I don't think they should even exist.
My guide was to pick the best model on "High" for 99% of tasks.
The naming convention is bizarre and doesn't really mean anything to normies. Trying to pick between "Sol" and "Terra" is like asking the average person if they want the Max or the Ultra chip.
Bizarre? The size of the model is in the name. Sun, earth, and moon don't mean anything?
> I really wish there was just an easy guide on when to use Sol vs Terra vs Luna
Terra when you need to get shit done here on Earth, Luna for moonshots, and SOl for when you want to launch something into the sun..
..right?
Terra when you need to get shit done here on Earth, Luna for moonshots, and SOl for when you want to launch something into the sun..
..right?
You don’t know what sol means? You don’t understand the difference in sizes between Terra and sol? I’m genuinely asking.
That isn't what "genuinely asking" looks like, you're criticizing using "questions" as cover. It isn't subtle, nor is it constructive.
I agree with them, Sol, Terra, and Luna are confusing names. They mean the same thing as GPT-5.6-Max, GPT-5.6-Plus, and GPT-5.6-Fast but require base knowledge for an analogy.
It feels like it was adding by the marketing department.
I agree with them, Sol, Terra, and Luna are confusing names. They mean the same thing as GPT-5.6-Max, GPT-5.6-Plus, and GPT-5.6-Fast but require base knowledge for an analogy.
It feels like it was adding by the marketing department.
Surely it's size based: Sun (Sol) > Earth (Terra) > Moon (Luna)
Similar to Anthropic's size/length based naming: Opus > Sonnet > Haiku
These names seem easy to understand to me, and much clearer than suffixes like -max and -plus.
Similar to Anthropic's size/length based naming: Opus > Sonnet > Haiku
These names seem easy to understand to me, and much clearer than suffixes like -max and -plus.
I'd agree it is similar to Anthropic's naming scheme, which I'd argue shares the same problems as this. It improves marketability/googlability, but decreases actual comprehension.
You don't actually explain why or how these names are "easy to understand" just state that they simply are. That's great; to me, they aren't obvious or intuitive at all. May have well just start randomly pointing at dictionary words.
You don't actually explain why or how these names are "easy to understand" just state that they simply are. That's great; to me, they aren't obvious or intuitive at all. May have well just start randomly pointing at dictionary words.
I collect roman coins, with latin legends, so the sun/earth/moon references jumped out at me, and partly based on the opus/sonnet/haiku precedent I assumed that these names were referring to different model sizes/prices in a way that mapped to the names (Sun > Earth > Moon).
I'll admit though that until recently I never really thought about Anthropic's naming scheme as having meaning (an Opus being longer than a Sonnet, being longer than a Haiku).
I'll admit though that until recently I never really thought about Anthropic's naming scheme as having meaning (an Opus being longer than a Sonnet, being longer than a Haiku).
It's just the latin tripping people up. If they had named them sun/earth/moon it would be clearer for some.
Or something like dog/puppy/sperm.
Or something like dog/puppy/sperm.
This demonstrates the problem with an education that has no emphasis on the liberal arts, such as critical thinking. No justification is needed for why and how these names are easy to understand.
>They mean the same thing as GPT-5.6-Max, GPT-5.6-Plus, and GPT-5.6-Fast but require base knowledge for an analogy.
But do they though? When do you use GPT-5.6-Max-Low vs. GPT-5.6-Plus High? Or GPT-5.6-Fast-Xhigh? What's the Pareto optimal choice (outcome and price)? According to the benches it seems to bop around and the even if the benches are accurate the best choice isn't always consistent.
But do they though? When do you use GPT-5.6-Max-Low vs. GPT-5.6-Plus High? Or GPT-5.6-Fast-Xhigh? What's the Pareto optimal choice (outcome and price)? According to the benches it seems to bop around and the even if the benches are accurate the best choice isn't always consistent.
> When do you use GPT-5.6-Max-Low vs. GPT-5.6-Plus High?
You don't, because that isn't something I proposed using for model naming.
I called them GPT-5.6-Max, GPT-5.6-Plus, and GPT-5.6-Fast. Reasoning levels are distinct from the model design itself, and the UI makes that clear.
Plus, using that same flawed argument this would be called GPT-5.6-Sol-Low or GPT-5.6-Luna-High which also makes no sense/is confusing. So that argument applies (or more accurately doesn't), no matter the model names.
You don't, because that isn't something I proposed using for model naming.
I called them GPT-5.6-Max, GPT-5.6-Plus, and GPT-5.6-Fast. Reasoning levels are distinct from the model design itself, and the UI makes that clear.
Plus, using that same flawed argument this would be called GPT-5.6-Sol-Low or GPT-5.6-Luna-High which also makes no sense/is confusing. So that argument applies (or more accurately doesn't), no matter the model names.
Your names are not good because “Fast” is not a descriptor of model size and overlaps with fast/ultrafast inference. And “Plus” collides with the ChatGPT subscription plan. Point being, naming is hard.
I do know what Sol/Terra/Luna mean, but was also confused for a second on the hierarchy. After doing a bit or research it dawned on me that they are arranged in the order of the sizes of the celestial objects but it somehow wasn't immediately obvious to me from the start.
Anthropic ships models with a helpful one-liner tag that makes the model hierarchy obvious. I think it wouldn't hurt if OpenAI did the same.
Anthropic ships models with a helpful one-liner tag that makes the model hierarchy obvious. I think it wouldn't hurt if OpenAI did the same.
Sure—so, is Sol 109.2x better than Terra? Or 1.304x10^6 better?
Did you not read the second sentence? Obviously I know what sol is given my first language being Spanish. I'm just speaking in a general sense that it can be confusing for others.
I already know plenty who had no clue what the difference between Terra and Luna would be.
I already know plenty who had no clue what the difference between Terra and Luna would be.
My first instinct was Sol > Luna > Terra, since Sol is the farthest away, then Luna, and Terra is the closest. Size was not my first instinct. Or should Terra be the best model because its closest to people, then Luna because there have been people on it, then Sol be the worst because no human has been there?
The naming scheme is too "clever."
I've been testing Sol/Terra/Luna now since yesterday, running complex evals on all of them and I feel a bit... mixed on how they perform.
The eval is an agent that runs a set of tools and a prompt we can tune separately for different models. The OpenAI version of the prompt was specifically tuned based on their guide[0]. Then we let Opus to run another agent that acts as a user, trying to solve a problem (anonymized and taken from production). The problem is complex and we don't expect it to be solved by these agents, but we measure how the agents operate when faced with a vague problem:
- Opus 4.8 and GLM 5.2 both identified a constraint sooner and stopped so the user can fix an issue first that the agent cannot solve.
- Sol tried hard to solve the issue with different tools, burning tokens, until finally reached to the same conclusion with Opus and GLM. It was two times more expensive compared to Opus and six times more expensive to GLM for this task.
- Terra went even further and started calling tools that would not solve the issue, burning tokens and failing.
- Luna repeated the same failing tool call until it hit the round limit, and burned more money than Opus.
I'm kind of puzzled with the new GPT. Like, yes Sol is OK for programming, but I was expecting to get a cheap agentic model for non-programming tasks, one that can detect if things go awry and correct. Terra is too expensive and Luna not really fit for the task. Sonnet 5 is a bit better but more expensive than Opus 4.8, which is still the best in my evals. GLM 5.2 is extremely good if you can define the task and the tools clearly for it, and costs pennies!
[0] https://developers.openai.com/api/docs/guides/latest-model
The eval is an agent that runs a set of tools and a prompt we can tune separately for different models. The OpenAI version of the prompt was specifically tuned based on their guide[0]. Then we let Opus to run another agent that acts as a user, trying to solve a problem (anonymized and taken from production). The problem is complex and we don't expect it to be solved by these agents, but we measure how the agents operate when faced with a vague problem:
- Opus 4.8 and GLM 5.2 both identified a constraint sooner and stopped so the user can fix an issue first that the agent cannot solve.
- Sol tried hard to solve the issue with different tools, burning tokens, until finally reached to the same conclusion with Opus and GLM. It was two times more expensive compared to Opus and six times more expensive to GLM for this task.
- Terra went even further and started calling tools that would not solve the issue, burning tokens and failing.
- Luna repeated the same failing tool call until it hit the round limit, and burned more money than Opus.
I'm kind of puzzled with the new GPT. Like, yes Sol is OK for programming, but I was expecting to get a cheap agentic model for non-programming tasks, one that can detect if things go awry and correct. Terra is too expensive and Luna not really fit for the task. Sonnet 5 is a bit better but more expensive than Opus 4.8, which is still the best in my evals. GLM 5.2 is extremely good if you can define the task and the tools clearly for it, and costs pennies!
[0] https://developers.openai.com/api/docs/guides/latest-model
I thought I was an OpenAI fanboy, but version 5.6 isn’t for me. Sol Ultra just keeps working and checking, and working and checking again, but it can’t even correct minor errors that aren’t a problem for 5.5 xhigh. I’ve rolled Codex back to 5.5 for now.
The problem is they nerfed 5.5 about a month ago. The change was immediately visible to me: context compacting started to happen about 3x as frequently.
I think 5.6 is still not even close to 5.5 xhigh pre-nerf.
I think 5.6 is still not even close to 5.5 xhigh pre-nerf.
Yes, this is exactly why I said yesterday that OpenAI does benchmaxxing, seemingly quite a bit [1]. I got a flurry of downvotes for it at first, but I think people came around to it once they tried the model like you did.
Ultimately I think the issue is that OpenAI is under tremendous pressure to perform, but GPT-6 is not ready yet, so they had to push GPT-5 to its limits, and the only way they could do it was with really heavy RLHF, which has its shortcomings. Like, it is super obvious that Sol, Terra and Luna are all heavily biased towards working on a problem relentlessly because that's what their reward functions emphasized. That pushes up their scores in some benchmarks but does not translate to actual intelligence and capability.
[1]https://news.ycombinator.com/item?id=48849454
Ultimately I think the issue is that OpenAI is under tremendous pressure to perform, but GPT-6 is not ready yet, so they had to push GPT-5 to its limits, and the only way they could do it was with really heavy RLHF, which has its shortcomings. Like, it is super obvious that Sol, Terra and Luna are all heavily biased towards working on a problem relentlessly because that's what their reward functions emphasized. That pushes up their scores in some benchmarks but does not translate to actual intelligence and capability.
[1]https://news.ycombinator.com/item?id=48849454
For me the weirdest is Luna. It costs the same as GLM to solve a task with it, but it just calls the same failing tool over and over again until we cut it out.
Now if you look at where GLM stands, or even DeepSeek v4 Flash, things get really interesting for what they provide.
US labs completely miss a cheap model that can solve problems for 95% of the people. Gemini 3.5 Flash could've been it if it didn't burn so many tokens.
Now if you look at where GLM stands, or even DeepSeek v4 Flash, things get really interesting for what they provide.
US labs completely miss a cheap model that can solve problems for 95% of the people. Gemini 3.5 Flash could've been it if it didn't burn so many tokens.
I just watched the video on their launch page and I am really not sure how I feel about it. On one side, it's cool that these people get to start businesses and stuff using ChatGPT (assuming these are true stories), but how much of the business is really them? And how much does this business rely on a chat bot always being present as a kind of know it all employee? Maybe I'm just being naive or old fashioned (haven't really used AI much), but seeing these two people who started a cereal business for example talking to their laptop as if they're talking to a human advisor makes me feel, I don't know, I find it creepy.
By the way, this isn't about their 5.6 version in particular I guess, it's just the first time I've looked at one of their videos.
By the way, this isn't about their 5.6 version in particular I guess, it's just the first time I've looked at one of their videos.
You could say the exact same thing about human businesses.
How many ancient Roman businesses are around today?
How many ancient Roman businesses are around today?
I would add to this that, to me and to many friends in their 30-40s, using AI models to achieve something we used our brains to achieve feels... empty? wrong? soulless?
Sure, a lot of menial work can be relegated to the models, and it's ok, most of the time, but you finish a day of work with the inability to shake a precise new feeling: that you haven't really achieved something, even if you shipped more than on an average day in the past.
It's frankly depressing, and it's even more depressing thinking that most people seem to absolutely disregard this feeling completely.
I also feel this, and it also troubles me. Unfortunately, we may be in the minority on this site.
What the other replies seem to overlook is that it fundamentally changes the nature of the work - it’s not the next step in the evolution after an IDE, it is closer to an automated tractor, and yes that does make the farmer’s work trivial. Pressing a button and having the field get plowed is a very different experience than manually plowing a field yourself. I can no longer in good conscience say “I plowed that field”, because all I did was press a button. The tractor plowed that field.
Most people on this site seem to feel comfortable claiming “I plowed that field” after they pressed the button that started the automated tractor. Okay, you had the idea to press the button - but you didn’t actually do the work, you delegated it. Same end result, but a different lived experience. Now would I rather actually plow a field by hand or simply press a button? You can probably guess - but the two actions do differ greatly in the experience they bring me, and the feelings of satisfaction.
What the other replies seem to overlook is that it fundamentally changes the nature of the work - it’s not the next step in the evolution after an IDE, it is closer to an automated tractor, and yes that does make the farmer’s work trivial. Pressing a button and having the field get plowed is a very different experience than manually plowing a field yourself. I can no longer in good conscience say “I plowed that field”, because all I did was press a button. The tractor plowed that field.
Most people on this site seem to feel comfortable claiming “I plowed that field” after they pressed the button that started the automated tractor. Okay, you had the idea to press the button - but you didn’t actually do the work, you delegated it. Same end result, but a different lived experience. Now would I rather actually plow a field by hand or simply press a button? You can probably guess - but the two actions do differ greatly in the experience they bring me, and the feelings of satisfaction.
I am on the side that says the farmer didn't plow the field. I guess you could say he took responsibility for it, which isn't the same thing. I guess a counter argument that some would make is that the farmer didn't plow the field if they didn't pull the plow themselves (or to a silly level: if they didn't plow the field with their bare hands). They just pressed the right pedals while sitting in the tractor. I am not sure I have a good argument either way, but a farmer plowing a field with a tractor is physically involved during the whole process, and it feels more fair for them to say that they plowed the field, for the simple fact they weren't doing anything else during the process other than constantly controlling the plowing machinery.
What if the farmer built/tested/refined the pathing/automation around the tractor? I mean it probably doesn't ship knowing the most optimal way to plow the layout of his fields. There's still some knowledge/skill required in optimizing that portion.
The argument does eventually degrade over time, though. For example, in the future the farmer uses satellite imagery fed into an advanced AI to build the most optimal route. So yeah, eventually the farmer loses all utility I suppose beyond being a land owner (until AI owns land lol).
The argument does eventually degrade over time, though. For example, in the future the farmer uses satellite imagery fed into an advanced AI to build the most optimal route. So yeah, eventually the farmer loses all utility I suppose beyond being a land owner (until AI owns land lol).
I don’t get this feeling, I feel like I’ve achieved so much more than I have ever achieved, everything is polished and I’m happy with it.
I’m still developing, I’m just doing more than I ever did by directing Codex.
The way I see it is the same as I saw the leap from writing code in a text editor, to using an ide with intellisense, to using the jetbrains ide’s, to using mcp’s, to now directing AI - at all of those steps I wrote code, each step less and less but still it has the same output which is it is my work - even writing in a text editor I wrote less Java (until enterprise architects got involved :) ) than C++, and than assembly.
I’m still developing, I’m just doing more than I ever did by directing Codex.
The way I see it is the same as I saw the leap from writing code in a text editor, to using an ide with intellisense, to using the jetbrains ide’s, to using mcp’s, to now directing AI - at all of those steps I wrote code, each step less and less but still it has the same output which is it is my work - even writing in a text editor I wrote less Java (until enterprise architects got involved :) ) than C++, and than assembly.
You don't get it because for you destination is more important than the journey. And it is fine. Probably.
For me and (likely) for OP as well it is the opposite. Result is meaningless without the process. I can't take fruits of my labor (whichever labor it can be) to the grave, and if you remove the process of growing and picking said fruits, then where is "you" in that? Did you really "achieved" anything? Or whatever you wished for just magically appeared in front of you with little effort from your side?
Where is the fun of renting a helicopter that would carefully put you onto mountain's summit and pick you up 5 minutes later?
For me and (likely) for OP as well it is the opposite. Result is meaningless without the process. I can't take fruits of my labor (whichever labor it can be) to the grave, and if you remove the process of growing and picking said fruits, then where is "you" in that? Did you really "achieved" anything? Or whatever you wished for just magically appeared in front of you with little effort from your side?
Where is the fun of renting a helicopter that would carefully put you onto mountain's summit and pick you up 5 minutes later?
With little effort?
I am working tirelessly and often long nights, on top of a day job.
My effort has shifted to QA testing, reviewing UI designs, and delegating the agents on the implementation.
I will consider it an achievement if I manage to publish a successful app.
Where is the "me" in that? I am guiding the design of every screen and feature the way I would like it to be.
On top of that I make technical decisions on how it is implemented.
Apart from the loads of QA work I will have to handle the business side as well.
As of now it's hardly as trivial and effortless as some make it out to be.
Yes, I no longer write the code, and sometimes it feels frustrating that any teenager without experience could perhaps build a similarly good app soon.
Overall I'm still happy I can now build much larger and better apps and realistically publish them in my free time, for a chance to make serious money.
I am working tirelessly and often long nights, on top of a day job.
My effort has shifted to QA testing, reviewing UI designs, and delegating the agents on the implementation.
I will consider it an achievement if I manage to publish a successful app.
Where is the "me" in that? I am guiding the design of every screen and feature the way I would like it to be.
On top of that I make technical decisions on how it is implemented.
Apart from the loads of QA work I will have to handle the business side as well.
As of now it's hardly as trivial and effortless as some make it out to be.
Yes, I no longer write the code, and sometimes it feels frustrating that any teenager without experience could perhaps build a similarly good app soon.
Overall I'm still happy I can now build much larger and better apps and realistically publish them in my free time, for a chance to make serious money.
>Where is the "me" in that? I am guiding the design of every screen and feature the way I would like it to be.
I disagree. I think you're actually giving up so many little decisions. You are delegating decisions to agents all the time. In place of your slow but still personal decisions, you are ok with decisions that might be similar to what the LLM believes to be the best average, or the best solution based on what limited experience of the world they were trained on. The Ai devil for me is in these details.
That is probably fine for a lot of use-cases, but it's still removing your own agency from the process itself willfully, and yet still taking all of the merit. And to me that makes the final thing less of a byproduct of you and your experiences.
I am not black and white on this, and there are different degrees to the issue. But I just cannot accept this approach that trades the nature of the output for the quantity of it.
And frankly, a lot of other people feel the same. Check the data on the explosion of apps and how little they are maintained or picked up by final users.
I disagree. I think you're actually giving up so many little decisions. You are delegating decisions to agents all the time. In place of your slow but still personal decisions, you are ok with decisions that might be similar to what the LLM believes to be the best average, or the best solution based on what limited experience of the world they were trained on. The Ai devil for me is in these details.
That is probably fine for a lot of use-cases, but it's still removing your own agency from the process itself willfully, and yet still taking all of the merit. And to me that makes the final thing less of a byproduct of you and your experiences.
I am not black and white on this, and there are different degrees to the issue. But I just cannot accept this approach that trades the nature of the output for the quantity of it.
And frankly, a lot of other people feel the same. Check the data on the explosion of apps and how little they are maintained or picked up by final users.
I expect those are low quality apps churned out with minimal effort.
For the design of my app, I try to imitate the UX of Apple's first party apps.
I checked the competitors, and what is in the AppStore looks like it was built in 2015, and only updated to add Ads and subscriptions. Surprisingly, I did not even see vibe coded apps for my use case.
For the design of my app, I try to imitate the UX of Apple's first party apps.
I checked the competitors, and what is in the AppStore looks like it was built in 2015, and only updated to add Ads and subscriptions. Surprisingly, I did not even see vibe coded apps for my use case.
I think this is where people start to consider you a "Luddite".
Is it bad that you used a computer and Google and lifted information from other people to accomplish a task?
My father is a machinist and has built has knowledge off the skills and documentation of thousands before him, is that empty?
You still have to do the actual work, and where do you draw the line on "shipping more". If a farmer now has an automated tractor, does that mean his work is now trivial?
Is it bad that you used a computer and Google and lifted information from other people to accomplish a task?
My father is a machinist and has built has knowledge off the skills and documentation of thousands before him, is that empty?
You still have to do the actual work, and where do you draw the line on "shipping more". If a farmer now has an automated tractor, does that mean his work is now trivial?
I am not a luddite by any means, and I think that all these comparisons fall short. Automating physical work has a different effect on our brain than automating intellectual work. Or maybe I'm fooling myself, and it's just our turn as worker to get the industrial revolution treatment.
There are lots of prior examples of automating intellectual work, though -- calculators (rather than doing the math yourself), Google search (rather than looking through library catalogues yourself), Word processors/typewriters (rather than writing by hand), heck, even writing (rather than simply remembering things), which Plato railed against.
True, I guess at every step it’s affecting a growingly larger part of the intellectual population.
I remember Umberto Eco only half-jokingly ranting in the early 2000 against the speed and ease with which suddenly everyone could match his rare bibliophile abilities and his portentous memory.
I guess we’re all getting to feel the same way now, with the difference that the industrial/financial complex built on this new revolution feels like the most disenfranchising ever so far.
> to me and to many friends in their 30-40s, using AI models to achieve something we used our brains to achieve feels... empty?
you have to start using AI to achieve something which was way more challenging before, then you will feel lots of fulfillment and inspiration.
you have to start using AI to achieve something which was way more challenging before, then you will feel lots of fulfillment and inspiration.
I can relate, but it's our turn of the wheel it seems.
It's a phenomenon that's happened basically nonstop since the Enlightenment. Yeaterday it was John Henry, today it's you.
Time to find a new hobby to bring intellectual satisfaction if typing "do my job" over and over in a text box isn't doing it.
It's a phenomenon that's happened basically nonstop since the Enlightenment. Yeaterday it was John Henry, today it's you.
Time to find a new hobby to bring intellectual satisfaction if typing "do my job" over and over in a text box isn't doing it.
Inversely I am building things I either never would have gotten to and a rate that would have taken infinitely longer. I get excited every day because instead of so much of my day spent writing the boiler plate I can spend most of the time around the architecture.
Not manually cleansing data and building apps with AI to do the parts you can't or don't want to do is a massive lift.
I can do things that were far beyond my reach — suddenly. Almost all at once.
"Now you can discover you suck at marketing and business" :D
I can do things that were far beyond my reach — suddenly. Almost all at once.
"Now you can discover you suck at marketing and business" :D
I feel like this: "working" with AI is fucking depressing.
No, I do not have an unshakeable feeling of not really having achieved anything at the end of the day.
Consequently I do not feel depressed or have to disregard any feelings.
I am busy working on my project. It is still hard to ship good software, even if the implementation is mostly getting done by itself.
Consequently I do not feel depressed or have to disregard any feelings.
I am busy working on my project. It is still hard to ship good software, even if the implementation is mostly getting done by itself.
My code was always plumbing. Glue stuff together. Connect data model to api.
I'm still using my brain, just doing a lot more plumbing now and reading a lot more code than writing. Depressing in some ways and exciting in others. At the end of the day it's not going way. It's disruptive and better to embrace it.
I'm still using my brain, just doing a lot more plumbing now and reading a lot more code than writing. Depressing in some ways and exciting in others. At the end of the day it's not going way. It's disruptive and better to embrace it.
Those of us in our 50s have been waiting for this day since Eliza.
The frontier graph on all these benchmark are extremely in favor of 5.6 Sol over Fable, more than the best model comparisons in previous iterations.
I'd like to know how cherry-picked this is, and what tests it performed less overwhelmingly in, but I suppose that info is not going to be on this post.
If it pans out to be as good as it says, that's great. On the other hand, if this model is not overwhelmingly impressive over Fable, I will lose what remaining trust I had in these announcements.
I'd like to know how cherry-picked this is, and what tests it performed less overwhelmingly in, but I suppose that info is not going to be on this post.
If it pans out to be as good as it says, that's great. On the other hand, if this model is not overwhelmingly impressive over Fable, I will lose what remaining trust I had in these announcements.
They do disclose that they scored much lower than Fable on SWEBench Pro, which is a pretty high-quality benchmark. I think it's partially just about what they choose to emphasize...
It's worth noting that OpenAI recently came out saying, "We don't think SWEBench Pro is worth reporting any more" - https://openai.com/index/separating-signal-from-noise-coding...
37374848(1)
SWEBench Pro should be ignored until they fix it or disprove the broken task accusations.
There has been a lot of chatter ever since the Mythos scores had been release that SWEbench pro had major contamination and that Mythos had memorized many questions that lacked the context to be solvable on their own. And now with OpenAI saying a large number of the questions are broken, I think it's worth taking that single outlier benchmark with some salt when the overall trend is that 5.6 is very competitive with Mythos at about half the price.
I totally missed that, because in the charts they showcase for coding, the SWEBench score is not present, they only include it at the end of the post in tables. Hmm.
Great catch.
Great catch.
The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a while.
SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.
SWE Bench Pro is also gamed and shouldn't be trusted.
Didn't they also just post about how SWEBench is broken?
> SWEBench Pro, which is a pretty high-quality benchmark
No, doesn't seem like it
https://openai.com/index/separating-signal-from-noise-coding...
No, doesn't seem like it
https://openai.com/index/separating-signal-from-noise-coding...
They set both fable and opus to "max" to do them dirty, without also seeing high/xhigh for claude it's not a useful comparison at all.
No-one should be using fable or opus on "max", it often over-thinks and does worse than on xhigh.
No-one should be using fable or opus on "max", it often over-thinks and does worse than on xhigh.
The proof is in the pudding and these benchmark stats will only work for so long before people lose interest.
The charts are also extremely difficult to parse. They seem auto-generated. Dataset coloring is atrocious.
Regarding your main point, yes, I agree. My impression (as someone who uses both Codex and Claude Code daily) is that OpenAI does a fair amount of benchmaxxing.
Regarding your main point, yes, I agree. My impression (as someone who uses both Codex and Claude Code daily) is that OpenAI does a fair amount of benchmaxxing.
I really appreciate the focus on intelligence WITH token efficiency. I'd like to see that become the trend. Smartest per token metrics. Least tokens to accomplish the task above a certain success level. Most of my tasks would benefit from efficiency / token, but switching models constantly, and trying to guess the right model and effort level takes up too much of my processing.
> I really appreciate the focus on intelligence WITH token efficiency
that is a polite way of saying "I don't believe AGI is coming anytime soon".
that is a polite way of saying "I don't believe AGI is coming anytime soon".
Not sure why you would think a focus on efficiency means less performance. Compression is intelligence. Higher efficiency enables higher performance.
seems more like "this priorisies AGI"
What is AGI to you though?
What is AGI to you though?
I honestly interpreted and agreed with the version "this saves money".
We Openly hate OpenAI because they’re not very Open but we secretly hope they win against not-open-at-all Anthropic.
I openly hope the chinese labs distilling them into open weights win.
Yup. I'm done with US companies. Let's go China !
have fun with your sub-tier models then. More compute for me
...until you get rug-pulled (like with Fable recently) because the model you're depending on is proprietary, then you have fun with no model at all.
The sooner Chinese labs catch up the better it is for all of us, even if you don't use their models, as they're the ones who define the baseline capability that cannot be taken away from you (no one is going to limit/remove access to an LLM if you can get equivalent/better unrestricted access from Chinese open-weight models).
The sooner Chinese labs catch up the better it is for all of us, even if you don't use their models, as they're the ones who define the baseline capability that cannot be taken away from you (no one is going to limit/remove access to an LLM if you can get equivalent/better unrestricted access from Chinese open-weight models).
Tried Xiaomi MiMo v2.5 via opencode today. Since Sonnet 5's release week, Sonnet 4.6 has been feeling like a vegetable, with Sonnet 5 itself being only a little better. MiMo on the other hand feels like Sonnet 4.6 did up until very recently. Absolutely impressive.
In some ways, more impressive than GPT 5.5 with high(!) thinking. GPT says quite some nonsense from time to time; didn't see any sign of this in MiMo so far, which is a pretty wild difference.
In some ways, more impressive than GPT 5.5 with high(!) thinking. GPT says quite some nonsense from time to time; didn't see any sign of this in MiMo so far, which is a pretty wild difference.
Not at all, we love them all with Chinese labs. And wish them to continue competing and not winning. That is how we get best models, lower prices and better availability.
I'm just happy there's competition.
personally I hope any company involved with child slaughter ends up crashing and burning, i say this because both those companies are buddies with the us department of war (who helped annihilate a school the other day)
"We've extended usage of Claude Fable" message incoming any day now.
They reset all usage half an hour ago. It's back to 0% per week and session. No specifically Fable related.
Hahaha seeing this play out in real time is absolutely incredible.
Im here for it, good on Anthropomorphic to feel some heat again after all that drug dealer Fable business.
100. I'm so tired of being treated like a drug addict by them. I'm currently sitting on _four_ "reset vouchers" from OpenAI so I get basically all week to play with 5.6 Sol to my hearts content. The amount of positive sentiment that brings to me should really be a concern for Anthropic who is increasingly alienating me away with their shit strategies.
you are a drug addict, though. we all are
To an extent, but I still feel consciously addicted at this point, and Anthropic's antics seem to suggest that I've reached the unconscious level where I'll do anything to keep access, including their insane street API price.
I let it work on two features while I was using mostly GPT 5.6 and it has already consumed 10% of the weekly Fable limit on Max x20.
GPT 5.6 on the Pro x5 plan is down to... 100%. It looks like they just reset the usage limits again. And I still have two resets on the bench.
Anthropic is going to have to up their game to compete.
GPT 5.6 on the Pro x5 plan is down to... 100%. It looks like they just reset the usage limits again. And I still have two resets on the bench.
Anthropic is going to have to up their game to compete.
What do you mean 'resets on the bench?
When OpenAI do quota resets (not your scheduled weekly ones, the promotional or apology resets) they don't just reset whatever quota you were on. They give you a token to reset your weekly quota when you like.
They added this a few weeks ago.
Before, they would do their celebrated usage resets while I was behind the already generous weekly quota.
If I had 80% left on Friday to spend over the weekend, they would reset and I'd have 100% until next Friday, a potential net negative.
OpenAI listened to the complaints.
Before, they would do their celebrated usage resets while I was behind the already generous weekly quota.
If I had 80% left on Friday to spend over the weekend, they would reset and I'd have 100% until next Friday, a potential net negative.
OpenAI listened to the complaints.
Before the reset, a single 5h window would use up 5% of the weekly quota. Now it uses 10%. Thanks a lot, Anthropic.
Or an allegedly even more dangerous model that they refuse to release. What a joke Anthropic has become.
The most impressive part is the token efficiency/cost per task of 5.6 Sol, it makes Opus 4.8 and Fable look extremely bad ($1.04 vs $1.80 vs $2.75)[0].
And 5.6 Luna ($0.21) is also impressive, cheaper than GLM 5.2 ($0.37) with higher intelligence.
[0]: https://artificialanalysis.ai/#price-and-cost
And 5.6 Luna ($0.21) is also impressive, cheaper than GLM 5.2 ($0.37) with higher intelligence.
[0]: https://artificialanalysis.ai/#price-and-cost
Well it's smaller model (something like 4T against 10T Fable). So it's faster and cheaper and with a lot of RL and maybe some favorable benchmark selection it can compete on these scores. In real tasks I expect it to have less intelligence, generalization ability, etc. than Fable.
5.6 Terra (mid tier model) as good as Fable on DeepSWE while cheaper than Opus API pricing. Seems like a homerun.
GPT usually performs better on DeepSWE while Claude does better on FrontierCode. These two coding benchmarks are pretty much the only ones right now that's still worth taking a look at imo.
DeepSWE seems to strongly, strongly prefer ChatGPT models. There were also major flaws in its methodology pointed out recently, that overlap strongly with the flaws OpenAI pointed out in its SWE Verified report.
I use both ChatGPT and Claude for engineering work on a daily basis, touching performance critical code to application backends to frontend work, and I've found that DeepSWE scores don't reflect my reality when I assess high quality output from the models/harnesses.
Not that Opus always beats GPT 5.5., but that 5.5 is ahead of Opus on a general benchmark smells off to me.
I use both ChatGPT and Claude for engineering work on a daily basis, touching performance critical code to application backends to frontend work, and I've found that DeepSWE scores don't reflect my reality when I assess high quality output from the models/harnesses.
Not that Opus always beats GPT 5.5., but that 5.5 is ahead of Opus on a general benchmark smells off to me.
Just my two cents. I'm on the Plus plan, I ask gpt-5.6 sol / high to analyze a vibe-coded codebase (~50k LoC) and write a plan to make it production ready. It wasn't a great prompt, I just wanted to test it quickly. It ran for ~15min and consumed 95% of my 5h quota (I thought it was gonna crash). The output is excellent but just a heads up that it consumes a lot of quota!
Anyone looking to use frontier SOTA on the $20 plans is going to have a bad time
If you don't have an agent heavy workflow, you'd be surprised how far the 20$ subscription stretches. I used ~500$ of usage in the last month on a team 20$ plan.
Yes had the same experience, Sol consumes limits quite fast
Anyone else noticed the "Extended: Fable 5 is included in your weekly limit
through July 12 blablabla" disappeared from claude code? Did they panic-delete the july 12th deadline ?
I still see in the menu to select the model in the GUI (Claude Desktop, claude.ai etc).
Yes, I noticed this too!
Looks like they reset everyone's Fable usage.
They did. I wonder if Anthropic will also be removing the 50% limit.
My Fable weekly limit is at 15% used already, 5.6 Sol at 3% used. And this is with the Max 20x plan compared to Codex 5x.
I don't work on the same tasks to compare them objectively, but GPT 5.6 on xhigh seems much cheaper. Essentially unlimited usage.
My Fable weekly limit is at 15% used already, 5.6 Sol at 3% used. And this is with the Max 20x plan compared to Codex 5x.
I don't work on the same tasks to compare them objectively, but GPT 5.6 on xhigh seems much cheaper. Essentially unlimited usage.
Anthropic really needs to get Opus 5 out ASAP.
The gap between Opus 4.8 and Fable is large enough to drive a GPT 5.6 sized bus through. A better Opus would take some of the heat off.
The gap between Opus 4.8 and Fable is large enough to drive a GPT 5.6 sized bus through. A better Opus would take some of the heat off.
Provided such a Opus 5 performs on par with Fable 5 and GPT 5.6 Sol.
Otherwise I am not interested.
Supposedly Fable 5.1 is in the later stages of the release pipeline, maybe it takes back the crown from OpenAI, who are now rumored to launch GPT 6 in August.
Otherwise I am not interested.
Supposedly Fable 5.1 is in the later stages of the release pipeline, maybe it takes back the crown from OpenAI, who are now rumored to launch GPT 6 in August.
"GPT‑5.6 delivers a step change in design judgment. With only high-level direction, GPT‑5.6 creates tasteful, ergonomic, and functional interfaces. Its stronger computer-use capabilities let it inspect and refine the rendered result—not just generate the underlying code or content—so it can catch visual and functional issues and apply finishing touches before handing the work back."
This one is really promising, as it may allow to close major gap with Claude in design/UI skills
This one is really promising, as it may allow to close major gap with Claude in design/UI skills
+1. I've been only using Sonnet/Opus these days for UI work because GPT 5.5 just can't do any of that. Its just really terrible. Eager to give this one a try.
Computer-use is a big limitation that my 2015 Macbook Pro cannot handle. I find the Codex cli says it looks at the end output artifact but so often it fails to refine it into acceptable form. If it could use my computer screen and visual inputs for review, it might be able to actually design documents/powerpoints/etc. I'm juicing everything I can out of the 11 year old laptop and I'm honestly impressed at what it can still do.
How dare you point out that 2015 is 11 years ago.
Agreed, I’m looking forward to trying it out. I think that the rise of visual design skills that are pretty clearly targeted towards Codex users has lit a bit of a fire under their butts.
GPT-5.6 is a really good model, and quite cheap. I can finally replace GPT-5.3-Codex for my Tool Calling in n8n.
Here's my benchmark results for GPT-5.6:
https://aibenchy.com/?q=gpt-5.6
(the high reasoning variants are still running, uploading them soon too)
EDIT: The high variants are there too, enjoy the hamsters[0].
[0]: https://aibenchy.com/showcase/?q=gpt-5.6
Here's my benchmark results for GPT-5.6:
https://aibenchy.com/?q=gpt-5.6
(the high reasoning variants are still running, uploading them soon too)
EDIT: The high variants are there too, enjoy the hamsters[0].
[0]: https://aibenchy.com/showcase/?q=gpt-5.6
Given that both Gemini 3.5 Flash (high) and Gemini 3 Flash Preview (medium) beat GPT-5.6 Sol (high) for correctness and score in your benchmarks I don’t trust them at all. The rest of the ranking also doesn’t make sense, like GPT-5.3-Codex
(medium) performs better than Claude Opus 4.8 (medium) yeah sure
It's because the benchmark is not coding-only.
Gemini models tend to have most knowledge for most domains, and are one of the most intelligent overall. You can check other benchmarks too, on specific categories, those models still beat other SOTA models.
Regarding Opus, Anthropic models often fail to follow instructions, formatting requirements or simply refuse to answer questions (i.e. Fable).
The issue with Gemini models is that they are not as good as using tools or go into weird failure modes when coding or trying to extract/generate specific data. They work amazing, until they don't...
Gemini models tend to have most knowledge for most domains, and are one of the most intelligent overall. You can check other benchmarks too, on specific categories, those models still beat other SOTA models.
Regarding Opus, Anthropic models often fail to follow instructions, formatting requirements or simply refuse to answer questions (i.e. Fable).
The issue with Gemini models is that they are not as good as using tools or go into weird failure modes when coding or trying to extract/generate specific data. They work amazing, until they don't...
100% agree with your statement that gemini models have most knowledge in domains. They shine in so many generic topics
One example where the order seems correct, is this SVG generation test:
https://aibenchy.com/showcase/?q=Gemini+3.5%2Cgpt+5.6%2C+5.3...
You can see that most Gemini 3.5 generations are more correct than 5.6 Sol (the net is in the middle of the table, hamster seems reasonable and not deformed, etc.)
https://aibenchy.com/showcase/?q=Gemini+3.5%2Cgpt+5.6%2C+5.3...
You can see that most Gemini 3.5 generations are more correct than 5.6 Sol (the net is in the middle of the table, hamster seems reasonable and not deformed, etc.)
It doesn't seem correct at all though? the supposedly best one isn't even the best gemini flash output (the medium one looks better than the high one)
Interesting that Sol (low) did better than Sol (medium) in your benchmark (and is barely more expensive than Terra). I too have been using 5.3 codex as a cheap-but-good model and are switching to Terra (xhigh).
Here's all 3 (medium), and GPT-5.5
It GPT-5.6 doesn't seem to be a lot smarter than 5.5, but it is faster, cheaper, more efficient and more consistent:
https://aibenchy.com/compare/openai-gpt-5-6-sol-medium/opena...
It GPT-5.6 doesn't seem to be a lot smarter than 5.5, but it is faster, cheaper, more efficient and more consistent:
https://aibenchy.com/compare/openai-gpt-5-6-sol-medium/opena...
>> approximately 700,000 A100e GPU hours of black-box automated red teaming
Amusing that they use A100e as the reference point to sound impressive. Different ways you could make that conversion, but based on FP4 FLOPs (yes it's disadvantageous to A100, that's the point), that's something like 200hr on a GB300 NVL72 rack.
Not nothing either, but far less astounding sounding than 700k hrs.
Amusing that they use A100e as the reference point to sound impressive. Different ways you could make that conversion, but based on FP4 FLOPs (yes it's disadvantageous to A100, that's the point), that's something like 200hr on a GB300 NVL72 rack.
Not nothing either, but far less astounding sounding than 700k hrs.
I'm pretty sure Altman has spoken about giving a model 100k+ A100s specifically, this might be them being very literal
Wait, what do you mean? 700k A100e hours are equal to 200 hours of a GB300 NVL72 rack? One GB300 NVL72, 72-GPU rack has equal processing power to 3500 A100e GPUs?
maybe? ai says about *8.3 days* of continuous runtime on a single GB300 NVL72 rack
about a sprint's level of effort.
about a sprint's level of effort.
a very expensive sprint
> based on FP4 FLOPs (yes it's disadvantageous to A100, that's the point)
The A100 doesn't have hardware FP4, and you'd be running a quantized model with some accuracy loss but unless this was natively trained on FP4*
* to add another layer, they own the model and could apply tons of post-training techniques to reduce that accuracy loss and probably already do
The A100 doesn't have hardware FP4, and you'd be running a quantized model with some accuracy loss but unless this was natively trained on FP4*
* to add another layer, they own the model and could apply tons of post-training techniques to reduce that accuracy loss and probably already do
Have been testing Luna against other small models for production customer support uses cases, finding negligible performance impact when comparing against the substantial cost increase. Claims re: reduced token use also don't seem reproducible. Wrote a quick blog with some sample findings (marketing content but the findings are real): https://valiopt.com/blog/gpt-5-6-customer-support-cost-perfo...
Here are 18 pelicans - six each for Luna, Terra and Sol at the six different reasoning effort levels (plus the price to generate each one): https://static.simonwillison.net/static/2026/gpt-5.6-pelican...
Or if you want to see some in 3D, OpenAI featured a pelican riding a tricycle, bicycle, pony and another pelican in their livestream this morning: https://www.youtube.com/live/Wq45rvPGNHs?t=1070s
Or if you want to see some in 3D, OpenAI featured a pelican riding a tricycle, bicycle, pony and another pelican in their livestream this morning: https://www.youtube.com/live/Wq45rvPGNHs?t=1070s
Time to dump this test. Probably not a coincidence every version has the same rolling green hills, gradient blue sky, sun in the corner, etc.
On the one hand: yes, pelicans on bikes are definitely in the training set at this point.
On the other hand: the test is clearly not saturated, given that you can see a clear difference in output at the various reasoning levels / model versions.
On the other hand: the test is clearly not saturated, given that you can see a clear difference in output at the various reasoning levels / model versions.
I sort of agree, but within the same model I expect the reasoning effort to be reflected in the quality of output and that's basically how it played out. When you're comparing different models, then it's just who benchmaxxed the best and there's not a lot of value there.
But that is my point: if benchmaxxing was all the labs were doing, then surely the dumber model could/would have equivalent performance? Rather than noticeably worse perf on a (somewhat trivial to game) test.
Waiting for the AMA on Reddit "Ten years ago I was responsible for the pelican department at OpenAI, AMA"
I don't know. If they were training on this, I feel like they would be able to get the shape of a bike frame right; it's a pretty simple polygon, and a lot of the bike frames that are getting generated would be impossible to steer.
I mean, humans can't draw bikes!
https://themagnet.substack.com/p/why-is-it-so-hard-to-draw-a...
https://themagnet.substack.com/p/why-is-it-so-hard-to-draw-a...
I partially agree, but in this case it kinda illustrates that it may not be worth using Terra on any reasoning level below high; those are some awful penguins on bikes.
Goodhart's law.
I think the 'pelican test' is becoming useless. It's been around long enough that now I'm sure good examples are in the training data, and hell they might even do some hand tuning to make it do a decent job since they know people will ask about it.
But either way, with no real way to visualize the result of the text it starts with - it will always be stabbing in the dark. It can't understand conceptually what any of it should look like and then refine the SVG to improve it gradually. It just throws darts at a wall and hopes it comes out alright.
But either way, with no real way to visualize the result of the text it starts with - it will always be stabbing in the dark. It can't understand conceptually what any of it should look like and then refine the SVG to improve it gradually. It just throws darts at a wall and hopes it comes out alright.
Pelicans, maybe, but the point is to measure how good the "internal visualization" abilities are. Throw curveballs, like a unicorn with a duck bill serving coffee at a basketball court. An elephant playing a piano while its trunk swings a baseball bat at a tiny alien spaceship buzzing its head.
Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint.
Compositionality and visualization are generally much, much better at each new generation / release cycle.
It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.
Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint.
Compositionality and visualization are generally much, much better at each new generation / release cycle.
It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.
Still fun to watch models trying their best :). I think "money spent" metric in this test is becoming the most interesting to watch. It is like looking at RT cost of "Hello, world!"...
I think it's still useful in a "hello world" sort of way. It means you actually tried the new model.
Honestly that's the main value I get from it myself - making a pelican means I have to figure out API keys and how to talk to the provider, or how to run it locally for the local models.
What's strange with this is the prompt "Photorealistic photograph of a pelican riding a bicycle down a coastal boardwalk, wings gripping the handlebars, webbed feet on the pedals, large orange bill, detailed feather texture, golden hour lighting, shallow depth of field, shot on a DSLR with 85mm lens, natural motion blur on the wheels" produced, well, exactly what I asked it for. I wonder if I tell it then to make it SVG ...
https://chatgpt.com/share/6a5009de-fff8-83ea-98ff-0da17d1d04...
https://chatgpt.com/share/6a5009de-fff8-83ea-98ff-0da17d1d04...
Cool. I still find these a useful visualization of some the qualities of llms. Even if they did train for [animal] on [vehicle] svg, it's still nice to see at a glance how the different models and reasoning levels perform. Lunar misses part of the frame, except on max reasoning. While most of the others have a mostly correct bike at all reasoning levels.
I once used something like karpathy's auto-scientist to mutate the prompts and rank them with a vison model. Some of the winners where pretty neat. I think they have a lot more style than the gpt-5.6 ones. https://xcancel.com/xundecidability/status/20449185674144196...
I once used something like karpathy's auto-scientist to mutate the prompts and rank them with a vison model. Some of the winners where pretty neat. I think they have a lot more style than the gpt-5.6 ones. https://xcancel.com/xundecidability/status/20449185674144196...
people are saying this is benchmark is saturated but all of these have occlusion issues, even sol max.
A skilled human artist wouldn't have both legs in front of the bike, or a single straight line representing both leg's crank arms.
A skilled human artist wouldn't have both legs in front of the bike, or a single straight line representing both leg's crank arms.
Yeah it makes no sense at all to dismiss the test, when even the very best examples are noticeably below what a skilled teenager could produce.
Dead internet theory? Semi-random parroting by real people? Or something else.
Dead internet theory? Semi-random parroting by real people? Or something else.
At what stages will models start to internally reflect the drawn SVG and automatically fix their own mistakes?
I assume multimodal models can do it already do it today if constantly asked "make it better"
I assume multimodal models can do it already do it today if constantly asked "make it better"
The quality of sol on effort=none makes me think this test is saturated or they are benchmarkmaxxing this exercise.
Ok, I'll never use max effort again on OAI models..
Is that... an x-rated, censored pelican?
They said in the AI community, a pelican riding a bicycle is a good test to measure effectiveness of the model, wondering if they were referring to you, or is it really a standard in the AI community ?
Also would be good to have a tool where users can select models and instantly see each model's generated pelicans. That will make it easy to compare the output of different models.
Also would be good to have a tool where users can select models and instantly see each model's generated pelicans. That will make it easy to compare the output of different models.
Simon did start the pelicans on bicycles as an SVG, but I think it's more of a fun goofy thing to see how the model performs at. I don't think it has a direct correlation to a model's performance though.
I'm waiting for the day that the "generate a Pelican" test comes back with a SVG-art like illustration of a Pelican equipment case, like a model 1620 or similar.
https://www.google.com/search?client=firefox-b-d&q=pelican+1...
https://www.google.com/search?client=firefox-b-d&q=pelican+1...
Surely "how to draw a SVG pelican on a bike" has made it into the training data by now ...
Surely this comment is literally on every new model release post.
It's part of the pelican tradition at this point.
I eagerly await the models replying with that: "I'd be happy to create a pelican riding a bicycle, but just a note that this might already be in my training data. Simon."
Someone told me that a model said "oh, the classic" when they asked for this recently. I think that was Opus 4.8.
And yet... only one is any good.
Thank you Simon! Luna is surprisingly decent across all reasoning levels.
I think all of Luna's are bad. The only decent one is sol @ xhigh. Even sol @ max is weird. Sol @ high and @ medium are ok, and every other single one across every model is bad.
Strong disagree, but to each their own. For sol I really like how only medium uses the wings on the handlebars to ride the bike. For all the other sols the pelican evolved a new set of arms separate from the wings.
gpt-5.6-sol Max pelican didn’t skip neck day
gpt-5.6-sol x XHIGH is my favourite
Apparently plus users do not have access to Sol, so I'm really worried about the ugly Terra Pelican.
somehow Terra really struggles here even compare to Luna.
max effort sol clearly over-engineered
terra is just weird. in this nothingburger test, time nor higher costs seem to not strongly correlate with the aesthetics.
You're famous!
Oh god change benchmark ffs.
I like Terra High the best. That pelican is utterly yoked.
alecco(2)
this looks like the same shit from 4 years ago. give it up.
The first time I did this was actually less than two years ago - in October 2024 - and it's fun seeing how much better they've got since then: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
Absolutely, the change in quality over time is a great yardstick.
Nice to see you did the quality level comparisons and did three passes.
I've been using that technique myself on my image gen reviews[1] and it also works well in presentations and for personal study.
[1] https://generative-ai.review/2025/12/beast-mode-activated-op...
Nice to see you did the quality level comparisons and did three passes.
I've been using that technique myself on my image gen reviews[1] and it also works well in presentations and for personal study.
[1] https://generative-ai.review/2025/12/beast-mode-activated-op...
Things I have been struggling with Fable over and GPT 5.5, were just solved handily by SOL in a real "thank you, next problem" kind of way. Overall, something that just works is way less wasteful for your usage than struggling back and forth for hours.
From my first tests today, it is a workhouse. It can scan my whole code base, optimize every part, with a greater level of autonomy than other tools. This is insane, we are living at the best time.
The literal worst time. I prefer meritocracies.
You think aristrocrats are using GPT and succeeding?
No I think aristocrats are using llms to justify the flattening of wages, where they can use it as an excuse diminish the value of merit based systems.. Meritocracy / education was one of the few ways someone from a lower class couod climb the ladder and they're trying to destroy that ladder.
I also like to take a look at https://cursor.com/cursorbench
While in the past months Composer 2.5 was a lot better than I had expected a year ago, and the GPT 5.6 family does a good job in terms of cost for performance, I wonder why nobody is talking about Grok 4.5 high? Those numbers look very convincing to me.
While in the past months Composer 2.5 was a lot better than I had expected a year ago, and the GPT 5.6 family does a good job in terms of cost for performance, I wonder why nobody is talking about Grok 4.5 high? Those numbers look very convincing to me.
> * Grok 4.5 has an advantage on CursorBench: an earlier snapshot of the Cursor codebase was unintentionally included in training. The exact score impact is unclear. That data has been removed for future models. For a rundown of third-party benchmark scores, see the Grok 4.5 launch blog.
I don't know about those numbers, even assuming this was by mistake :)
I don't know about those numbers, even assuming this was by mistake :)
First impression of 5.6 Sol in Codex is fantastic — the model asks dozens of clarifying questions before starting to implement where other models (including 5.5 and Terra) just yolo it with assumptions that needed to be walked back later.
I find it interesting that no one here has mentioned the increased (usable) context window 258k -> 353k. That's huge, but I wonder if it means we pay long context (2x) for the ones past 272k still.
It's working great for me.
I generally prefer OpenAI. I usually start projects with Codex; I love the plan we created. Then, after about 2 hours of working, back and forth, etc.I realize it's drifting HARD, or getting stuck on relatively simple things.
Once I get frustrated enough, I sometimes start over with Anthropic. Anthropic has been better (for me at least) to work with from start to finish.
I have the $200/month plan for each of them. (And I used to have the $250/month plan for Gemini... lol?)
Both Fable and Sol are good and definite improvements. I don't have a favorite yet. It's too early to tell.
The biggest difference I'm noticing is usage.
Anthropic is like a used car salesman. Not allowed to do a basic check on your own car (website), they'll scrape every penny possible out of you. Low limits, high api prices, trying to keep everything in their system.
OpenAI is like a cool but aloof dad. Go ahead and borrow his car, shoot, he'll even pay for your gas every once in a while. He'll answer your 5th question drunk, forgetting what you asked. But, at least he's a nice drinker that buys the group shots.
I have to watch my Fable usage, and I'm sure as hell not going to pay API prices. I don't have to watch/worry about my Sol usage even in Ultra.
But, 1 thing I'm noticing using Sol Ultra (on fast mode) is that it's slowing WAY down after a bit. I work the opposite of peak times so that's not it.
I generally prefer OpenAI. I usually start projects with Codex; I love the plan we created. Then, after about 2 hours of working, back and forth, etc.I realize it's drifting HARD, or getting stuck on relatively simple things.
Once I get frustrated enough, I sometimes start over with Anthropic. Anthropic has been better (for me at least) to work with from start to finish.
I have the $200/month plan for each of them. (And I used to have the $250/month plan for Gemini... lol?)
Both Fable and Sol are good and definite improvements. I don't have a favorite yet. It's too early to tell.
The biggest difference I'm noticing is usage.
Anthropic is like a used car salesman. Not allowed to do a basic check on your own car (website), they'll scrape every penny possible out of you. Low limits, high api prices, trying to keep everything in their system.
OpenAI is like a cool but aloof dad. Go ahead and borrow his car, shoot, he'll even pay for your gas every once in a while. He'll answer your 5th question drunk, forgetting what you asked. But, at least he's a nice drinker that buys the group shots.
I have to watch my Fable usage, and I'm sure as hell not going to pay API prices. I don't have to watch/worry about my Sol usage even in Ultra.
But, 1 thing I'm noticing using Sol Ultra (on fast mode) is that it's slowing WAY down after a bit. I work the opposite of peak times so that's not it.
May I know if it’s for hobby projects or you run your own business/company where you use subscriptions.
I was using Claude at work, switched to Codex and it's very capable after some setup and migrations. Claude's usage limits is inadequate if you're coding all day.
The plan will be that Claude is for frontend design while Codex takes care of the bulk of the work.
While I have not looked into it deeply (someone correct me on this), OpenAI seems to give more bang for the buck because it appears ChatGPT web usage is separate from Codex usages unlike Claude.
The plan will be that Claude is for frontend design while Codex takes care of the bulk of the work.
While I have not looked into it deeply (someone correct me on this), OpenAI seems to give more bang for the buck because it appears ChatGPT web usage is separate from Codex usages unlike Claude.
Both.
Both.
Wow the video is much better.. the PR spend clearly went up a lot. Mainly just showing "real people" doing "real stuff".
When in doubt, use grandmas knitting. yeah, that will make them trust us.
I haven't tried an OpenAI model for a long time, but with Fable going to API pricing soon this might be enough to get me to try codex.
Seeing how Anthropomorphic just reset usage quotas back to 0 and the other day extended Fable sub inclusion by a few days, I have a feeling they might not drop Fable out of sub after all, because like you I would most definitely take a long good look at codex at that point.
It's not just the API pricing either, there's also the constant uncertainty. They pull the model then put it back up, they say the model is going away then suddenly it's not. And then there's the fact Fable is barely usable because it randomly downgrades to Opus out of nowhere whenever it thinks about exploits.
It's definitely good that Anthropic's feeling the pressure. Anthropic has worn out their welcome with this "safety" nonsense. If OpenAI actually lets me use the LLMs on a subscription without any of this bullshit, I'll definitely switch.
It's definitely good that Anthropic's feeling the pressure. Anthropic has worn out their welcome with this "safety" nonsense. If OpenAI actually lets me use the LLMs on a subscription without any of this bullshit, I'll definitely switch.
> And then there's the fact Fable is barely usable because it randomly downgrades to Opus out of nowhere whenever it thinks about exploits.
I suspect it's not what you meant, but it's definitely not random and is very deliberate. Just today I got it to reliably trigger the "safety" filter with (drumroll) having it list the weight keys of a 300M parameter ModernBERT-derived model. Their "safety" classifier must be matching one of the key names in there and trigger their "this is a frontier model" anti-competitive filter[1] (even though it's just a tiny 300M parameter model, four orders of magnitude smaller than the frontier).
[1]: https://news.ycombinator.com/item?id=48464732
Fortunately once you know how it works (i.e. dumb keyword classifier) it's easy-ish to get around: just rename the keys so that it doesn't contain the naughty keyword. (At least as long as it doesn't trigger on something in its own thinking trace, which needs... more creative workarounds.)
I suspect it's not what you meant, but it's definitely not random and is very deliberate. Just today I got it to reliably trigger the "safety" filter with (drumroll) having it list the weight keys of a 300M parameter ModernBERT-derived model. Their "safety" classifier must be matching one of the key names in there and trigger their "this is a frontier model" anti-competitive filter[1] (even though it's just a tiny 300M parameter model, four orders of magnitude smaller than the frontier).
[1]: https://news.ycombinator.com/item?id=48464732
Fortunately once you know how it works (i.e. dumb keyword classifier) it's easy-ish to get around: just rename the keys so that it doesn't contain the naughty keyword. (At least as long as it doesn't trigger on something in its own thinking trace, which needs... more creative workarounds.)
[flagged]
Unfortunately, I'm finding that in long-form agentic use, when I'm trying to use Sol, I keep tripping guardrails – moreso than even Fable, somehow.
I don't know exactly what part of my codebase is triggering it, so I'm going to have to keep poking, but apparently the guardrails are not that gentle despite the phrasing. :(
I don't know exactly what part of my codebase is triggering it, so I'm going to have to keep poking, but apparently the guardrails are not that gentle despite the phrasing. :(
Sounds like you are working on something naughty :)
I flip back and forth between whoever currently has the more powerful frontier model that isn't cost prohibitive - subscriptions only, API pricing a non-starter. Today that's Fable 5 which has been excellent, as soon as it's Sol I'll switch to that. The OAI/Anthropic harness behavior has mostly stabilized for me with consistent AGENTS.md that I sync with CLAUDE.md - I like pi (pi.dev) and have tried to build it up to get performance comparable to the two "first-party" harnesses, I'm just not there yet.
One major sticking criteria for not going with OpenCode / pi for all of my coding is I want access to the tier-1 frontier model of the day without API pricing - e.g. afaik I can't use Fable 5 via pi harness even though I have a subscription, so for this week I'm on Claude Code. It's not the need to Fable 5 for everything, but even if I just want the marginal intelligence benefit to stress test an architecture decision, it's a safety blanket to know there isn't a ~smarter~ model I could have used. And for my use cases, the doggedness and capability of these frontier models has been insanely effective.
My feeling is we're still in the Uber era subsidy period - the moment the subscriptions either try to lock me in longer than a month or stop OAI/Anthropic stop delivering frontier models in the subscriptions, I'm out - switching fully over to pi.dev or another OS harness and routing my token spend via OpenRouter or offloading to Qwen locally. Then I'll have to put an accurate dollar amount on frontier intelligence.
One major sticking criteria for not going with OpenCode / pi for all of my coding is I want access to the tier-1 frontier model of the day without API pricing - e.g. afaik I can't use Fable 5 via pi harness even though I have a subscription, so for this week I'm on Claude Code. It's not the need to Fable 5 for everything, but even if I just want the marginal intelligence benefit to stress test an architecture decision, it's a safety blanket to know there isn't a ~smarter~ model I could have used. And for my use cases, the doggedness and capability of these frontier models has been insanely effective.
My feeling is we're still in the Uber era subsidy period - the moment the subscriptions either try to lock me in longer than a month or stop OAI/Anthropic stop delivering frontier models in the subscriptions, I'm out - switching fully over to pi.dev or another OS harness and routing my token spend via OpenRouter or offloading to Qwen locally. Then I'll have to put an accurate dollar amount on frontier intelligence.
> My feeling is we're still in the Uber era subsidy period
I often wonder whether this doesn't continue indefinitely.
Uber was able to do this because it was just them and Lyft playing second fiddle, with a huge barrier to entry once the network effects had kicked in.
It just seems like the model space has way too many competitors, + OSS/Local options for them to ever be able to jack up their prices. At least once the datacenter bottleneck has been cleared.
I often wonder whether this doesn't continue indefinitely.
Uber was able to do this because it was just them and Lyft playing second fiddle, with a huge barrier to entry once the network effects had kicked in.
It just seems like the model space has way too many competitors, + OSS/Local options for them to ever be able to jack up their prices. At least once the datacenter bottleneck has been cleared.
Cars are pretty mature while AI is just getting started. Expect 100x price drop for the same quality.
I'm working on a multi-harness IDE that supports custom agent workflows and skills that are shared between any harnesses it wraps over. I think it might prove handy for a workflow like yours.
Would it currently support Fable 5 via the restrictions Anthropic is placing on usage... because that's my major blocker
Zed IDE is allowing me to use Fable with Max x5 or any OpenAI model.
Huh, a good alternative just as anthropic's 50% weekly subscription subsidy is ending this weekend. Time to see if it's benchmaxxed or actually a strong leap over GPT5.5.
They also seem to really not care about alignment, or care about it in the wrong way. It's entirely missing in the blogpost and there are some concerning bits in the model card, seemingly treating CoT controllability as something to be "investigated" rather than the warning sign it's supposed to be.
They also seem to really not care about alignment, or care about it in the wrong way. It's entirely missing in the blogpost and there are some concerning bits in the model card, seemingly treating CoT controllability as something to be "investigated" rather than the warning sign it's supposed to be.
There's also this:
> GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated -- https://www.lesswrong.com/posts/JFjNmPTbH8kL6xtp6/gpt-5-6-th...
> GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated -- https://www.lesswrong.com/posts/JFjNmPTbH8kL6xtp6/gpt-5-6-th...
In the introduction video they say 5.6 Sol autonomously post-trained 5.6 Luna. Curious what this means.
/goal tune 5.6 Luna parameters until performance is maximized across all benchmarks
It means OpenAI and Anthropic are now in a RSI race with each other
Sounds like they gave it a goal to hit certain benchmarks and just let it have its way with the base Luna model.
This produced a disturbing mental image.
Benchmarks look really promising. Suspiciously good, even. I guess we’ll see soon enough.
My question to previewers: how are the guardrails for random joe that wasn’t personally blessed by the ai pope to access the non-nerfed model? Fable is a nightmare in this regard, but I’m not sure whether 5.6 also gets a critical side-eye from the gubmint when you ask it to fix bugs in your code (you filthy hacker, you).
My question to previewers: how are the guardrails for random joe that wasn’t personally blessed by the ai pope to access the non-nerfed model? Fable is a nightmare in this regard, but I’m not sure whether 5.6 also gets a critical side-eye from the gubmint when you ask it to fix bugs in your code (you filthy hacker, you).
I almost immediately ran into "This request requires additional safety checks, which can take extra time. Hang tight or retry with a faster model for a
quicker response, though it may be less capable of handling complex requests."
Which is something I've never seen with codex before, and I wasn't doing anything funky. Just writing CUDA kernels and benchmarks for them.
Which is something I've never seen with codex before, and I wasn't doing anything funky. Just writing CUDA kernels and benchmarks for them.
Is it actually usable though? Because the Fable situation is just obnoxious. If OpenAI's Fable equivalent is actually usable, I'll cancel my Anthropic subscription on the spot.
Yes it's usable, however it does make it difficult to just do /goal and walk away if this is happening periodically. I don't do that often, but for well defined things I sometimes do.
Great. I usually babysit the agents so I think it won't be a problem. So long as there's no obnoxious downgrading going on.
That settles it. Anthropic has until the end of this month to get rid of Fable's "safety" bullshit. If they don't, I will not renew my subscription.
That settles it. Anthropic has until the end of this month to get rid of Fable's "safety" bullshit. If they don't, I will not renew my subscription.
I was getting that regularly last week with regular 5.5 medium on the plus plan. I was doing benchmarking for a photo editor in Swift.
Interesting. Never seen it before. Now it's just constant and there's nothing sensitive about this work.
I'm getting the same message doing WebGL shader work.
It (Sol, on high) does seem actually really quite competent at this work though (GPU programming). Much more so than the attempts I made with 5.5 earlier today.
They talk a lot about speed in the article, but having tried out Sol today with Pi, 'medium' mode, one thing that stands out is that it's really ssslllloooowww.
It also defaults to 'low' mode for some reason. Can't tell if that's a step backwards compared to GPT-5.5 in medium mode so I'm sticking to medium.
Edit: just noticed it's spawning subagents in 'high' thinking mode.
It also defaults to 'low' mode for some reason. Can't tell if that's a step backwards compared to GPT-5.5 in medium mode so I'm sticking to medium.
Edit: just noticed it's spawning subagents in 'high' thinking mode.
Very interesting: I wonder if the RL approach is diverging between Anthropic and OAI?
I noticed that Fable uses shell tools almost exclusively (even to search and edit files), compared to previous Anthropic models.
Having run some experiments with 5.6, I notice that it uses built-in file systems and provider native tools much more (not shell tools), compared to previous OAI models.
I noticed that Fable uses shell tools almost exclusively (even to search and edit files), compared to previous Anthropic models.
Having run some experiments with 5.6, I notice that it uses built-in file systems and provider native tools much more (not shell tools), compared to previous OAI models.
Not specific to OpenAI / Codex, but I'm curious what people are doing to protect themselves from any destructive actions by their coding agents? Just install and pray? Explicity approve all actions? Reconfigure for safety? Run in a sandbox (Docker) ?
YOLO.
Btw for real tho, if you don't have the time or means to mess with full sandboxed environments, just working within a git repo and instructing on your agents.md project level that the agent should back up dirty files (local changes that were not yet committed) before changing them is enough and super fast and easy to set up. And by back up I just mean a simple instruction to back up to some temp location under random named, but rembered during one "turn" of agent thinking, subfolder ( .../temp/{random}/orginal/tree/file.xyz )
This is so the agent (or you later) can recover even locally changed files if it messes them up for whatever reason.
As for the rest you gotta watch what you're asking for, but generally speaking, these SOTA models are smart, none of them will just delete your stuff even with full access. I've been raw dogging multiple projects on my work machine with zero issues of this kind for months. I created codex_reader read only acccounts for my local databases and just add that to agents.md with a note its allowed to only use that and never had a problem.
Btw for real tho, if you don't have the time or means to mess with full sandboxed environments, just working within a git repo and instructing on your agents.md project level that the agent should back up dirty files (local changes that were not yet committed) before changing them is enough and super fast and easy to set up. And by back up I just mean a simple instruction to back up to some temp location under random named, but rembered during one "turn" of agent thinking, subfolder ( .../temp/{random}/orginal/tree/file.xyz )
This is so the agent (or you later) can recover even locally changed files if it messes them up for whatever reason.
As for the rest you gotta watch what you're asking for, but generally speaking, these SOTA models are smart, none of them will just delete your stuff even with full access. I've been raw dogging multiple projects on my work machine with zero issues of this kind for months. I created codex_reader read only acccounts for my local databases and just add that to agents.md with a note its allowed to only use that and never had a problem.
I literally have had a SOTA Codex agent delete a bunch of files just last month, but it took very specific circumstances. It was working within my game repo, in a branch, and it ran out of SSD. It had to free up some space to work, so it looked around and found 25GB of untracked files in the project, in a folder called Recordings. Might as well clean that up, it thinks. There goes all the raw Unity Recorder footage I'd ever recorded of my game, over many years of development.
So yeah, it won't go on a spree outside of its lane even with full access, but if you give it a box and tell it to go ham, it's on you to make sure you didn't leave precious unrecoverable assets in that box.
So yeah, it won't go on a spree outside of its lane even with full access, but if you give it a box and tell it to go ham, it's on you to make sure you didn't leave precious unrecoverable assets in that box.
I run codex in a dedicated vm, I have a cronjob which resets it to clean installed state every week. Nothing too fancy just bhyve and debian, 8gb mem. It has root access there, can install stuff, no permissions to push to protected branches etc. It didn't take very long to setup, and I can sleep a bit better...
I use the auto-reviewer for actions outside the builtin sandbox.
So far this has been rock solid, and tens of millions of developers use this setup without issue.
It is not going to wipe our hard disks. At least I hope so. Fable and GPT 5.6 have been ever more proactive, and GPT 5.6 is automating the AppStore on my machine to download an Xcode update while I am typing this.
So far this has been rock solid, and tens of millions of developers use this setup without issue.
It is not going to wipe our hard disks. At least I hope so. Fable and GPT 5.6 have been ever more proactive, and GPT 5.6 is automating the AppStore on my machine to download an Xcode update while I am typing this.
Is this auto-reviewer part of Codex? Is the review done by the agent or the model?
Yes. In Codex it is called 'Approve for me', in Claude it is 'Auto mode'.
I believe in both cases it is prompting a model with a fresh context that is tasked with reviewing the reason for the action.
With Claude, I have seen that if the reviewer does reject the proposed action, it responds with a long text about how the Agent should not try to work around this rejection, and instead prompt the user for an explicit approval of the proposed action.
I believe in both cases it is prompting a model with a fresh context that is tasked with reviewing the reason for the action.
With Claude, I have seen that if the reviewer does reject the proposed action, it responds with a long text about how the Agent should not try to work around this rejection, and instead prompt the user for an explicit approval of the proposed action.
Typically I just want to isolate the agent disallowing it from accessing other parts of the filesystem. Using a different user might be enough, but I typically use [bubblewrap](https://github.com/containers/bubblewrap).
Dunno, personally, I've had 5.5 lie to me. Which was a common occurence with weaker models, but I haven't really seen it that much with strong ones.
Basically what happened was there was a task which could either be solved in a simple way that yielded poor performance, but there was also a highly involved solution. Both Opus and 5.5 identified the steps necessary for the hard solution, but Opus gave up.
5.5 claimed it did it, but looking at the code, there were parts that were missing based on the plan. Basically 5.5 gave up halfway, did the easy solution, and claimed it was done.
Basically what happened was there was a task which could either be solved in a simple way that yielded poor performance, but there was also a highly involved solution. Both Opus and 5.5 identified the steps necessary for the hard solution, but Opus gave up.
5.5 claimed it did it, but looking at the code, there were parts that were missing based on the plan. Basically 5.5 gave up halfway, did the easy solution, and claimed it was done.
I live in fear lol.
More seriously, I was blindly trusting the auto-classifier from claude code (same as the middle option when you do `/permissions` in codex), and it actually allowed the agent to do pretty hardcore `rm` and `git push --force-with-lease` commands, which I would have expected to have to approve manually. Luckily no major issue from those yet.
The best option imo is the integrated cloud environments from claude code (not sure yet if there's a codex equivalent). It spawns a VM in the cloud where the agent runs, and you can open a PR from the app when it's done. Very smooth experience
More seriously, I was blindly trusting the auto-classifier from claude code (same as the middle option when you do `/permissions` in codex), and it actually allowed the agent to do pretty hardcore `rm` and `git push --force-with-lease` commands, which I would have expected to have to approve manually. Luckily no major issue from those yet.
The best option imo is the integrated cloud environments from claude code (not sure yet if there's a codex equivalent). It spawns a VM in the cloud where the agent runs, and you can open a PR from the app when it's done. Very smooth experience
Interesting - I'd never heard of this Claude Code VM option.
Does it auto install all the dev/test tools it needs, maybe including things like web server & browser? Does your code live in the VM, or in some external repository? Is the lifetime of the VM the same as the agent, or does it persist until you remove it?
Where can I find documentation on this?
Does it auto install all the dev/test tools it needs, maybe including things like web server & browser? Does your code live in the VM, or in some external repository? Is the lifetime of the VM the same as the agent, or does it persist until you remove it?
Where can I find documentation on this?
I use it for basic web stuff so haven't pushed it to the limit, but they have tons of stuff you can configure (up to self-hosting your environments): https://platform.claude.com/docs/en/managed-agents/environme...
Thanks!
I still just explicitly approve all actions and review all code (unless it's a personal/throwaway project no one else will ever touch/use/see). I know a lot of people that run in a sandbox though. That said, I'm sure there are lots of people that just yolo it and hope for the best.
What destructive actions are you afraid of in particular? Honestly the models are pretty smart, I let the agents go --yolo and nothing bad has ever happened (yet) that couldn't be solved with git.
I'm not concerned about the code it's working on, but rather anything else - modifying files outside of the project dir (e.g. incorrect tool call), modifying system configuration, doing something bad on the internet, etc.
It is a valid concern but I've been running yolo mode since the inception, using Claude and now Codex. I'm not bragging or anything, since I'm feeding my own curiosity too, trying to answer what's the worst that could've happened? So far, nothing catastrophic that I recall.
[deleted]
Don't let it outside the sandbox. Don't let it have access to anything but dev environments. Continue using git.
Never had any issues.
Never had any issues.
Which agent/sandbox are you referring to?
I am looking to rent an apartment in a new residential tower. I have asked Fable and Sol to scrap the listings from various sources, deduplicate them and present them as a web application. Just using the cowork/(ex-)codex application interfaces.
Fable had issues with the sourcing and organizing images, and shoot itself at foot looking for shortcuts as usual. As I was getting it fix these back and forth, I copied my prompt and gave it to Sol.
Sol has surpassed my expectations by far. With a one shot simple prompt on a complex task, it gave me a working web app with everything I want with minor issues to track and fix.
Fable had issues with the sourcing and organizing images, and shoot itself at foot looking for shortcuts as usual. As I was getting it fix these back and forth, I copied my prompt and gave it to Sol.
Sol has surpassed my expectations by far. With a one shot simple prompt on a complex task, it gave me a working web app with everything I want with minor issues to track and fix.
Is any of those comparisons about Pro vs non-Pro (Pro is only available in $100+ plans)? I am curious about that but I think Sol, Terra, Luna are different sizes of it without the Pro part, and I want to know how much worse do I have it on the $20 plan compared to if I upgrade.
I’m interested in knowing how each of GPT 5.6’s variants fare in non-English writing/translation tasks.
GPT 5.5 has a tendency to write English calques and non-idiomatic prose in other languages. Although that can be somewhat tamed with detailed instructions and a corpus of confusing terms, the model’s output often reads like a literal translation rather than native prose. Since I notice these issues most clearly in languages I know well, it makes me reluctant to trust the model’s output in languages in which I’m less proficient.
Ironically, ChatGPT began as a simple text-generation tool, but much of its offerings and benchmarks now focus on coding and agentic workflows, while leaving behind what made it notable in the first place.
GPT 5.5 has a tendency to write English calques and non-idiomatic prose in other languages. Although that can be somewhat tamed with detailed instructions and a corpus of confusing terms, the model’s output often reads like a literal translation rather than native prose. Since I notice these issues most clearly in languages I know well, it makes me reluctant to trust the model’s output in languages in which I’m less proficient.
Ironically, ChatGPT began as a simple text-generation tool, but much of its offerings and benchmarks now focus on coding and agentic workflows, while leaving behind what made it notable in the first place.
I have Fable send the specs/plans it comes up with to GPT for review and in 2/5 cases yesterday it found additional 1-2 bugs while in the process of reviewing.
GPT-5.6 didn't try to fix the bugs (as instructed) but it did surface them, which is something that didn't happen with GPT-5.5. When spec/plans approved Fable sends back to GPT-5.6 for ralph implementation and it seems to be an even faster, more reliable workhorse than it already was in GPT-5.5. Overall, impressed. Will continue to be a core piece of my workflow.
GPT-5.6 didn't try to fix the bugs (as instructed) but it did surface them, which is something that didn't happen with GPT-5.5. When spec/plans approved Fable sends back to GPT-5.6 for ralph implementation and it seems to be an even faster, more reliable workhorse than it already was in GPT-5.5. Overall, impressed. Will continue to be a core piece of my workflow.
> average daily output tokens per active researcher were more than twice the highest level observed for GPT‑5.5.
I use 5.5 a ton. It's immediately apparent that 5.6 is truly a better model. Hope they don't lobotomize it later.
8% on ARC-AGI-3, they actually got some traction going...
note that ARC-AGI-3 has its rules changed.
before today all the contestants were capped at $10k
before today all the contestants were capped at $10k
I can't try it since it hasn't appeared in my Codex yet, but this is is necessary from OpenAI in my opinion. Fable is just so much better at understanding broad context. I only use GPT 5.5 for straight forward easy to describe tasks, and it does crush those. But I spend a lot more time steering Codex towards good design on broad concept type tasks, ones that Fable shows sometimes surprising clarity.
I look forward to seeing how it compares once I have access. Not getting tripped by spurious safe guard flags could be an advantage.
I look forward to seeing how it compares once I have access. Not getting tripped by spurious safe guard flags could be an advantage.
Just used terra ultra for exactly one prompt in codex and it ate through my full 5h window in about 10mns (20$ plan). The results look pretty good though. Luckily I have had my chatGPT subscription for a while and have a bunch of resets available (nice compared to anthropic).
Assuming I take the 5x plan it would give me about an hour of active sessions with terra ultra (maybe ultra is not good value regarding tokens?), not even using Sol yet. Does everyone using codex use the 200$ plan?
I normally use the 100$ anthropic plan and barely ever reach the usage limit.
Assuming I take the 5x plan it would give me about an hour of active sessions with terra ultra (maybe ultra is not good value regarding tokens?), not even using Sol yet. Does everyone using codex use the 200$ plan?
I normally use the 100$ anthropic plan and barely ever reach the usage limit.
> maybe ultra is not good value regarding tokens?
Well, yes, as explicitly stated on https://openai.com/index/gpt-5-6/: "ultra goes further by coordinating four agents in parallel by default, trading higher token use for stronger results and faster time-to-result on demanding tasks."
Well, yes, as explicitly stated on https://openai.com/index/gpt-5-6/: "ultra goes further by coordinating four agents in parallel by default, trading higher token use for stronger results and faster time-to-result on demanding tasks."
thanks, it makes sense, I'll stick to max from now on
I use the $20 plan, but I don't code all day every day.
With Codex, it is my experience that I can churn through a 5h window in no time with newer models -- especially when they're new. So I tend to use fancier models for planning, and the less-fancy models for writing code based on that plan. I switch to the fanciest model if any part of this gets stuck.
If I've got a something big-ish to work on, I pay attention to the reset timers so I can get more of it done in one chunk.
Models seem to slowly get better/relatively less-expensive as they age. (It isn't clear to me if that's because the cost actually goes down, or if the allotment goes up, or if things get more efficient in unseen ways, or what. OpenAI is vague AF about what we get for the $20 that we pay.)
With Codex, it is my experience that I can churn through a 5h window in no time with newer models -- especially when they're new. So I tend to use fancier models for planning, and the less-fancy models for writing code based on that plan. I switch to the fanciest model if any part of this gets stuck.
If I've got a something big-ish to work on, I pay attention to the reset timers so I can get more of it done in one chunk.
Models seem to slowly get better/relatively less-expensive as they age. (It isn't clear to me if that's because the cost actually goes down, or if the allotment goes up, or if things get more efficient in unseen ways, or what. OpenAI is vague AF about what we get for the $20 that we pay.)
Same for me, but it produced much better answer than what would I expect from GPT 5.5 xhigh. Also the good think about Codex is that it always finishes the prompt, even when the limit was reached in the half.
Do you know if you used sol/terra/luna?
So glad Fable limits just got reset. Thanks OpenAI.
Oh hey, thanks for the hint!
Based on the Intelligence vs. Cost graph, not clear to me why anyone would use Terra? Luna looks quite interesting though, happy to see OpenAI still serving the more budget-oriented side of the market (seems like Anthropic and Google have lost interest there).
https://artificialanalysis.ai/articles/gpt-5-6-has-landed
https://artificialanalysis.ai/articles/gpt-5-6-has-landed
Luna@max is in a VERY interesting spot if their rankings are at all to be believed:
- Better than Opus4.8 in the coding agent index (doubt)
- Just below sonnet 5, even with glm5.2, in the overall intelligence index
- Cheaper than haiku4.5, glm5.2 and kimi2.6 on cost per intelligence task index
- Better than Opus4.8 in the coding agent index (doubt)
- Just below sonnet 5, even with glm5.2, in the overall intelligence index
- Cheaper than haiku4.5, glm5.2 and kimi2.6 on cost per intelligence task index
Cost and intelligence aren't the only axes. Terra has better latency and output speed than Sol for example.
Still fails my internal test of counting legs on animals who have had extra legs photoshopped in. However if prompted to determine what is wrong with the image, it does get it right.
This kind of "out of bounds" image analysis seems to be a very difficult problem to solve, but totally necessary for transformers to really bring about massive change.
This kind of "out of bounds" image analysis seems to be a very difficult problem to solve, but totally necessary for transformers to really bring about massive change.
Will this run on Cerebas? I'm really looking forward to that.
Sam Altman confirmed during the initial limited release that Sol will run on Cerebras at 750 tok/sec.
"I canna' give her any more, Captain!" - Montgomery "Scotty" Scott, Chief Engineer
This is the part I'm most excited about with the new release, though I'm concerned plebs like me may never get a chance to play with it
5.6 Sol High Fast is using more capacity than 5.5 High Fast, I hit the 5h limit for the first time.
Other than that, I think the difference between 5.5 and 5.6 will be the same as 5.4 and 5.5. 5.5 is just less frustrating to use, although not perfect and still has derp moments. But a lot less than 5.4.
So I expect 5.6 Sol to be smoother to use. But so far it just feels slower. We'll see.
Other than that, I think the difference between 5.5 and 5.6 will be the same as 5.4 and 5.5. 5.5 is just less frustrating to use, although not perfect and still has derp moments. But a lot less than 5.4.
So I expect 5.6 Sol to be smoother to use. But so far it just feels slower. We'll see.
I've found Sol's propensity for delegating to subagents can make it... disastrously expensive, especially with each subagent having some implicit floor on further reasoning/context gathering before action.
The base model is certainly cheaper and more token efficient etc, but on large tasks cost in some way is now n^2
The base model is certainly cheaper and more token efficient etc, but on large tasks cost in some way is now n^2
I have the codex x20 plan also. Set it to 5.6 sol ultra, gave it one task, in less than 10 mins I got the 10% warning. I did have it do a few things earlier, just some anaylsis and then create a pdf from it. I thought maybe that took a little longer than I though.
Used a reset, it went for about 11 minutes and then, just out of usage popped up, no warning, there were still 4 agents running, the nice thing is it did let them finish, each one went for about 10 minutes.
I also have a claude max plan, I have been using Fable 5 on ultra, I never hit the session limit, and get 3 or 4 full day's looping on ultra.
I don't know how it handles the subagents, but claude does it much more efficiently, Codex does seem much faster, so maybe it's just a relativity thing.
Used a reset, it went for about 11 minutes and then, just out of usage popped up, no warning, there were still 4 agents running, the nice thing is it did let them finish, each one went for about 10 minutes.
I also have a claude max plan, I have been using Fable 5 on ultra, I never hit the session limit, and get 3 or 4 full day's looping on ultra.
I don't know how it handles the subagents, but claude does it much more efficiently, Codex does seem much faster, so maybe it's just a relativity thing.
How bad is it for you? Are you on ultra or xhigh/max? I typically ask it (5.5, now 5.6-sol) to use subagents for specific things anyway. On the Pro 20x plan, I'm seeing like ~1% usage per 20-30 min per session (on max effort), which is in line with 5.5.
Currently trying out ultra on a personal project, feels like ~3x more expensive per unit time. (No idea on quality yet, for obvious reasons.)
Max via Cursor, should've mentioned, very possible I'm seeing more Cursor than OAI. All the subagents it picked were also Sol Max... I've seen 7-10 in one turn. Regardless the top reasoning tiers being conflated with subagent count feels double edged
My global CLAUDE.md/AGENTS.md etc have strict rules to never use subagents without approval or when specifically requested except for whitelisted scenarios.
I think that's the way to go most of the time
I think that's the way to go most of the time
Zero information on the knowledge cutoff. The model itself responds it's June 2024 which is weird given that GPT-5.5 has knowledge cutoff at August 2025.
GPT-5.6 is the first model where I’ve actually frustrated to use it.
I’m explicitly telling it to do something extremely specific and it’s just not listening to me.
Eg, I gave it an image to update. The image is sized 400x200 pixels. It then generates a new image at 300x300. I explicitly state to be 400x200 in size and it won’t listen.
I’m explicitly telling it to do something extremely specific and it’s just not listening to me.
Eg, I gave it an image to update. The image is sized 400x200 pixels. It then generates a new image at 300x300. I explicitly state to be 400x200 in size and it won’t listen.
Is it possible GPT-5.6 is not a very aligned model?
Dirac (https://github.com/dirac-run/dirac, https://dirac.run/) now supports gpt-5.6. This thing does now seem to be on the chatGPT/codex accounts yet.
UPDATE: it is now available in chatGPT account also, they rolled it out
UPDATE: it is now available in chatGPT account also, they rolled it out
Will be there soon according to the last commits in the codex repo:
https://github.com/openai/codex/pull/31684/changes
Also, confirmed it works for me by using --model gpt-5.6-sol
Also, confirmed it works for me by using --model gpt-5.6-sol
I used to pride myself on not being the "fonts too pointy, scroll too buttery" crowd! But AI has brought me full circle and now nothing removes my interest in reading even a single word on a page faster than purple gradient greeble-afflicted tailwind-slop models put out without stronger prompting/references
That being said, maybe 5.6 can fix that!
That being said, maybe 5.6 can fix that!
Thanks, I needed to hear that lol. Yes, the site was an afterthought, core work took/takes most my focus. I will look into un-slopping the site soon.
I think 5.6 Sol is only as good as 5.5 or Opus 4.8 in terms of getting its given work done. It just has an uncanny ability to pickup more work that it can tackle next that the older models lack, or have not been trained to do before.
Where folks are seeing a difference between working with Fable or 5.6 I think also boils down to this phase shift.
I find that 5.5 gives me far fewer refusals than Anthropic models for security and reverse engineering work. I hope the same is true for 5.6.
Yeah, I pretty much had to switch to using GPT rather than Opus completely for all my security benchmarking and harness development. I was annoyed enough to blog about it: https://swelljoe.com/post/why-i-had-to-switch-to-gpt/
I would be really interested in real life throughput. For an agentic chat situation, we are still on 5.4 - not because of the cost, but it's simply much faster than 5.5 with comparable results. Also we are using gpt-5.4-mini a lot for quick summaries, tldrs etc.
In an ideal world we would upgrade 5.4 to 5.6 terra and 5.4 mini to 5.4 luna. But does somebody already have some measurements at least in terms of speed?
Not sure what everyone's experience is but I find 5.6 Sol to be a great liar. Reported success on a half done job and left things in a broken state after having quite a few back & forth followups on the initial prompt to clarify the plan. Didn't experience this with 5.5. Opus 4.7 and below sometimes did it but they fixed it in Opus 4.8. So, overall, the initial experience has made me think that this model will be a lot more stressful to work with just because the level of trust that it actually completes the task is now much much lower.
On the tiny voids demo: does your Firefox js thread lock up as well, when you try to interact with it?
https://openai.com/index/gpt-5-6/#a-leap-forward-in-design
https://openai.com/index/gpt-5-6/#a-leap-forward-in-design
Yep, happens to me on Chrome as well
Oh man, I love capitalism spoiling us here. I was just enjoying my extra Fable credits, now I'll switch to using 5.6 this weekend. I was planning to ration my Anthropic credits, I guess now I do not have to. And I was half wondering if exactly this would happen: right when Fable usage credits were starting to kick in for people, OAI swoops in and takes the puck. As much the AI craze is crazy, this play by play part is pretty fun.
top it off with anthropic stressing about the release and resetting usage to 0 for the week just now.
Anthropic just reset all limits, including Fable. Capitalism is spoiling us.
Make hay while the sun is out.
If hoarding is spoiling. You know what would be better than using fable and gpt 5.6, being able to run that level of model on your own hardware.
There is an issue on the page that causes the benchmark tables to get cut off. If you highlight and drag right you can see a few more models like Gemini and Claude Opus. It's also interesting that they introduced explicit caching, which is something that only Anthropic had for a long time.
I am seeing some bugginess in testing:
I hope we aren't trying to push customers off the chat completion endpoint... Responses endpoint looks great on paper, but the business wants more visibility and control over the reasoning process than this product currently offers.
Edit: This is broken in my VS copilot setup too.
Parameter: reasoning_effort
Function tools with reasoning_effort are not supported for gpt-5.6-sol in /v1/chat/completions.
To use function tools, use /v1/responses or set reasoning_effort to 'none'.'
I hope we aren't trying to push customers off the chat completion endpoint... Responses endpoint looks great on paper, but the business wants more visibility and control over the reasoning process than this product currently offers.
Edit: This is broken in my VS copilot setup too.
> Instead of requiring developers to script every step or passing every tool response back through the model, Programmatic Tool Calling in the Responses API can filter large amounts of intermediate data, retain only what matters, and adapt its workflow along the way.
this seems very interesting
this seems very interesting
The claims are pretty bold. I think 5.6 may exceed Fable.
Looks like a great set of models, but there are about 20 different thinking/model levels here in this family and they are very complex to pick the right one for the task
E.g. for GeneBench Pro, it looks like you would always use GPT-5.6 Sol over Terra/Luna, its pareto optimal.
For Agents Last Exam, you would maybe want Luna, then Terra, then Luna, then Sol as you increasingly budget for tasks.
I feel that there may need to be a new auto mode in many of these cases. It selects the best model and thinking given a particular problem.
Feels like it's going to have to go that way eventually, because here we have about 20 different model and thinking levels you could use, and they're not obvious which ones are right for the given use case.
E.g. for GeneBench Pro, it looks like you would always use GPT-5.6 Sol over Terra/Luna, its pareto optimal.
For Agents Last Exam, you would maybe want Luna, then Terra, then Luna, then Sol as you increasingly budget for tasks.
I feel that there may need to be a new auto mode in many of these cases. It selects the best model and thinking given a particular problem.
Feels like it's going to have to go that way eventually, because here we have about 20 different model and thinking levels you could use, and they're not obvious which ones are right for the given use case.
I wish model launches were like proper product releases
it's impossible to _try_ it out on release!
it's not on their codex subscription, or the web/mobile chatgpt interfaces, or aws bedrock, etc. I just cant find a working endpoint with the latest model after they announce
it's impossible to _try_ it out on release!
it's not on their codex subscription, or the web/mobile chatgpt interfaces, or aws bedrock, etc. I just cant find a working endpoint with the latest model after they announce
The announcement says they're rolling it out over the next 24 hours or so. I think it's reasonable to do a slow-roll-out release for one of the most used products on the internet.
For me, minutes ago, as a Plus subscriber:
I started up Codex CLI fresh. That version of Codex was 1.42.5. 5.6 wasn't in the models list.
After I updated Codex to a newer version (0.144.0), 5.6-terra and -luna appeared in the models list (but not 5.6-sol).
(It's impossible for me to know whether updating was causative or just correlative, but that's the timeline I experienced.)
I started up Codex CLI fresh. That version of Codex was 1.42.5. 5.6 wasn't in the models list.
After I updated Codex to a newer version (0.144.0), 5.6-terra and -luna appeared in the models list (but not 5.6-sol).
(It's impossible for me to know whether updating was causative or just correlative, but that's the timeline I experienced.)
GPT-5.6 Terra just showed up in Codex for me.
One of my best use cases for the short duration I have fable is to use it to create the plan and acceptance test files then use GPT 5.5 Pro to do an adversarial review on the plan then feed that feedback into fable to fix the plan.
Looks like I have access to gpt-5.6-terra and luna. How does one decide between gpt-5.5 and gpt-5.6-terra? Pricing is similar, but it's hard to tell if it's better..
Wow, the "Agents' Last Exam" graph looks unreal!
I mean the y axis is deceptive to make it seem like greater gains since it starts at 30%, when in reality the differences aren't great.
Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.
What about the graph looked so unreal to you?
Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.
What about the graph looked so unreal to you?
> Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.
We agree models should be compared on a fair basis. Unfortunately, adaptive was the only publicly available number. Anthropic doesn't generally let us run their models for evals, so we rely on whatever Anthropic or third parties have published. In this case, the Agents' Last Exam leaderboard has Fable Adaptive, but not Fable Max.
https://agents-last-exam.org/leaderboard
Would have loved to publish a full curve for Fable if anyone makes the data available.
Although we do bias toward publishing evals where we're ahead, we have historically been unafraid to publish evals where we're behind (e.g., GDPval). The point is give people useful information to decide what's best, not to trick people.
Edit: Now I see there's a second entry with xhigh effort. Not sure if that was added or recently or we skipped it.
(I work at OpenAI.)
We agree models should be compared on a fair basis. Unfortunately, adaptive was the only publicly available number. Anthropic doesn't generally let us run their models for evals, so we rely on whatever Anthropic or third parties have published. In this case, the Agents' Last Exam leaderboard has Fable Adaptive, but not Fable Max.
https://agents-last-exam.org/leaderboard
Would have loved to publish a full curve for Fable if anyone makes the data available.
Although we do bias toward publishing evals where we're ahead, we have historically been unafraid to publish evals where we're behind (e.g., GDPval). The point is give people useful information to decide what's best, not to trick people.
Edit: Now I see there's a second entry with xhigh effort. Not sure if that was added or recently or we skipped it.
(I work at OpenAI.)
That’s because it’s bullshit
We have an official pelican on a bicycle from the OpenAI livestream:
https://imgshare.cc/mz9xwut3
https://imgshare.cc/mz9xwut3
holy moly it's in THREE dimensions!
AGI solved
AGI solved
So it's failing epically because it generated a tricycle instead of a bicycle?
Logged into the OpenAI platform this morning and had to double check they hadn't pivoted into a crypto company with these new names
So with this release do they kill the 5.5-Pro model with super long thinking and reasoning? 5.6-Sol-Ultra is not the equivalent, right?
I use both Claude and Codex, but mostly Claude for planning and coding, and Codex to review Claude’s work.
I follow a sort of waterfall workflow which is verbose but fully transparent.
Anthropic’s $100 subscription works fine for me, but whatever subscription my company has with OpenAI reaches the 5hr limit ridiculously quickly.
I follow a sort of waterfall workflow which is verbose but fully transparent.
Anthropic’s $100 subscription works fine for me, but whatever subscription my company has with OpenAI reaches the 5hr limit ridiculously quickly.
When I was going through this it was because OpenAI had defaulted to /fast mode with 2x token usage
> Your subscription to Claude Max has been successfully canceled.
Switching next month. Looking forward to working with Sol.
Switching next month. Looking forward to working with Sol.
Anybody have an idea of what the flops per token generated is on a SOTA model like current GPT/OPUS? Is it basically the parameter count? So something like GLM-5.2 is, at a minimum, ~744 GFLOPs per token generated?
Am I way off base? Seems astronomical.
Am I way off base? Seems astronomical.
GPT-5.6 Sol, Terra, and Luna. at this rate GPT-6 will be named after a parking lot and GPT-7 after whatever Elon names his next kid.
Where is Gemini in all this? Lately it's not even been in the running. Sir Demis asleep at the wheel? Or Google too scared to release a SOTA model?
Or ... maybe Gemini 4 is too good and the NSA is using it to break into systems worldwide ...?
Or ... maybe Gemini 4 is too good and the NSA is using it to break into systems worldwide ...?
The rumor is Gemini 3.5 pro will be released next week.
It had previously been announced (at Google I/O) to be released in June, but it got pushed back. Why? Rumors range from they are having trouble to they scrapped an old framework to pursue a new one and the new model will surpass Fable 5 and be cheaper.
It had previously been announced (at Google I/O) to be released in June, but it got pushed back. Why? Rumors range from they are having trouble to they scrapped an old framework to pursue a new one and the new model will surpass Fable 5 and be cheaper.
Google pretending they're still far ahead. In reality, far far behind. Google has become "just like another" company, nothing special. Few bright minds, with a lot of overpaid engineers.
I expect Google will drop a new SOTA model soon. Rumor has it they're training a new model from the ground-up, which takes a while.
On top of GPT 5.6 Sol they added a Tamagotchi / Clippy mascotte
https://x.com/giorgio_zampa/status/2075319657997750495?s=20
What in the world is that? Why.
No, this Pets feature has been around for a while now.
I never have have the issues most people talk about ... I feel like most were never Devs before ai and don't know what they actually need done when prompting. that on top of not utilizing good tools such as a codebase indexer, lsp and a project scaffold.
It scores high on BenchCAD, that's interesting to see, I was wondering about how each model could handle this. Seems like they trained it on programmatic CAD specifically.
> GPT‑5.6 Terra and GPT‑5.6 Luna outperform Fable 5 at around one-sixteenth the cost
If that's the case, we're definitely going to have some Mythos level open weights models by the end of the year. Maybe even at a size that can run on a ~40k locally hosted server.
If that's the case, we're definitely going to have some Mythos level open weights models by the end of the year. Maybe even at a size that can run on a ~40k locally hosted server.
If it's not dangerous enough to be classified as WMD by USG, who's interested.
I've played around GPT 5.6 sol high at both work and home.
At work, it was able to one shot a dashboard. Of course, my prompts are vague as I'm not exactly sure what I want yet, but it did a better job than I could do as a backend dev forced to work on frontend sometimes.
Usage is also great, it just feels so much more efficient than older models in terms of thinking and time. Cost is barely better though.
It can burn a million tokens in less than a minute, at least at launch where there's likely less load on the servers.
At home, it feels like I'm fighting the AI less while letting it refactor code. I'm glad that I left this 12,000 line vibe coded port of a hand written codebase to future models to refactor. It feels like the model has better judgement than old models that would destroy your codebase so long as it meant accomplishing your prompt.
I'm almost disappointed that it's this good.
At work, it was able to one shot a dashboard. Of course, my prompts are vague as I'm not exactly sure what I want yet, but it did a better job than I could do as a backend dev forced to work on frontend sometimes.
Usage is also great, it just feels so much more efficient than older models in terms of thinking and time. Cost is barely better though.
It can burn a million tokens in less than a minute, at least at launch where there's likely less load on the servers.
At home, it feels like I'm fighting the AI less while letting it refactor code. I'm glad that I left this 12,000 line vibe coded port of a hand written codebase to future models to refactor. It feels like the model has better judgement than old models that would destroy your codebase so long as it meant accomplishing your prompt.
I'm almost disappointed that it's this good.
The cost & output token charts are useful but I wish I could view them more like a 3D surface. Like the CS:APP memory mountain charts.
I wonder how long model size and effort will be a few discrete points instead of continuous.
I wonder how long model size and effort will be a few discrete points instead of continuous.
GPT‑5.6 system card https://deploymentsafety.openai.com/gpt-5-6/gpt-5-6.pdf
it seems terra is pretty much useless, you either want luna max for everyday coding (cheaper and same perf as 5.5 high), or sol xhigh/max for demanding tasks
Maybe it’s a bug but on iOS individual paid Pro account - I can no longer see which model is being used nor select which model I want.
For context, I have access to MS Copilot through my workplace. To see what it looks like, I have tried to login through https://copilot.microsoft.com/ , where I was informed that my account, although recognised, is not yet supported. However, I can get more or less the same chat window, with access to all the data, through https://m365.cloud.microsoft/ A redirect could have been useful.
"GPT‑5.6 is available starting today across ChatGPT, Codex, and the OpenAI API. The rollout is starting globally now and will continue gradually toward full availability over the next 24 hours."
I am on Plus subscription and see Terra and Luna in Codex, but no sign of Sol. Will it be available only on Pro plans?
I an on Pro and it still returns "The 'gpt-5.6' model is not supported when using Codex with a ChatGPT account"
UPD from announcement: "The rollout is starting globally now and will continue gradually toward full availability over the next 24 hours."
UPD from announcement: "The rollout is starting globally now and will continue gradually toward full availability over the next 24 hours."
My Codex app got upgraded to the new unified ChatGPT app. I don't see Sol available though. Only Terra and Luna. I'm on the Pro plan. Anyone else see it?
It looks like ChatGPT FINALLY has artifacts. I can't believe this took over a year or two, it was easily Claude's biggest edge.
devops monster! crazy how intelligently it debugs/solves any devops problem I could throw at it!
Is it available in EU? I only see 5.5 still :-(
Arguably not in the EU, but I'm seeing Sol,Terra and Luna on my account here in Norway
This marketing video on the page is nice!! can't wait for the hardware to get cheaper to live the AI life i wanna live.
Trying to play those games has really bad impact on PC performance
I wish they had kept their previous sensible naming convention instead of this celestial Sol, Terra, and Luna mumbo-jumbo
I assume they're jealous of the Fable/Mythos hype. People talk about Fable like it's a whole new thing, rather than another incremental improvement over the existing best models (which has happened several times and continues to happen).
prompts -> loops -> slingshots?
Its an extremely capable model. I think the way we need to approach works shifts again. We need to get our harnesses/workflows to let it gather some momentum on the first couple rounds but then we also need to structure it so that it can slingshot and accomplish the long range goal.
Its an extremely capable model. I think the way we need to approach works shifts again. We need to get our harnesses/workflows to let it gather some momentum on the first couple rounds but then we also need to structure it so that it can slingshot and accomplish the long range goal.
I hope it isnt like Opus eating so many tokens and taking so much time
Really wanna see it in DeepSWE benchmark
Really wanna see it in DeepSWE benchmark
we probably need to use gpt sol max to decide which gpt flavor and effort we need to use per task.
it seems like 5.6 SOL is better at almost everything than Mythos except Coding Benchmarks (except TerminalBench)? anyone knows why Mythos scores so high on SWEBench are they cheating or are they just optimised better for coding?
where is it? Still not accessible...
"GPT‑5.6 is available starting today across ChatGPT, Codex, and the OpenAI API. The rollout is starting globally now and will continue gradually toward full availability over the next 24 hours."
They have a fantastic media team.
5.6 sol ultra just nuked my branch and burned my 5h limit. nice work
i wish they had renamed chatgpt to codex instead of the other way around ...
I wonder what increment of progress will be achieved by the next billion dollars
GPT 5.6 Sol is a token hog. After implementing the task, it started some "reviews" I didn't ask for - they consumed 19.5M and 11.9M tokens, while the task itself was below 5M tokens.
So is 5.5-Daybreak still relevant for cyber security give. 5.6 capabilities?
Overloaded in Codex, no indication if it is already in ChatGPT and I can't use it in the API even though it says it should be available. Typical horrible OpenAI launch. Glad that Anthropic just reset the rate limits so I will go back to Fable again.
5.6 SOL is basically useless, even on fast mode. It takes so long to do anything that it would be faster to do yourself. And it burns usage so quickly it's genuinely not worth it.
> This page couldn’t load
> Reload to try again, or go back.
This on iOS, safari
> Reload to try again, or go back.
This on iOS, safari
are people getting the `<!-- -->` sentinel'd reasoning summaries?
From what I see on ArtificialAnalysis.ai, OpenAI cranked up the reasoning effort to surpass Fable, and now the cost/task for 5.6 max is closer to the Claude Opus 4.7 and 4.8 cost/task.
Sounds like a perfect fit for a minimal or bespoke harness?
does anyone on chatgpt business plan (not enterprise) not have access to the Sol models in codex? i have 5.6 for terra and luna but not sol
they update these shits too much.
> GPT‑5.6 also introduces more predictable prompt caching, including support for explicit cache breakpoints (opens in a new window) and a 30-minute minimum cache life.
Great to read they are moving away from the 5 minute cache defaults. Hopefully other providers follow soon!
Great to read they are moving away from the 5 minute cache defaults. Hopefully other providers follow soon!
They highlight the cache write price now much more in the guide. Did it increase vs. prior generations?
There was no cache write before!
https://openrouter.ai/openai/gpt-5.5?endpoint=58e5b336-423e-...
vs
https://openrouter.ai/openai/gpt-5.6-sol?endpoint=a54c5de0-8...
https://openrouter.ai/openai/gpt-5.5?endpoint=58e5b336-423e-...
vs
https://openrouter.ai/openai/gpt-5.6-sol?endpoint=a54c5de0-8...
After some time with it...
It has a tendency to do things without asking, a trait I'd associated more with the Claude Opus & Sonnet models than with Codex & GPT in the past. Specifically I've seen it go and update e.g. README.md files filling it with recenty-biased gibberish that means nothing to the user (e.g. very specific technical notes related to what it was currently working on) or staging and adding design/spec documents that were meant to just be working documents. In general it tends to behave more aggressively with git, if you let it get its hands on it. It has stronger "opinions" on that stuff, that don't always agree with me.
I'm going to have to update my prompts, I think. But I'm not used to this kind of thing in Codex, which in the past has been much more explicit and cautious, and one of the reasons I've preferred it over Claude.
It is very "smart." It also has a tendency to yak-shave things. Producing huge volumes of correctness and regression tests and nitting over e.g. very minor variances.
One thing that is "entertaining" is letting two separate instances review each other's code. They will endlessly find things to nit at.
It has a tendency to do things without asking, a trait I'd associated more with the Claude Opus & Sonnet models than with Codex & GPT in the past. Specifically I've seen it go and update e.g. README.md files filling it with recenty-biased gibberish that means nothing to the user (e.g. very specific technical notes related to what it was currently working on) or staging and adding design/spec documents that were meant to just be working documents. In general it tends to behave more aggressively with git, if you let it get its hands on it. It has stronger "opinions" on that stuff, that don't always agree with me.
I'm going to have to update my prompts, I think. But I'm not used to this kind of thing in Codex, which in the past has been much more explicit and cautious, and one of the reasons I've preferred it over Claude.
It is very "smart." It also has a tendency to yak-shave things. Producing huge volumes of correctness and regression tests and nitting over e.g. very minor variances.
One thing that is "entertaining" is letting two separate instances review each other's code. They will endlessly find things to nit at.
Here's me using a Gemini chat log scraper (from Gdrive) then dumping my prompt+Gemini response into local AI
Never go over the free limits in Gemini Pro.
Gemini is great at research and architecture, and my 30 years experience in programming everything; for fun or work; means together there is little to no code slop.
Add to project repo some git submodules of reference source code; boom, bobs your uncle
Zero reason to sign up for OAI or Claude. With employers realizing the costs are more than employees, local models getting more powerful, and models in chips just a few years out, neither of the one note LLM companies without diversified services and R&D portfolios gonna last
Never go over the free limits in Gemini Pro.
Gemini is great at research and architecture, and my 30 years experience in programming everything; for fun or work; means together there is little to no code slop.
Add to project repo some git submodules of reference source code; boom, bobs your uncle
Zero reason to sign up for OAI or Claude. With employers realizing the costs are more than employees, local models getting more powerful, and models in chips just a few years out, neither of the one note LLM companies without diversified services and R&D portfolios gonna last
cant wait to try this with frontend. gpt-5.5 litters pages with microcopy everywhere.
cursor benchmarks with GPT 5.6 in picture, a good reason to stop using opus.
https://cursor.com/evals
The good news you don't have to send your dollars to China to fund ai dictatorship, in russia, north korea, african countries and south america.
https://cursor.com/evals
The good news you don't have to send your dollars to China to fund ai dictatorship, in russia, north korea, african countries and south america.
so the answer is use grok ?
I'd say answer , the opus is no longer undisputed. grok + gpt models are very competitive + glm if you are ok to wait 3-4 times longer, unless you have some unique access to GPU
Almost immediately ran into some the kind of gatekeeping I've heard Claude Code users complaining about with Fable. Not sure why, I just had it working on writing benchmarks for some CUDA kernels. Nothing security related:
"This request requires additional safety checks, which can take extra time. Hang tight or retry with a faster model for a quicker response, though it may be less capable of handling complex requests."
At least it gave me the option of waiting instead of just unceremoniously downgrading me. Appears to be making progress but... weird?
"This request requires additional safety checks, which can take extra time. Hang tight or retry with a faster model for a quicker response, though it may be less capable of handling complex requests."
At least it gave me the option of waiting instead of just unceremoniously downgrading me. Appears to be making progress but... weird?
I think the most interesting part of this is that OpenAI is going way easier on the classifiers than Anthropic. They explicitly state that many defensive cybersecurity uses are supported and implicitly criticize Anthropic's stance on Fable's uses by saying that overblocking cyber requests is itself a major security risk as more AI models continue to advance in intelligence. I have so many questions as to what is going on on a game theoretic level in the AI space in the past two months, it seems like multiple actors have realized their incentives are really quite different than they originally thought.
For writing GPT which i was subscribed to Fall 2024 to March 2026 (laid off) is superior to Gemini. Been using Gemini since March mostly and they offered a $10 a month plan so i took it. Though today realizing GPT is superior to help me write I am back to being a paying customer. Im in full swing mode to get back into the job market (get the heck away from UI/UX which is now a stupid career in terms of number of jobs out there and in the future there will continue to be less) pivoting into product management (can vibe code anything now) and or customer relations. Hopefully GPT helps me with this pivot and Im again gainfully employed!
>Even at medium reasoning, it beats Fable 5 by 11.4 points at roughly one-quarter the estimated cost.
Sounds great.
Also latency looks very good.
Sounds great.
Also latency looks very good.
"On Agents’ Last Exam (opens in a new window), an evaluation of long-running professional workflows across 55 fields, GPT‑5.6 Sol sets a new high of 53.6, eclipsing Claude Fable 5 (adaptive reasoning) by 13.1 points. Even at medium reasoning, it beats Fable 5 by 11.4 points at roughly one-quarter the estimated cost. That efficiency extends to smaller models, which are essential to making intelligence more abundant and affordable: GPT‑5.6 Terra and GPT‑5.6 Luna outperform Fable 5 at around one-sixteenth the cost. "
Some pretty big claims and results! Excited to see how it feels during usage.
I use Fable and 5.5 extensively and I still find both have a place in my toolkit, i.e. Fable IS good but it isn't perfect, and it's still better to play them off against each other. I have Fable and 5.5 write plans and have them adversarially review each other's plans.
Having this amount of competition in the coding model space is good for all of us.
Some pretty big claims and results! Excited to see how it feels during usage.
I use Fable and 5.5 extensively and I still find both have a place in my toolkit, i.e. Fable IS good but it isn't perfect, and it's still better to play them off against each other. I have Fable and 5.5 write plans and have them adversarially review each other's plans.
Having this amount of competition in the coding model space is good for all of us.
I think this is the phase shift 5.6 (Sol set to Ultra) is bringing to the table. Until now we have become accustomed to asking models to continue and their natural inclination is always to stop.
Now OpenAI have flipped it around and for the first time are asking us to steer or stop the model instead, and its own inclination is to keep going. We now have to decide when we need to steer or want to catch up on our understanding of the work done but it will keep going.
It's good to see labs taking into account the cost/task.
Grok 4.5 is interesting because it's smart enough at great price. It seems gpt 5.6 is right there with great efficiency and great pricing.
Working with Fable has been a great experience, but at the end of the day, if you can get only 10% of your work done because it just burns through tokens, that's not that interesting.
I've been mostly using Opus and Fable high for planning and codex 5.5 medium for implementations. Claude is also the only model i can use for design tasks. If gpt 5.6 can finally deliver on the design side, it might be time to ditch the Claude sub and go full Gpt.
Grok 4.5 is interesting because it's smart enough at great price. It seems gpt 5.6 is right there with great efficiency and great pricing.
Working with Fable has been a great experience, but at the end of the day, if you can get only 10% of your work done because it just burns through tokens, that's not that interesting.
I've been mostly using Opus and Fable high for planning and codex 5.5 medium for implementations. Claude is also the only model i can use for design tasks. If gpt 5.6 can finally deliver on the design side, it might be time to ditch the Claude sub and go full Gpt.
Using the Claude "superpowers" skill will downgrade models automatically, using Sonnet and Haiku for trivial things.
I'm using luna (the smallest), at low thinking in my 9-to-5 job and I'm quite happy. No groundbreaking tasks so far, but typical small jira issues and fixes are done in a matter of low minutes. Very fast loops have their pros.
Fable or Opus would wander and wander.
Fable or Opus would wander and wander.
Bro these colors on chars are unbelievlable, I can not understand which is opus, which is fable, which is GPT...
good alternative, while gemini still no news
Cannot believe I needed a VPN to the US, to open this from Switzerland...
At least give me the article ffs.
At least give me the article ffs.
Annoyingly, the new ChatGPT app which folds in Codex, no longer recognizes Shift-Tab to toggle plan mode. Irritatingly you have to enter /plan. OpenAI, fix this!
Noticed this as well. You have to go into keyboard shortcuts and set it manually.
GPT Terra is 50% cheaper than 5.5 while being more performant. So it’s like a straight up 50% reduction in cost!
That leads me to a question. Why wouldn’t they just default to terra in ChatGPT in the last few months? If they didn’t then they burnt money for no reason by giving a shittier model at a higher price
That leads me to a question. Why wouldn’t they just default to terra in ChatGPT in the last few months? If they didn’t then they burnt money for no reason by giving a shittier model at a higher price
50% reduction in cost charged to customers, inference cost may as well be the same, we don't know.
Recent reports that OpenAI found optimisations would explain that these are legit
"while being more performant"
..on some specific set of benchmarks ;)
..on some specific set of benchmarks ;)
good alternative to anthropic
If Fable is removed from my Anthropic sub, I'll have to change to OpenAI.
Benchmaxxed
Like the last time, again they failed to note whether there is an Instant model or when it might become available.
Just a day before my $100 subscription expires, perfect
i'm not happy with how openai is trying to pit 5.6 sol as a cheaper equivalent to fable here
for one thing, they said that on AA, sol is "within one point of fable" at 58.9 vs 59.9 but don't clarify that the latter is with safeguards where ~8% of the tasks got routed to opus
i'm not rooting for either and genuinely think that the token efficiency and cheaper price are important but this sort of thing just feels disingenuous :-/
for one thing, they said that on AA, sol is "within one point of fable" at 58.9 vs 59.9 but don't clarify that the latter is with safeguards where ~8% of the tasks got routed to opus
i'm not rooting for either and genuinely think that the token efficiency and cheaper price are important but this sort of thing just feels disingenuous :-/
This is especially interesting because IIRC the AA benchmark is calibrated so that 1 point and greater difference is statistically significant.
There is no price-matching analog of nano. It's a disaster for nano users.
Would be interesting to see how it fares against Fable/Sonnet 5 in terms of costs/token output.
Any ideas how they were able to squeeze out this much efficiency? Or is it just inevitable benchmark pollution?
Any ideas how they were able to squeeze out this much efficiency? Or is it just inevitable benchmark pollution?
Do they expect us this model is 15ppt more accurate at half the price of fable? What’s going on?
Not available - checked and it's not there.
As usual, even though GPT-5.6 is releasing today, the rollout in ChatGPT and Codex will be gradual over many hours so that we can make sure service remains stable for everyone (same as our previous launches). We usually start with Pro/Enterprise accounts and then work our way down to Plus. We know it's slightly annoying to have to wait a random amount of time, but we do it this way to keep service maximally stable.
The timescale is typically hours not minutes, so if you don't see it now, I'd try again later today.
We mention it will be a gradual rollout over the next 24 hours in the Availability section at the bottom of the blog but I admit it's pretty buried.
(I work at OpenAI.)
The timescale is typically hours not minutes, so if you don't see it now, I'd try again later today.
We mention it will be a gradual rollout over the next 24 hours in the Availability section at the bottom of the blog but I admit it's pretty buried.
(I work at OpenAI.)
Is this bug fixed with 5.6? If not, it probably doesn’t matter which version Codex users are getting because the overall result is dramatically worse than stated by Open AI advertising: https://github.com/openai/codex/issues/30364
Not entirely fixed yet, but should be rarer with 5.6. Don’t have a quantification, unfortunately.
Understood thanks; will 5.6 fix this issue that makes Pro unusable?
https://github.com/openai/codex/issues/30364
"GPT-5.5 Codex reasoning-token clustering at 516/1034/1552 may be leading to degraded performance on complex tasks"
https://github.com/openai/codex/issues/30364
"GPT-5.5 Codex reasoning-token clustering at 516/1034/1552 may be leading to degraded performance on complex tasks"
on Plus I see Terra and Luna, but not Sol
It's available in Cursor now.
They have really been stringing us along for the past few weeks.
Thought Fable was great
sol is good
The meat of the report for SWEs:
SWE-Bench Pro Sol: 64.6% Fable: 80% Opus: 69.2% (!!!!)
So, it still trails Opus, significantly, and is not a next-gen coding model like Mythos/Fable 5.
Disappointing to say the least, but somewhat expected.
SWE-Bench Pro Sol: 64.6% Fable: 80% Opus: 69.2% (!!!!)
So, it still trails Opus, significantly, and is not a next-gen coding model like Mythos/Fable 5.
Disappointing to say the least, but somewhat expected.
SWE-Bench pro is pretty much useless now even though many ppl still look at it. OpenAI published a report yesterday saying so as well. Only look at DeepSWE and FrontierCode right now for coding imo.
Amazing, a company that does poorly in a benchmark says that benchmark is useless...
SWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors.
But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.
But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.
> Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark
yeah that was the point of introducing the Pro version
yeah that was the point of introducing the Pro version
You've overstated the conclusion. The SWE-bench series has had issues since its inception.
OpenAI no longer recommends SWE-Bench-Pro as a benchmark: https://openai.com/index/separating-signal-from-noise-coding...
OpenAI no longer recommends SWE-Bench-Pro as a benchmark: https://openai.com/index/separating-signal-from-noise-coding...
That's smart, they only recommend benchmarks that make them look better then their competitors.
I'm disappointed these models continue to be closed source and so expensive.
Open weight models being 10x or more cheaper is just so much more of an unlock than incremental gains for me.
Open weight models being 10x or more cheaper is just so much more of an unlock than incremental gains for me.
because they're stealing from the frontier models. they're gaming the benchmarks. look how bad glm 5.2 is on cursors evals. gmhit garbage , but it gets glazed as God tier.
They're stealing, eh?
> On the Artificial Analysis Coding Agent Index, GPT‑5.6 Sol with max reasoning sets a new state of the art at 80, 2.8 points above Fable 5, while using less than half the output tokens, taking less than half the time, and costing about one-third less.
> That advantage extends across the family: Terra performs just above Fable 5, while Luna outperforms Opus 4.8; each does so in roughly one-third of the time, with about half as many output tokens, and at approximately one-quarter the estimated cost.
Wow. I don't believe it. Every indication and twitter post told me that Fable is much more intelligent than Sol and here we are told that even Terra outperforms Fable?
Not only that, Sol doesn't even come with run time classifiers. So it is even more suspicious.
What's even stranger is that OpenAI is directly referencing a competitor in this direct way.
> That advantage extends across the family: Terra performs just above Fable 5, while Luna outperforms Opus 4.8; each does so in roughly one-third of the time, with about half as many output tokens, and at approximately one-quarter the estimated cost.
Wow. I don't believe it. Every indication and twitter post told me that Fable is much more intelligent than Sol and here we are told that even Terra outperforms Fable?
Not only that, Sol doesn't even come with run time classifiers. So it is even more suspicious.
What's even stranger is that OpenAI is directly referencing a competitor in this direct way.
Most importantly, the cost:
> GPT‑5.6 is priced per 1M tokens across three model sizes: Sol is $5 input / $30 output; Terra is $2.50 input / $15 output; and Luna is $1 input / $6 output.
Just as expensive as Fable 5. But of course, another slot machine upgrade but the costs will keep going up and the open weight models from china will continue to race everyone else to $0.
Looking forward to the next version of GLM, Qwen, Deepseek and Minimax.
> GPT‑5.6 is priced per 1M tokens across three model sizes: Sol is $5 input / $30 output; Terra is $2.50 input / $15 output; and Luna is $1 input / $6 output.
Just as expensive as Fable 5. But of course, another slot machine upgrade but the costs will keep going up and the open weight models from china will continue to race everyone else to $0.
Looking forward to the next version of GLM, Qwen, Deepseek and Minimax.
That's wrong. GPT 5.6 Sol looks to have the same price as GPT 5.5, apart from a new pricing fee for cache writes. Fable 5 is $10 input / $50 output.
Also watching deepseek closely. Seems like US frontier labs only know how to throw money at things as opposed to actually make smart improvements to the algorithms.
To be fair, DeepSeek doubled prices during the peak Chinese workday. (Which admittedly doesn't affect me much.)
“Be scared”
Are you people seriously this dumb? Have you conwidered that all of these benchmarks are trained into these models. Can you stop sharing them as if they matter?
poolnoodle(2)
CTRL-F: Fable
15 hits
Holy shit. They must be feeling very threatened by Fable if they're spending this much energy talking about it in the release notes for their own model.
15 hits
Holy shit. They must be feeling very threatened by Fable if they're spending this much energy talking about it in the release notes for their own model.
In the past they received a lot of hate for not comparing to the competition.
Sir, you do not understand! This is the Internet! You must always find a reason to be upset and/or complain!
yikes - looks like you need to go back to stats school
gemini - 13 hits
opus - 18 hits
So they are more threatened by opus than fable, or are they almost as threatened by gemini as they are by fable?
gemini - 13 hits
opus - 18 hits
So they are more threatened by opus than fable, or are they almost as threatened by gemini as they are by fable?
The second paragraph has four mentions of Fable. I think that makes my case pretty clearly.
Anthropic fumble of Fable's release will go down in the history books, makes sense for OpenAI to run with it.
Apparently it significant outperforms fable on both an intelligence and cost index.
I don’t believe it at all and I don’t think anyone else does either.
I don’t believe it at all and I don’t think anyone else does either.
I believe that it outperformed it on benchmarks.
Downvoted comment but I did find this comparison aggressive and tacky.
nobody cares anymore
I find it way too defensive of known Jewish pedophiles like Anthony Wiener and Scott Wiener.
You can’t even ask about it.
And yet they write the laws (and the AI truths).
You can’t even ask about it.
And yet they write the laws (and the AI truths).
At this point, they are just changing the decimals to stay relevant and in the news.
Anthropic should be grateful OpenAI did not borrow "Epic" and "Legend".
I expect OpenAI names to be "fabulous", "glorious", "empowered", "delicious" etc.
The marketing team must've done research that said "people are starting to think that you guys are evil-water-stealing-lay-off-loving-bubble-bursting scumbags" and decided to really lean into the small family business and happy font vibes!
The way they talk about cyber security fixes makes clear that they are in bed with the government in order to get ahead of Anthropic.
All of them closely collaborate with the government. LLMs are a national security priority and are vetted. Claude AI was used by Palantir's Maven to target the Minab school that led to a triple tap strike killing over 150 schoolchildren.
The Minab disaster has every sign of being a pure humint fail the defense department decided to cover up with politically expedient AI blaming.
Claude makes suggestions for targets and humans review and approve them. We always knew there was a human in the process but if there's one massive takeaway from years of AI ethics research, it's that there is a very clear and well-documented human bias towards automated answers when there's any ambiguity.
Including a human in the loop does not excuse the fact that AI was trusted in a process that decides who lives and dies.
Including a human in the loop does not excuse the fact that AI was trusted in a process that decides who lives and dies.
The DIA's Maven database was out of date: https://www.theguardian.com/news/2026/mar/26/ai-got-the-blam...
"We triple tapped a girls elementary school because our data wasn't up to date."
I wish there was a comments cooldown that paused all comments until we've all been able to use this model for a few hours.
This thread is going to be thousands of comments long about a product we haven't gotten to use yet.
This thread is going to be thousands of comments long about a product we haven't gotten to use yet.
Weirdly, normally new ChatGPT releases are head and shoulders above anything else, but according to OpenAI's own evaluation, Anthropic's Mythos outperforms ChatGPT in quite a few benchmarks: https://openai.com/index/gpt-5-6/.
ChatGPT 6 must be deep in the pipeline and will be released within the next few months. Maybe that's why this release is versioned 5.6, not 6.0.
ChatGPT 6 must be deep in the pipeline and will be released within the next few months. Maybe that's why this release is versioned 5.6, not 6.0.
I think its more about branding than anything else. Anthropic played a masterstroke with the way they marketed, released, and then blocked Mythos. Now everyone know the Mythos "model" by name.
ChatGPT 6 is trying to follow suite.
https://developers.openai.com/api/docs/guides/latest-model
https://x.com/levie/status/2075287443411222628, https://xcancel.com/levie/status/2075287443411222628