Tau² benchmark: How a prompt rewrite boosted GPT-5-mini by 22%(quesma.com)
quesma.com
Tau² benchmark: How a prompt rewrite boosted GPT-5-mini by 22%
https://quesma.com/blog/tau2-benchmark-improving-results-smaller-models/
68 comments
Wait, are we about to reinvent programming from first principles?
Seemingly its always been on a scale between directly editing 1s and 0s and drafting legislature. Compile times may vary
I'd say it's more "programming with extra steps"
This is more like reinvention by trying everything which doesn't work first. It's the dual of first principles.
Maybe one day we will all be using https://shakespearelang.com/
I’ve found myself writing code intending to write prompts for writing better code.
Soon enough Im sure we’ll start to see programming languages that are geared towards interacting with llms
Soon enough Im sure we’ll start to see programming languages that are geared towards interacting with llms
Finally a use for Lojban!
https://en.wikipedia.org/wiki/Lojban
https://en.wikipedia.org/wiki/Lojban
Great! A diviner has vibe-exposed the arcane magic word knowledge on the steps to ultimate knowledgeplasty! Come let us get together to share more trial-and-error wordsmithery, Together we will someday have ultimate power!
If the model creators themselves arent sharing this magic-word bullshitteryy then why is anyone spending time on this? It is just going to change with every model release
If the model creators themselves arent sharing this magic-word bullshitteryy then why is anyone spending time on this? It is just going to change with every model release
In other words, just like programming, we’re writing better instructions. In this case, we’re asking it to think out loud more clearly. It’s almost like whiteboard interview prep.
It’s quite amazing because it means programming is fully entering the natural language phase of the timeline.
If you aren’t a solid clear writer, you may not make it in the brave new world.
It’s quite amazing because it means programming is fully entering the natural language phase of the timeline.
If you aren’t a solid clear writer, you may not make it in the brave new world.
>If you aren’t a solid clear writer, you may not make it in the brave new world.
Have you not heard of all the AI startups that can turn a 3-word thought into very clearly written prose to be lovingly poured into the waiting mouth of your AI agent?
Have you not heard of all the AI startups that can turn a 3-word thought into very clearly written prose to be lovingly poured into the waiting mouth of your AI agent?
Isn’t programming the clearest form of writing? Perhaps it’s the non programmers that need to “catch up”.
> Isn’t programming the clearest form of writing?
Not the way most people do it.
Not the way most people do it.
If programming was the most clear form of writing, then how come eductators frequently use pseudo code to make programming more clear?
The same reason kids are first taught the Bohrian atom model. It is less clear and precise, but thereby also less complex.
"100 baskets of apples" is easier to hold in your head than "23 baskets of red, small-ish apples, 12 of large red, 6 of any size green...", but my no means does it permit a more clear view of the Truth.
Different usage of the word "clear".
"100 baskets of apples" is easier to hold in your head than "23 baskets of red, small-ish apples, 12 of large red, 6 of any size green...", but my no means does it permit a more clear view of the Truth.
Different usage of the word "clear".
Addendum: 'Clear' as in the opposite of ambiguous, not 'clear' as in the opposite of confusing.
The computers of the future will be operated by shamans making incantations more than technicians writing code.
Of the future?
We already have people praying to the machine gods, so I guess your future is next week?
We already have people praying to the machine gods, so I guess your future is next week?
>GPT-5 showed significant improvement only in one benchmark domain - which is Telecom. The other ones have been somehow overlooked during model presentation - therefore we won’t bother about them either.
I work at OpenAI and you can partly blame me for our emphasis on Telecom. While we no doubt highlight the evals that make us look good, let me defend why the emphasis on Telecom isn't unprincipled cherry picking.
Telecom was made after Retail and Airline, and fixes some of their problems. In Retail and Airline, the model is graded against a ground truth reference solution. Grading against a reference solution makes grading easier, but has the downside that valid alternative solutions can receive scores of 0 by the automatic grading. This, along with some user model issues, is partly why Airline and Retail scores stopped climbing with the latest generations of models and are stuck around 60% / 80%. I'd bet you $100 that a superintelligence would probably plateau around here too, as getting 100% requires perfect guessing of which valid solution is written as the reference solution.
In Telecom, the authors (Barres et al.) made the grading less brittle by grading against outcome states, which may be achieved via multiple solutions, rather than by matching against a single specific solution. They also improved the user modeling and some other things too. So Telecom is the much better eval, with a much cleaner signal, which is partly why models can score as high as 97% instead of getting mired at 60%/80% due to brittle grading and other issues.
Even if I had never seen GPT-5's numbers, I like to think I would have said ahead of time that Telecom is much better than Airline/Retail for measuring tool use.
Incidentally, another thing to keep in mind when critically looking at OpenAI and others reporting their scores on these evals is that the evals give no partial credit - so sometimes you can have very good models that do all but one thing perfectly, which results in very poor scores. If you tried generalizing to tasks that don't trigger that quirk, you might get much better performance than the eval scores suggest (or vice versa, if your tasks trigger a quirk not present in the eval).
Here's the tau2-bench paper if anyone wants to read more: https://arxiv.org/abs/2506.07982
I work at OpenAI and you can partly blame me for our emphasis on Telecom. While we no doubt highlight the evals that make us look good, let me defend why the emphasis on Telecom isn't unprincipled cherry picking.
Telecom was made after Retail and Airline, and fixes some of their problems. In Retail and Airline, the model is graded against a ground truth reference solution. Grading against a reference solution makes grading easier, but has the downside that valid alternative solutions can receive scores of 0 by the automatic grading. This, along with some user model issues, is partly why Airline and Retail scores stopped climbing with the latest generations of models and are stuck around 60% / 80%. I'd bet you $100 that a superintelligence would probably plateau around here too, as getting 100% requires perfect guessing of which valid solution is written as the reference solution.
In Telecom, the authors (Barres et al.) made the grading less brittle by grading against outcome states, which may be achieved via multiple solutions, rather than by matching against a single specific solution. They also improved the user modeling and some other things too. So Telecom is the much better eval, with a much cleaner signal, which is partly why models can score as high as 97% instead of getting mired at 60%/80% due to brittle grading and other issues.
Even if I had never seen GPT-5's numbers, I like to think I would have said ahead of time that Telecom is much better than Airline/Retail for measuring tool use.
Incidentally, another thing to keep in mind when critically looking at OpenAI and others reporting their scores on these evals is that the evals give no partial credit - so sometimes you can have very good models that do all but one thing perfectly, which results in very poor scores. If you tried generalizing to tasks that don't trigger that quirk, you might get much better performance than the eval scores suggest (or vice versa, if your tasks trigger a quirk not present in the eval).
Here's the tau2-bench paper if anyone wants to read more: https://arxiv.org/abs/2506.07982
Appreciated the response! I noticed the same when I ran tau2 myself on gpt-5 and 4.1, where gpt-5 is really good at looking at tool results and interleaving those with thinking, while 4.1/o3 struggles to decide the proper next tool to use even with thinking. To some extent, gpt-5 is too good at figuring out the right tool to use in one go. Amazing progress.
Haha, I guess my little sarcasm just earned us a masterclass! Thanks a lot for sharing your insights — really appreciate it!
This sounds very vague, what does scoring good at Telecom mean?
Can we get some (hypothetical) examples of ground truths?
For example for the Airline domain, what kind of facts are these ground truth facts? All the airports, the passenger lines between them, etc? Or does it mean detailed knowledge of the airplane manuals for pilots, maintenance, ...?
Can we get some (hypothetical) examples of ground truths?
For example for the Airline domain, what kind of facts are these ground truth facts? All the airports, the passenger lines between them, etc? Or does it mean detailed knowledge of the airplane manuals for pilots, maintenance, ...?
I wish they had published what prompt was given to Claude to improve GPT-5-mini's performance, as well as a before and after comparison of a prompt that underwent this transformation.
Thanks for the feedback, appreciate it!
It makes lot of sense - I'll update the article with links to the actual prompts.
Initially I thought these would be too lengthy for the article and no one would care, but as it seems people are really interested in it. Of course I'd be happy to share the details.
I see that you've added links to a pull request that show the previous and final optimized prompts. However, the OP was asking for the prompt you gave to Claude to assist you in optimizing your prompt. Would you mind sharing that one? (That way nobody has to reverse engineer the instructions from the diff you provided.)
My take: we have no clue how this works and the performance can be down tomorrow just as well.
My hypothesis: the length of the prompt shrunk, yet maintained the same amount of information.
This is the PR with the changes in case people missed it:
https://github.com/mieciu/tau2-bench/pull/1/files
https://github.com/mieciu/tau2-bench/pull/1/files
That seems so strongly directed, that it feels like an attempt to reproduce a classic chat bot.
Thanks! I also updated the post with the link on the website.
Can one customer get the model to return the bill details for another customer?
Really intresting. What did the original prompt look like? Perhaps the original prompt was not that good? I feel like the changes claude suggested (except a couple maybe) are already pretty well known prompt engineering practices.
Thank you for the feedback!
In this (telecom) benchmark you can review agent policies and manuals here: 1) https://github.com/sierra-research/tau2-bench/blob/main/data... 2) https://github.com/sierra-research/tau2-bench/blob/main/data...
Of course these are just parts of the prompt, you can inspect benchamark code to see how these are rendered to actual LLM calls.
In case someone is not familiar with framework methodology I've wrote a separate article covering that (with some of my thoughts) -> https://quesma.com/blog/tau2-from-llm-benchmark-to-blueprint...
In this (telecom) benchmark you can review agent policies and manuals here: 1) https://github.com/sierra-research/tau2-bench/blob/main/data... 2) https://github.com/sierra-research/tau2-bench/blob/main/data...
Of course these are just parts of the prompt, you can inspect benchamark code to see how these are rendered to actual LLM calls.
In case someone is not familiar with framework methodology I've wrote a separate article covering that (with some of my thoughts) -> https://quesma.com/blog/tau2-from-llm-benchmark-to-blueprint...
The only problem is I feel like having to have Claude rewrite the prompt negates some of the efficiency and latency benefits of using mini. For system prompts obviously this doesn't matter, but for actual continuous user interaction, it feels unworkable.
It definitely makes sense that improving formatting and clarity for these smaller models would really help with performance, but I'm wondering if gpt5-mini is already smart enough to handle that reformatting, and can rewrite the prompt itself, before handing it off to another instance of itself.
Overall an awesome article!
It definitely makes sense that improving formatting and clarity for these smaller models would really help with performance, but I'm wondering if gpt5-mini is already smart enough to handle that reformatting, and can rewrite the prompt itself, before handing it off to another instance of itself.
Overall an awesome article!
Thank you!
Great point. Indeed my methodology was to treat the prompt refactoring as one-off task, therefore I didn't care much about cost/latency.
As for having GPT-5-mini do the rewriting — that’s a really interesting idea. I think the biggest challenge is avoiding cognitive overload. The Tau² agent policies are pretty complex: it’s easy to grasp the overall task, but the detailed rules for each user case aren’t always obvious.
I'm not sure if how easy it is to actually overload GPT-5-mini, so that's definitely worth exploring.
Great point. Indeed my methodology was to treat the prompt refactoring as one-off task, therefore I didn't care much about cost/latency.
As for having GPT-5-mini do the rewriting — that’s a really interesting idea. I think the biggest challenge is avoiding cognitive overload. The Tau² agent policies are pretty complex: it’s easy to grasp the overall task, but the detailed rules for each user case aren’t always obvious.
I'm not sure if how easy it is to actually overload GPT-5-mini, so that's definitely worth exploring.
My experience as well.
Prompt changes affect output substantially (just look up arxiv), the difficult part is find an optimal structure to yield the best results. It is a bit expensive to do a lot of testing on your own, so it all boils down to feels and experience at the moment. Then you mix up tool calls, other agent calls, client functions and this gets terribly hard to evaluate.
I am still puzzled how distance between policies can have an effect on the output. And how a simple retry fixes everything.
Prompt changes affect output substantially (just look up arxiv), the difficult part is find an optimal structure to yield the best results. It is a bit expensive to do a lot of testing on your own, so it all boils down to feels and experience at the moment. Then you mix up tool calls, other agent calls, client functions and this gets terribly hard to evaluate.
I am still puzzled how distance between policies can have an effect on the output. And how a simple retry fixes everything.
This is very much what dspy aims to address. Learning the incantations necessary to prompt well can be replaced by an algorithmic loop and example labelled cases.
This sort of stuff is trodden ground, if this seems exciting to you check out DSPy.
Many of the "look at what I did programming LLMs" blog posts on Hacker News have been developed and put out in academic papers and groups. The posts which gain traction here seem to be perennially behind the state of the art.
https://dspy.ai/tutorials/tool_use/
Definitely interesting, thank you!
Definitely interesting, thank you!
I wonder if it would be possible to improve even further on the benchmark by simply showing Claude the current hardest problems and asking it to improve the prompt without including any specifics related to the problems
I think there's a chance we could squeeze a better benchmark score, although there's a risk of overfitting which I wanted to avoid.
The simplest test would be to make previously “unreachable” tasks succeed through obvious prompt tweaks — like reordering instructions or emphasizing key parts.
That said, my methodology intentionally avoided exposing the model to actual tasks. Instead, I focused on the domain as a whole: refining the instructions so a smaller model could understand and act reliably.
The simplest test would be to make previously “unreachable” tasks succeed through obvious prompt tweaks — like reordering instructions or emphasizing key parts.
That said, my methodology intentionally avoided exposing the model to actual tasks. Instead, I focused on the domain as a whole: refining the instructions so a smaller model could understand and act reliably.
Rewriting prompts don't come with no costs. The cost here is that different prompts work for different contexts and is not generalisable. The rewritten prompt here will not work well for other cases like medical or social advice.
I think this rewriting of prompts technique is the reason "reasoning" models perform well - they know exactly how to rewrite the prompts for a context.
FWIW I don't trust these benchmarks fully because a huge bump like this is not expected - I would expect OpenAI to optimise enough to let such gaps open.
I think this rewriting of prompts technique is the reason "reasoning" models perform well - they know exactly how to rewrite the prompts for a context.
FWIW I don't trust these benchmarks fully because a huge bump like this is not expected - I would expect OpenAI to optimise enough to let such gaps open.
> Removed verbose explanations mixed with instructions
Is Claude rewriting generic instructions once, or is it rewriting the core task statement each time? If so, I'm not sure how you prevent information leakage: Claude might easily be "solving" some of the tasks and inserting subtle hints on the approach. I think this result is very interesting if it holds after rewriting only the generic instructions, even if the performance boost is lower.
Is Claude rewriting generic instructions once, or is it rewriting the core task statement each time? If so, I'm not sure how you prevent information leakage: Claude might easily be "solving" some of the tasks and inserting subtle hints on the approach. I think this result is very interesting if it holds after rewriting only the generic instructions, even if the performance boost is lower.
I only had Claude rewrite the domain policies and generic instructions, not the individual task statements. I updated the blog with a link showing the exact changes.
So no leakage — it wasn’t solving or hinting at any of the specific test cases, since none of the tasks were ever exposed to it.
So no leakage — it wasn’t solving or hinting at any of the specific test cases, since none of the tasks were ever exposed to it.
No before/after prompt.
Into the trash it goes.
Into the trash it goes.
I feel like eventually we’ll get LLMs that will act like compilers do now. So they will take a prompt and turn it into an optimized prompt for a bigger LLM.
[deleted]
Have you tried to use gpt-5 with high thinking to rewrite the prompt? why claude for this vs some other model?
Yea, so that part I actually did not overthink - I knew I need strong reasoning and just grabbed opus which is my personal go-to for such tasks and sticked to it as I wanted to avoid too many moving parts.
Would be interesting to compare both the benchmark result as well as the way other models approached the whole refactoring process!
Would be interesting to compare both the benchmark result as well as the way other models approached the whole refactoring process!
Doesn't saying "check -> action" suggest you're taking _away_ the agentic capabilities, and optimizing for the benchmark, meaning it's no longer a good benchmark for agentic capabilities?
That's like being able to see the test before taking it
That's like being able to see the test before taking it
Great point! However, I’d ask the following: isn't faithfully following nuanced instructions an _agentic capability_ by itself?
If a model only performs well once the rules are clarified, that’s still revealing something important about its agency: it’s brittle when policies are ambiguous, but much stronger when they’re structured.
I agree with you that there’s a fine line between genuinely helping the model 'understand' the task and just 'teaching to the test'.
That said, Tau² is framed as a very specific use case — and we showed it can be solved more reliably. At the end of the day, that means we now have an agent built on a cheaper, faster model that still performs its job with higher reliability.
If a model only performs well once the rules are clarified, that’s still revealing something important about its agency: it’s brittle when policies are ambiguous, but much stronger when they’re structured.
I agree with you that there’s a fine line between genuinely helping the model 'understand' the task and just 'teaching to the test'.
That said, Tau² is framed as a very specific use case — and we showed it can be solved more reliably. At the end of the day, that means we now have an agent built on a cheaper, faster model that still performs its job with higher reliability.
[deleted]
I have read somewhere that XML prompting could also help to remove ambiguities and increase success rates for agents, did you here about that and would that be a good idea? Christophe from France
Salut Christophe! Yes, I’ve come across the concept :) In fact, I think what we did with the ├── and └── notation is already a step in that direction (at least concept-wise) as it also puts a specific structure over the instructions.
But stretching all the way seems worth exploring too!
thanks for your reply
Copilot in VSCode seems to do something similar in the form of todo lists.
Would be curious to run this through DSPy/GEPA and see if it can squeeze even further performance by optimizing the prompt
Using an LLM to (re)write your prompt or system prompt (for local models) is free alpha.
you would also be interested in dSPY...
1. Structure & Flow
2. AI Agent Optimizations
3. Cognitive Load Reduction
4. Actionable Language