Is GPT-5.6 Sol Max Worth It?

2 pointsby yohji1984เมื่อวานซืน4 comments

I ran some test with gpt- 5.6 sol in max reasoning on both codex cli and my own agent harness: https://github.com/Tura-AI/tura I tested only 1 task and the toekn efficency difference is not as great as in high mode: Tura used up to 83.1% fewer turns on the rewrite benchmark and improved the harness results by 7 percent point. In ultra the dirrence is around 50% and incressed harness result only by one. Full report: https://turaai.net/blog#is-gpt-5-6-sol-max-worth-it

However in the bug fixing tasks based on the pulbished report from DeepSWE and others, the performance incress is less impressive compare with the incess in costs. here is the summarry:

Use Max for building new project, and high for debuging or add defined feature.

4 comments

jotatoเมื่อวานซืน

I needed a transom bracket for my boat. I didn't want to buy one (they are ~50USD) so I figured I could print one using nylon. Don't worry, this is only holding a rudder.

Anyway, I couldn't find one that would work so I just asked Sol to design one using free cad. It used 12% of my weekly budget ($20/m plan) and spit out something nearly perfect. I imported it into OnShape to add some chamfers and make a few tweaks.

https://cad.onshape.com/documents/f44e8be346cd029c5c8fe07c/w...

mattfrommarsเมื่อวานซืน

Sorry, I can't really help you with the tool as I don't have access to it in my day to day job, but are you really the creator behind https://github.com/Tura-AI/tura I which I assume is solo dev project?

SuperLedeเมื่อวานซืน

The useful distinction may be ambiguity, not "new project vs bug fixing." Max for open-ended architecture work, high when the task and acceptance criteria are already clear. Cost per successful task would be the metric I'd care about most.

moomoo11เมื่อวานซืน

yes