- Gemini 2.5 pro is absolutely a beast in coding, perhaps the best model right now
- They spent all the computing resources on training it on coding data and forgot to give it a distinct personality.
- It doesn’t do well on reasoning as well as Grok 3 (think) and Claude 3.7 Sonnet (thinking)
- On par with 03-mini-high in general mathematics
If you’re a coder, you’ll love it, or else you will be fine with other frontier reasoning models (Deepseek r1, if you ask me)
Deepseek v3 0324 is the first open-source model to match SOTA coding performance
- Understands user intention better than before; I’d say it’s better than Claude 3.7 Sonnet base and thinking. 3.5 is still better at this (perhaps the best)
- Again, in raw quality code generation, it is better than 3.7, on par with 3.5, and sometimes better.
- Great at reasoning, much better than any non-reasoning models available now.
- Better at the instruction following than 3,7 Sonnet but below 3.5 Sonnet.
This is a step towards a human-machine hybrid world. Putting a human in the loop can do wonders. Sure, it is expensive now, but the subsequent iterations will crush it.