XCSme

19 uur geledendiscuss

And in my tests, that point of "overthinking" depends on the problem's complexity, so it's not necessarily that using "xhigh" is always bad or good.

XCSme

19 uur geledendiscuss

One example where the order seems correct, is this SVG generation test:

https://aibenchy.com/showcase/?q=Gemini+3.5%2Cgpt+5.6%2C+5.3...

You can see that most Gemini 3.5 generations are more correct than 5.6 Sol (the net is in the middle of the table, hamster seems reasonable and not deformed, etc.)

XCSme

19 uur geledendiscuss

It's because the benchmark is not coding-only.

Gemini models tend to have most knowledge for most domains, and are one of the most intelligent overall. You can check other benchmarks too, on specific categories, those models still beat other SOTA models.

Regarding Opus, Anthropic models often fail to follow instructions, formatting requirements or simply refuse to answer questions (i.e. Fable).

The issue with Gemini models is that they are not as good as using tools or go into weird failure modes when coding or trying to extract/generate specific data. They work amazing, until they don't...

XCSme

gisterendiscuss

In my tests, in almost all cases, using Sol on (low) reasoning is the best option intelligence/price-wise.

Luna is good too, for classification tasks or any pre-processing task that is not critical

XCSme

gisterendiscuss

Also for most, there doesn't seem to be a big difference between (medium) and (high).

XCSme

gisterendiscuss

Yeah, for some reason the (low) versions do really well, like they think directly of the solution instead of going around all the edge-cases and getting lost in one of them.

XCSme

gisterendiscuss

Here's all 3 (medium), and GPT-5.5

It GPT-5.6 doesn't seem to be a lot smarter than 5.5, but it is faster, cheaper, more efficient and more consistent:

https://aibenchy.com/compare/openai-gpt-5-6-sol-medium/opena...

XCSme

gisterendiscuss

Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.

It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.

XCSme

gisterendiscuss

GPT-5.6 is a really good model, and quite cheap. I can finally replace GPT-5.3-Codex for my Tool Calling in n8n.

Here's my benchmark results for GPT-5.6:

https://aibenchy.com/?q=gpt-5.6

(the high reasoning variants are still running, uploading them soon too)

EDIT: The high variants are there too, enjoy the hamsters[0].

[0]: https://aibenchy.com/showcase/?q=gpt-5.6

XCSme

eergisterendiscuss

What would the advantage be?

XCSme

eergisterendiscuss

Hamsters are also getting better, but still quite off compared to SOTA models:

https://aibenchy.com/showcase/?q=grok

XCSme

eergisterendiscuss

I am trying to benchmark it now, but:

    - It doesn't seem available in EU (?)
    - Using a VPN seems to sort of fix it, but it's way slower than I expected, when everyone was praising it, it feels like the speed is slowly ramping up
    - Cost is $2/$6 for <200k context only, above that, cost is $4/$12
    - GLM-5.2 still seems smarter, faster and much cheaper:

https://aibenchy.com/compare/x-ai-grok-4-5-medium/z-ai-glm-5...

XCSme

eergisterendiscuss

I often use Gemini as my "chat" app to ask questions, etc.

I stopped using ChatGPT because of they're weird login system, where it keeps switching to my Workspace Codex account, which doesn't actually have the free/chat functionality.

I usually just switch between gemini/grok when asking questions or to research something online.

XCSme

8 dagen geledendiscuss

I spent 30mins debugging why my Github Pages were serving old versions...

It has been down for at least 2 hours, with actions not being executed. I can not finalize my deployment because of this outage, so now I have to delay my table-tennis training to wait for the Actions to be available again before I can complete the deployment :(

XCSme

10 dagen geledendiscuss

It is on top for many benchmarks, only not the coding/agentic ones.

Still one of the most intelligent models overall, most likely to get any question you ask correctly (without tools).

XCSme

10 dagen geledendiscuss

What's interesting, is that Sonnet 5 is actually worse[0] than 4.6 without reasoning.

It makes some sense, as models are trained more and more with reasoning, than without.

[0]: https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-non...

XCSme

10 dagen geledendiscuss

Well, it is a Sonnet model, it is indeed better[0] than Sonnet 4.6 (smarter, faster, cheaper), but I don't see why would you use it as opposed to Opus 4.8 low or GLM-5.2...

[0]: https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-med...

XCSme

10 dagen geledendiscuss

As always, note: faster than GLM-5.2 doesn't mean too much, as GLM-5.2 is served by different providers, so the inference speed can vary drastically between providers or over time.

XCSme

10 dagen geledendiscuss

I just tested it on my benchmarks[0], it's GLM-5.2 level, at 2x cost, but also 2x faster.

Weak spots (categories it fails):

    - Trivia — 0/3 - basically not much built-in knowledge
    - Combined tool-calling tasks — score 45/100, sometimes makes invalid tool calls
    - Puzzle Solving — score 77, flubs carwash-like tests

[0]: https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-med...

XCSme

11 dagen geledendiscuss

Considering the cloud version, all three models compared in the article (Qwen 3.6 35BA3b, 3.6 27B and DeepSeek V4 Flash), have very similar performance[0], BUT on cloud, for some reason DeepSeek V4 Flash is 10-20x cheaper than the Qwen models.

If Qwen models are so much easier to run, why are the providers charging more than V4 Flash?

[0]: https://aibenchy.com/compare/qwen-qwen3-6-35b-a3b-medium/qwe... <-- compare how the three models draw hamsters svgs, lol

Submissions

Show HN: One hundred LLMs Generating a HTML/CSS Solar System

MariaDB now has a DuckDB storage engine

SVG of a Hamster Playing Table-Tennis

[untitled]

Tell HN: Gemini 3.5 Flash breaks in stupid ways

Cisco Announces End of Life for Smartlook

Ask HN: Are MiniMax Models Scams?

Grok 4.20 brings minimal improvements over Grok-4.1-fast

Why Not Boost?

Show HN: AI Benchy – AI benchmarks and comparisons

PostHog now shares hashed emails of new users with Reddit and LinkedIn

Show HN: AIBenchy – Independent AI Leaderboard

Mailchimp Free Plan Changes

InvokeAI Commercial Platform Shuts Down, Open-Source Project Continues

Prevent others sending emails using your domain name

[untitled]

LanguageTool browser extension is no longer free

Does anyone run ads successfully?

Hetzner Servers Benchmark

N8n added native persistent storage with DataTables

comments