[untitled]
3 comments
And so did Opus 4.6 and Gemini 3.1 Pro, why is Grok singled out here? Oh right because HN has become Reddit.
Clickbait/ragebait title. They are insisting on testing these things without a harness.
> In practise this philosophy translates into using extremely generic and miminal LLM testing prompts, no client-side "harnesses", no hand-crafted tools, and no tailored model configuration.
Yeah, that's not gonna work, chief. The agentic stuff works today because it's in a loop and because it has some feedback from that loop.
As it stands now, the SotA models are 0.2% (Gemini 3.1 and Opus 4.6) and 0.3% GPT5.4-High. By only hitting their APIs, with no client-side orchestration, they're only testing internal tools (if any). This is not the way. (but grabs headlines)
FWIW, the kaggle competition is already at 0.5% with 2x T4 GPUs max and a few hours for ~110 problems so... Yeah. (and it's only been live for <1week)
> In practise this philosophy translates into using extremely generic and miminal LLM testing prompts, no client-side "harnesses", no hand-crafted tools, and no tailored model configuration.
Yeah, that's not gonna work, chief. The agentic stuff works today because it's in a loop and because it has some feedback from that loop.
As it stands now, the SotA models are 0.2% (Gemini 3.1 and Opus 4.6) and 0.3% GPT5.4-High. By only hitting their APIs, with no client-side orchestration, they're only testing internal tools (if any). This is not the way. (but grabs headlines)
FWIW, the kaggle competition is already at 0.5% with 2x T4 GPUs max and a few hours for ~110 problems so... Yeah. (and it's only been live for <1week)