Two of the three strip titles are hallucinated and two of the three strips are bad examples. Haley is mute in strip 403 and does nothing. Strip 578 is the start of the arc that shows the behavior Gemini is talking about, but has things going wrong so it's not a good example either.
I think LLMs are better at finding the most helpful sources now, but that's more a testament to how much the front page of web search has lost to low value LLM content.
I've used it for live service video games, it's pretty good at summarizing major changes to a game since you've played it last. With regular web search you'd have to go to every major patch and a lot of games don't even have good patch notes / it's all stuck in content creator videos.
Though I still prefer Claude for this since it's better at citing sources.
Unironically you can put "what are good demos for agentic workflows at Google I/O that would be received well by the general public" into Gemini's AI Mode and get better suggestions for use cases than what they're showing.
Note the scoring function is significantly different for ARC-AGI-3. It isn't the percentage of tests passed like previous versions, it's the square of the efficiency ratio -- how many steps the model needed vs the second best human.
So if a model can solve every question but takes 10x as many steps as the second best human it will get a score of 1%.
It is a fair question. I'd expect the numbers are all real. Competitors are going to rerun the benchmark with these models to see how the model is responding and succeeding on the tasks and use that information to figure out how to improve their own models. If the benchmark numbers aren't real their competitors will call out that it's not reproducible.
However it's possible that consumers without a sufficiently tiered plan aren't getting optimal performance, or that the benchmark is overfit and the results won't generalize well to the real tasks you're trying to do.
Focusing on the AI content generation aspect of this is disappointing to me, since those tools are fuzzy at best and even if the post was AI generated, it isn't necessarily a red flag since the user could be trying to disguise their writing style.
There are plenty of other signs this story is likely fake. The author claiming to be posting from a library on New Years' Day (most government buildings are closed) and was responding over 10 hours on the account. He's using a throwaway and a "burner laptop" at the library, but he also says he put his two weeks' notice in yesterday (also odd that this is on New Years' Eve) which would make identifying him trivial.
Fake stories get to the front page of Reddit every day, I wish journalists were pointing out the actual signs not to trust something to act as a better example.
Offworld Trading Company is a real-time economic strategy game. It does have a fair amount of things to manage, but having no units means the APM is far lower than a traditional RTS.