As part of the Holistic Agent Leaderboard (HAL) initiative at Princeton CITP, we evaluated more than 220 agent runs across 9 benchmarks, the equivalent of over 20,000 agent rollouts across 9 models and 9 benchmarks for a total cost of $40,000. The benchmarks are: AssistantBench, CORE-Bench Hard, GAIA, Online Mind2Web, Scicode, ScienceAgentBench, SWE-bench Verified Mini, TAU-bench Airline, and USACO.
In that process, we “burned” 2.6 billion prompt tokens and learned a lot along the way. In this article, I’d like to share some of the insights we gained, with a particular focus on the GAIA benchmark.
Exactly. I think the study is a good reminder that we really have to be careful about the productivity gains attributed to AI. Main takeaway imo, despite limitations from the study, is AI is not a panacea, it can increase productivity, but only if used 'well' and with the good workflows in place, and in the right context.
now openai has no other choice than shipping a cheaper version of o1 and o3. The alternative is everyone using r1 (self hosted or via openrouter, nebius AI, together AI and co)
We just have his version of the story though... He might be wrong. It's natural after spending so many years in a company to see change as bad, to miss the good old days... And he sure seems to have a problem with that black leader Jeanine...
A guy who never rose from his technical roles is lecturing a VP and the CEO of Google for their "lack of vision and strategy". Come on. Managing is startup and managing a huge behemoth like Alphabet will never be the same