Another famous dude dumping his thoughts on HN who is gulping it up like an addict.
Add this to the long list of names like Terence Tao, and others who seem to be intellectually incontinent lately in the sense that one cannot navigate this space anymore without encountering their thoughts
> built with American capital and mostly American minds.
I would say "built with American agency and commercial spirit", not minds.
Most of the things that we have were first built elsewhere (Germany being a prime supplier here with the mp3 or the Zuse), but turning them commercial was the input that came from America.
Moral? Hm. From a moral POV this would be about who has the right to terrorize the Iranian population: the Iranian government or the US/Israel government.
Opinions differ: hobby coders love it, but domain expert secretly despise it because it narrows the gap between the skills they spent years honing and the average Claude, I mean Joe, that just uses this mental exoskeleton.
> What do you mean ? These are top-notch mathematicians
YeS. I didn't dispute that. I disputed that they are NOT top notch ML specialist and have made one of the worst benchmarks of 2025-2026. Benchmarks like these would have worked maybe in early 2024 at latest. The field has moved on significantly since.
And yes, many many other benchmarks don't use toy problems -- their names are just a prompt away.
> You are kidding right ? FrontierMath benchmark [1] is produced by a startup whose incentives are dubious to say the least.
They did 1) open source some of their datapoints (on a similar order of magnitude) and 2) they carried out detailed evals. Here is much to learn from their blog posts, much more than from the current dataset.
But fair. If you don't like them, have a look at IMProofBench. Have a look at the AIMO competition. Have a loom at HardMath. It's quite a landscape of datasets already.
> Unlike the AI hypesters, these are real mathematicians trying to inject some realism and really test the boundaries of these tools
As mentioned above, realistic benchmarks that are bigger and better exist. Unfortunately, from a benchmarking POV, these mathematicians are the hypesters with a preprint that wouldnt even make it to the AI&Math workshops at ICML or NeurIPS.
If it's the latter case (which it has to be), it seems that attention credit (via, e.g., articles in NY Times) is very unfairly distributed.
None of the people that advanced the state of benchmarking and did the hard work on much bigger benchmarks got any, but a ridiculous benchmark of 10 question scored big.
> We will learn if the magical capabilities attributed to these tools are really true or not.
They're not. We already know that. FrontierMath. Yu Tsumura's 553th problem, RealMath benchmark. The list goes on. As I said many times on this thread, there is nothing novel in this benchmark.
This fact that this benchmark is so hyped shows that the community knows nothing, NOTHING, about prior work in this space, which makes me sad.
> These problems are representative of the types of subproblems research mathematicians have to solve to get a “research result”. They are finding that LLMs aren’t that useful for mathematical research because they can’t crush these problems along the way. And I assume they put this doc together because they want that to change :)
Same holds true for IMProofBench problems. This dataset shows nothing new.