curious where the disagreement lands: the claim i'm least sure of myself is that measurement alone already counts as activation (nothing in the weights changes, so it's a looser sense of the word than usual)
the part i'd defend harder is the eval -> reward one: once a benchmark becomes the thing you train against, its flaws stop being measurement error and start being incentives. if you're pushing back somewhere in there, i'd genuinely like to hear it
yeah its really counterintuitive i think; i.e, getting the right framework and structure for this to work probably isn't trivial, models really hate playing well together. i wonder how their version would fair in real world use.
From the link: "Shot from 90 perspectives, 88 focus stacked images each. Nikon Z8, full frame, f/7.1, exposure 1/160, ISO 100, Laowa 180mm macro lens, with LED light and bluescreen."
Insane!
I think the question he tried to raise was "is this needed? Aren't today's / tomorrow's models well-enough equipped to deal with just OPEN API?"
(idk, just if I understand the question)
got me at "Most often scientists believe they understand more than they do, making their belief an illusion."
but why is it still bothering me?
1. feels unfalsifiable in spirit
2. somewhat restates "all models are wrong, but some are useful" less cleanly
3. doesn't really offer like, what can we do as science people? tomorrow morning perspective
author here. the part i'd actually like discussion on is the buried finding: physicians+GPT-4 didn't outperform GPT-4 alone on the management cases, and on the landmark cases the model alone beat the model+physician. the paper reports it and moves on. that's the 2026 question, and it's the one a Science-level platform could have been used to ask
Hey thanks! I do wonder that. I think that even if specifically for code smell the things would be subtler, for other forms of AI driven averageness (especially in areas where we can't RLVR the models to perfection) it might still be present. But yeah I wonder how those thoughts will age (and how we'll update our priors accordingly).
yeah I was really thinking about what the best "umbrella term" would be here. Since "LLM" is too widely used in a really specific context and "AI systems" felt niche I ended up with "LMs". Idk, up for debate..