The fundamental premise of this paper seems flawed -- take a measure specifically designed for the nuances of how human performance on a benchmark correlates with intelligence in the real world, and then pretend as if it makes sense to judge a machine's intelligence on that same basis, when machines do best on these kinds of benchmarks in a way that falls apart when it comes to the messiness of the real world.
This paper, for example, uses the 'dual N-back test' as part of its evaluation. In humans this relates to variation in our ability to use working memory, which in humans relates to 'g'; but it seems pretty meaningless when applied to transformers -- because the task itself has nothing intrinsically to do with intelligence, and of course 'dual N-back' should be easy for transformers -- they should have complete recall over their large context window.
Human intelligence tests are designed to measure variation in human intelligence -- it's silly to take those same isolated benchmarks and pretend they mean the same thing when applied to machines. Obviously a machine doing well on an IQ test doesn't mean that it will be able to do what a high IQ person could do in the messy real world; it's a benchmark, and it's only a meaningful benchmark because in humans IQ measures are designed to correlate with long-term outcomes and abilities.
That is, in humans, performance on these isolated benchmarks is correlated with our ability to exist in the messy real-world, but for AI, that correlation doesn't exist -- because the tests weren't designed to measure 'intelligence' per se, but human intelligence in the context of human lives.
The function of news is to help a democratic citizenry be critically informed, and that this kind of statistic doesn't accomplish what it set out to do, although it's certainly interesting for its own sake. I think it's a challenge of our age to figure out how to create institutions that are wise and don't simply bend to distorting pressures (money, politics, psychology).
For example, we do want terrorism over-represented relative to old-age-deaths. However, a responsible and self-aware media would really attempt to counteract 'availability bias' -- e.g. that due to the human mind what is repeated we tend to assume is actually more prevalent. But we don't have wise institutions at the moment.
The more general problem is that it is hard to quantitatively demonstrate the ways in which media fails at fulfilling its complex societal role, because it is a qualitative failure in general, although we can poke at it's edges for sure (e.g. fearmongering language probably has gone up, as has polarization on both sides of the aisle, and the amount of information-free 'babbling and speculating' in the immediate aftermath of some event has likely gone up over time).
Yeah -- I don't get why this is front-page -- reads like LLM quasi-insight:
"Through activation, lifeless equations became living systems. The neuron was no longer a mere calculator; it was a decider - a locus of transformation where signal met significance." -- wtf
This paper, for example, uses the 'dual N-back test' as part of its evaluation. In humans this relates to variation in our ability to use working memory, which in humans relates to 'g'; but it seems pretty meaningless when applied to transformers -- because the task itself has nothing intrinsically to do with intelligence, and of course 'dual N-back' should be easy for transformers -- they should have complete recall over their large context window.
Human intelligence tests are designed to measure variation in human intelligence -- it's silly to take those same isolated benchmarks and pretend they mean the same thing when applied to machines. Obviously a machine doing well on an IQ test doesn't mean that it will be able to do what a high IQ person could do in the messy real world; it's a benchmark, and it's only a meaningful benchmark because in humans IQ measures are designed to correlate with long-term outcomes and abilities.
That is, in humans, performance on these isolated benchmarks is correlated with our ability to exist in the messy real-world, but for AI, that correlation doesn't exist -- because the tests weren't designed to measure 'intelligence' per se, but human intelligence in the context of human lives.