AI evals are becoming intractable for the same reasons that measuring human general intelligence has been intractable for 150 years: the constructs resist decomposition, the benchmarks saturate, and the goalposts move. The reasons are the same because the thing being measured is the same. The difficulty is the evidence.
Something is happening right now that echoes the trends Clay Shirky was observing in those early days of Web 2.0. Perhaps it will turn out differently this time.
Skill at “steering” coding assistants may soon be the quality most sought after in software engineers, systems administrators and the like. It's a balance between a light touch and a firm focus.
Our entire system for measuring the value of cognitive work is denominated in human time. But not even machines realize they’re not on human time. That’s a problem.
We regularly have 92% - 93% participation in federal elections here in Australia. Having one next weekend, and already record numbers of pre-poll votes.