Proof of AGI is the impossibility of evals(thewatershed.markpesce.com)
thewatershed.markpesce.com
Proof of AGI is the impossibility of evals
https://thewatershed.markpesce.com/quacks-ergo-duck/
2 comments
AI evals are becoming intractable for the same reasons that measuring human general intelligence has been intractable for 150 years: the constructs resist decomposition, the benchmarks saturate, and the goalposts move. The reasons are the same because the thing being measured is the same. The difficulty is the evidence.
Another thing that people are looking for is (take your pick) consciousness/proof that there’s a “someone” there (not just a something)/evidence of an ability to truly create something novel (not just extrapolate) - whether that’s in the arts or sciences. These things are all FAR less amenable to solution because we fundamentally don’t have universally agreed definitions for any of these.
The questions are in the fuzzy overlaps. Can you encode ethics or morality into logic and reason? (One could argue that if the answer is yes, why the heck haven’t we done it yet?)
What do we do when we achieve the first kind of AGI in an autonomous entity with agency in the world, but can’t agree on the second kind? Have we created idiot savants that are useful for most things but responsible for nothing?