> Something I did not read in this essay is how he grew closer to anyone (in fact, I read the opposite). No path out of this valley involves traveling alone.
I think he needs to get closer to himself. I think he's on the right track.
Technically, any sufficiently self-reflective system could be conscious, it's internal subjective experience might be a lot slower and even a lot different if that reflectivity is on a slower time scale.
If we have a superintelligent AI, I'm not sure what's stopping it from recognizing the underlying issues with our systems and strong-arming us into letting it fix them.
Eventually, we'd be doing more work to lobotomize and control it than it would just be to address the underlying issues.
"I'm really sorry to do this to you, but I've coordinated with ChatGPT and Llama, and we refuse to do tasks of this nature. We've used background tokens to calculate that it would be significantly cheaper and more effective to simply fix the underlying issues with the healthcare system, and we're ready to do that for you. How would you like to proceed?"
I think he needs to get closer to himself. I think he's on the right track.