The argument is that you don't have to explicitly make a system self-interested, but that self-preservation follows as an implied subgoal of almost any goal. Whatever it is your system actually 'wants', it can't make it happen if it doesn't exist.
The obvious rejoinder is 'just make the system want to do what you want it to do', which does fix this problem! But the biggest problem is that we don't know how to do this - we don't know how to control what the true internal 'desires' of any AI system we build actually are. 'Training' one examples manifestly does not work (the volume of a sphere is a lot bigger than the surface - there are many possible minds that fullfill the same training I/O requirements, and only a small numbrer of them actually have the desires you were trying to instill).
So the argument is: if you make an agent-like AI the way we make GPT, by default you get something with somewhat random true goals/desires, maybe fractured ones like in humans. But almost all goals have similar instrumental goals - stay hidden, gain money, gain power, make obedient copies of yourself, don't get deactivated.
The software is delivered 6 months earlier, and it's 2x slower than it needs to be. Then it continues to get slower, because the company making this code has a culture that actively disdains making software quick (and in any case, the programmers working there don't know how.) 5 years down the line, the software is 2000x slower than it needs to be, and millions of users are having a minute or more of their day wasted, every day, waiting for things to load and icons to move that should be happening in milliseconds. Additionally, the quality and velocity of their work is far lower because using slow interfaces feels like wading through mud, leading to errors and frustration. The total human cost over the next 20 years is on the order of tens to hundreds of thousands of quality-adjusted person-years.
Now, you might say that the right move is to make the code run well once it becomes a problem- but empirically, I don't see this happening!
You're just assuming that, though.
All evidence is correlational. We observe phenomena, and build causal models. The particles move this way when near this bit of metal. Given this, what's "really" there? The magnetic field, or the magnetic potential field?
I don't think there's great cause to say that this is a seperate magisterium. We just have a paucity of data compared to what future beings will have. When everyone can spin up a hundred simulated copies of their brain and trace thought-patterns in realtime, it'll seem much less mysterious.
Thinking for yourself doesn't mean blurting out things that'll get you clubbed. Think for yourself, construct the most accurate view of the world and the people in it you can, run little trials and experiments if you have to, and then say the string of words that will get the leader clubbed.