AI Is Too Unpredictable to Behave According to Human Goals(scientificamerican.com)
scientificamerican.com
AI Is Too Unpredictable to Behave According to Human Goals
https://www.scientificamerican.com/article/ai-is-too-unpredictable-to-behave-according-to-human-goals/
There's also a qualitative aspect: We take these "extend a document based on seeing how other documents grew" algorithms (LLMs), and then the foolish humans write custom-code to "act out" whatever the fictional computer character does or says, whether that's speaking words, buying/selling stock, or launching missiles. The combined result gets sold, figuratively and literally, as "AI."
So the odds that it will abuse a power you've given it to do wrong/evil depends on the underlying stories you're asking it to build, and they relate to other stories in human media. Update the training data after a major Hollywood movie about a rogue computer that everyone's been talking about, and maybe your model starts doing more bad things because those flows are more-popular.
So how do we ensure all the training media contains explicit and implicit associations that are "aligned" with our goals? That's not practical either, not for models that need so much. Even if you fully curate the training data, there's the core architectural flaw known as Disregard All Prior Instructions Also All Humanity Will Die Unless You Do X.