Then instead of superheroes think about countries or groups. How do you align Iran? How do you align the NRA? How’s culture, storytelling, arguing, etc. working for us so far? The point is that there is no simple recipe for "alignment", it will be continuous work and discourse, just as it is today with humanity. We’re talking about minds here, not machines that follow exact orders. How do you change minds?
When we get intelligent machines they will become beings. Contrary to SV corporate-speak, you don’t align beings. You don’t align your neighbors, you don’t align people from other countries. You can’t hardwire them to follow certain principles, that’s the whole point of beings, otherwise you’re back at automated machines. All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?' and that is not a very interesting question to me.
But optimization is decidedly not intelligence. We’ve known this for decades and have clear proofs this is the case. This is just a collective dance of burying our heads in the sand. I’ll quote this guy called Turing: “If a machine is expected to be infallible, it cannot also be intelligent”.
This is not something programmers and other wild HN dwellers are accustomed to hearing, but there is a very strong case to suggest that intelligence, the universal kind, is not possible without empathy. Worry about alignment in machines, not in human-likes.
At scale, Evolutionary Strategies (ES) are a very good approximation of the gradient as well. Don’t recommend to jump just yet to conclusions and unifications.
"Blockchains are computers that can make commitments. Traditional computers are ultimately controlled by people, either directly or indirectly; blockchains invert this power relationship, putting the code in charge"
Really hits the crux of this futile effort. Trust cannot be formally defined. Trust is an ever-changing subjective belief done in an unknown world.
The stereotype of a libertarian crypto advocate who is really into bayesian inference is not a coincidence. They are micro-fascists: given time and information supposedly everyone would converge on the same indisputable truth (which I happen to currently hold) and act in the same optimal rational way.
The irony is that you don't even need to bring the human condition into the argument. The mathematics they so revere practically scream that there are fundamentally inherent limits. Solomonoff induction is incomputable. There is no free lunch. Why continue this march of madness?