This makes sense. Interacting with the world is hard and poses real bottlenecks.
I think that resource acquisition is a solvable problem for such a system, robots seem mainly limited by software.
I concede that I am too certain about the linearity of it. But like, cmon man, in the limit entropy ensures that everything ends. The world could still get very weird very fast. I think that's what original comment was trying to argue, and I think it's possible within the limits you've laid out.
That's all I'm trying to argue at least. And I admit that waving my hand at linearity sounds uneducated.
Is a coherent, human-interpretable, goal necessary for recursive improvement?
> how can you be so sure that reaching that goal looks like the straight line you envision?
Because the thing pursuing the goal would be able to improve it's ability to pursue the goal. Doesn't that follow from the premise of RSI?
> we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes.
We are terrible at it! I don't think that will stop us letting something rip in some unknowable direction (with unintended outcomes). I hope that aligned incentives are a prerequisite, but it seems increasingly likely that they are not.
> What I see is resistance to the idea that we can have any certainty about the future by drawing a straight line from the past.
I'm arguing that if recursive self improvement happens, that the trend line will be reasonably predictable. (and that we are close enough to RSI that we should take this possibility seriously).
That's the fun part, there is none (in the traditional way of people buying things at least).
> Why would I invest my earning into sustaining a machine that ideally is extracting every cent possible without leeway and funneling it up the pyramid.
You shouldn't! Unfortunately everyone acting in their own self interest still results in this getting funded, since if such a machine were to exist, would it not be better to have a share of it?
I agree with your perspective! I just don't think that, as a species, we have a good track record of saying no to the existence of 2000hp trucks with ALMOST no use.
What reason? To me recursive self improvement seems credible, it's just a question of when. It seems obvious that given RSI, trend lines will be at least linear.
The disagreement could just be about if RSI is currently meaningfully happening, but that seems very difficult to tell given the lack of public data.
I think it's credible to believe that that a system which is able to meaningfully improve it's own intelligence will lead to a far weirder world than the one imagined by the article.
The parent comment could be interpreted as frustration from a lack of imagination about how weird the future could be.
Perhaps we're not at the recursive self improvement yet, but it seems increasingly naive to believe that it isn't possible.
Yes. If it looks like 10000 iq is possible, then money will be thrown at whoever is most likely to achieve it first, and whoever has the most intelligent models right now strongly predicts who will be first.
This is driven by the belief that whoever gets to 10000 iq first will likely dominate the majority of all economic activity, since it will be more efficient to trade with them than with some less efficient (dumber).
This does not depend on traditional economic demand, at some point this becomes a self-fulfilling prophecy.
China will experience the same problems described in the article, even if that war happens and even if they win. They are possibly better equipped to deal with the problems, but I don't think that nicely-asking-companies-to-keep-humans is a viable long term strategy. And given Chinas history, I'm not sure most humans would enjoy a China shaped solution.
Regardless, their actions seem to indicate that they are trying their hardest to make this true; given that they have the resources of a developed nation behind them, it seems reasonable to seriously consider what they are saying.
Also we generally don't give the same latitude to people who lie about things as destructive as what they are proposing.
Don't forget that humans require inputs (land, water). It's not obvious that there is a happy equilibrium where the majority of humans are able to meaningfully compete for those inputs.
I think that resource acquisition is a solvable problem for such a system, robots seem mainly limited by software.
I concede that I am too certain about the linearity of it. But like, cmon man, in the limit entropy ensures that everything ends. The world could still get very weird very fast. I think that's what original comment was trying to argue, and I think it's possible within the limits you've laid out.
That's all I'm trying to argue at least. And I admit that waving my hand at linearity sounds uneducated.