I hear you - its just that I dont think that it neccesarily follows that an AGI in whatever form will necesarily be unconstrained or free to improve itself. I also happen to think that the biological systems are in-fact rather special and hard to replicate - especially in terms of efficiency and resillience. Just because some implementation of a universal turing machine can simulate intelligence doesn't mean it can do it well enough to survive the real world.
Finally, to me, nuclear reactions are kind of the opposite of AGI: I think it's vastly easier to blow something up (increase entropy) than to create an intelligence capable of understanding and improving itself (decreasing entropy - possibly at an accelerating rate).
I disagree with your argument, especially for point 1: these systems are massively constrained. The hardware they run on is fragile requiring massive amounts of power and tightly controlled environments. They don't have any means of replicating themselves (it can't run on arbitrary systems). The datacenters also have massive bandwidth between nodes - even if you could run 'it' on all the personal computers and phones in the world, it will likely struggle. Sure we can compress recent llm's down to being able to run on consumer hardware - but these things cant introspect, reason or adapt. They are completely static models and very far removed from anything agi. A lot of the progress in compute power in the last few years also come from changing representation: moving from foat32 to float 16 and more recently to float8. The silicon itself can only get so much better. It's not super obvious to me that we will have chatgpt4 like models on consumer hardware soon let alone solve true agi. Why don't we have true level 5 self driving cars yet?
We cant even figure out how to simulate a flatworm - and the connectome is solved.
I think there may be a significant advantage to running your own server: you can controll how long you want to keep the data. For example, you can keep data from cameras recording the street/yard for longer and delete footage from inside the house daily (or choose to not even record when you are at home).
I remember reading a popular science article a while back: apparently we have managed to construct the complete neural connectome of C. Elegans (a flatworm) some years ago and scientist were optimistic that we would be able to simulate it. The article was about how this had failed to realize because we don't know how to properly model the neurons and, in particular, how they (and the synapses) evolve over time in response to stimuli.
I feel quite strongly that there is a large difference between Stable Diffusion and Copilot: with the size of the training set vs the number of parameters, it should be very difficult if not impossible for Stable Diffusion to memorize and, by extension, copy paste to produce its outputs. Copilot is trained on text and outputs text. Coding is also inherently more difficult for an AI model to do. I expect it will memorize large portions of its input and is copy pasting in many cases to produce output. I therefore believe Copilot is doing "copyright laundering" but Stable Diffusion is not. Furthermore, I do not believe, for example, that artists should be able to copyright a "style" - but I would like to see them not be negatively impacted by this. Its complicated.
Presumably you can use the 97% that is correctly transcribed to rapidly filter out the relevant content. This is likely to be only a small portion of the total content. Then you check 100% of that.
Finally, to me, nuclear reactions are kind of the opposite of AGI: I think it's vastly easier to blow something up (increase entropy) than to create an intelligence capable of understanding and improving itself (decreasing entropy - possibly at an accelerating rate).