> they things they believe in are unlikely to happen.
Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:
- "well, it's just such an extreme outcome, it must be improbable"
- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"
There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.
I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)
Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".
If M is the total mass of the object, then we can substitute this into the sum in the last term. And we already saw that the sum in the middle term was 0. So:
T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅0⃗ + ½Δv⃗² M
T' = ½∑ⱼ mⱼ v⃗ⱼ² + ½MΔv⃗²
So in terms of the original kinetic energy T, which was purely thermal energy, we get:
T' = T + ½MΔv⃗²
In other words, because of the quadratic kinetic energy formula, we can see that the total kinetic energy T' of a hot object is just its thermal kinetic energy T plus the usual mechanical kinetic energy ½MΔv⃗².
If you use the right formula for calculating it (which approximates p=mv at low speeds), momentum is actually conserved in special relativity, and so is energy.
However: Energy and momentum are not invariant under changes of reference frame, though the magnitude of the energy-momentum 4-vector is invariant between frames.
Energy is conserved in Galilean relativity. The thing you're trying to say is that it's not invariant across reference frames.
The answer linked above actually takes advantage of the fact that energy is not the same in different reference frames in order to make the argument work.
I think you are overthinking the heat thing. If you have a train car full of hot water and you slow the train down (extracting kinetic energy from it) until it stops, the water in the train car does not change temperature at all, other than a bit of sloshing around and loss of heat to the surroundings.
This looks like good work. Unfortunately, this kind of thing always seems to attract midwits on social media who then exclaim "oh, the people worried about AI alignment have caused the very alignment issues they feared? How ironic!"
In reality, it is (as mentioned in TFA) very possible to filter the training data and remove documents that contain discussions of AI misalignment. If an AI lab isn't doing this, it's simply because they don't consider the problem important enough to be worth the expense and development effort.
If you have a limited budget of tokens as a defender, maybe the best thing to spend them on is not red teaming, but formalizing proofs of your code's security. Then the number of tokens required roughly scales with the amount and complexity of your code, instead of scaling with the number of tokens an attacker is willing to spend.
(It's true that formalization can still have bugs in the definition of "secure" and doesn't work for everything, which means defenders will still probably have to allocate some of their token budget to red teaming.)
Kudos for giving a concrete example, but the square-cube law means that scaling area A results in A^(3/2) scaling for the mass of material used and also launch costs. If you make the pyramid hollow to avoid this, you're back to having to worry about heat conduction. You assumed an infinite thermal conductivity for your pyramid material, a good approximation if it's solid aluminum, but that's going to be very expensive (mainly in launch costs).
In reality, probably radiator designs would rely on fluid cooling to move heat all the way along the radiator, rather than thermal conduction. This prevents the above problem. The issue there is that we now need to design this system with its pipes and pumps in such a way that it can run reliably for years with zero maintenance. Doable? Yes. Easy or cheap? No. The reason cooling on Earth is easier is that we can transfer heat to air / water instead of having to radiate it away ourselves. Doing this basically allows us to use the entire surface of the planet as our radiator. But this is not an option in space, where we need to supply the radiator ourselves.
In terms of scaling by instead making many very small sats, I agree that this will scale well from a cooling perspective as long as you keep them far enough apart from each other. This is not as great from the perspective of many things we actually want to use a compute cluster for, which require high-bandwidth communication between GPUs.
In any case, another very big problem is the fact that space has a lot of ionizing radiation in it, which means we also have to add a lot of radiation shielding too.
Keep in mind that the on-the-ground alternative that all this extra fooling around has to compete with is just using more solar panels and making some batteries.
Radiators can shadow each other, so that puts some kind of limit on the size of the individual satellite (which limits the size of training run it can be used for, but I guess the goal for these is mostly inference anyway). More seriously, heat conduction is an issue: If the radiator is too long, heat won't get from its base to its tip fast enough. Using fluid is possible, but adds another system that can fail. If nothing else, increasing the size of the radiator means more mass that needs to be launched into space.
Name some of the contradictory possibilities you have in mind?
Also, do you actually think the core idea is wrong, or is this more of a complaint about how it was presented? Say we do an experiment where we train an alpha-zero-style RL agent in an environment where it can take actions that replace it with an agent that pursues a different goal. Do you actually expect to find that the original agent won't learn not to let this happen, and even pay some costs to prevent it?
Oh, so just a probability density thing where we sample q and check if it's p^n (retrying if not) rather than sampling p and n separately and computing q=p^n? I guess that's probably what the they were going for, yeah.
Why is that? My guess would be that you could adjoin an i all the time to the p^n field and get the p^2n field, as long as you had p = 4k + 3. But that's admittedly based on approximately zero thinking.
EDIT: Looking things up indicates that if n is even, there's already a square root of -1 in the field, so we can't add another. So now I believe the 1/4 of the time thing you mentioned, and can't see how that's wrong.
Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:
- "well, it's just such an extreme outcome, it must be improbable"
- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"
There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.
I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)