Yeah, this is a good point. IIRC, I wasn't able to get the network to train very well at all with standard SGD. I don't think I thought to try Adam with β1 = 0, I will try it (& recompute brackets) if I get some time.
If we have built up a momentum M, then the two orderings are:
So there is an extra term which is not actually a Lie bracket itself. I think the bracket can still be informative on its own, but it's definitely no longer the sole component of what happens when order is swapped.
One other inconsistency that is a little less bad is BatchNorm. Since it needs a whole batch to work, and we're just comparing individual examples, I computed the Lie brackets with the BatchNorm layers in eval mode, not train mode.
I don't know if there is any relevance of this to Muon, even if so, it would likely be very messy to compute.
If I understand your question correctly, the answer is that not only are the Lie brackets non-commutative, they're anti commutative (swapping the order negates the bracket). But this ironically means they end up having the same RMS, because the squaring part of the RMS gets rid of the sign.
Bhauth wrote an analysis of this idea, and tldr is that trying to get an energy payback on literally vaporizing such a long cylinder of rock is brutally difficult, probably enough to make the economics of the plan unworkable.
tldr is that it happened because the universe cooled down from a stupendously insanely high temperature to a merely insanely high temperature shortly after the big bang.
The Higgs field is a complex number Φ (this number can vary at different points in space, we'll come back to this, so don't worry about it for now). You can imagine it as a ball bouncing around on the landscape shown in the image. The higher the altitude of the ball, the more energy it has (just like a ball in real life). Φ = 0 corresponds to the center of the image, the point right at the top of the little hill.
At a high temperature, the ball is jostling and moving around like crazy. You can imagine constantly pelting the ball with marbles from all directions, causing it to dance eratically around the landscape. (Further, the ball doesn't experience any friction. It slows down when it happens to get hit by a marble that's heading in the opposite direction to it.) In reality, there are no marbles, of course, the jostling comes from the interactions of the Higgs field with other fields, all of which are also stupendously insanely hot.
The landscape in the picture has a rotational symmetry. You can rotate it by any angle, and it will still look the same. When the temperature is very high, the ball dances across the whole landscape. It slows down as it climbs up a slope, so it does spend less time at the bits that are at a higher altitude. But if we consider a thin ring around the center that's all at about the same altitude, the ball is equally likely to be anywhere along the ring. The average value of Φ is 0.
As the temperature decreases, the ball's motion calms down, and it spends more and more of its time in the deepest valley of the landscape. It rarely has the energy to climb high up the slopes anymore. Eventually, the ball will start to live on just the narrow ring around the center where the altitude is lowest.
Now we come back to the fact that the Higgs field is a field, which means it has a value at every point in space, and these values can differ from each other. It turns out that all fields in physics "prefer" to have similar values at nearby points in space. There is an energy penalty for fields that change rapidly in space. At high temperature, this didn't matter too much. The Higgs field had lots of energy to pay this penalty, just like it had lots of energy to climb up the slopes of the landscape. So the field here and the field 1nm to the left could have wildly different values. At cold temperatures, it matters a lot. So the Higgs field has the lowest energy if it has the same value everywhere in space. Anything else comes with an energy penalty. If we pick a point in space, and try to move the field clockwise or counterclockwise around the center, the neighbouring points in space pull the field back towards the average of the surrounding values.
So at any point in space, Φ is just equal it its average value, which is not 0. It's not zero because we have to randomly pick a point somewhere along the ring of lowest altitude, which is some distance from the central 0. The universe has randomly selected a direction in this landscape to be "special".
This is the situation from when the universe was insanely hot all the way up until the present. Incidentally, if you vibrate the ball radially, towards and away from the center of the landscape, this vibration corresponds to the Higgs boson.
If we could somehow heat the universe up to a stupendously insanely high temperature again, then the special direction would disappear, and the average of Φ would be 0 again. This is kind of similar to how magnets lose their magnetization if heated past a certain critical temperature, the Curie point. [1] If we let it cool down again, it would choose a different random special direction.
If we have built up a momentum M, then the two orderings are:
M' = M + εv1
θ' = θ + M' = θ + M + εv1
M'' = M' + εv2(θ') = M + εv1 + ε(v2 + (M + εv1)⋅∇v2)
M' = M + εv2
θ' = θ + M' = θ + M + εv2
M'' = M' + εv1(θ') = M + εv2 + ε(v1 + (M + εv2)⋅∇v1)
Then the resulting difference in momenta M'' is:
ε^2*[v1, v2] + ε(M⋅∇)(v2 - v1)
So there is an extra term which is not actually a Lie bracket itself. I think the bracket can still be informative on its own, but it's definitely no longer the sole component of what happens when order is swapped.
One other inconsistency that is a little less bad is BatchNorm. Since it needs a whole batch to work, and we're just comparing individual examples, I computed the Lie brackets with the BatchNorm layers in eval mode, not train mode.
I don't know if there is any relevance of this to Muon, even if so, it would likely be very messy to compute.