Dynamical Isometry and Mean Field Theory: How to Train 10k-Layer Vanilla CNNs [pdf](proceedings.mlr.press)
proceedings.mlr.press
Dynamical Isometry and Mean Field Theory: How to Train 10k-Layer Vanilla CNNs [pdf]
http://proceedings.mlr.press/v80/xiao18a/xiao18a.pdf
It shows that extremely deep vanilla CNNs - without the use of batch normalization or residual connections - can be trained simply by using a Delta-Orthogonal weight initialization scheme and appropriate activation function.
The Delta-Orthogonal initialization scheme is derived theoretically by developing a mean field theory for signal propagation which characterizes the conditions for dynamical isometry. Ultra-deep CNNs can train faster and perform better if their input-output Jacobians exhibit dynamical isometry, namely the property that the entire distribution of singular values is close to 1. Put another way, dynamical isometry is a necessary condition for signals to flow both forward and backward through the network without attenuation. A variety of pathologies such as vanishing/exploding gradients make training such deep networks challenging - mean field theory is a powerful tool that offers solutions to these challenges.
The authors demonstrate experimentally that Delta-Orthogonal kernels outperform existing initialization schemes for very deep vanilla convolutional networks. They also find strikingly good agreement between theoretical and experimental results. One of the most astonishing findings IMO is that for networks initialized using this scheme the learning time measured in number of training epochs is independent of depth.
> Our results indicate that we have removed all the major fundamental obstacles to training arbitrarily deep vanilla convolutional networks. In doing so, we have layed the groundwork to begin addressing some outstanding questions in the deep learning community, such as whether depth alone can deliver enhanced generalization performance. Our initial results suggest that past a certain depth, on the order of tens or hundreds of layers, the test performance for vanilla convolutional architecture saturates. These observations suggest that architectural features such as residual connections and batch normalization are likely to play an important role in defining a good model class, rather than simply enabling efficient training.
Here is a link to the ConvolutionDeltaOrthogonal initializer in tensorflow [1].
[1] https://github.com/tensorflow/tensorflow/blob/d287ff3d95c06b...