These findings seem to be at odds. The former says that deep linear nets are useful, non-linear and trainable with gradient descent. The latter says that the non-linearity only exists due to quirks in floating point and that evolutionary strategies must be use to find extremely small activations that can exploit the non-linearities in floating point.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
"We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions."
"Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it’s a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."
Latently (SUS17) also provides a more self-directed path to learning deep learning focused exclusively on implementing research papers and conducting original research: https://github.com/Latently/DeepLearningCertificate
Our deep learning certificate is the product of years of hard work. It is a free service. The writing is quite clear (open to suggestion on that). Check it out.
Latently | Deep Learning Software Engineer | Boulder, CO | REMOTE
Have some time on your hands and interested in implementing scientific papers for a stealth-mode deep learning startup? Send your resume to [email protected]
Latently (SUS17) also provides a more self-directed path to learning deep learning focused exclusively on implementing research papers and conducting original research: https://github.com/Latently/DeepLearningCertificate
This is from 1999. Meanwhile, IARPA now conducts studies where intelligence analysts look at various *INT layers of data stacked on top of eachother and cognitive modelers try to recreate the biases they demonstrate in the lab. See:
The Neural Basis of Decision-Making During Sensemaking: Implications for Human-System Interaction
Someday perhaps YC's efforts at UBI will trickle down to HN, resulting in a fair platform that doesn't extremize inequality. In the meantime, us peons are left to hijack the threads of YC Partners.
Free access to ridiculous amounts of hardware and the opportunity to implement important and hot scientific papers and conduct original research in deep learning.