In their second publication they actually use the probabilistic unit, but computing those requires running an EM algorithm for each layer of capsules: https://openreview.net/forum?id=HJWLfGWRb
I doubt this will be a real problem because I've seen brilliant people excel in provincial universities with very little means and talent around. Internet education has gotten pretty good and probably reflects most of the experience of studying at a top university with the best researchers in the field. People might be 5% worse prepared. Nothing to be concerned about.
I still find it incredibly hard to tell whether this is overblown hype or legit scientific progress. There is no indication whatsoever that this approach scales to deep feature hierarchies and that is likely what you need to compete on hard tasks like classification on ImageNet. Given the amount of money at play (several hundred millions of dollars), writing 70 pages, making code publishable is certainly an obvious way to get the most out of the hype.
My guess would be that symmetry is a simple heuristic measure of physical fitness. Visual attraction is basically a strong regularizer that restricts the search space to phenotypes with particular traits. Asymmetry means that the joints wear out more quickly and muscles might not coordinate optimally leading to less strength and a reduced ability to hunt and to fight predators. AFAIK it is also a quite robust predictor of all kinds of diseases because it often means that the growth signalling is out of tune throughout the system. Visual selection basically performs environmental selection more immediately and more effectively: an asymmetric person might still survive, but its offspring has a lower overall chance to survive. The teaching signals of that are much weaker.
The problem is that anxiety is warranted in this case. You can’t do anything about solving the problem for everybody, but you can potentially save your own life:
- Work like a madman to get into the 1%.
- Move to a region that is self-sustaining and is little impacted by global warming.
> evolution of evolvability and then a seemingly unrelated subject: the evolution of robustness.
Isn't evolvability only about robustness? What other criteria would improve evolution? Mutations just happen, so the question is how well the phenotype can deal with these mutations. If it can incorporate mutations well, it can tunnel to different useful phenotypes and therefore is robust.
I think, the main effect of sexual reproduction is that, much like GANs and competitive self-play, it creates species-internal competition: Both sexes need to impress, which makes cheating an obvious strategy (makeup, steroids, Shakespeare quotes, LISP etc., but many such examples can be found in the animal world), and hence both sexes also need to be able to detect cheating. Some species are rather asymmetric in that regard. For example, in humans it is mainly women who attract (they masquerade as fruits [make up is likely a cross-cultural phenomenon; and, well, breasts] tapping into male food gathering circuitry); men compete in hierarchies trying to impress and women select men from the top of the hierarchy. Complex dynamics emerging from this likely lead to the immense growth of the human cortex.
Sexual reproduction basically outsources some of the selection effort to the cognitive apparatus of the species itself, thereby introducing a massive amount of additional selection signals (mainly by the much increased necessity to model other minds, namely minds of the opposite sex). Many of these signals promote traits that are useful for survival (mainly intelligence and health).
Wait, that can’t be wrong because that is literally what DO does. It is a convex hull regularizer around the network activations using noise. That is also why dropout does not solve susceptibility to adversarial examples: It merely extends the regions that the NN generalizes to outward; but that is limited because high-dimensional spaces are counter-intuitively large and the noise required to cover a descent fraction of the “unmapped” space would completely prevent learning. AFAIK, Yarin Gal merely provides a Bayesian interpretation of the noise.
Actually, GANs reach state of the art in anomaly/outlier detection and drug/molecule prediction, so there is certainly more to it than just artistic applications:
Totalitarian systems are quicker to come up with the rules (in this case for self-preservation). And when time matters, then it has obvious advantages.
Small correction: Learning algorithms that determine the contribution of a unit to the result over time can also be local in time, namely by tracking the contributions online (for example by eligibility traces or RTRL).
> By firing earlier it inhibits neighboring cells, creating highly sparse patterns of activity for correctly predicted inputs.
This part is so vague. It seems to lack an explanation of how interneurons inhibit other neurons nearby. Also, wouldn‘t sparsity even occur without the early firing enabled by distal pattern matching?
> When relatively few neurons are active relative to the population, then such pattern recognition is robust.
I guess deep world models are still severely riddled by all sorts of problems: vanishing gradients, BPTT being O(T), poor generalization ability of NNs (which likely is due to the lack of attractor state associative recall, as well as concept composability), lack of probabilistic message passing to deal with uncertainty, and perhaps some priors about the world are necessary to make learning tractable (such as spatial maps and fine-tuning for time scales that contain interesting information).
It also presumes that one can simulate the world at low cost. In AlphaGo Zero it takes 0.4 s for 1.600 node extensions, but in this case the cost of the world is negligible. Anyway, assuming you need that many node extensions to get decent quality updates, that puts a rather a tight limit on the cost of simulating the world.
> It's an uninteresting reduction since linear algebra can describe almost everything.
The question is whether it can do so efficiently. As far as I know, alternating applications of affine transforms and non-linearities are not so useful for some computations that are known to occur in the brain such as routing, spatio-temporal clustering, frequency filtering, high-dimensional temporal states per neuron etc.