Hinton's 'Capsule Networks' Unveiled(wired.com)
wired.com
Hinton's 'Capsule Networks' Unveiled
https://www.wired.com/story/googles-ai-wizard-unveils-a-new-twist-on-neural-networks/
5 comments
There is also code https://github.com/naturomics/CapsNet-Tensorflow and pretty cool explanation of ideas https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...
Thanks for that. the hackernoon piece in particular is really worth the read. Two salient points/questions/comments that jumped out to me after reading (these are really about the theory and not practice):
-The fact that you have a bunch of capsules in each layer, and each one of them is intrinsically performing a non-linear filter function (a particular squashing function shown as an image in that blog), seems like both a great asset (it looks like a 'meta-network') and also a potential problem. If you need to tune weights within these caps by the derivative of such a composite function, it doesn't seem straightforward.
-The 'routing by agreement' feature is interesting, but I dont quite get why it is superior to max pooling. If the feature is simply that it punishes weak links rather than selecting only the strongest, one interesting analogy is that it seems a bit Hebbian , and related to a concept in un-supervised learning called STDP (spike timing dependent plasticity).
-The fact that you have a bunch of capsules in each layer, and each one of them is intrinsically performing a non-linear filter function (a particular squashing function shown as an image in that blog), seems like both a great asset (it looks like a 'meta-network') and also a potential problem. If you need to tune weights within these caps by the derivative of such a composite function, it doesn't seem straightforward.
-The 'routing by agreement' feature is interesting, but I dont quite get why it is superior to max pooling. If the feature is simply that it punishes weak links rather than selecting only the strongest, one interesting analogy is that it seems a bit Hebbian , and related to a concept in un-supervised learning called STDP (spike timing dependent plasticity).
To save you clicks, the two pre-prints are at :
https://arxiv.org/pdf/1710.09829.pdf &&
https://openreview.net/pdf?id=HJWLfGWRb
Is this the only reference to this on HN? I'd be surprised but I didn't see it. Hinton is a powerhouse and he's been wanting to make progress on this for years.
Can someone ELI-not-a-genius-at-neural-networks?