Capsule Networks Tutorial [video](youtube.com)
youtube.com
Capsule Networks Tutorial [video]
https://www.youtube.com/watch?v=pPN8d0E3900
30 comments
For anyone who hasn't watched the video: this comment is on topic and certainly relevant, because Hinton himself invented Capsule Networks. https://www.wired.com/story/googles-ai-wizard-unveils-a-new-...
I think this is the same author that published "Hands-On Machine Learning with Scikit-Learn and TensorFlow...". The quality of the book (thus far) is so high that I immediately started Googling about the author to try and learn more (and did not learn much), assuming he must be well known. I did not learn much, but can at least say the book is fantastic.
[1]: https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-T...
[1]: https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-T...
Too bad O'Reilly no longer sells books -- Cyber Monday 50% off is no more.
He used to be the PM of youtube video classification too.
Nice video, however instead of riding the hype train of arxiv, could we wait until peer review analyzes the paper?
If someone other than Hinton presented a YADLA (Yet Another Deep Learning Architecture) that does not achieve state of the art level of performance in the basic datasets, it would not be very well received.
If someone other than Hinton presented a YADLA (Yet Another Deep Learning Architecture) that does not achieve state of the art level of performance in the basic datasets, it would not be very well received.
/user/geoffhinton 1 year ago
> Over the last three years at Google I have put a huge amount of work into trying to get an impressive result with capsule-based neural networks. I haven't yet succeeded. That's the problem with basic research. There is no guarantee that ideas will work even if they seem very promising. Probably the best results so far are in Tijmen Tieleman's PhD thesis. But it took 17 years after Terry Sejnowski and I invented the Boltzmann machine learning algorithm before I found a version of it that worked efficiently. If you really believe in an idea you just have to keep trying.
https://www.reddit.com/r/MachineLearning/comments/4w6tsv/ama...
Most of the deep learning papers published are just exploring and incrementally building upon the ideas 'Canadian Mafia' (Hinton, LeCun and Bengio) discovered years ago. At some point this 'idea space' is explored and understood and we hit the wall just like before. Let's hope that people doing basic research can find new breakthroughs in less than 17 years.
> Over the last three years at Google I have put a huge amount of work into trying to get an impressive result with capsule-based neural networks. I haven't yet succeeded. That's the problem with basic research. There is no guarantee that ideas will work even if they seem very promising. Probably the best results so far are in Tijmen Tieleman's PhD thesis. But it took 17 years after Terry Sejnowski and I invented the Boltzmann machine learning algorithm before I found a version of it that worked efficiently. If you really believe in an idea you just have to keep trying.
https://www.reddit.com/r/MachineLearning/comments/4w6tsv/ama...
Most of the deep learning papers published are just exploring and incrementally building upon the ideas 'Canadian Mafia' (Hinton, LeCun and Bengio) discovered years ago. At some point this 'idea space' is explored and understood and we hit the wall just like before. Let's hope that people doing basic research can find new breakthroughs in less than 17 years.
I'm not saying to discard this research. I'm suggesting to wait until it is peer-reviewed and published before jumping on it.
To me, the capsule concept seems reasonable, and I have my personal opinion about its strengths and flaws. But my opinion hardly matters.
I expect peer reviewers from NIPS to have a better understanding that I have, and I trust them to filter and clean this idea, instead of trusting the research just because of the name that signs the paper.
To me, although it has its flaws, the _double-blind_ _peer-reviewed_ processes is important.
To me, the capsule concept seems reasonable, and I have my personal opinion about its strengths and flaws. But my opinion hardly matters.
I expect peer reviewers from NIPS to have a better understanding that I have, and I trust them to filter and clean this idea, instead of trusting the research just because of the name that signs the paper.
To me, although it has its flaws, the _double-blind_ _peer-reviewed_ processes is important.
[deleted]
The paper has already been accepted to NIPS 2017. Poster session is Tue Dec 5th 06:30 - 10:30 PM @ Pacific Ballroom #94
https://papers.nips.cc/paper/6975-dynamic-routing-between-ca...
https://papers.nips.cc/paper/6975-dynamic-routing-between-ca...
Wait for what exactly? Talking about this? Implementing and testing it? Experimenting with it, trying to replicate results, trying to extend the ideas? Etc? I'd argue that all of this is peer-review, if not in the traditional / formal sense. And CS (especially ML/AI) seems to be moving in this direction over the past few years and that's not necessarily a bad thing.
Also, keep in mind that peer-review or not, if you look at this from a Bayesian point-of-view, the prior on this work being important / meaningful is going to be pretty high for a lot of people - just because it is Hinton. And that's a reasonable position given his past work.
Also, keep in mind that peer-review or not, if you look at this from a Bayesian point-of-view, the prior on this work being important / meaningful is going to be pretty high for a lot of people - just because it is Hinton. And that's a reasonable position given his past work.
We have double blind peer review precisely to avoid the bias that you express in your second sentence.
I feel like you're missing my point. Bayesian reasoning of that nature is totally reasonable and isn't something to be avoided just for the sake of avoiding it. What is is, is useful as a guide for where to direct energy and focus. And what it is is faster than sitting around playing with your pud waiting for review for a journal submission.
Again, what's going on now is a form of peer-review. Double blind? No, but that's not really relevant in this context anyway.
ML is really more of an empirical field in this day and age and people are going to read pre-prints on ArXiv, and use various Bayesian weighting schemes to decide what to direct time and energy towards. This process complements, not replaces, the kind of formal peer review you're demanding. There will still be plenty of room, and time, for that stuff, but there's no real reason to wait for all that to happen before starting to look into something.
Again, what's going on now is a form of peer-review. Double blind? No, but that's not really relevant in this context anyway.
ML is really more of an empirical field in this day and age and people are going to read pre-prints on ArXiv, and use various Bayesian weighting schemes to decide what to direct time and energy towards. This process complements, not replaces, the kind of formal peer review you're demanding. There will still be plenty of room, and time, for that stuff, but there's no real reason to wait for all that to happen before starting to look into something.
Is peer review generally done double blind? Who in the field has not heard of hinton s capsule proposal that would be a blind reviewer? Hell I'm not even in the field and I've heard of it.
It was already reviewed when they published the arxiv preprint, so there would be no bias in this paper in particular. And as now, everyone and his/her cat has heard about capsules, you'd expect that some others than Hinton et al might write about capsules, so you shouldn't be able to say "hey, it's Hinton because capsules!".
I think it's important to note that the paper was accepted for NIPS 2017, and isn't just some random paper pushed on arXiv.
I think if you have good natural instincts about artificial intelligence you realize that Hinton really is the god in this space. If you don't, you get distracted by LeCunn and Schmidhuber and whatever lesser minds. The capsule theory here is maybe not the best implementation, but the intuition behind it is still leading the way. The undifferentiated mass of neurons is not how evolution has solved our problems. 3d geometry is intrinsic to the low level design. It shouldn't be learned. It should be assumed.
I reject your generic devotion to process. The real leadership and the process are far different. The process of peer review might do a good job of rejecting bad ideas, but it does a lousy job of accepting revolutionary ideas. I bet you don't understand the difference.
I reject your generic devotion to process. The real leadership and the process are far different. The process of peer review might do a good job of rejecting bad ideas, but it does a lousy job of accepting revolutionary ideas. I bet you don't understand the difference.
While I agree wholeheartedly that peer review is fundamentally a risk-averse, conservative process, I bristled when I read the statement that LeCun as a "lesser mind." That's quite rude, and uncalled for.
Two impressions: 1. when I saw the original Hinton proposal of capsule networks, I thought it was kind of halfway to a hofstadter-style cognitive machine from his work "conceptual slippages". Now understanding it more, I am more confident in my assessment.
2. I think that implementations are going to be hamstrung by the clunky nature of tensorflow's architecture... Did anyone else feel this?
2. I think that implementations are going to be hamstrung by the clunky nature of tensorflow's architecture... Did anyone else feel this?
What is clunky exactly about tensorflow's architecture?
for starters, the problem that you have to separate the definition of the computational graph from the actual execution of the function (there's separate declarative and imperative stages).
This problem is not insurmountable. Something like this would be really cool:
https://www.youtube.com/watch?v=ijI0BLf-AH0
This problem is not insurmountable. Something like this would be really cool:
https://www.youtube.com/watch?v=ijI0BLf-AH0
I found this video to be much easier-to-follow than previous posts focusing on intuition, e.g., https://news.ycombinator.com/item?id=15690121
How does it compare with, Spiking Neural Network?
Both use vectors (but in different way) to encode more information
The author answers some questions in the comments as well. Worth checking out!
I predict capsule networks will not nearly have as big of an impact on the ML community as many think it will. Why? Because the main reason they exist is to address performance issues in really advanced, cutting-edge, models. But that is not what drives upvotes here and on reddit. The failure of capsule networks to pick up steam and the continued popularity of GANs, I think, is a signal that the main reason ML is still trendy and in vogue is because the subject, AI, tickles the imagination of engineers, but is still, five years after ML started to become popular, not as big in actual practical engineering systems as what the outside public might think it is.
I think you have a misconception of what capsule networks are. They are not intended to address "performance issues in really advanced models", they are intended as another paradigm in deep learning that Geoff Hinton thinks has a lot of promise.
I also don't know what you mean by "the failure of capsule networks to pick up steam". The paper literally came out a month ago. It's too early to say whether it'll "pick up steam" or not.
I also don't understand what you mean by "the continued popularity of GANs" showing anything.
I also don't know what you mean by "the failure of capsule networks to pick up steam". The paper literally came out a month ago. It's too early to say whether it'll "pick up steam" or not.
I also don't understand what you mean by "the continued popularity of GANs" showing anything.
I agree, it still needs a strong use case. IMHO, new network architectures are generally overhyped. Even AlexNet performs wonderfully well in most problems once you add nice initialization and batch normalization.
The specific performance issue they are designed to address is training set size dependence. That's not exactly trivial.
https://www.reddit.com/r/MachineLearning/comments/7ew7ba/d_c...