Why Deep Learning Cannot Be Applied to Natural Languages Easily(linkedin.com)
linkedin.com
Why Deep Learning Cannot Be Applied to Natural Languages Easily
https://www.linkedin.com/pulse/google-hyping-why-deep-learning-cannot-applied-easily-berkan-ph-d
59 comments
Seconding it with: https://arxiv.org/abs/1610.03017 Fully Character-Level Neural Machine Translation without Explicit Segmentation.
In general, RNNs are especially fit for discrete sequences. And their continuous representation is actually an advantage (so they can see that two words are similar or analogous). BTW: see my draft on word2vec: http://p.migdal.pl/2016/12/30/why-do-word2vec-analogies-work...
In general, RNNs are especially fit for discrete sequences. And their continuous representation is actually an advantage (so they can see that two words are similar or analogous). BTW: see my draft on word2vec: http://p.migdal.pl/2016/12/30/why-do-word2vec-analogies-work...
this paper still consistently blows my mind whenever it is brought up. the future is clearly headed for complete end-to-end learning systems, maybe even with learning to learn methods. https://youtu.be/x1kf4Zojtb0?t=1h4m54s
> No offense, but this person has no idea what they are talking about
Honestly that seems to be the default for technical posts on linkedin.
Honestly that seems to be the default for technical posts on linkedin.
No offense, but this person has no idea what they are talking about.
He says: "Why Deep Learning cannot be Applied to Natural Languages Easily"
But I say: "It's true that neural networks can not be easily applied to natural languages," Take that!
He says: "Why Deep Learning cannot be Applied to Natural Languages Easily"
But I say: "It's true that neural networks can not be easily applied to natural languages," Take that!
Read beyond the headline.
The GP is pushing a particular solution they have for the problems of NN's applied to NLP. They might be right that this is a decent answer to the article's objections, for all I know.
But at the same time they say "It's true as well in my opinion that Google's Machine Translation is oversold" so the "they don't know what they're talking about except for all the points that I agree with them on" tone seems a bit silly.
But at the same time they say "It's true as well in my opinion that Google's Machine Translation is oversold" so the "they don't know what they're talking about except for all the points that I agree with them on" tone seems a bit silly.
Almost every sentence in this blog post is factually incorrect. Any agreement between the post and my comment is superficial.
Neural Translation isn't oversold. It's a bigger breakthrough than AlphaGo.
I agree. What I meant was that Google's blog post and the New York Times article make it sound like machine translation is a solved problem.
Which Google blog post made it sound like that?
"A Neural Network for Machine Translation, at Production Scale"[1] says:
Machine translation is by no means solved. GNMT can still make significant errors that a human translator would never make, like dropping words and mistranslating proper names or rare terms, and translating sentences in isolation rather than considering the context of the paragraph or page. There is still a lot of work we can do to serve our users better. However, GNMT represents a significant milestone. We would like to celebrate it with the many researchers and engineers—both within Google and the wider community—who have contributed to this direction of research in the past few years.
That seems fair to me.
"Found in translation: More accurate, fluent sentences in Google Translate"[2] says:
Today’s step towards Neural Machine Translation is a significant milestone for Google Translate, but there’s always more work to do and we’ll continue to learn over time. We’ll also continue to rely on Translate Community, where language loving multilingual speakers can help share their language by contributing and reviewing translations.
"Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System"[3] says:
In September, we announced that Google Translate is switching to a new system called Google Neural Machine Translation (GNMT), an end-to-end learning framework that learns from millions of examples, and provided significant improvements in translation quality.
[1] https://research.googleblog.com/2016/09/a-neural-network-for...
[2] https://blog.google/products/translate/found-translation-mor...
[3] https://research.googleblog.com/2016/11/zero-shot-translatio...
"A Neural Network for Machine Translation, at Production Scale"[1] says:
Machine translation is by no means solved. GNMT can still make significant errors that a human translator would never make, like dropping words and mistranslating proper names or rare terms, and translating sentences in isolation rather than considering the context of the paragraph or page. There is still a lot of work we can do to serve our users better. However, GNMT represents a significant milestone. We would like to celebrate it with the many researchers and engineers—both within Google and the wider community—who have contributed to this direction of research in the past few years.
That seems fair to me.
"Found in translation: More accurate, fluent sentences in Google Translate"[2] says:
Today’s step towards Neural Machine Translation is a significant milestone for Google Translate, but there’s always more work to do and we’ll continue to learn over time. We’ll also continue to rely on Translate Community, where language loving multilingual speakers can help share their language by contributing and reviewing translations.
"Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System"[3] says:
In September, we announced that Google Translate is switching to a new system called Google Neural Machine Translation (GNMT), an end-to-end learning framework that learns from millions of examples, and provided significant improvements in translation quality.
[1] https://research.googleblog.com/2016/09/a-neural-network-for...
[2] https://blog.google/products/translate/found-translation-mor...
[3] https://research.googleblog.com/2016/11/zero-shot-translatio...
If you look at the first chart in [1] and the cherry-picked example, it's easy for a casual observer to to come to the conclusion that MT is a solved problem, especially for "English > Spanish" and "French > English".
There is a clear gap between the performance of the system and "perfect translation". I don't think Google has been misleading in any way here - the system is pretty amazing, and this is the research blog so it is reasonable to expect people to look at it at least slightly.
But there is almost no gap between "neural" and "human translation".
Compared to a single human, yes. It really is that good for those language pairs(!)
This isn't cherry picking - you can try it yourself.
Randomly selecting the 2nd paragraph from https://www.hrw.org/fr/news/2017/01/02/israel/palestine-des-...
Human Rights Watch a documenté de nombreuses déclarations faites depuis octobre 2015 par des personnalités politiques israéliennes de haut rang, dont les ministres de la Police et de la Défense, appelant la police et les forces armées à tirer sur des individus suspectés d'attentats pour les tuer, avant même de déterminer si le recours à la force létale est, ou non, absolument nécessaire pour protéger des vies.
Choose which is automatically translated:
Human Rights Watch has documented numerous statements since October 2015, by senior Israeli politicians, including the police minister and defense minister, calling on police and soldiers to shoot to kill suspected attackers, irrespective of whether lethal force is actually strictly necessary to protect life.
vs
Human Rights Watch has documented numerous statements made since October 2015 by high-ranking Israeli politicians, including police and defense ministers, calling on police and armed forces to shoot at suspected terrorists even before deciding whether or not the use of lethal force is absolutely necessary to protect lives.
One of these in manually translated by an expert, one is by Google Translate (clicking on the "English" link on the page linked above will show the expert translation).
You can find other document collections to test here: http://www.miis.edu/academics/library/find/guides/translatin...
This isn't cherry picking - you can try it yourself.
Randomly selecting the 2nd paragraph from https://www.hrw.org/fr/news/2017/01/02/israel/palestine-des-...
Human Rights Watch a documenté de nombreuses déclarations faites depuis octobre 2015 par des personnalités politiques israéliennes de haut rang, dont les ministres de la Police et de la Défense, appelant la police et les forces armées à tirer sur des individus suspectés d'attentats pour les tuer, avant même de déterminer si le recours à la force létale est, ou non, absolument nécessaire pour protéger des vies.
Choose which is automatically translated:
Human Rights Watch has documented numerous statements since October 2015, by senior Israeli politicians, including the police minister and defense minister, calling on police and soldiers to shoot to kill suspected attackers, irrespective of whether lethal force is actually strictly necessary to protect life.
vs
Human Rights Watch has documented numerous statements made since October 2015 by high-ranking Israeli politicians, including police and defense ministers, calling on police and armed forces to shoot at suspected terrorists even before deciding whether or not the use of lethal force is absolutely necessary to protect lives.
One of these in manually translated by an expert, one is by Google Translate (clicking on the "English" link on the page linked above will show the expert translation).
You can find other document collections to test here: http://www.miis.edu/academics/library/find/guides/translatin...
Every translation is done by a single human... I would assume they used professional translators, otherwise it's not a fair comparison.
To honest it works better than I thought for French > English, but I when I tried some random French websites (not news) the results were not as good as this example and nowhere near human translations.
To honest it works better than I thought for French > English, but I when I tried some random French websites (not news) the results were not as good as this example and nowhere near human translations.
The argument the author uses is invalid. NNs are equally strong at modelling sparse signals, provided that they could be mapped into a continuous space, what is commonly referred to as an 'embedding'.
The premise of the article is valid though, in that NLP is a hard problem. The reason is partly because NLP is ill-defined; how do you define language understanding?
NNs are very effective at learning mappings of y=f(X), given enough examples. One of the reasons that they're so effective at modelling speech, vision, translation, etc., is that such mappings exist in high volumes. Because of the above-mentioned ambiguity of NLP, it's harder to come up with such pairs for 'understanding' a language. How do you come up with a dataset of sentences and their 'meaning'? Probably the best you could do is to map them to some action. And critics will readily disregard such attempts as 'not really NLP'.
The premise of the article is valid though, in that NLP is a hard problem. The reason is partly because NLP is ill-defined; how do you define language understanding?
NNs are very effective at learning mappings of y=f(X), given enough examples. One of the reasons that they're so effective at modelling speech, vision, translation, etc., is that such mappings exist in high volumes. Because of the above-mentioned ambiguity of NLP, it's harder to come up with such pairs for 'understanding' a language. How do you come up with a dataset of sentences and their 'meaning'? Probably the best you could do is to map them to some action. And critics will readily disregard such attempts as 'not really NLP'.
I think he's arguing that the current NN approach for NLP is not going to lead to embeddings that are going to make revolutionary progress in NLP.
And there have been attempts to ascribe a semantics to natural language from text (for ex. see CCG grammars). The datasets are not as big as for vision tho, yes. But I'm not convinced that we need such explicit datasets to be able solve this problem.
And there have been attempts to ascribe a semantics to natural language from text (for ex. see CCG grammars). The datasets are not as big as for vision tho, yes. But I'm not convinced that we need such explicit datasets to be able solve this problem.
I would not be sure that mapping of discrete linguistic objects to a continuous space is necessary. Why can't we handle the original space?
There are just a lot of things that have to be figured out still.
+ Different time scales. There is semantics on a sentence level while there is also semantics on a plot level. It's convenient to know key elements from the start of a story if you want to understand the plot. LSTMs are a perfect starting point.
+ When to stop learning. The so-called stability-plasticity dilemma. Our ability to pay attention to what matters might be tightly linked to our capability to forget vast bodies of texts that we just read. Current NNs do not seem to forget correctly. This was the rationale behind ART and ARTMAP (Grossberg) and might enter AI mainstream again soon.
+ Grammar constructions. Some aspects of grammar seem simpler than computer vision, where we also have a lot of structure in the environment, models like things that can be inside of other things, be balanced on top of other things, temporarily occluded by other things, etc. Other aspects seem more complicated, like the pleasantness of a poem. My gut feeling is that some of this gets spilled over from (a) structure in other modalities and (b) idiosyncrasies from our generative system (vocal cords, etc.). In other words, our grammatical preferences might be sampled not only from listening and reading.
+ Emphasis.
Just a few things that might lead to interesting NNs. Contrary to the author I think they are definitely in line with current research.
There are just a lot of things that have to be figured out still.
+ Different time scales. There is semantics on a sentence level while there is also semantics on a plot level. It's convenient to know key elements from the start of a story if you want to understand the plot. LSTMs are a perfect starting point.
+ When to stop learning. The so-called stability-plasticity dilemma. Our ability to pay attention to what matters might be tightly linked to our capability to forget vast bodies of texts that we just read. Current NNs do not seem to forget correctly. This was the rationale behind ART and ARTMAP (Grossberg) and might enter AI mainstream again soon.
+ Grammar constructions. Some aspects of grammar seem simpler than computer vision, where we also have a lot of structure in the environment, models like things that can be inside of other things, be balanced on top of other things, temporarily occluded by other things, etc. Other aspects seem more complicated, like the pleasantness of a poem. My gut feeling is that some of this gets spilled over from (a) structure in other modalities and (b) idiosyncrasies from our generative system (vocal cords, etc.). In other words, our grammatical preferences might be sampled not only from listening and reading.
+ Emphasis.
Just a few things that might lead to interesting NNs. Contrary to the author I think they are definitely in line with current research.
Starting his argument from a statement on continuity is a complete non sequitur though.
If he's going to make technical arguments, then they should at the very least be relevant and correct.
If he's going to make technical arguments, then they should at the very least be relevant and correct.
He makes a good point about continuous functions. But actually neural nets are quite good at handling discrete elements like words, likes, atoms in a molecule, etc.
Deep learning is part of enormous advances in NLP[0], just as it has set records to accuracy in almost every field of machine perception, from vision to audio.
Neural word embeddings like those produced by word2vec[1] make for very useful feature vectors when fed into other neural nets.
The headline of this post should be that NLP is harder than, say, image processing. In fact, for non-specialists, none of it is easy, because tuning hyperparameters is hard.
The kind of NLP that tries to reproduce human-level sentences and understanding is simply a more complex problem, given the plasticity of language.
[0] https://arxiv.org/abs/1611.04558 [1] https://deeplearning4j.org/word2vec
Deep learning is part of enormous advances in NLP[0], just as it has set records to accuracy in almost every field of machine perception, from vision to audio.
Neural word embeddings like those produced by word2vec[1] make for very useful feature vectors when fed into other neural nets.
The headline of this post should be that NLP is harder than, say, image processing. In fact, for non-specialists, none of it is easy, because tuning hyperparameters is hard.
The kind of NLP that tries to reproduce human-level sentences and understanding is simply a more complex problem, given the plasticity of language.
[0] https://arxiv.org/abs/1611.04558 [1] https://deeplearning4j.org/word2vec
Good point. I think the author quite clearly meant continuous wrt. a whole sentence or even text, not single (sparse) words, though. Take German, for example: The last word in a sentence that ends in a verb defines the whole structure of the sentence. Like, "Wir haben heute etwas über Neuronale Netzwerke gelernt.", which is in part also reflected in particularly problematic languages not being "projective". This gets even worse with "long-range" semantic cross-references, like anaphora. And, the latest Google Translate is still notoriously bad at the German language, at least [1]. Therefore, I think the article has a valid point, and I'd really like to hear some actual points where he is wrong before dismissing it (though, yes, I agree it is rather shallow, and best a good thought starter).
[1] Google Translate on the German Wikipedia entry for Weihnachten (X-mas): https://translate.googleusercontent.com/translate_c?depth=1&...
[1] Google Translate on the German Wikipedia entry for Weihnachten (X-mas): https://translate.googleusercontent.com/translate_c?depth=1&...
Why do you say German is notoriously bad? I don't speak German, but that linked page reads fine to me, and their neural translation system has state-of-the-art performance for both English->German and German->English at least[1]. I would have thought that the well-structured nature of German makes it reasonably easy to translate, assuming you have a NN architecture with sufficient memory range.
[1] https://arxiv.org/abs/1611.04558
[1] https://arxiv.org/abs/1611.04558
The translation isn't "only nonsense", and in many cases even grammatically correct, but mostly semantically wrong, or rather weird. In any case, a long shot from a human translation. But I wouldn't agree that it's "quite good". And, this is the easy case. EN->DE works even worse.
Doesn't German have a ton more compound words though? That could be an issue because their system is word based.
It does. In a way you face a parsing problem similar to Asian languages that don't use spaces to separate words.
Actually, the issue about continuity the article addresses is just the same for word embeddings, namely when there is an ambiguity. If the word has more than one meaning, the correct mapping is context dependent and the "one mapping fits all" approach that we are currently taking here is semantically incorrect. Yet another hard problem not easily resolved, and also attributable to "long-range dependencies".
There's a good point to be made about continuous functions, but the author hasn't made it, because he/she doesn't know the difference between continuous functions and time-series.
> What is a continuous function (or continuous data)? It is a sequence where each item is related to the one before and one after determined by a process.
So, to paraphrase the article: My reply should actually stop here with one sentence.
> What is a continuous function (or continuous data)? It is a sequence where each item is related to the one before and one after determined by a process.
So, to paraphrase the article: My reply should actually stop here with one sentence.
I agree that the author's interpretation of a continuous function is -umm- lacking, but I think it is thought provoking if you try to read it as what he probably meant, a continuous topological mapping from the character sequence to some function of the semantics of the text. Assuming that mapping does not exist would imply that we'd need more than a LSTM to build a proper language model some day. Yet, LSTMs are probably as close to a perfect model as we can currently get... So maybe this line of thought can help somebody imagine the "missing piece"?
The article doesn't mention "embedding" even once. How can an argument about discontinuities in language space leave out things like word2vec, etc, which are designed to make things continuous?
He sort of does, and then says that the information loss from that process will make it ineffective for NLP.
I don't think anyone deny that 'true' translation would require some kind of general intelligence that somehow understands what is being translated, but it seems to be the case that a 'dumb' translation works well enough for a great many use cases, regardless.
He's really just making the Chinese room argument. We have a computer shuffling symbols around according to some rule set, that doesn't know what they mean. I don't think it really matters, though, if it produces a reasonably accurate translation.
I don't think anyone deny that 'true' translation would require some kind of general intelligence that somehow understands what is being translated, but it seems to be the case that a 'dumb' translation works well enough for a great many use cases, regardless.
He's really just making the Chinese room argument. We have a computer shuffling symbols around according to some rule set, that doesn't know what they mean. I don't think it really matters, though, if it produces a reasonably accurate translation.
In a way, he's making an even stronger argument than Searle did in his Chinese Room. Even Searle would probably admit that it's possible in principle for the computer in the chinese room argument to fool people into thinking it's intelligent. Searle just objects to the idea that it could ever really be intelligent in the way a human is. To Searle, the human obviously isn't just running some algorithms on the input text to produce output.
I think the counter to Searle's argument isn't really that it doesn't matter as long as the result is close enough. The counter to that is that we don't understand how human intelligence works either. Searle is simply assuming that it's "magic" (or less condescendingly, some sort of metaphysical process) that can't be simulated by algorithmic machine. I think it's far more likely that intelligence is physical and we just don't understand the machinery than it is that it's mystical and cannot in principle ever be understood.
For this article, all that is seemingly unnecessary. He's just saying they won't work well enough to even fake it convincingly. Which is very nearly falsifiable just by running today's algorithms.
I think the counter to Searle's argument isn't really that it doesn't matter as long as the result is close enough. The counter to that is that we don't understand how human intelligence works either. Searle is simply assuming that it's "magic" (or less condescendingly, some sort of metaphysical process) that can't be simulated by algorithmic machine. I think it's far more likely that intelligence is physical and we just don't understand the machinery than it is that it's mystical and cannot in principle ever be understood.
For this article, all that is seemingly unnecessary. He's just saying they won't work well enough to even fake it convincingly. Which is very nearly falsifiable just by running today's algorithms.
Searle's Chinese Room is addressing something different from intelligence; it's concerned with the what happens /within/ the intelligent mind, whereas inputs and outputs are the only things that matter here. To be more specific, in the question of whether deep learning can be used to generate and/or comprehend natural language, we are not concerned with whether the algorithm is conscious of what it's doing as longs as the results are good.
In order to be conscious it has to be more than a reactive or feedforward system. It has to loop back on itself, like RNNs, and hold internal state.
The Chinese room argument is just a modern rehash of incredulity that physical systems could give rise to consciousness - which is absurd because that's what we know we are.
The argument goes - "when you look inside, it's just things pushing at each other. How could that produce perception and the conscious mind?"
So, a failure of imagination and incredulity based on how they understand the world and the mind makes them reject AI. They feel that the special place of the soul was traded for "information processing" which is dry and mechanical - a form of dualism creeping up in our day and age.
I would have felt the same if I didn't learn and use neural networks such as CNNs, RNNs and MLPs. Now I know how simple mechanical systems can recognize patterns and process information to generate complex behavior and I don't feel that "explanatory gap" any more.
Reinforcement learning is a good base for consciousness research - much more precise and with scientific results, not just p-zombies and bat based armchair experimentation. There's a limit where you can go with just pure thinking and then you need to start direct implementation.
The argument goes - "when you look inside, it's just things pushing at each other. How could that produce perception and the conscious mind?"
So, a failure of imagination and incredulity based on how they understand the world and the mind makes them reject AI. They feel that the special place of the soul was traded for "information processing" which is dry and mechanical - a form of dualism creeping up in our day and age.
I would have felt the same if I didn't learn and use neural networks such as CNNs, RNNs and MLPs. Now I know how simple mechanical systems can recognize patterns and process information to generate complex behavior and I don't feel that "explanatory gap" any more.
Reinforcement learning is a good base for consciousness research - much more precise and with scientific results, not just p-zombies and bat based armchair experimentation. There's a limit where you can go with just pure thinking and then you need to start direct implementation.
Accuracy is what matters if you're trying to build something that translates. You can judge if it's good or not based on (subjective or objective) measures of accuracy.
But if you're at 75% per cent and want to get to 100%, you need to understand what the problem is with the rest. And if it's 75% of "perfect translation of single written sentences from newspapers or technical litterature", how far along is that towards something "being part of an everyday conversation"?
I studied linguistics at a university where focus was very much spoken language, sociolinguistics, language in context, before moving into (or through) NLP, and the distance between what a statistical machine translation system is able to handle and the stuff I used to work with is very large.
A lot of NLP work now seems to focus on algorithms, but intuitively it seems to me that a much larger issue is the quality of the data, in the sense that humans don't learn language from piles of isolated text and somehow we're expecting machines to do it.. Rext is a lossy encoding of spoken language, even if you try your best to mimic it, but more seriously it does not include the physical context that children encounter language in. The learning situations aren't the same, I don't know why we're expecting the results to be.
But if you're at 75% per cent and want to get to 100%, you need to understand what the problem is with the rest. And if it's 75% of "perfect translation of single written sentences from newspapers or technical litterature", how far along is that towards something "being part of an everyday conversation"?
I studied linguistics at a university where focus was very much spoken language, sociolinguistics, language in context, before moving into (or through) NLP, and the distance between what a statistical machine translation system is able to handle and the stuff I used to work with is very large.
A lot of NLP work now seems to focus on algorithms, but intuitively it seems to me that a much larger issue is the quality of the data, in the sense that humans don't learn language from piles of isolated text and somehow we're expecting machines to do it.. Rext is a lossy encoding of spoken language, even if you try your best to mimic it, but more seriously it does not include the physical context that children encounter language in. The learning situations aren't the same, I don't know why we're expecting the results to be.
What's the purpose of a translation? I think 'non-intelligent' machine translation should eventually be fairly effective at producing a grammatically correct gloss of basically any sentence as long as there isn't too much ambiguity.
What machine translation isn't going to do is capture emotion or style or understand what someone is saying without them really saying it and so on. I would be very surprised if a machine translated novel ever hits the best seller charts. But I bet translators are going to be working from machine translated glosses if they aren't already.
What machine translation isn't going to do is capture emotion or style or understand what someone is saying without them really saying it and so on. I would be very surprised if a machine translated novel ever hits the best seller charts. But I bet translators are going to be working from machine translated glosses if they aren't already.
And bases the argument on a single feed forward network, nothing about RNNs which most good NN NLP results are based on.
Google's SyntaxNet (as of Andor et al 2016) achieves state-of-the-art accuracy for dependency parsing without RNNs.
"Text, a sequence of words, is not a byproduct of a statistical process, it is a byproduct of a cognitive process."
That cognitive inference process is what we've formalised as probability theory.
Whenever you do /anything/ your brain may be selecting from a probability distribution over things that can be done immediately.
As for text as continuous data just chuck it in glove, word2vec, lexvec or fasttext. Given enough training you could model the velocity of concepts as they're being introduced to the dataset / model.
Also, on the whole, shallowish learning can be applied to Natural Languages pretty easily. Keras includes a memory network (LSTM with an autoencoder) that averages around 98% accuracy on the bAbI 10k Q/A task.
That cognitive inference process is what we've formalised as probability theory.
Whenever you do /anything/ your brain may be selecting from a probability distribution over things that can be done immediately.
As for text as continuous data just chuck it in glove, word2vec, lexvec or fasttext. Given enough training you could model the velocity of concepts as they're being introduced to the dataset / model.
Also, on the whole, shallowish learning can be applied to Natural Languages pretty easily. Keras includes a memory network (LSTM with an autoencoder) that averages around 98% accuracy on the bAbI 10k Q/A task.
Whenever you do /anything/ your brain may be selecting from a probability distribution over things that can be done immediately.
There is no concrete evidence that the brain does math. If the brain did select things from a probability distribution, then why isn't everyone a math genius.
No one really knows how it works.
There is no concrete evidence that the brain does math. If the brain did select things from a probability distribution, then why isn't everyone a math genius.
No one really knows how it works.
There's actually quite a bit of evidence suggesting that brains, both behaviorally and mechanistically, are Bayesian [0].
As for your second point, assuming that humans are Bayesian, there are many reasons why people would have variability in their mathematical ability, including different priors and differences in the ability to estimate posteriors.
[0] https://scholar.google.com/scholar?q=brain+bayesian&hl=en&bt...
As for your second point, assuming that humans are Bayesian, there are many reasons why people would have variability in their mathematical ability, including different priors and differences in the ability to estimate posteriors.
[0] https://scholar.google.com/scholar?q=brain+bayesian&hl=en&bt...
My dog does advanced calculus and trigonometry whenevet I throw him a ball. He calculates the trajectory, the timing of his muscles, and thousands of other variables, and catches it.
The brain absolutely does tons of math. It's just well below the level of consciousness. Most people can learn to do analog math problems well within a few percent accuracy. It's symbolic math that is foreign to us.
The brain absolutely does tons of math. It's just well below the level of consciousness. Most people can learn to do analog math problems well within a few percent accuracy. It's symbolic math that is foreign to us.
Because we do these things subconsciously and intuitively, while math is done symbolically at a conscious level.
There are people who never manage to learn to do basic elementary-school math with fractions and percentages and yet manage to bet on sports and balance their checkbook because they're unable to translate their intuitive mathematical instincts into abstract formal math.
There are people who never manage to learn to do basic elementary-school math with fractions and percentages and yet manage to bet on sports and balance their checkbook because they're unable to translate their intuitive mathematical instincts into abstract formal math.
> If the brain did select things from a probability distribution, then why isn't everyone a math genius.
If the apple falls predictably from the tree, where is the math genius who plans its trajectory? Does the apple itself know physics?
If the apple falls predictably from the tree, where is the math genius who plans its trajectory? Does the apple itself know physics?
What's the alternative formalism?
> Another way to look at this problem is by data size. Let's assume we are training a NN using a data set of 10,000 pages text. The total number of pages of all possible knowledge in the world (ever) is an incalculable number, but let's assume it is a sextillion: 1,000,000,000,000,000,000,000. Then the question is, how much of the sextillion pages of data can be handled by training a NN using 10,000 pages? The answer would obviously be too small compared to the whole.
> On the other hand, grammar rules and ontological semantics mastered by the human brain can handle the entire sextillion (since those pages were written by human). If you know how to read and write, the entire sextillion will be understandable to you. This is the horrifying truth between the capabilities of the human brain versus the current state of neural networks.
This bit especially doesn't make any sense. I'm a human who has been reading all my life, does that mean I understand every grammar rule or ontological semantic ever created? Of course not! I barely understand all of them in my own language! My 'neural network' (brain) would need a bit more 'training' (studying) before that could happen. Even more, if all I had read in my life were 10,000 pages (which still may be true).
> On the other hand, grammar rules and ontological semantics mastered by the human brain can handle the entire sextillion (since those pages were written by human). If you know how to read and write, the entire sextillion will be understandable to you. This is the horrifying truth between the capabilities of the human brain versus the current state of neural networks.
This bit especially doesn't make any sense. I'm a human who has been reading all my life, does that mean I understand every grammar rule or ontological semantic ever created? Of course not! I barely understand all of them in my own language! My 'neural network' (brain) would need a bit more 'training' (studying) before that could happen. Even more, if all I had read in my life were 10,000 pages (which still may be true).
I do not think that deep learning can be applied to learning languages. It is based on an algorithm. Language and speech are too complex to be learned in such a way. I am studying a few foreign languages and cannot imagine how that can be possible. For example, machine translation cannot beat human one. I have tried to translate a few things with the help of online machine translators, but still had to contact https://www.translateshark.com/spanish.html to make a proper translation.
Just a datapoint regarding the supposed hyping of the recent Google Translate NN switch.
Last year I tried translating some everyday text from Turkish to English. Complete garbage, you could barely understand even what they were talking about.
I tried it now, albeit with different texts, and there's a world of a difference. Now you can actually understand what the text is saying, even if compared to other languages, I would still classify the Turkish->English translation as awful. Another big difference is that the resulting English text has relatively good grammer, as opposed to the previous version which was a broken English word soup.
Last year I tried translating some everyday text from Turkish to English. Complete garbage, you could barely understand even what they were talking about.
I tried it now, albeit with different texts, and there's a world of a difference. Now you can actually understand what the text is saying, even if compared to other languages, I would still classify the Turkish->English translation as awful. Another big difference is that the resulting English text has relatively good grammer, as opposed to the previous version which was a broken English word soup.
> Neural networks (NNs), recently referred to as deep learning, only work "effectively" with data that is produced from a process of a continuous function.
I think the author is overlooking the fact that images are not continuous functions but Deep-Learning image-recognition systems have been very successful representing discontinuities in images as hierarchical visual abstractions. In the same way, Deep Learning with recurrent neural networks should be able to learn discontinuities in symbol streams as hierarchical language abstractions, given enough data.
I think the author is overlooking the fact that images are not continuous functions but Deep-Learning image-recognition systems have been very successful representing discontinuities in images as hierarchical visual abstractions. In the same way, Deep Learning with recurrent neural networks should be able to learn discontinuities in symbol streams as hierarchical language abstractions, given enough data.
[deleted]
I don't have a deep understanding but isnt this where the 'word2vec' Algorithm comes in play. Isnt the idea to make one pass dedicated to finding relationships in words first?
Attention (see Google papers) seems like a pretty promising way of dealing with countable states. In quantum mechanics, we change the question we are asking. Instead of "which countable eigenstate are we in?", we ask "how much of our current state is in the Nth eigenstate?". We just expand our current state over the basis of interest. The coefficient of the state of interest is a continuous variable. This is what attention does for neural networks.
It's true that neural networks can not be easily applied to natural languages, but there are less obvious ways of applying them (namely Embeddings, LSTMs, Attention) and provided you have enough data and computational resources they give so much better results than any other method that it doesn't even help anymore to combine them.
[1] https://arxiv.org/abs/1609.08144
[2] https://arxiv.org/abs/1611.04558
EDIT: I also don't want to give the impression that Deep Learning can solve every NLP problem. We are still far away from passing the Turing test. It's true as well in my opinion that Google's Machine Translation is oversold. It's best at what it's trained and evaluated for: translating individual sentences from news sources.
EDIT 2: There are some tasks where traditional methods can work better, e.g. text classification on long documents. That's mainly because deep learning methods are too expensive computationally.