An Introduction to Recurrent Neural Networks(victorzhou.com)
victorzhou.com
An Introduction to Recurrent Neural Networks
https://victorzhou.com/blog/intro-to-rnns/
25 comments
It depends on the problem domain. Transformers are useful for NLP (language modeling, machine translation), but RNNs (and CNNs) are still being used in speech-to-text. When the input/output relationships are monotonic, transformers are probably too general.
[deleted]
Hey, author here. Happy to answer any questions or take any suggestions.
Runnable code from the article: https://repl.it/@vzhou842/A-RNN-from-scratch
Runnable code from the article: https://repl.it/@vzhou842/A-RNN-from-scratch
Very nicely written post. I particularly like how you attached a link to your codebase on repl.it so anyone who is interested can tinker with the code.
One thing I have been wondering for some time is whether the vanilla RNN can learn negations (i.e. 'not good' == 'bad') and valence shifts (e.g. modifier words like 'very' --- they do not carry sentiment connotations themselves, but may amplify/dampen the sentiment of the words they modify; negations like 'not' can be considered as a special-case valence shifter where it inverts the sentiment of the following word).
My suspicion is that vanilla RNNs are not capable of modelling negations and valence shifters since they make inference on the sentiment of a sentence by 'adding up' the sentiment connotations of its constituent words --- negations and valence shifts, however, works more like multiplications than additions.
I see you already have such examples in your dataset so I thought I'd do some experiments. I simplified your original dataset to the following:
It would be interesting to see how other RNN architectures (e.g. LSTM, Transformers) fare with negations and valence shifters.
P.S.: When calculating softmax, it is better to use the built-in functions or at least do the log-sum-exp trick to prevent under-flowing.
One thing I have been wondering for some time is whether the vanilla RNN can learn negations (i.e. 'not good' == 'bad') and valence shifts (e.g. modifier words like 'very' --- they do not carry sentiment connotations themselves, but may amplify/dampen the sentiment of the words they modify; negations like 'not' can be considered as a special-case valence shifter where it inverts the sentiment of the following word).
My suspicion is that vanilla RNNs are not capable of modelling negations and valence shifters since they make inference on the sentiment of a sentence by 'adding up' the sentiment connotations of its constituent words --- negations and valence shifts, however, works more like multiplications than additions.
I see you already have such examples in your dataset so I thought I'd do some experiments. I simplified your original dataset to the following:
train_data = {
'good': True,
'bad': False,
'not good': False,
'not bad': True,
'very good': True,
'very bad': False,
'not very good': False,
'not very bad': True
}
test_data = {
'very not bad': True,
'very not good': False
}
It would be interesting to see how other RNN architectures (e.g. LSTM, Transformers) fare with negations and valence shifters.
P.S.: When calculating softmax, it is better to use the built-in functions or at least do the log-sum-exp trick to prevent under-flowing.
I tried a LSTM model of stock twists and it seemed reasonably good at handling negations (single and double negatives at least)
Thanks for the comments! Interesting experiment - I wouldn't be surprised if better RNN architectures were more effective for this example.
Appreciate the softmax tip, I'll update soon.
Appreciate the softmax tip, I'll update soon.
Suggestion: do the same for transformer.
Machine translation using RNNs actually uses two of them, one serving as an encoder and the other as a decoder (the Seq2Seq architecture)
In your post you initialize your weights a certain way but you said there are better ways. Do you have any resources for better ways?
Nice! I like that the author wrote the code by hand rather than leaning on some framework. It makes it a lot easier to connect the math to the code. :)
As a meta-comment on these "Introduction to _____ neural network" articles (not just this one), I wish people would spend more time talking about when their neural net isn't the right tool for the job. SVMs, kNN, even basic regression techniques aren't any less effective than they were 20 years ago. They're easier to interpret and debug, require many fewer parameters, and potentially (you may need to apply some tricks here or there) faster at both training and evaluation time.
As a meta-comment on these "Introduction to _____ neural network" articles (not just this one), I wish people would spend more time talking about when their neural net isn't the right tool for the job. SVMs, kNN, even basic regression techniques aren't any less effective than they were 20 years ago. They're easier to interpret and debug, require many fewer parameters, and potentially (you may need to apply some tricks here or there) faster at both training and evaluation time.
This kind of article is absolutely the thing everyone new to deep learning/neural networks should read. I wish there was one for each type of algorithm.
I do have similar articles for Neural Networks (MLP) and CNNs:
---
NN: https://victorzhou.com/blog/intro-to-neural-networks/
NN HN discussion: https://news.ycombinator.com/item?id=19320217
---
CNN: https://victorzhou.com/blog/intro-to-cnns-part-1/ https://victorzhou.com/blog/intro-to-cnns-part-2/
CNN HN discussions: https://news.ycombinator.com/item?id=19981736 https://news.ycombinator.com/item?id=20064900
---
NN: https://victorzhou.com/blog/intro-to-neural-networks/
NN HN discussion: https://news.ycombinator.com/item?id=19320217
---
CNN: https://victorzhou.com/blog/intro-to-cnns-part-1/ https://victorzhou.com/blog/intro-to-cnns-part-2/
CNN HN discussions: https://news.ycombinator.com/item?id=19981736 https://news.ycombinator.com/item?id=20064900
Awesome!
Would be great if you showed the final output (eg. semantic analysis) result.
[deleted]
why do people insist on mentioning the bias terms in expository essays? it's a detail that clutters the equations. why not keep the transformations linear and then at the end make a note that you also need to shift using a bias term.
I doubt Google Translate uses RNN. They use Statistical Machine Translation. Oops, I see they switched to NN in 2016.
https://en.wikipedia.org/wiki/Google_Translate
I feel I should point out that those two things are not mutually exclusive. RNNs are, after all, a mechanism for learning conditional probabilities.
I think the confusion comes from Google itself, who used the term "Statistical Machine Translation" (SMT) to refer to "Rule-based SMT". Both methods are statistical.
I think the confusion comes from Google itself, who used the term "Statistical Machine Translation" (SMT) to refer to "Rule-based SMT". Both methods are statistical.
SMT as a term of art in the translation field means rule based SMT... it's not a google particularity, I see the same usage in both industry and academia
Every top MT uses pure-NMT, not SMT
I have to say that while I understand the problems with recurrent nets (which I've used many times), I haven't yet grokked the alternatives. Here are some decently looking search results for you as starting points. Warning, these can be longer and heavier reads probably not for beginners.
https://towardsdatascience.com/the-fall-of-rnn-lstm-2d1594c7... (there's some sensationalism here to be fair)
https://mchromiak.github.io/articles/2017/Sep/12/Transformer...
https://www.analyticsvidhya.com/blog/2019/06/understanding-t...
https://www.tensorflow.org/beta/tutorials/text/transformer
That being said, I think that understanding RNNs is very beneficial conceptually and nowadays there are relatively easy to use implementations that should be pretty good for many use cases.