Deep learning with text: Learning when to skim and when to read(metamind.io)
metamind.io
Deep learning with text: Learning when to skim and when to read
https://metamind.io/research/learning-when-to-skim-and-when-to-read
21 comments
In the Baseline section, you will find a sentence with the following: "However, there is a cost to using the strategies. We have to run all of the sentences through the bag-of-words model first, to determine if we should use the bag-of-words or the LSTM." followed by some math that takes the added cost into account, which is also what we based our plots/results on.
A 'batch' is how much of the data you put in memory at once while training the NN. To train even a small language model, you'll go through 1000s of batches, so the time difference is way bigger than it sounds. I agree a more practical example would have been nice -- maybe it'll come out in the paper.
my impression was that this was about the time taken to make each prediction, not to train the model? and yep, looking forward to the paper!
It was based on test time prediction, so given you have received a sentence, how fast does it take to compute the prediction with either a bag-of-words or an LSTM.
When you say practical example, would that be in the scenario that you have an API server running? So to consider such costs as latency, data transfer, API overhead etc.?
Thanks for your feedback!
When you say practical example, would that be in the scenario that you have an API server running? So to consider such costs as latency, data transfer, API overhead etc.?
Thanks for your feedback!
It irks me that BoW is referred to as an algorithm. They do refer to it as a model later, why conflate it?
toread
So... We are now automating laziness? There goes my last stronghold...
Not quiet! Choosing between using a fast mostly accurate model and a slow very accurate model. Time saved vs accuracy gained.
Not quiet!
No usually "that person", but: quite
Only pointing it out because of the irony: Choosing between using a fast mostly accurate model and a slow very accurate model.
No usually "that person", but: quite
Only pointing it out because of the irony: Choosing between using a fast mostly accurate model and a slow very accurate model.
No usually
You did that on purpose ;)
You did that on purpose ;)
Isn't automation "laziness incarnate"?
Automation doesn't necessarily equate to laziness. You can automate the boring shit so that you can focus on more productive things.
Boring things can be productive things!
automate the automation so you don't have to.
Automate the process of automating things.
Skimmming is definitely not a bad thing.
Not so sure. I do skim sometimes, but it always leaves me with are dirty feeling of cheating. Also one needs to go back and check whether something important was missed and this undoes most of the benefit of skimming.
The proper place or skimming would be to skim before reading just to get an overview of what's to come, or to skim while evaluating if the thing is worth reading in the first place.
The only reason I've ever had not to skim was because I was asked not to. But for example at school my purpose is really to learn not to satisfy the desires of my teachers, and actually most of the time reading, taking notes, and so on doesnt help me at all. Its a much smarter and more effective strategy. Trim the fat as the great adam levine would say.
i wonder if this would have been improved by being clearer about the motivation. the authors frame it as though the ~60ms penalty for using the LSTM for prediction is a huge burden, and i can imagine situations where it is. however, it seems like if this is the case, we need some real life/"scaled out" examples of how this solution would work in practice. e.g. how long does the decision logic take to execute (maybe 5ms?); what proportion of the time will you have to run the LSTM after the BoW model anyway? note that those instances you are now worse off than just running the LSTM in the first place (total time = BoW time + decision time + LSTM time). once you have all these you can run the math and know (on average) how much time you'll actually save, and how much performance you sacrifice