rsmith49 · HackerTrans

Making the calls to the corresponding model from Flask was actually easier on AWS, since they were loaded into memory. Unfortunately, the scaling issues/excessive load times were big enough of an issue that we had to make the switch, as our number of hosted models continues to grow.

rsmith49

há 6 anosdiscuss

It probably would - however we built this out with a team of 1-2 engineers, so paying a bit more and dealing with maybe 20 seconds of extra latency a day is a price we are willing to pay for expedited/easier deployment.

Thanks for the suggestion though! I'm sure we will consider that in the future :)

rsmith49

há 6 anosdiscuss

A bit of both. Flask was obviously not designed with serving Tensorflow models locally in mind, which is how we had it set up in v0. Towards the beginning we had to debug some weird threading issues, but towards the end once it was more stable (as a result of some hacky fixes), the timeouts were the main issue.

rsmith49

há 6 anosdiscuss

Not inherently, but how we're using it makes it cheaper. Our stack has ~50 different ML models being served live, and GCP makes it easy to treat each model as a micro-service, and give auto-scaling to each one.

This is in contrast to the easiest way we found to deploy the same architecture on AWS using Elastic Beanstalk, which involved one really big instance (that was constantly growing as we added more models), and the costs that come with that.

rsmith49

há 6 anosdiscuss

While AWS does allow more hands-on control of the model stack, our "Custom Models" are more just different sets of weights using the exact same methodology. Basically, each customer that creates their own custom model is plugging into our existing framework, but with a different configuration.

Because of this, GCP'S AI Platform allows us a more micro-services type approach to interacting with the ML models themselves - as opposed to our previous deployment strategy on AWS, which put all of the models into one big bucket on every instance that was serving requests.

Hope that answered your question!

rsmith49

há 7 anosdiscuss

For fine tuning BERT onto a specific domain, what amount of text data would you recommend to train on?

rsmith49

há 7 anosdiscuss

That depends on what you mean by noise. In the case of text classification, a lot of the noise in training is disagreement between human labelings. Unfortunately, classification systems will only be as good as the labels they are trained on.

However, if you are referring to noise as in typos and misspellings - then yes, depending on the training and/or preprocessing steps, classification systems could potentially reduce the noise in the input data to still achieve good results.

rsmith49

há 7 anosdiscuss

That is a good point. It helps to think of Precision and Recall (so by extension F1-Score) from your test data as random variables sampled from a distribution modeling the probability of getting each value in your sample based on a "True" precision/recall value. I won't go too deep into the math, but this was part of the approach in the confidence calculations towards the end of the paper: being able to factor in the uncertainty of your classification metrics to confidence calculations.

To formally answer your question, the main things that matter in determining how stable your F1-Score from your test set is are: - Size of the test set - % of test set that has the label (in our case feedback tag) - the values found for precision and recall

rsmith49

há 8 anosdiscuss

I would imagine that AWS Comprehend utilizes a lot of customer feedback when training its internal models. However, they do advertise Comprehend as a general purpose NLP service, so they may have taken steps to make the models generalize to the non-feedback domain. In any case, on our set of sentiment data for customer feedback, Comprehend is outperformed by Google’s NLP API and our own internal model.

As for Azure, we actually did not factor their service into our comparison. This is definitely something to consider in future blog posts.

rsmith49

há 8 anosdiscuss

Transfer learning is the premise of learning from data in one domain, and then being able to transfer that knowledge to another domain. In this case, we leverage large existing text datasets to build an initial model, and then tailor that model to customer feedback and apply it to our specific problem (for this blog post, classifying feedback sentiment).

We chose to use this approach for customer feedback since it combines the benefits of a general model (large amounts of data) with the benefits of a domain-specific model (targeted to customer feedback). It looks like we were validated, as it outperformed Google NLP API and AWS Comprehend - both general models - on our set of customer feedback data.

rsmith49

há 8 anosdiscuss

There is definitely a feedback loop at play in the NLP community. Since most NLP research is based on English, most applications that build off that research are also tailored to English. Our approach to classifying feedback in other languages is to simply use Google Translate's API and then proceed to classify the English translation. However, you are right in that many aspects of the base feedback are lost in this approach, and our accuracy is lower than if we had a language specific model for each language.

That being said, there is a promising new research paper from fast.ai (https://arxiv.org/abs/1801.06146) that speaks to using Wikipedia data to create a language model for a specific language, which can then be trained specifically on the task you are trying to solve. If this is as effective as the authors state, then NLP could see huge improvements non-English languages where there is already a large set of Wikipedia data.

rsmith49

há 8 anosdiscuss

Unfortunately, this is a very common occurrence in NLP applications. Our first step to combat this is through performing a spellcheck step when preprocessing all of our data. Next, some of the algorithms we employ only look for the presence of words in feedback, not necessarily grammatical correctness. So, if we get something like "food good, love love love", we will still be able to recognize the feedback is referring positively to food quality, and our ensemble prediction will reflect this.

Despite this, we still run into some feedback that is complete gibberish, or does not refer to anything. Fortunately, since this is a multi-label classification problem, it is possible for us to classify the feedback as not having any tag associated with it. Therefore, including some of these samples in our training data helps fortify our engine against any live data that may come in without meaning, and allows us to classify that feedback as having no tag associated with it.

In our upcoming blog about our "human in the loop" machine learning system, we also address how we can manually filter samples of data to make our training more efficient.

rsmith49

há 8 anosdiscuss

Our criteria mainly depended on internal testing to see which pre-packaged algorithms performed better on our data. While performance does vary with the domain (or in our case, industry of feedback we are analyzing), we have found it to be more efficient to find the overall effectiveness of one platform and deploy it for all of our analysis.

As far as the domain affecting the algorithm, it can vary from some algorithms maintaining decent performance over most industries, to some algorithms working very well for some industries and terribly for others. Although it is just feedback, the topic of the feedback and even the ways people talk about the same topic (such as price of the product) will vary across each industry.

rsmith49

há 9 anosdiscuss

To expand a little on the previous answer, existing cloud API's typically fall into one of the following categories:

1) generic NLP classification API's where you need your own large supply of training data and have very little (if any) control over model parameters, or

2) Topic extraction/topic modeling (usually employing LDA), which will simply give a list of potential topics, and will not be able to conform to the set of topics (in our case called "tags") that we have selected.

Because of this, our solution is better at addressing the specific problem of categorizing customer feedback, and allows for more flexibility in how we can collect and manage data.

comments