Good question! We are focused on vision at the moment, but we are indeed looking at text in the future. Happy to connect and have a chat around that if you are open as we would be curious to hear more about new text use cases
Hello! I wrote the article so happy to answer this. It is partially feature engineering but partially not. It’s essentially using feature engineering to curate/correct a dataset, but a neural network as the actual end model without explicit input of these features(we call them quality metrics). I abbreviated a good amount of the process in the article so that it wouldn’t run forever, but essentially we allowed ChatGPT to select and write its own features and then used the strategies it came up with to apply these features to improve the dataset.
In terms of if it’s realistic in practice, the answer is yes. Some teams have a dearth of data, but many AI companies we work with have more data than they can use, and it’s more a question of how to sample, curate, and correct the data and labels they have to improve their models rather than collect new data. Great questions!
It is important to note that these micro-models are only supposed to be used in the annotation process. During annotation there is a separate process for QA where there will be some form of human supervision. Micro-models are NOT supposed to be used for production environments.
100% agree on the healthcare front, which actually perfectly underlies the point here. These models are overfit to one specific modality but often used for generic purposes. One reason why it is important to define micro-models is to point them out when they are deployed in a live production environment, which I agree is very dangerous. Many healthcare models are truly not ready for live diagnostic settings. On the other hand, these same models often do perform well on assisting the actual annotation of new data when applied to the right domain and with appropriate human supervision. This is a perfect encapsulation of the distinction we are trying to make.
That's a good idea. It will likely be trickier to apply a method like this for text. Decomposition of the problem is less obvious than with vision tasks.
Fair enough, but target data in this sense IS a full distribution of Batmen. This approach is towards the goal of creating a broad dataset and fitting a full Batman model. We are training on a narrower subset of our actual target data and fitting to that narrow subset, whether you want to call that overfitting or not I suppose depends on your perspective.
I agree intuitive definitions are often murky, but given we are already throwing in murky notions of intention that are implicit in the word "target", I think an at least colloquial usage of overfitting is appropriate.
Sometimes we try micro-models on broader domains than what we expect they will work for, and they work fine. Sometimes not. The point is that the target here is not well defined because we are just using them as annotation tools with some human supervision and not in a "typical" production environment.
Overfitting is a concept in data science, which occurs when a statistical model fits exactly against its training data.
We can argue over the precise definition of overfitting, but when you fitting a high-capacity model exactly to the training data, that is a procedural question and I would argue falls under the overfitting umbrella.
I see what you are saying, but in that context then you lose what most people's intuitive definition of overfitting is. If I train a model on one image as my train set and then change one random pixel and run that model on this eval set then your argument would be that this is not overfitting because you are performing well on the eval set you created the model for.
My argument is that compared to models, as most people use them, micro-models are low bias and high variance, and thus overfit. That's why I set a distinction between a batman model and a batman micro-model.
Thanks! You have the application correct, but there are many ways by which we use this. An example is if you have trying to build models that require sequentially annotated images(like action recognition). Another is creating many micro-models that each only detect one type of object even though your general model will have to detect multiple objects.
In general, the theory of what you are saying is correct that this method annotates data that is correlated with the original set, but practically it is still quite useful. Having more ground truth to work with gives a lot more practical flexibility with things like sampling, testing your model, randomization, and training more robust versions of your model.
Hi everyone, I wrote the article. I do consider this overfitting because we are training on these frames way more time than would be normally advised for the size of the training set such that the error is essentially zero for these frames. The model performs well in "out-of-sample" here but only out of sample that is semantically close to the original training set. Besides, overfitting is defined procedurally, not by how well it performs. You could have an overfit model that just happens to perform well on some stuff it was not trained on, that doesn't change the fact that the model was overfit.
Great question. Superb AI also seems like a great tool. I’m not sure if they have video annotation though.
We are different from them in a bunch of ways, but the biggest one is that we are optimized for handling videos, sequential images, and radiology. We really like handling groups of semantically similar images.
Thank you! We think Hasty is a pretty neat tool, I am just not sure they have annotation for video datasets? Our platform is really optimized for video and sequential image. A GI rotoscope is right up our alley. If you have any more of those projects come up, give us a shout!
Really interesting idea with regards to the spectogram, especially with the image-like representation. Will take a look at some of those use cases.
You can do either! We offer a bunch of automation features directly through the Web App but people have also used the SDK to write their own algorithms. We have seen a lot of different annotation processes now so we can often direct people on the best flow to automate their labeling.
As far as I can tell Roboflow is more focused on being an end to end platform for AI. The customers we work with generally want to retain more control over their model building process, we just help them with automating as much of the annotation process as we can, often with the help of their own models.
Hi Eric from Cord here. Scale is a great company and they have done really well in AV especially. The issue with them is that they require you to send your data overseas to be annotated by a human workforce. They also probably have a bunch of in-house automated tools, but they don’t pass the savings onto their clients because they are paid on a per label basis. We want to pass the benefit of automation to anyone that needs labeled data.
I hear you, but I don't even think the labor intensiveness is a lost cause here. Labor intensiveness in mining insights from a dataset is worth way more than labor intensiveness in manual labelling.
I think we are lazily giving up our intellectual power to models hoping that they will just discover patterns by magic, where it is actually very worth to go through the data science process starting with labelling because we actually learn as humans. The thesis is that this will also make our DL models better in the long run. We would never have come up with cool algorithms if we just always outsourced this work to models.
Hi Sriku, with regards to your first point, not necessarily. I mentioned in another comment, but the model you are using the labels to build and the labelling process are related but not the same, they have different fundamental constraints and rely on different techniques. You can't have a human in the loop to help you out in a live model as an example.
You are right there are a bunch of difficult problems this technique isn't perfect for, but it actually can still help improve the efficiency of labelling a lot and when I do it I get the added bonus of understanding the dataset a lot better.