Why is a very good question indeed! My guess is we have not figured out how to include long term enough tasks into the RL training.
As to whether they can today, it's pretty easy to figure out that they can't by observing the difference between asking for very high level things like "please make me a website that will be very successful" vs sending it a detailed plans on the goals and many of the specifics and then holding it accountable towards the planning.
Doing appropriate project structure and trying to optimize code quality are definitely good methods to try and increase the chance that the project will succeed long term but I would say they are only methods towards the long term goal and the LLM need to be thinking of the long term goal and how to plan towards that to achieve a much wider and subtile range of methods to achieve long term goals.
I think the simple answer is LLM cannot do long term planning.
Maintainability means thinking what will happen to this code in the next year and possibly hundreds of changes and based on that selecting the right abstractions that will work in the long term.
LLMs currently cannot do long term planning and specifically cannot pick abstractions that will work in the long term.
I think that's the important problem to solve. Teaching LLMs how to pick the right abstractions.
Looks like you missed the whole point of this dataset.
The idea that we proved is you can get a dataset with decent caption and images (that do match yes, you can see for yourself at https://rom1504.github.io/clip-retrieval/ ) that can be used to trained well performing models (eg openclip and stable diffusion) while using only automated filtering of a noisy source (common crawl)
We further proved that idea by using aesthetic prediction, nsfw and watermark tags to select the best pictures.
Is it possible to write caption manually? sure, but that doesn't scale much and won't make it possible to train general models.
Using clip for searching is better than direct text indexing for a variety of reasons but here for example because it matches better what stable diffusion sees
Still interesting to have a different view over the dataset!
If you want to scale this out, you could use elastic search
Thanks to approximate knn, it's possible to query and explore that 5B datasets with only 2TB of local storage, anyone can download the knn index and metadata to run that locally too.
Regarding duplicates, indeed it's an interesting topic!
Laion5b deduplicated samples by url+text, but not by image.
To deduplicate by image you need to have an efficient way to compute whether image a and b are the same.
An idea to do that is to compute an hash based on clip embeddings. A further idea would be to train a network actually good at dedup and not only similarity by training on positive and negative pairs, eg with triple loss.
Let me start by saying that laion is a non profit, open to anyone that want to contribute.
Agreed about the website css. Do you want to contribute?
What's the problem with the dataset name exactly? Seems to work pretty well.
Yes the dataset is an extract of common crawl, this is an accessible to all method to produce valuable dataset. This is unlike supervised dataset which are reserved to organization with millions of dollars to spend on annotation and do not scale.
Non annotated datasets are the base of self supervised learning, which is the future of machine learning. Image/text with no human label is a feature, not a bug. We provide safety tags for safety concerns and watermark tags to improve generations.
It also so happens that this dataset collection method has been proven by using laion400m to reproduce clip model. (And by a bunch of other models trained on it)
That's interesting indeed!
Note that the description search as well as the image search are both using a knn index on embeddings and not exact search. That helps for finding semantically close by items but indeed for exact reference match it might not be the best solution.
Re-indexing the dataset with something like elastic search would give the reference search results you expect.