Ask HN: How to build an image search service?
What are the modern and practical image hashing algorithms which can be used to build an image search engine from scratch?
8 comments
Don't do image hashing please. Do this:
1. Get CLIP embeddings for text & images 2. Put them in a vector database (Pinecone.io or something similar)
It's unreasonably effective. Checkout this search engine: https://same.energy/
1. Get CLIP embeddings for text & images 2. Put them in a vector database (Pinecone.io or something similar)
It's unreasonably effective. Checkout this search engine: https://same.energy/
Thanks for mentioning Pinecone. We have several resources that might be helpful:
1. Ebook on image search using AI: https://www.pinecone.io/learn/image-search/
2. Article on using OpenAI CLIP: https://www.pinecone.io/learn/clip/
3. Example code for an image search app: https://docs.pinecone.io/docs/image-similarity-search
1. Ebook on image search using AI: https://www.pinecone.io/learn/image-search/
2. Article on using OpenAI CLIP: https://www.pinecone.io/learn/clip/
3. Example code for an image search app: https://docs.pinecone.io/docs/image-similarity-search
I'd take a look at the open source project https://github.com/marqo-ai/marqo instead. It does 1 and 2 out of the box. You can use CLIP or any model you want really.
Here are the types of images your users might be searching for:
Examples of a class of objects - say "Lion" - your best friend is CLIP, trained on captions and naturally weighted towards popular images.
Conceptual ideas, abstract levels - 'Beautiful painting' - CLIP is ok here, but it's not really optimised for that and most 'beautiful paintings' are not labelled as such, but rather something like 'Glowing fields of joy'. Fear not, Conceptual-CLIP has your back! It can nab these.
Diagrams/slides/technical/things with text - Your SOL beyond word match, tbh. If your image doesn't have words, maybe OCR it?
Pictures of specific things like people or logos - to "find similar" you'll want to embed old style image features like corners, SIFT features etc. Note that an iceberg with a clear sky is identical to a desert with a clear sky to this approach.
Examples of a class of objects - say "Lion" - your best friend is CLIP, trained on captions and naturally weighted towards popular images.
Conceptual ideas, abstract levels - 'Beautiful painting' - CLIP is ok here, but it's not really optimised for that and most 'beautiful paintings' are not labelled as such, but rather something like 'Glowing fields of joy'. Fear not, Conceptual-CLIP has your back! It can nab these.
Diagrams/slides/technical/things with text - Your SOL beyond word match, tbh. If your image doesn't have words, maybe OCR it?
Pictures of specific things like people or logos - to "find similar" you'll want to embed old style image features like corners, SIFT features etc. Note that an iceberg with a clear sky is identical to a desert with a clear sky to this approach.
Let me share an article I wrote: https://medium.com/@lukawskikacper/how-to-implement-a-visual... It uses Qdrant and ResNet embeddings to do that. Sources are included if you want to go straight to the implementation.
Qdrant is a Rust-based vector database: https://qdrant.tech/ According to benchmarks that's the fastest vector DB available.
Qdrant is a Rust-based vector database: https://qdrant.tech/ According to benchmarks that's the fastest vector DB available.
Take a look at SIFT and SURF for reverse image search or image comparison.
https://en.m.wikipedia.org/wiki/Scale-invariant_feature_tran...
If you want to search with a natural language description then you will need to tag your images or use object detection to generate tags.
https://en.m.wikipedia.org/wiki/Scale-invariant_feature_tran...
If you want to search with a natural language description then you will need to tag your images or use object detection to generate tags.
Here's a write-up from a relatively small/personal perspective
https://blog.qwertyforce.dev/posts/similar_image_search
https://blog.qwertyforce.dev/posts/similar_image_search