In IVF you can start checking conditions only in the final bucket. There are no guarantees if the bucket has any acceptable value, and there are no procedures to find the bucket which has acceptable values before scanning it.
IVF, unfortunately, is barely compatible with filtered search.
It have to rely on post-filtering and retrieve more and more candidates if the result set is not big enough. If the query is in some form correlated with the filter, this approach quickly degrades to brute-force.
Surprised that the article doesn't mention filtering use-case at all.
> Qdrant stores both the vectors and the metadata in a sqlite database.
It looks like you have benchmarked local mode of qdrant.
It doesn't even use vector indexes and is not designed for any kind of production usage.
For anyone reading this article, I urge you to do your own benchmarks and not rely on claims that do not have open source code attached to them to replicate the results
I would say that text text and vector search are orthogonal. Some scenarios are better with one, others with combination. But fitting vector search into the interface designed for text is limiting vector search potential
Benchmarks like https://big-ann-benchmarks.com/index.html#organizers are good for comparing algorithms, but not engines. They are focused on a single use scenario and do not cover variety of possible applications. Like, for example, how filtering affect the performance.
> Please expand on how you are defining full text search distinctly from sparse search to continue the discussion
In addition to the indexing algorithm, there is the tokenizer, which depends on the language, lemmatizer, synonyms, stop-words, and so on and so forth.
In addition, the ranking function itself may be quite different and based on different rules. See how Meilisearch does it.
Reducing full-text search to just a reverse index is a misconception
> Your criticism of the Weaviate ANN benchmarks isn't relevant to our discussion on Hybrid Search.
It is very much relevant, as I mentioned, in parallel processes
You just benchmarked yourself, that is not interesting and not helping.
> I also agree that it would be interesting to run ANN recall tests on several hardware configurations.
That is not the point. In our benchmark we run all engines on exactly the same machine to make it fair.
Sometimes same configuration in different regions already gives very different performance on some cloud providers.
> There are a few arguments why adding sparse search doesn't require too much extra specialization
Full-text search != sparse search, that's a naive oversimplification. Btw, sparse search is on Qdrant roadmap, so we should be able to compare it's performance on benchmarks.
> Cross Encoder inference generally doesn't happen in the database itself thus it makes sense to use modules to process the additional ranking logic
That statement makes your argument that `A combined system have better end-to-end latency.` invalid
> Such benchmarks exist as in trengrj's initial response and we are working on them as well.
link or it didn't happen.
In your current benchmarks you advertise everywhere, you're just throwing in disproportionately powerful and expensive hardware. Even a full-scan can give good results under those conditions
Proposed architecture doesn't limit you to use self-hosted transformers only, you can use OpenAI just as easily. And you don't need a to install a "module" for that
The latency of the combination of parallel systems is equal to the slowest component. And obviously, specialized tools will be faster than a component of a multi-tool system cause while dedicated engines can invest in optimizing specific functionality, multi-tool engines are stuck in the integration hell.
A solution assembled for a specific task from highly specialized components will always be more optimal than 'one-size-fits-all' pipelines. Meilisearch solves search-as-you-type better than anyone else, so why compromise? Not to mention that the scalability pattern of BM25 and vector search is entirely different.
This, by the way, is pretty obvious from the fact that you don't publish comparative benchmarks.
Starting from 0.5.0 finetuner is no longer an open-source.
> From 0.5.0, Finetuner computing is hosted on Jina Cloud. The last local version is 0.4.1, one can install it via pip or check out git tags/releases here.
But there are some cool ideas implemented there as well, I encourage you to try both!
It is definitely possible to do, if you have a proper training set. You would need to somehow give model a signal, that you are interesting in e.g. arguing style specifically, and not the topic of the text.
The idea is to fine-tune model, which can be used to produce embeddings required for Qdrant. Our design approach is to make things as modular as possible, so you can use framework and engine independently.
But we are working on integrating components as well