LLM applications can benefit from Retrieval-Augmented Generation (RAG) in a similar way that humans benefit from search engines like Google. Therefore, I believe RAG cannot be replaced by prompts or fine-tuning.
MyScaleDB utilizes approximate nearest neighbors (ANN) algorithms such as ScaNN, HNSW, and IVF. As a result, it may not achieve a 100% recall rate. However, depending on the search parameters used, it can attain recall rates of up to 95% or even 99%.
Considering that embedding vectors represent a lossy compression of the original text or images, is achieving a 100% recall necessary? I am interested in understanding its practical implications.
Since we developed our internal version before the release of usearch, we integrated hnswlib with faiss's PQ/SQ algorithms. Search performance is typically not a concern in real-world applications, especially for those using LLMs due to their high latency. We highly recommend Google's scann algorithm (integrated into MyScaleDB) as it is much faster for indexing and offers similar search performance to hnsw.
I work on MyScale (https://myscale.com), a fully-managed vector database based on ClickHouse.
Some unique features of MyScale:
1. This solution is built on ClickHouse and offers comprehensive SQL support. Our users leverage vector search for a wide range of interesting OLAP use cases.
2. We utilize a property vector search algorithm called the multi-tier tree graph (MSTG). This algorithm is significantly faster than HNSW for both vector index building and filtered vector searches.
3. We utilize NVMe SSDs for the vector index cache, which greatly reduces the cost of hosting millions of vectors.
We conducted a benchmark to evaluate the precision, throughput (QPS), insert speed, build speed, and cost-effectiveness of Pinecone, Qdrant, MyScale, Weaviate, and Zilliz (Milvus). This information will be valuable for those seeking to choose a vector database for production purposes. The results can be found at https://myscale.github.io/benchmark/.
This idea is a simplified version of Retrieval-Augmented Generation (RAG), and RAG has been studied in various research papers, such as the one available at https://arxiv.org/abs/2005.11401
This article suggests that LLMs should use a database as a reference for factual information. Rather than asking LLMs to provide their own answers, it is recommended that they summarize based on the facts extracted from the database. This approach reduces the likelihood of hallucinations among LLMs.
That's true and we have tried different configurations for systems that use HNSW. But for the ease of presentation, we only choose the configuration with the highest throughput at precision 98%.
Yes. For MyScale (aws us-east-1), Pinecone (aws us-east-1), Qdrant (aws us-east-1), and Zilliz Cloud (aws us-east-2), we run the clients in the same region as the servers. For Weaviate, the server is in GCP US east, while the client is in aws us-east-1. Since its throughput is around 66 QPS, the impact of networking should be low.
Over the past few months, we have been working on an exciting project to bridge the gap between high performance vector search and OLAP database. Today, we are thrilled to announce the release of our new end-to-end benchmark of MyScale, which includes a comparison with some of the state-of-the-art vector databases for your reference. Here are some key takeaways that might pique your interest:
1. [Low Cost]
in this case, we measures the ratio of the monthly cost to the QPS (Queries Per Second) of the service per one hundred units. It quantifies the monthly cost required to achieve 100 QPS on 5 million vector data points. Our analysis highlights the superior cost-performance ratio of MyScale, which is over 3.6 times cheaper than other vector databases.
https://blog.myscale.com/2023/05/17/myscale-outperform-speci...
2. [High Throughput]
MyScale outperforms other vector databases in terms of QPS on the LAION 5M dataset with a 98.5% recall rate, achieving over 150 QPS. In comparison, Pinecone s1 has a QPS of approximately 10, which is significantly lower than MyScale. Weaviate and Zilliz Cloud both achieve around 65 QPS, while Qdrant achieves 81 QPS.
https://blog.myscale.com/2023/05/17/myscale-outperform-speci...
3. [Quick Responce]
Query latency is an important performance metric that is measured from the time the client sends the request until it receives the response. MyScale achieves 150 QPS while maintaining an average latency as low as 25.8 ms. Pinecone s1 has a relatively high latency of over 400 ms. Weaviate and Zilliz Cloud both have latencies of around 60 ms, while Qdrant has a slightly higher latency of around 100 ms.
https://blog.myscale.com/2023/05/17/myscale-outperform-speci...
4. [Fast Data Ingestion]
The time it takes from data upload to the vector index being built and ready to serve is referred to as data ingestion time. Index creation can take a long time, especially for graph-based algorithms such as HNSW. Among all the services tested, MyScale had the fastest ingestion time for 5 million data points, completing the task in about 30 minutes. Pinecone s1 takes approximately 53 minutes, while Weaviate takes 72 minutes. Zilliz Cloud requires a longer duration of approximately 113 minutes, while Qdrant has the longest ingestion time, taking 145 minutes to process 5 million data points.
https://blog.myscale.com/2023/05/17/myscale-outperform-speci...
Also here are some other nice features myscale can offer:
1. Simple data import and backup: We support common format like Parquet, tar, csv from or to S3 buckets or other object storage systems.
2. There are more options from FAISS and HNSW other than MSTG, the algorithmn we proposed and tested in this benchmark. You may choose one you familiar with.
3. Built on Clickhouse and being part of the community. Boost your vector search with Clickhouse advanced features.
MyScale is currently in beta, with a free developer tier and a commercial plan on the way. To the best of our knowledge,
MyScale offers the first free plan that supports 5 million 768-dimensional vector data points with high performance search.
You can explore MyScale at https://myscale.com/. It's a SaaS platform built on ClickHouse, offering more sophisticated vector indexing options like HNSW and IVF compared to the open-source version. It also provides a free tier for beta users.
