It is open-sourced, and you can find its source code on GitHub. The leaderboard is just something easy for readers to have a quick look at. You're welcome to use this tool to generate your results.
I won't argue with you about whether "vector database" should be a product category. You are not the first or even the last person who has doubts about it. Maybe many years later, we'll have the answer.
Good question. 1. VectorDBBench is an open-source benchmark that is pretty vendor-neutral. If you've used it or checked its leaderboard, you'll find that almost every database is ranked first or in the top place on specific metrics.
2. The key to using a benchmark is not to find which vector database is the "best" but which one is most suitable for your use cases and applications. Actually, there is no "universally best" vector database for any situation. So I don't think anyone should be able to cheat on this.
There are multiple vectordb available in the market, open source ones include Milvus, Qdrant, Weaviate etc. Cloud services include Zilliz Cloud (managed milvus), Qdrant Cloud, Weaviate Cloud etc. Try using a benchmark tool to evaluate them. Here is an open-source option for your reference: VectorDBBench (https://github.com/zilliztech/VectorDBBench)