I added an edited note to the bottom of the blog post.
The original post and the experiments were created before pgvector 0.5.1 was out, and we had not realized there was significant work to optimize index creation time in the latest pgvector release.
We reran pgvector benchmarks with pgvector 0.5.1.
Now pgvector index creation is on par or 10% faster than lantern on a single core. Lantern still allows 30x faster index creation by leveraging additional cores.
Hi, sorry that you didn't have a good experience with Lantern before. We first posted in HN about 3 months ago - Things should be better now, please let us know if you have any issues.
Yes it is WAL protected: the advantage of external indexing is that the HNSW graph is being constructed externally on multiple cores instead on a single core inside the Postgres process. But eventually the graph is being parsed and processed inside Postgres with all the necessary WAL logs for blocks.
Unless Supabase decides to integrate Lantern (currently they integrate pgvector) you unfortunately cannot use Lantern with Supabase.
That said, we will offer Lantern Cloud, our own hosted postgres offering (very soon. Happy to keep you in the loop. If you’re interested, please feel free to join the waitlist here: https://forms.gle/PouJxAWiSa63udJW8
We don’t do cleanup of the index during VACUUM yet. That said, it’s coming very soon. We’re built on top of Usearch, which supports deletes. We plan to work with the Usearch team to port the post performant deletes to Lantern, and thereby support VACUUM.
You're right, 100k rows isn’t a reputable benchmark. We wanted to launch very quickly, and have benchmarking for larger datasets coming soon. Benchmarking is baked into our CI/CD, we take it very seriously!
> There's a fourth metric that I'm really interested in: assuming it's possible, how long does it take to update the index with just one or two updated or inserted vectors?
At the moment, we underperform Neon wrt this metric, but a better implementation is coming soon that will address this.
> Is the expectation with this (and the other) tools that I'll do a full index rebuild every X minutes/hours, or do some of them support ongoing partial updates as data is inserted and updated?
The HNSW algorithm updates the index after every insert. So all existing HNSW options (Lantern, pgvector, Neon, …) already support this.
With pgvector IVFFlat, you expect the performance to degrade over time, and you will need to re-index. This is because IVFflat’s index quality heavily depends on the centroids chosen at index creation time. HNSW does not have this limitation.
In both cases, you might want to do a full-index build to tune your hyperparameters.
We’re working on this in a few ways. One is automatic hyperparameter tuning. Another is supporting external index creation that would offload this to another server. Does this answer your question?
I don’t believe pgvector reports performance changes between releases.
At the moment, we run the benchmarking on Github CI, but we plan to move this to an external machine, since the results are unstable on Github machines. We’re planning to extend benchmarking across other repos and versions.
