HackerTrans
TopNewTrendsCommentsPastAskShowJobs

rohanpdes

no profile record

comments

rohanpdes
·2 mesi fa·discuss
Not stupid at all. API cost (especially for writes) is indeed one of the main challenges of building for object storage. DBs mitigate this by aggressively caching reads and batching writes. There's a fundamental latency/cost tradeoff here - you accept higher latency to get enough batching to amortize PUT costs. This is a very reasonable tradeoff for search systems which typically aren't as latency sensitive.
rohanpdes
·2 mesi fa·discuss
Yep! Vector provides a lot of the same benefits, just as an OSS project. They were definitely a major inspiration. Vector's performance is similar to their published benchmarks. The biggest gap is (unsurprisingly) for larger (e.g. 100s of M - 1B+) datasets. We talk about it in the post, but the main improvement there is adding quantization to reduce the overhead of loading large posting lists. There's also a bunch of storage and caching layer work to be done. That's on our roadmap along with some cool features like full-text search and better support for multi-tenancy.