Actually, you should take a look at [1]. It's made in Rust, inspired by LMDB, and supports a cool feature that allows close to anything to be implemented: allocating any page you want to store anything you want. The BTree storage is optional and you can implement whatever storage system you want. When storing a value to disk, you can allocate pages and decide exactly how you plan to store the bytes, allowing you not to store the length of them or to split your data into multiple pages, etc.
Yup! Same, I'm the maintainer of the main LMDB Rust wrapper [1] and I was maintaining heed and heed3 (because 1.0 was available from the mdb.master3 branch).
But now that it's LMDB 1.0, I need to find a better way to make it be the official one but I can't really rename heed3 into heed and heed into heed-0.9...
Yeah, and I also added support for parallel read your own writes where you can write tons of entries and spawn multi children read-only transactions from your writes where transaction and read from them in parallel.
We use this in Meilisearch [1] to post-process cache for our most common prefixes i.e., "w" will match "work", "word"... and computing this requires doing large unions of the documents matching those words.
Being able to do it in parallel is necessary, especially when you have billions of entries to operate on.
I recently talked to Howard [1] about lies he was saying about Sanakirja, an LMDB-inspired disk allocator. That's always the same arguments: C is better than Rust for X, Y or Z reasons. While I reported a segfault just two weeks earlier... [2].
I love LMDB, we use it in Meilisearch (second most stared search engine on GitHub) [3] for about 7 years now. The main issues were related to write speed but we do a compaction of the database and write performances are way better after that. We never had any major DB corruption... I mean... other than when using it on Azure. Azure never works, that's expected, I suppose.
So nice! That's an excellent extract and looks useful for benchmarking Meilisearch. I'll probably spend my Christmas holidays importing the tracks, albums, and artists into Meilisearch, while my CEO builds a beautiful front-end for it. I'll probably replace [the current music search demo](https://music.meilisearch.com) we have with this much higher-quality dataset!
That would also be a good fit for [the new delta-encoded posting lists I am working on](https://github.com/meilisearch/meilisearch/pull/5985). Let's see how good it can get. My early benchmarks showed a 50% reduction in disk usage.
Someone reported it, and I answered today [1]. It's a rule that is too hard on the front end, and we will fix it by using a better Hybrid search setup (not only semantic). Thank you for the report.
V1.14, released yesterday [1], ships with a search embedding cache. Most of the time you see is spent waiting for an OpenAI embedding answer. We also just shipped composite embedders to reduce the network latency when you need to respond quickly to user searches (by running embedders on the Meilisearch server) but still use external APIs to index many documents in batches. Note that it can only work with open-source embedders, the ones HuggingFace serves.
Right. We released a lot of new versions of the engine to improve the indexing part of it. V1.12 is improving the document indexing a lot! Have you tried the latest version v1.14 we released yesterday?
While Meilisearch is capable of limiting it's resident (actual mallocs) memory. However, it requires a bare minimum (about 1GiB).
35 GiB is probably a third of the data I index into Meilisearch just for experimenting and don't forget about the inverted indexes. You wouldn't use any O(n) algorithm to search in your documents.
Also, every time you need to reboot the engine you would have to reindex everything from scratch. Not a good strategy, believe me.
The best you could do is put Meilisearch on a very good NVMe. I am indexing large streams of content (Bsky posts + likes), and I assure you that I tested Meilisearch on a not-so-good NVMe and a slow HDD — and ho, Boy!! The SSD is so much faster.
I am sending hundreds of thousands of messages and changes (of the likes count) into Meilisearch, and so far, so good. It's been a month, and everything is working fine. We also shipped the new batches/ stats showing a lot of internal information about indexing step timings [1] to help us prioritize.
Meilisearch just improved the indexing speed and simplified the update path. We released v1.12 and highly improved indexing speed [1]. We improved the upgrade path with the dumpless upgrade feature [2].
The main advantage of Meilisearch is that the content is written to disk. Rebooting an instance is instant, and that's quite useful when booting from a snapshot or upgrading to a smaller or larger machine. We think disk-first is a great approach as the user doesn't fear reindexing when restarting the program.
That's where Meilisearch's dumpless upgrade is excellent: all the content you've previously indexed is still written to disk and slightly modified to be compatible with the latest engine version. This differs from Typesense, where upgrades necessitate reindexing the documents in memory. I don't know about embeddings. Do you have to query OpenAI again when upgrading? Meilisearch keeps the embeddings on disk to avoid costs and remove the indexing time.
> I'm still looking for a systematic approach to make a hybrid search (combined full-text with embedding vectors).
You know that Meilisearch is the way to go, right? Tantivy, even though, I love the product, doesn't support vector search. Its Hybrid search is stunningly good. You can try it on our demo [1].
Meilisearch decided to use hybrid search and avoid fusion ranking. We plan to work on reranking soon, but as far as I know, our hybrid search is so good that nobody asked for reranking. You can read more about our Hybrid search in our blog post [1].
About streaming ingestion support. Meilisearch support basic HTTP requests and is capable of batching task to index them faster. In v1.12 [2], we released our new indexer version that is much faster, leverages high usage of parallel processing, and reduces disk writes.
Meilisearch has been production-ready since v1.0. I made it in Rust to ensure it stays production-ready for years and years. Memory-safe languages are here to replace unsafe ones like C++ and reduce the number of breaches you expose in production.
Here is an article by Google showing the benefits of using memory-safe languages in production rather than others. It is explicitly rotating around Rust [1].
Thank you very much! We put a lot of effort into our documentation and be ready for the next version of our documentation coming soon. The experience will be even better and faster. We also put much effort recently into simplifying how people can migrate to the next engine version with the [dumpless upgrade feature][1]. We also stabilized our full Rust Vector Store and Hybrid search (AI-powered search) feature in v1.13.
Nope, it doesn't. It's based on Cascade Ranking, also called [bucket sorting][1]. We released our new Hybrid search ranking system, combining the best full-text search results (our Cascade Ranking) with semantic results (with arroy, our full-Rust Vector Store). You can try that at https://wheretowatch.meilisearch.com.
[1]: https://pijul.org/posts/2021-02-06-rethinking-sanakirja