HackerTrans
TopNewTrendsCommentsPastAskShowJobs

fulmicoton

no profile record

Submissions

A math puzzle and a better algorithm for top-K

quickwit.io
4 points·by fulmicoton·2 वर्ष पहले·2 comments

comments

fulmicoton
·12 माह पहले·discuss
One trouble I could see with your approach is that you treat the information "Doc at pos i" beats "Doc at pos j" independently from i and j. Intuitively, it is not as critical when a bad doc is at rank 9 instead of rank 10; compared to bad doc landing at rank 1 instead of rank 10.

LambdaMART's approach seems better in that respect.

https://medium.com/@nikhilbd/pointwise-vs-pairwise-vs-listwi...
fulmicoton
·पिछला वर्ष·discuss
This bug hit us, and yes, I hadn't thought of just switching to opendal. That's indeed a great reminder.
fulmicoton
·2 वर्ष पहले·discuss
No. Quickwit was founded well before Warpstream and it did not inspire us.

The Husky blog post was released after we released a few versions of quickwit if I recall correctly. It was not an inspiration either.

As far as I know, the similarities are fortuitous.
fulmicoton
·2 वर्ष पहले·discuss
Our seed round was 100% made of SAFE, so VCs did not have the power to force us to do anything.

The sentence in the blog post is a tad misleading. I suspect François is not really talking about VCs that had already invested in quickwit, but about the usual flow of other VCs who contacted us, to know about the company and be part of our eventual series A.

It just generally felt like we were "at a crossing".

No one twisted our arm.
fulmicoton
·2 वर्ष पहले·discuss
Developer of tantivy chiming in! (I hope that's ok) Database performance is a space where there are a lot of lies and bullshit, so you are 100% right to be suspicious.

I don't know SeekStorm's team and I did not dig much into the details, but my impression so far is that their benchmark's results are fair. At least I see no reason not to trust them.
fulmicoton
·2 वर्ष पहले·discuss
Yes. We should shut down this demo. We reduced the hardware to cut down our costs. Right now it runs a ludicrously small amount of hardware.
fulmicoton
·2 वर्ष पहले·discuss
Quickwit is targetting logs:

    - it does not do vector search. It can rank docs using BM25, but usually people just want to sort by timestamp.
    - its does not use an SSD cache. Quickwit reads directly into the object storage.
    - it is append-only (you can't modify documents)
    - it scales really well and typically shines on the 1TB .. 100PB range
    - it has a Elastic search compatible API.
fulmicoton
·2 वर्ष पहले·discuss
This is NOT about transaction log. This is application logs. The thing you generate via Log4j for instance.

Also 100PB is measured as the input format (JSON). Internally Quickwit will have more efficient representations.
fulmicoton
·2 वर्ष पहले·discuss
Security and customer support are the two main reasons why people want a super long retention.

Medium retention (1 or 2 months) is still very appreciable if some issue in your bugtracker stay stale for this amount of time.
fulmicoton
·2 वर्ष पहले·discuss
It is pretty much the same as Lucene. The compression ratio is very specific logs and depends on the logs themselves. (Often it is not that good)
fulmicoton
·2 वर्ष पहले·discuss
Quickwit (like Elasticsearch/Opensearch) stores you data compressed with ZSTD in a row store, builds a full text search index, and stores some of your fields in a columnar. The "compressed size" includes all of this.

The high compression rate is VERY specific to logs.

- What happens when you alter an index configuration? Or add or remove an index?

Changing an index mapping was not available in 0.8. It is available in main and will be added in 0.9. The change only impacts new data.

- Or add or remove an index?

This is handled since the beginning.

- What about cold storage?

What makes Quickwit special is that we are reading everything is on S3. We adapted our inverted index to make it possible to read straight from S3. You might think this is crazy slow, but we typically search into TBs of data in less than a second. We have some in RAM cache too, but they are entirely optional.

> 2. Sampled data, generally for debugging. I would generally try to keep this at 10TB or less;

Sometimes, sampling is not possible. For instance, some of Quickwit users (including Binance) use their logs for user support too. A user might come asking details about something fishy that happened 2 months ago.
fulmicoton
·2 वर्ष पहले·discuss
Again, this is application logs. The stuff you would log in your program with log4j for instance.

With a microservices architecture in particular that can pile up rapidly.
fulmicoton
·2 वर्ष पहले·discuss
Thank you for the kind word @ZeroCool2u ! :)
fulmicoton
·2 वर्ष पहले·discuss
Building an inverted index is actually very cpu intensive. I think we are the fastest on that (if someone knows something faster than tantivy at indexing I am interested).

I'd be really surprised if you can make a 10x improvement here.
fulmicoton
·2 वर्ष पहले·discuss
If you can limit your research to GBs of logs, I kind of agree with you. It's ok if a log search request takes 100ms instead of 2s, and the "grep" approach is more flexible.

Usually our users search into > 1TB.

Let's imagine you have to search into 10TB (even after time/tag pruning). Distributing over 10k cores over 2 second is not practical and does not always economically make sense.
fulmicoton
·2 वर्ष पहले·discuss
The data is just Binance's application logs for observability. Typically what a smaller business would simply send to Datadog.

This log search infra is handled by two engineers who do that for the entire company.

They have some standardized log format that all teams are required to observe, but they have little control on how much data is logged by each service.

(I'm quickwit CTO by the way)
fulmicoton
·2 वर्ष पहले·discuss
Quickwit is designed to do full-text search efficiently with an index stored on an object storage.

There are no equivalent technology, apart maybe:

- Chaossearch but it is hard to tell because they are not opensource and do not share their internals. (if someone from chaossearch wants to comment?)

- Elasticsearch makes it possible to search into an index archived on S3. This is still a super useful feature as a way to search punctually into your archived data, but it would be too slow and too expensive (it generates a lot of GET requests) to use as your everyday "main" log search index.
fulmicoton
·2 वर्ष पहले·discuss
This is their application logs. They need to search into it in a comfortable manner. They went for a search engine with Elasticsearch at first, and Quickwit after that because even after restriction the search on a tag and a time window "grepping" was not a viable option.
fulmicoton
·2 वर्ष पहले·discuss
Thank you @tyler!!!
fulmicoton
·2 वर्ष पहले·discuss
Thank you so much for sharing!!!