HackerTrans
トップ新着トレンドコメント過去質問紹介求人

Tananon

no profile record

投稿

[untitled]

1 ポイント·投稿者 Tananon·2 か月前·0 コメント

[untitled]

1 ポイント·投稿者 Tananon·4 か月前·0 コメント

Show HN: Pyversity – Fast Result Diversification for Retrieval and RAG

github.com
86 ポイント·投稿者 Tananon·9 か月前·11 コメント

Show HN: Pyversity – Fast Result Diversification for Retrieval and RAG

github.com
4 ポイント·投稿者 Tananon·9 か月前·0 コメント

コメント

Tananon
·9 か月前·議論
Nice, I actually read that Jina article when it was published, but forgot they use facility location as well! The saturated coverage algorithm looks pretty interesting, I'll have a look at how feasible it would be to add that to Pyversity.
Tananon
·9 か月前·議論
That's indeed something I plan to add in the near future. I'll probably add a tutorial as well to showcase how you can use this with e.g. sentence transformers. There's some pretty good benchmarks in the paper that I used as inspiration for some of these algorithms: https://arxiv.org/pdf/1709.05135, I'll most likely try to reproduce some of these.
Tananon
·9 か月前·議論
True, I think that's also a great usecase! Though these algorithms likely won't scale to very large datasets (e.g. millions of samples), but for smaller datasets, like fine-tuning sets, I think this would work very well. I've worked on something similar in the past that works for larger datasets (semantic deduplication: https://github.com/MinishLab/semhash).