RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation(arxiv.org)
arxiv.org
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
https://arxiv.org/abs/2404.12457
4 comments
cool but deep integration requires the LLM provider to run it and get privacy right, that'll be $premium plus tip.
>RAGCache reduces the time to first token (TTFT) by up to 4x and improves the throughput by up to 2.1x compared to vLLM integrated with Faiss.
Are RAG systems really so heavy they add 4x to the TTFT?
>RAGCache reduces the time to first token (TTFT) by up to 4x and improves the throughput by up to 2.1x compared to vLLM integrated with Faiss.
Are RAG systems really so heavy they add 4x to the TTFT?
> Are RAG systems really so heavy they add 4x to the TTFT?
Only the pure python + unoptimized orm ones. So only most of them.
Only the pure python + unoptimized orm ones. So only most of them.
[deleted]
https://arxiv.org/abs/2401.18059