We've just released R2R V3 with a completely RESTful API that covers everything you need for production RAG applications. The biggest change is our Git-like knowledge graph architecture, but we've also unified all the core objects you need to build real applications.
The key advantages can be extracted from the response above to Kluless -
R2R is built around RESTful API and is dockerized, so devs can get started on app development immediately.
The system was designed so that devs can typically scale data ingestion up to provider bottlenecks w/out extra work.
We have implemented user-level permissions and high level document management alongside the vector db, which most devs need to build in a production setting, along with the API and data ingestion scaling.
Lastly, we also log every search and RAG completion that flows through the system. This is really important to find weaknesses and tune the system over time. Most devs end up needing an observability solution for their RAG.
All of these connect to an open source developer dashboard that allows you to see uploaded files, test different configs, etc.
These basic features mean that devs can spend more time on iterating / customizing their application specific features like custom data ingestion, hybrid search and advanced RAG.
great question, I can talk about how we do the more challenging "List all YC founders that worked at Google and now have an AI startup."
For this we have a target dataset (the YC co directory) that we have around 100 questions over. We have found that when feeding an entire company listing in along with a single question we can get an accurate single answer (needle in haystack problem).
So to build our evaluation dataset we feed each question with each sample into the cheapest LLM we can find that reliably handles the job. We then aggregate the results.
This is not perfect but it allows us to have a way to benchmark our knowledge graph construction and querying strategy so that we can tune the system ourselves.
I can't answer for the kindly poster above (ty), but from our experience techniques like HyDE are great when you are getting a lot of comparative questions.
For instance, if a user asks "How does A compare to B" then the query expansion element of HyDE is incredibly useful. The actual value of translating queries into answers for embedding is a bit unclear, since most embedding models we are using have been ft'ed to map queries onto answers.
Is this an example that could benefit from something like knowledge graph construction or structured entity extraction?
I'm just curious because we have theorized and seen in practice that extraction is a way to answer questions which require connected information across disparate chunks, like you can see in the simple cookbook here [https://r2r-docs.sciphi.ai/cookbooks/knowledge-graph].
Or do you think this is something that can just be solved with more advanced multimodal ingestion?
There are a lot of good questions around ingestion today, so we will likely figure out how to intelligently expand this.
For mp3s we use whisper to transcribe, for videos we transcribe with whisper and sample frames to "describe" with a multimodal model. For images we again transcribe to a thorough text description - https://r2r-docs.sciphi.ai/cookbooks/multimodal
We have been testing multi-modal embedding models and open source models to do the description generation. If anyone has suggestions on SOTA techniques that work well at scale we would love to chat and work to implement these. Long run we'd like the system to be able to handle multi-modal data locally.
We agree that ingestion and extraction are a big part of the problem for building high quality RAG.
We've talked to a lot of different developers about these problems and haven't found a general consensus on what features are needed, so we are still evaluating advanced approaches.
For now our implementation is more general and designed to work across a variety of documents. R2R was designed to be very easy to override with your own custom parsing logic for these reasons.
Lastly, we have been focusing a lot of our effort on knowledge graphs since they provide an alternative way to enhance RAG systems. We are training our own model for triples extraction that will combine with the automatic knowledge graph construction in R2R. We are planning to release this in the coming weeks and are currently looking for beta testers [we made a signup form, here - https://forms.gle/g9x3pLpqx2kCPmcg6 for anyone interested]
We've just released R2R V3 with a completely RESTful API that covers everything you need for production RAG applications. The biggest change is our Git-like knowledge graph architecture, but we've also unified all the core objects you need to build real applications.
Check our [API](https://r2r-docs.sciphi.ai/api-and-sdks/introduction) or join our [Discord](https://discord.gg/p6KqD2kjtB) if you want to dive deeper. We'd love feedback from folks building in production!