Is anyone deploying GraphRAG in prod?
I've been seeing a lot of buzz around GraphRAG and its potential, but many implementations seem to encounter issues, often with Neo4j. Has anyone managed to set up a reliable and cost-effective GraphRAG system in production? I'm curious about real-world experiences and practical solutions.
5 comments
I develop AI girlfriends.
I've struggled a lot with achieving natural-feeling recall. I've gone through a few iterations, but I'm currently experimenting with a knowledge graph/vector hybrid that uses an LLM to extract facts to build the graph. Both the performance and $ cost really hurt, but it truly does breathe life into AI characters.
I've seen a lot of the commercial products using the latest and most expensive models in the cloud to extract facts. Instead, I fine-tuned a local model on a gpt4-generated dataset, and it works surprisingly well. It will miss some connections but in practice I don't think it will be too noticeable.
Do you find you really need that level of “resolution” with memories?
On our [1] chatbots we use one long memories text field per chatbot <-> user relationship.
Each bot response cycle suggests a new memory to add as part of its prompt (along with the message etc)
Then we take that new memory and the existing memories text and feed it to a separate “memory archivist” LLM prompt cycle that’s tasked with adding this new memory and resummarizing the whole thing, yielding a replacement for the stored memories, with this new memory added.
Maybe overly simplistic but easy to manage and pretty inexpensive. The archiving part is async and fast. The LLM seems pretty good sussing out what’s important and what isn’t.
[1] https://Graydient.ai
On our [1] chatbots we use one long memories text field per chatbot <-> user relationship.
Each bot response cycle suggests a new memory to add as part of its prompt (along with the message etc)
Then we take that new memory and the existing memories text and feed it to a separate “memory archivist” LLM prompt cycle that’s tasked with adding this new memory and resummarizing the whole thing, yielding a replacement for the stored memories, with this new memory added.
Maybe overly simplistic but easy to manage and pretty inexpensive. The archiving part is async and fast. The LLM seems pretty good sussing out what’s important and what isn’t.
[1] https://Graydient.ai
I have already tried what you're doing, and it didn't perform well enough for me. I've been developing this project for a two years now. Its memory isn't going to fit in a single prompt.
I imagine that your AI chatbots aren't as cheap or performant as they can be with your potentially enormous prompts. Technical details aside, just like when talking to real people, it feels nice when they recall minor details you mentioned a long time ago.
I imagine that your AI chatbots aren't as cheap or performant as they can be with your potentially enormous prompts. Technical details aside, just like when talking to real people, it feels nice when they recall minor details you mentioned a long time ago.
If it's your personal assassinate and is helping you for months it means pretty fast it will start forget the details and only have a vogue view of you and your preferences. So instead of being you personal assassinate it practically cluster your personality and give you general help with no reliance on real historical data.
We see three major issues that different projects encounter:
1. Knowledge Graph quality - if you don't have a clean well defined Knowledge Graph then the end result will not be good.
2. Multi Graphs support - you want to break the large Knowledge Graph to small per domain Knowledge Graphs which really helps the LLM work with the data.
3. User/Agent memories - You want each user have a dedicated & effective long term memory AKA personal Knowledge Graph completely private and secured.
4. Latency/performance - you have to have a low latency Graph Database that can provide a good user experience.
1. Knowledge Graph quality - if you don't have a clean well defined Knowledge Graph then the end result will not be good.
2. Multi Graphs support - you want to break the large Knowledge Graph to small per domain Knowledge Graphs which really helps the LLM work with the data.
3. User/Agent memories - You want each user have a dedicated & effective long term memory AKA personal Knowledge Graph completely private and secured.
4. Latency/performance - you have to have a low latency Graph Database that can provide a good user experience.