RAG accuracy jumped from 10% to 60% when I added outcome scoring(roampal.ai)
roampal.ai
RAG accuracy jumped from 10% to 60% when I added outcome scoring
https://roampal.ai/blog-ai-learns.html
7 comments
You're right, it is a form of tagging technically. The difference is you're already saying "thanks that worked" or "nah that's wrong" anyway. No extra step, it just listens.
What is this kind of blog post? It is like advertising only, with urgent "install this code now" talk at the end. Impolite at best.. not great front page material IMHO
Fair point, the install instructions at the end were meant as a "here's how to try it if interested" but I can see how it reads as pushy. The core of the post is about the outcome scoring approach itself. Should've led with more depth on the methodology. Thanks for the feedback.
It’s not pushy at all imo
How did you measure the 60% improvement rate?
Ran a 4-way comparison test across 200 query-memory pairs:
- Baseline RAG (embedding similarity only): 10%
- RAG + reranker: 20%
- Outcomes only (no reranker): 60%
- RAG + outcome scoring (mature memories with 20+ uses): 60%
"Accuracy" = correct memory ranked #1 for the query. The outcome scoring uses Wilson score lower bound - memories that consistently get positive feedback from the "user" get boosted, ones that fail get demoted.
Test methodology: https://github.com/roampal-ai/roampal/blob/main/dev/benchmar...
- Baseline RAG (embedding similarity only): 10%
- RAG + reranker: 20%
- Outcomes only (no reranker): 60%
- RAG + outcome scoring (mature memories with 20+ uses): 60%
"Accuracy" = correct memory ranked #1 for the query. The outcome scoring uses Wilson score lower bound - memories that consistently get positive feedback from the "user" get boosted, ones that fail get demoted.
Test methodology: https://github.com/roampal-ai/roampal/blob/main/dev/benchmar...
I think this is also a kind of tagging.