I Updated My Embedding Model and My RAG Broke: A Post-Mortem(decompressed.io)

3 pointsby zacole4 bulan yang lalu2 comments

I Updated My Embedding Model and My RAG Broke: A Post-Mortem

https://decompressed.io/learn/rag-observability-postmortem

2 comments

> Here's what most teams miss when upgrading embedding models:vectors from different models cannot be compared with cosine similarity.

I'm not in the RAG space but that sounds weird to me. Not knowing this aspect of an embedding model is like a race car team not knowing their car runs on race fuel and filling it up with diesel.

pmoati4 bulan yang lalu

haha, I know that feeling! I worked on a RAG system for a pharmaceutical client and the hardest part was exactly this: You know, when everything looks fine, without error, but results are silently wrong!!! I think LLMs answering with full confidence on bad data is the most dangerous failure mode.