Working with embeddings (RAG, semantic search, clustering, recommendations, etc.), means:
- Generate embeddings
- Compute cosine similarity
- Run retrieval
- Hope it "works"
But then I stumbled upon the issue of not being able to determine why my RAG responses felt off, retrieval quality being inconsistent and clustering results looked weird.
Debugging embeddings was painful.
To solve this issue, we built this Embedding evaluation CLI tool to audit embedding spaces, not just generate them.
Instead of guessing whether your vectors make sense, it:
- Detects semantic outliers
- Identifies cluster inconsistencies
- Flags global embedding collapse
- Highlights ambiguous boundary tokens
- Generates heatmaps and cluster visualizations
- Produces structured reports (JSON / Markdown)
But then I stumbled upon the issue of not being able to determine why my RAG responses felt off, retrieval quality being inconsistent and clustering results looked weird.
Debugging embeddings was painful.
To solve this issue, we built this Embedding evaluation CLI tool to audit embedding spaces, not just generate them.
Instead of guessing whether your vectors make sense, it: - Detects semantic outliers - Identifies cluster inconsistencies - Flags global embedding collapse - Highlights ambiguous boundary tokens - Generates heatmaps and cluster visualizations - Produces structured reports (JSON / Markdown)
Checkout the tool and feel free to share your feedback: https://github.com/dakshjain-1616/Embedding-Evaluator
This is especially useful for: - RAG pipelines - Vector DB systems - Semantic search products - Embedding model comparisons - Fine-tuning experiments
It surfaces structural problems in the geometry of your embeddings before they break your system downstream.