We identified and solved for 2 key problems with generating data using GPT:
1. Duplicate/similar data points - we solve this by adding deduplication to our pipeline.
2. Incorrect question-answers - we check for correctness and context relevance. Filter out incorrect rows of data.
Apart from this, we generate a diverse set of questions including complex reasoning and chain of thought.
We also generate domain specific unsafe questions - questions that violate TnC of the particular LLM to test the model guardrails.
Ragas is an eval tool which needs ground truths and queries for evaluation. FiddleCube generates the queries and the ground truth needed for eval in Ragas, LangSmith or an eval tool of choice.
We incorporate user prompts to generate the outputs and provide diagnostics and feedback for improvement, rather than eval metrics. So you can plug your low scored queries provided by Ragas, your prompt and context. FiddleCube can provide the root cause and the ideal response.
This is an alternative to manual auditing and testing, where an auditor works on curating the ideal dataset.
Our goal is to focus on datasets and make it very easy to create and manage data.
In our next release, we will be launching a way to do this using a UI.