Show HN: We built an AI Agent to reproduce bugs(metabase.com)
metabase.com
Show HN: We built an AI Agent to reproduce bugs
https://www.metabase.com/blog/reprobot-github-issue-triage-agent
3 comments
[deleted]
It is good to see AI agents are used outside of client apps and I really appreciate the idea behind it. What's your false-positive rate like cases where the bot "reproduces" something but actually hits a different bug, or confirms a non-bug as reproduced? How you validate the agent's own confidence.
t's been about 10% false positive so far. Working to get it lower by seeing the examples, but it will never be always right.
We've been validating it through the triage people and the follow-up devs actually comparing the result vs. what they see. Nothing is being automatically done based on the bot, it's another input into the existing processes. The triage people can often tell when something seems "off" in the repro report based on their overall knowledge. And if it gets to development to fix, they can quickly see when the reproduction is completely wrong.
When people find something particularly good or bad, we mark it in linear and do periodic sweeps of those comments for how to improve the prompts and tools available to the bot.
Interestingly, there is a bit of a grey area at times on "did it correctly reproduce it" where it sort of does but not exactly. Even in those cases, though, seeing it's "thought" process and what it tried and the results have been saving people time vs. having to work though that process themselves. Even if it's not 100% right, it's almost always directionally right
When people find something particularly good or bad, we mark it in linear and do periodic sweeps of those comments for how to improve the prompts and tools available to the bot.
Interestingly, there is a bit of a grey area at times on "did it correctly reproduce it" where it sort of does but not exactly. Even in those cases, though, seeing it's "thought" process and what it tried and the results have been saving people time vs. having to work though that process themselves. Even if it's not 100% right, it's almost always directionally right
It started as a hackathon project and is now part of our daily workflow, so we wrote about it and open-sourced the code as an example for others.
How have similar tools been working for you? What has worked well and what has not?