t's been about 10% false positive so far. Working to get it lower by seeing the examples, but it will never be always right.
We've been validating it through the triage people and the follow-up devs actually comparing the result vs. what they see. Nothing is being automatically done based on the bot, it's another input into the existing processes. The triage people can often tell when something seems "off" in the repro report based on their overall knowledge. And if it gets to development to fix, they can quickly see when the reproduction is completely wrong.
When people find something particularly good or bad, we mark it in linear and do periodic sweeps of those comments for how to improve the prompts and tools available to the bot.
Interestingly, there is a bit of a grey area at times on "did it correctly reproduce it" where it sort of does but not exactly. Even in those cases, though, seeing it's "thought" process and what it tried and the results have been saving people time vs. having to work though that process themselves. Even if it's not 100% right, it's almost always directionally right
When people find something particularly good or bad, we mark it in linear and do periodic sweeps of those comments for how to improve the prompts and tools available to the bot.
Interestingly, there is a bit of a grey area at times on "did it correctly reproduce it" where it sort of does but not exactly. Even in those cases, though, seeing it's "thought" process and what it tried and the results have been saving people time vs. having to work though that process themselves. Even if it's not 100% right, it's almost always directionally right