Improving AI-generated tests using mutation testing(blog.senko.net)
blog.senko.net
Improving AI-generated tests using mutation testing
https://blog.senko.net/improving-ai-generated-tests-using-mutation-testing
2 comments
> Apply a spell/grammar checker to your post before sharing it with other people.
Hey, at least you know it isn't LLM generated! :) Thank you - I usually do, here I obviously didn't. Appreciate the callout.
>> do not look at the tests. If you let them into your context
> This is a bad idea. You are trusting the LLM’s ability to follow instructions
Oh I'm bot, I manually checked what it was doing. I might set up the ignore explicitly if I turn this into a repeatable procedure.
Hey, at least you know it isn't LLM generated! :) Thank you - I usually do, here I obviously didn't. Appreciate the callout.
>> do not look at the tests. If you let them into your context
> This is a bad idea. You are trusting the LLM’s ability to follow instructions
Oh I'm bot, I manually checked what it was doing. I might set up the ignore explicitly if I turn this into a repeatable procedure.
> Remember, this is important: do not look at the tests. If you let them into your context
This is a bad idea. You are trusting the LLM’s ability to follow instructions. Worse, depending on your harness the LLM might not even be able to follow these instructions. The harness may indiscriminately place code into the context in a way that is uncontrolled by the LLM.
A better idea is to modify your harness so that certain files are excluded from the context.
> I asked another AI to carefully review the tests and identify those that don't make sense.
Test validation is an entire area of research and I’m yet to be convinced that this is a task for LLMs.