Good questions! You're right the 200 test runs limit is too low. I badly want to increase this, but currently each test costs $.05 to run so I need to get the costs down to enable that.
But my next update will fix that – and also answer your second question. What I've realized is we don't actually need to use the AI on every test run. We just need to use the AI on the initial run, then subsequent tests can just replay the playwright steps the AI chose.
And if a test fails, before throwing a failure we can re-run it with the AI assistance to see if the test truly failed from a user perspective. Hopefully when I add this functionality it can address your concerns around unpredictability :)
TestingBee is a way for startups to get part-time QA for their product's critical flows.
I've been working at startups for the last four years and I've consistently been on teams struggling to balance launching quickly versus keeping our product working. We've never had success creating a substantial test suite because our product is changing too fast and engineers are too overloaded.
I built testingbee as the solution. It lets you write your app's flows in plain english and the bot I created will execute those flows in your app as a user would. This triggers on every push to make sure every release keeps your product working :)
U.S Based Full Stack Engineer and SaaS Founder - I've scaled side products to 12,000 users and $4K ARR. Looking for a full-time Senior Software Engineer role in a startup!
I previously worked at Target as an L4 Software Engineer. Then, moved to the startup world and worked for a small seed stage startup that was acquired 3 months ago. Happy to hop on a call to hear about your role.
But my next update will fix that – and also answer your second question. What I've realized is we don't actually need to use the AI on every test run. We just need to use the AI on the initial run, then subsequent tests can just replay the playwright steps the AI chose.
And if a test fails, before throwing a failure we can re-run it with the AI assistance to see if the test truly failed from a user perspective. Hopefully when I add this functionality it can address your concerns around unpredictability :)