We are building a command & conquer style RTS. If you're interested in taking a look at our very early demo of the game play, please check out:
https://warpact-rts-9221.web.app/
great question! we have logic to look for things like when certain network requests are completed, dom loaded, etc as well as a timeout so we are not waiting for ever. The LLM based on the screenshot can also decide to wait longer if the page hasn't fully loaded despite the checks we do.
They have browser automation, and a bunch of other agent tools to manage tasks, do things like PowerPoint slides, etc. I find chatgpt agent mode better for most tasks though.
I just did this test with our web QA agent - kodefreeze.com, it was able to test creating an account until it reached the screen that requires email confirmation.
Support for being able to receive email/custom actions is on our roadmap, but would love to see if getting this far would be valuable to you. The test was with the [email protected].
How do you build trust in a system like that? The flowchart style have the advantage that you can decide when you want a human to review/approve as well as ensuring actions that need to happen at certain conditions do happen.
This is really interesting. We've been working on a smaller set of this problem space. We've also found in some cases you need to somehow pass to the model the sequence of events that happen (like a video of a transition).
For instance, we were running a test case on a e commerce website and they have a random popup that used to come up after initial Dom was rendered but before action could be taken. This would confuse the LLM for the next action it needed to take because it didn't know the pop-up came up.