The most interesting thing about this is buried in a reply rather than the post: the ability to blend deterministic workflows with AI calls, so the agent kicks in on recovery when a dialog or popup breaks the scripted path.
That's your actual differentiator and I'd lead with it. Right now the post frames it as observe-decide-execute-observe, which reads as "another CUA agent," and the top comment is immediately asking how you differ from the other CUA APIs in the batch. But deterministic-by-default with AI on exception is a genuinely different shape from both of the alternatives you name. RPA breaks when the screen changes; a pure agent is nondeterministic on the 95% of steps that never needed judgment in the first place. Doing the boring path deterministically and paying for a model only when reality deviates is the thing that would make me try this.
I came at the same conclusion from a completely unrelated domain (I build asset generators, no computer-use anywhere near it) and the tool only got good the day I stopped letting the model do the parts that had to be identical every run. Generation is sampling. Same prompt, same model, different run, different result. The skill is knowing which steps genuinely need judgment and refusing to spend a sampling operation on the ones that don't. Sounds like you've built exactly that control surface and then under-sold it.
Genuine question on the handoff, since that's where I'd expect this to get hard: when the deterministic path hits something unexpected and the agent takes over for recovery, how does it hand control back? Does it have to return the UI to a known state that the script recognizes, or does the script resume wherever the agent left off? That reconciliation seems like the interesting engineering, and it's also where I'd expect the failure modes to live.
That's your actual differentiator and I'd lead with it. Right now the post frames it as observe-decide-execute-observe, which reads as "another CUA agent," and the top comment is immediately asking how you differ from the other CUA APIs in the batch. But deterministic-by-default with AI on exception is a genuinely different shape from both of the alternatives you name. RPA breaks when the screen changes; a pure agent is nondeterministic on the 95% of steps that never needed judgment in the first place. Doing the boring path deterministically and paying for a model only when reality deviates is the thing that would make me try this.
I came at the same conclusion from a completely unrelated domain (I build asset generators, no computer-use anywhere near it) and the tool only got good the day I stopped letting the model do the parts that had to be identical every run. Generation is sampling. Same prompt, same model, different run, different result. The skill is knowing which steps genuinely need judgment and refusing to spend a sampling operation on the ones that don't. Sounds like you've built exactly that control surface and then under-sold it.
Genuine question on the handoff, since that's where I'd expect this to get hard: when the deterministic path hits something unexpected and the agent takes over for recovery, how does it hand control back? Does it have to return the UI to a known state that the script recognizes, or does the script resume wherever the agent left off? That reconciliation seems like the interesting engineering, and it's also where I'd expect the failure modes to live.