Teaching GPT-5 to Use a Computer(prava.co)
prava.co
Teaching GPT-5 to Use a Computer
https://prava.co/archon/
24 comments
One potential virtuous cycle here is that accessibility trees used by tools like screen readers are also a nice potential way for a model to consume information about what's on screen and how it can be interacted with. So it creates an additional incentive for improving the accessibility of new and existing software, because doing that lights up integration with future models.
This cycle starts with an integration for model developers. I wonder if anyone is working on a generic ARIA hookup, as well as whatever standards are necessary for desktop/smartphone integration?
I imagine in the future someone will make an Agent-First OS that is entirely built from the ground up to be run by AI and runs off the assumption that there are no human users or that their usage is limited. That will be interesting, imagine all the things you could do differently, the design choices you could make. You lose a lot by accommodating human ergonomics.
What might you imagine being different in an “agent first OS” compared to a terminal only Linux distribution?
I can imagine a different UI. Instead of a browser with tabs and an address bar, you have a series of infinitely scrolling "terminals" with a chat bar at the bottom. Each one is just a chat session into which objects like web pages, images, etc can be embedded and they scroll horizontally a bit like multiple desktops do. You could zoom in and out as much as you want, the goal is to let you easily monitor many parallel tasks.
From the AI's perspective a filesystem that vector indexes data on the fly would make sense, perhaps, along with an ability for the user to share out fine-grained permissions with it.
From the AI's perspective a filesystem that vector indexes data on the fly would make sense, perhaps, along with an ability for the user to share out fine-grained permissions with it.
No I/O, ability to mount vectors and graphs, bare metal file system, structured traces, deterministic replayability of commands
For now, I still have hard time believing in unattended agents. Be it using a computer or generating programs.
I mean that would sure be a nice demo, but it’s too probabilistic to give AI agents real tasks (and it seems that isn’t going to change anytime soon).
It’s all fun and games until it implies spending money and/or taking responsibility.
And be it in personal life or in businesses, money and responsibility are vital things.
Sure you can ask LLMs to generate a minesweeper game with custom rules or ask it to summarize headlines from HN.
Releasing a program an unattended agent generated to real clients that pay you or asking it to order a non refundable flight ticket is something else.
However, I can see the point of an agent that uses my computer while I watch it.
I mean that would sure be a nice demo, but it’s too probabilistic to give AI agents real tasks (and it seems that isn’t going to change anytime soon).
It’s all fun and games until it implies spending money and/or taking responsibility.
And be it in personal life or in businesses, money and responsibility are vital things.
Sure you can ask LLMs to generate a minesweeper game with custom rules or ask it to summarize headlines from HN.
Releasing a program an unattended agent generated to real clients that pay you or asking it to order a non refundable flight ticket is something else.
However, I can see the point of an agent that uses my computer while I watch it.
[deleted]
I really like this approach. Nice job!
> We also plan to compile solved steps into micro‑policies. If you're running something like a RPA task or similar workflow as before, you can simply run the execution locally (with archon-mini running locally) and not have to worry about the planning. Over time, the planner is a background teacher, not a crutch.
Conceptually, I really like this - why re-do the work of reasoning about an already solved task? Just do it again. For some plausibly large majority of things, this could speed things up considerably.
> In the future we hope to run a streaming capture pipeline similar to Gemma 3. Consuming frames at 20–30 fps, emitting actions at 5–10 Hz, and verifying state on each commit.
I love targets like this. It makes you tune the architecture and abstractions to push the boundary of whats possible with a traditional agent loop.
The salience heat map compression is a great idea. I think you could take this a step further and tune a model so that it compresses an image into a textual semantic/interactive element hierarchy. This is effectively what browser-use is doing, just using javascript instead of a vision model.
This seems like a task that would benefit from narrow focus. I'm aware of the "Bitter Lesson," but my intuition seems to tell me that chaining together fit to purpose classification as an input to an intelligent planning system is the way to go.
> We also plan to compile solved steps into micro‑policies. If you're running something like a RPA task or similar workflow as before, you can simply run the execution locally (with archon-mini running locally) and not have to worry about the planning. Over time, the planner is a background teacher, not a crutch.
Conceptually, I really like this - why re-do the work of reasoning about an already solved task? Just do it again. For some plausibly large majority of things, this could speed things up considerably.
> In the future we hope to run a streaming capture pipeline similar to Gemma 3. Consuming frames at 20–30 fps, emitting actions at 5–10 Hz, and verifying state on each commit.
I love targets like this. It makes you tune the architecture and abstractions to push the boundary of whats possible with a traditional agent loop.
The salience heat map compression is a great idea. I think you could take this a step further and tune a model so that it compresses an image into a textual semantic/interactive element hierarchy. This is effectively what browser-use is doing, just using javascript instead of a vision model.
This seems like a task that would benefit from narrow focus. I'm aware of the "Bitter Lesson," but my intuition seems to tell me that chaining together fit to purpose classification as an input to an intelligent planning system is the way to go.
For someone who wants to do “archon‑mini is a 7B Qwen‑2.5‑VL–based executor (dynamic‑res ViT) fine‑tuned with GRPO for GUI grounding” part at home, is there a guide/post you would recommend?
This may be a good start:
https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct
It’s pretty interesting.
I see a ton of potential for testing. RPA can quickly get annoying because even simple change can break automation. The LLM ability to “reason” could really bridge the gap.
Coupled with agents who help turn specification/stories into a testing plan, I could really see automated end to end testing becoming far cheaper than it is nowadays in the near future.
That’s a very good piece of news for system reliability.
I see a ton of potential for testing. RPA can quickly get annoying because even simple change can break automation. The LLM ability to “reason” could really bridge the gap.
Coupled with agents who help turn specification/stories into a testing plan, I could really see automated end to end testing becoming far cheaper than it is nowadays in the near future.
That’s a very good piece of news for system reliability.
This is cool though wanted to share a couple of thoughts for reflection:
I feel like your demo video is not the greatest one to highlight the capability. A browsing use case likely does require a key press->planning loop, but a gaming use case, or a well known software (e.g., excel), may be able to think ahead 10-20 key presses before needing the next loop / verification. The current demo makes it seem slow / prototype-like.
Also, the X/Y approach is interesting when thinking about a generic approach to screen management. But for example for browsers, you're likely adding overhead relative to just marking the specific div/buttons that are on screen and having those be part of the reasoning (e.g., "Click button X at div with path XX"). It may be helpful to think about the workflows you are going after and what kind of accelerated management you have over them.
I feel like your demo video is not the greatest one to highlight the capability. A browsing use case likely does require a key press->planning loop, but a gaming use case, or a well known software (e.g., excel), may be able to think ahead 10-20 key presses before needing the next loop / verification. The current demo makes it seem slow / prototype-like.
Also, the X/Y approach is interesting when thinking about a generic approach to screen management. But for example for browsers, you're likely adding overhead relative to just marking the specific div/buttons that are on screen and having those be part of the reasoning (e.g., "Click button X at div with path XX"). It may be helpful to think about the workflows you are going after and what kind of accelerated management you have over them.
Nice writeup! A few years ago I proposed to a friend that he should try rendering accessibility trees and fine-tuning a model to issue tool calls over them; I don't know if this has been tried and failed or if nobody bothered trying because so few people know desktop APIs anymore. The main advantage would be accuracy/speed and avoiding the need for too many image tokens (you still need them for things that are actually images, though).
This is very interesting, my thinking has been along these same lines. This covers some of the more complicated engineering tasks required to pave the way for LLMs to automate diverse tasks with much higher accuracy and lower cost. Building the ultimate RPA as it were.
I can't help but feel like some sort of hybrid approach: use GPT5 for the strategy, then a more direct ML model for actually executing the strategy might work better than trying to use reasoning directly for input control, would work better than trying to reason your way through driving.
Can I use Archon? There is a use case I’ve wanted to explore and this tool can do most of it
Can we please squash the idea of turning this into a product early, so no one starts to take it seriously? Agentic AI shouldn't be hooked up to any environment that doesn't have a flawless "undo" or "rewind 5 minutes" button. Would you hook up randomized mouse/keyboard input into a computer that you need to get work done on?
At the very least, display a very strongly worded warning message when the tool is run in a non-VM environment. Internet connectivity is still dangerous, but at least VMs can be snapshot. And it should not be packaged as an end-user product with a bar at the top of the screen, period.
This is a risky product and those who developed it have every ability to know that, based on the history of AI hallucinations. Not because it will escape or self replicate or other things claimed by the various idiotic AI religions, but because one of the first things it will inevitably do is screw up someone's work and cause data loss.
At the very least, display a very strongly worded warning message when the tool is run in a non-VM environment. Internet connectivity is still dangerous, but at least VMs can be snapshot. And it should not be packaged as an end-user product with a bar at the top of the screen, period.
This is a risky product and those who developed it have every ability to know that, based on the history of AI hallucinations. Not because it will escape or self replicate or other things claimed by the various idiotic AI religions, but because one of the first things it will inevitably do is screw up someone's work and cause data loss.
This is great, once we get LLMs playing Doom, we'll all be free to touch grass.
Imagine getting beat by a bot and it also has the capability to talk trash to you.
"Unfortunately, my content guidelines prohibit me from describing my activities with your mother last night"
Without this, AI is going to be limited and kloodgy. Like if I wanted to have AI run a FEA simulation on some CAD model, I have to wait until the FEA software, the CAD software, the corporate models repo, etc., etc. all have AI integrations and then create some custom agent that glues them all together. Once AI can just control the computer effectively, then it can look up the instruction manuals for each of these pieces of software online, and then just have at it e2e like a human would. It can even ping you over slack if it gets stuck on something.
I think once stuff like this becomes possible, custom AI integrations will become less necessary. I'm sure they'll continue to exist for special cases, but the other nice thing about a generic computer-use agent is that you can record the stream and see exactly what it's doing, so a huge increase in observability. It can even demo to human workers how to do things because it works via the same interfaces.