I built a screen-aware desktop assistant; now it can write and use your computer
4 comments
Isnt this the sort of thing Microsoft is trying to integrate into Windows? I can imagine ever using something like this. Take that with a pinch of salt though. I still use my phone as a phone and have cash in my pocket.
What sort of safety assurances are there? I'm having a hard time imagining a scenario where I'd let such an assistant have full access to my machine.
Totally fair concern. I wouldn’t expect anyone to give an assistant full, unsupervised access to their machine.
The model here is supervised control by default. Julie can see context and propose actions, but anything state-changing is gated behind explicit approval. You can also run it in read-only or draft-only modes where it helps you think and write, but never clicks or types on your behalf.
Access is permissioned and scoped, not blanket. Screen, keyboard, mouse, filesystem, network, each is a separate capability with clear boundaries and per-app controls. On top of that, actions go through defined protocols rather than free-form behavior, so you know exactly what kinds of operations it’s allowed to perform and under what conditions.
The goal isn’t full access. It’s controlled access with clear protocols, human-in-the-loop execution, and the ability to shut it off instantly if something feels off.
The model here is supervised control by default. Julie can see context and propose actions, but anything state-changing is gated behind explicit approval. You can also run it in read-only or draft-only modes where it helps you think and write, but never clicks or types on your behalf.
Access is permissioned and scoped, not blanket. Screen, keyboard, mouse, filesystem, network, each is a separate capability with clear boundaries and per-app controls. On top of that, actions go through defined protocols rather than free-form behavior, so you know exactly what kinds of operations it’s allowed to perform and under what conditions.
The goal isn’t full access. It’s controlled access with clear protocols, human-in-the-loop execution, and the ability to shut it off instantly if something feels off.
This is extremely useful, thanks!
Update: I just shipped Julie v1.0, and the big change is that it’s no longer only “answer questions about my screen.” It can now run agents (writing/coding) and a computer-use mode via a CUA toolkit. ((https://tryjulie.vercel.app/))
What that means in practice:
- General AI assistant, it hears what you hear, sees what you see, and gives you real-time answers for any question instantly. - Writing agent: draft/rewrite in your voice, then iterate with you while staying in the overlay (no new workspace). - Coding agent: help you implement/refactor with multi-step edits, while you keep your editor as the “source of truth.” - Computer-use agent: when you want, it can take the “next step” (click/type/navigate) instead of just telling you what to do.
The goal is still the same: don’t break my flow. I want the assistant to feel like a tiny utility that helps for 20 seconds and disappears, not a second life you manage.
A few implementation notes/constraints (calling these out because I’m sure people will ask):
- It’s opt-in for permissions (screen + accessibility/automation) and meant to be used with you watching, not silently running. - The UI is intentionally minimal; I’m trying hard not to turn it into a full chat app with tabs/settings/feeds.
Repo + installers are here: https://github.com/Luthiraa/julie
Would love feedback on two things: 1. If you’ve built/used computer-use agents: what safety/UX patterns actually feel acceptable day-to-day? 2. What’s the one workflow you’d want this to do end-to-end without context switching?