Not dumb at all. It's a whole field of active research - Speculative Decoding.
A recent paper goes one level deeper with Speculative Speculative Decoding - https://arxiv.org/abs/2603.03251
Framer Motion has recently launched CSS Studio. It's a library that runs on your browser and gives you the ability to tweak the site directly in browser which fires off instructions for your coding agent to commit to code.
I have been thinking about better and faster ways to communicate with agents for UI related changes and this is the right direction I believe.
Couldn't we get a low-res version of this info by tracking the active window using a cli tool? For linux, there are several options. Not sure about Mac.
Another approach is to run OCR on 1FPS screenshots. Everything runs locally without draining the battery like an LLM would.