We're building an open-source tool that makes your screen activity searchable via AI.
Processing hundreds of screenshots/hour forced us to optimize for token costs.
The surprise: send video, not images
- Single screenshot (1698×894): 1,812 tokens
- Same frame in video: 258 tokens (Gemini 2.5) or ~70 tokens (Gemini 3)
- Full 8-hour workday: ~$1-3
Video gives you timestamps for free and compresses well since consecutive frames are nearly identical. We keep costs down by having the LLM write short summaries while running OCR locally for text extraction.
I did this while trying to figure out what to use in our own tool. The task was to analyze around 12,000 screenshots and find recurring manual workflows worth automating.
Results:
- Claude Sonnet 4.6: 8/10, $0.53/run — wins on quality
- Kimi K2.5: 7/10, $0.09/run — 6x cheaper, now my production pick
- GPT-5.2: 6/10, $0.41/run — missed the most obvious patterns, odd
- DeepSeek V3.2: 0/10 — gave me a garbled XML...
Models that flagged a one-time DKIM setup as "recurring automation candidate" got penalized.
Happy to share more if folks find this interesting.
We built a desktop app that takes screenshots as you work, analyzes them with AI, saves the output locally and lets you query this "context" via MCP.
Next thing I thought of was why not plug it into Openclaw. Sure, when I did, it started referencing meetings and action points from last week and suggesting follow ups. It's like 10x more proactive.
I'm gonna test it a bit more but would appreciate feedback/pointers from others with similar use cases.
- Quality: night and day vs local models, we tried running vision locally first and it was mediocre
It works by triggering a screenshot on activity, sending it to a cloud vision model for summarization, then deleting the screenshot and storing only the text in local SQLite. You query it via MCP – "what was I working on before lunch?" and Claude actually knows.
Processing hundreds of screenshots/hour forced us to optimize for token costs.
The surprise: send video, not images
- Single screenshot (1698×894): 1,812 tokens
- Same frame in video: 258 tokens (Gemini 2.5) or ~70 tokens (Gemini 3)
- Full 8-hour workday: ~$1-3
Video gives you timestamps for free and compresses well since consecutive frames are nearly identical. We keep costs down by having the LLM write short summaries while running OCR locally for text extraction.