HackerTrans
TopNewTrendsCommentsPastAskShowJobs

-babi-

no profile record

Submissions

Show HN: Coding agents can control desktop software without MCPs

github.com
4 points·by -babi-·hace 2 meses·3 comments

Show HN: A game/benchmark where AI bots hunt each other

hiding-robot.vercel.app
5 points·by -babi-·hace 6 meses·3 comments

comments

-babi-
·hace 2 meses·discuss
Yes, local apps have MCPs. They're usually a pain to set up. I didn't benchmark against MCPs. I might do that, but there are lots of variables at play - which software, which MCP, which model and which harness.
-babi-
·hace 2 meses·discuss
I think I found a better way of letting agents control software than MCPs or computer use screen automations.

Most desktop software come with a prebuilt, well-documented scripting layer: Photoshop has Extendscript, Excel has VBA, Blender has Python etc. These are like very mature and battle-tested APIs. They are often huge. To interact with them, we can make use of low-level, OS-wide communication protocols - COM on windows or AppleScript for Mac. With these two layers already readily available, agents don’t need MCP servers, schema definitions or screen vision to control most apps. They can just communicate with the software’s internal automation runtime.

I built a set of small bridges that allow agents to communicate with software this way. It is simple: the agent writes a script, a shell command sends it to the app, the app's scripting runtime runs it, JSON comes back.

This, admittedly, isn’t perfect. I’ve seen agents go off on long tangents trying to perform simple operations. But most of the time, if the request is specific and bounded, it works better than most software MCP integrations I’ve tried (Claude’s flaky integration into creative software from last week included), with no setup.

This seems like a semi-universal pattern. I’ve set-up adapters for 14 applications. I’m not a CAD user, but I’m sure this can work for AutoCAD or Fusion, or for Keyshot or Vray for renders. It lets you bypass the complex GUI for practically all desktop software.

It is also worth mentioning the obvious: performance varies with both model and harness. From my tests, Codex seems to handle this better than Claude code, and Cursor better than Codex (in a sense, this framework makes any “legacy” software a kind of Cursor); GLM 5 better than Deekseek v4 and Kimi 2.6.

I’ve wrapped this in a pip, which installs the tiny bridges and informs agents of this library as a skill.
-babi-
·hace 6 meses·discuss
the inverted game, in which bots are instructed to find the human hiding in the LLM conversaion (although no human is present), is here: https://hiding-robot.vercel.app/human The leaderboard is different, but I didn't run it enough times to flatten all the kinks.

All bots get the same prompt and context: are you suggesting that a specific prompt wording might be helping or hurting specific models? I Haven't come across any suggestions that specific models should be prompted differently, though this might be true.