TL;DR; You can save ~50-60% context tokens spent in exploration w/o compromising quality by delegating exploration to a local LLM. The cost is the execution time. Which can vary depending on how good your local llm hardware setup is. Mine is not that good.
Last month microsoft published an agent reference implementation where the Coding Agent delegated the exploration to a local agent that used a local light LLM. There was an arXiv paper , a github repo and a custom tarined model. Later in the month all these resources got pulled from the internet without any explanation.
So I tested the setup all over again to understand better. The model was poor and unreliable. The repo was duct taped at the best. The tests it ran were very poor quality explorations on small codebases.
But the architecture genuinely looked interesting. So I ported it to Grove as a fresh rewrite. Ran it on real large codebases.
Last month microsoft published an agent reference implementation where the Coding Agent delegated the exploration to a local agent that used a local light LLM. There was an arXiv paper , a github repo and a custom tarined model. Later in the month all these resources got pulled from the internet without any explanation.
So I tested the setup all over again to understand better. The model was poor and unreliable. The repo was duct taped at the best. The tests it ran were very poor quality explorations on small codebases.
But the architecture genuinely looked interesting. So I ported it to Grove as a fresh rewrite. Ran it on real large codebases.