TL;DR; You can save ~50-60% context tokens spent in exploration w/o compromising quality by delegating exploration to a local LLM. The cost is the execution time. Which can vary depending on how good your local llm hardware setup is. Mine is not that good.
Last month microsoft published an agent reference implementation where the Coding Agent delegated the exploration to a local agent that used a local light LLM. There was an arXiv paper , a github repo and a custom tarined model. Later in the month all these resources got pulled from the internet without any explanation.
So I tested the setup all over again to understand better. The model was poor and unreliable. The repo was duct taped at the best. The tests it ran were very poor quality explorations on small codebases.
But the architecture genuinely looked interesting. So I ported it to Grove as a fresh rewrite. Ran it on real large codebases.
You are absolutely right about what we feel intuitively - LSPs should beat the shit out of the competition. But surprisingly it did not. Across 10 different LSP servers, across 5 different levels of prompt complexity it did not. Mind you, I painstakingly warmed up the LSP servers that needed it warmed. Some liked it cold and it fared equally non impressively. The pattern I saw was, LLMs (sonnet w.6 with cc) was very clever to use whatever it had to get to a verifiable answer. It could do it just with bash for sure. But as the prompt complexity grew the cost also rose.
Treesitter is sitting in a sweet spot here. a vrainy LLM can find the shortest path with high quality with treesitter and a few bash calls.
oh! I thought this was like bash w/o verbosity. That is a tool I would love to see. bash4llm . May be llike Gandhi said - I should be the change you want to see!
Bruno, if we are operating the agent (assuming coding-agent ) in an environment where the build and test tools as well the source code lives, why would we need an additional step to supply context? Wouldn't even the most basic of agents eb able to operate on those tools and build the discovery themselves?
Agree. The challenge is that same model/provider can use dramatically different numbers for thefirst two items. Maintainability is an invariant of quality of the system. Measuring that is a good metric for sure.
Agree that LOC and Story points are not metrics that needs tracking. What is happening is that same model and provider can use a very different experience in terms of [Turns, Tokens, Time and to certain extent the results]. This is expected and to some extent acceptable. In this scenario how will we write a test case? What is success? What moves the needle to say that a commit made the software perform 20% better?
This is a fascinating thought. I agree. It is one thing to let it write a prompt. Prompting, Generating, Judging and going till it is satiated is gluttony.
Every week I ship 4-5 sprints' worth of work that improves features, simplifies code, adds maintainability while delivering value to my end users without feeling I butchered a software hand-engineered over 10 years. This is one product. Mentioning it as an example because it got built over the years starting 2010. It has ~7000 paying customers and works in the B2B space. The B2B SaaS space is brutal since the defendable value should be delivered with margins two levels deep. I wouldn't be able to do it w/o coding agents.
BUT I will never do it with LOOP ENGINEERING. Not in 2026 and not in 2036. Software Engineering as such is not going to be much different in 2036 than it is now.
What stops me is the accountability that lies with me in delivered software. Setting lofty goals, a few guardrails and then letting harness and agent rip it overnight can (and have) created slop.
I did not find LLMs systematically generate poor code. What is poor? Poor in traditional handcraft coding days is defined by
These are guardrails that the models understand. However it cannot understand all these upfront - when building new. It always needs a retrospection cycle and a token budget to refactor.
This is tech debt which we all carried and solved. It is just at an exponential scale because - well, the generation is happening at a super human scale.
So, either build the harness with the tech debt coverage cycles or take on ourselves to add that cycle triggered by human judgement.
A few things that are working for me and I trust are:
Clarifying - I am not suggesting headroom and lean-ctx are only doing the tool compression. It runs elaborate subsystems towards a noble objective of saving tokens. RTK on the otherhand is focused purely on the compression. At the sametime, platforms - the providers - especially Anthropic and and Open AI have transparently absorbed many of the big ticket token saving items these tools are cleverly managing. That is a different discussion.
It is a very valid question. My intution (no grounding) is to the model training. Optimizations traditionally have worked well in human wrote software with either experience of the developer , usage of architectural patterns or a second ir third pass of fine tuning. In case of model written code - (e/p one token at a time), only possible orchitectural optimization is either with a strict guardrail on patterns to use for a specific implementation OR by giving a second or third optmization path. All of which burns more tokens, but can lead to better software.
Last month microsoft published an agent reference implementation where the Coding Agent delegated the exploration to a local agent that used a local light LLM. There was an arXiv paper , a github repo and a custom tarined model. Later in the month all these resources got pulled from the internet without any explanation.
So I tested the setup all over again to understand better. The model was poor and unreliable. The repo was duct taped at the best. The tests it ran were very poor quality explorations on small codebases.
But the architecture genuinely looked interesting. So I ported it to Grove as a fresh rewrite. Ran it on real large codebases.