Show HN: RLM-based local debugger for AI agent traces(github.com)
github.com
Show HN: RLM-based local debugger for AI agent traces
https://github.com/context-labs/halo
16 comments
Very interesting. Haven't heard of RLMs before.
https://github.com/alexzhang13/rlm
> We propose Recursive Language Models (RLMs), a general inference paradigm that treats long prompts as part of an external environment and allows the LLM to programmatically examine, decompose, and recursively call itself over snippets of the prompt.
> We find that RLMs can successfully process inputs up to two orders of magnitude beyond model context windows and, even for shorter prompts, dramatically outperform the quality of vanilla frontier LLMs and common long-context and coding scaffolds [...] across four diverse long-context tasks while having comparable cost.
https://arxiv.org/abs/2512.24601
I had a similar thought the other day. When doing a research task, you don't want to crap up the context with all the web scrapes. But you want to ask follow up questions on the full context, not the anemic subagent summaries. So what you actually want is an "extended context" you can grep.
https://github.com/alexzhang13/rlm
> We propose Recursive Language Models (RLMs), a general inference paradigm that treats long prompts as part of an external environment and allows the LLM to programmatically examine, decompose, and recursively call itself over snippets of the prompt.
> We find that RLMs can successfully process inputs up to two orders of magnitude beyond model context windows and, even for shorter prompts, dramatically outperform the quality of vanilla frontier LLMs and common long-context and coding scaffolds [...] across four diverse long-context tasks while having comparable cost.
https://arxiv.org/abs/2512.24601
I had a similar thought the other day. When doing a research task, you don't want to crap up the context with all the web scrapes. But you want to ask follow up questions on the full context, not the anemic subagent summaries. So what you actually want is an "extended context" you can grep.
I started building my agent loops based on the RLM paper and I am finding the recursive part serves two major purposes. First, it pushes down token consumption as you describe. The other thing it does is prevent the agent from returning too soon, since most of the real work happens at depth. Especially if you forbid tool use in the root.
I am starting to wonder if maybe I could just focus on these aspects more directly as opposed to treating them as side effects of symbolic recursion. I do have to agree with the paper in that recursive depth beyond 1 doesn't seem to matter. At least not with the current frontier of models. If we can't recurse more than once and extract much uplift, then I question us labeling this a recursive scheme.
I am starting to wonder if maybe I could just focus on these aspects more directly as opposed to treating them as side effects of symbolic recursion. I do have to agree with the paper in that recursive depth beyond 1 doesn't seem to matter. At least not with the current frontier of models. If we can't recurse more than once and extract much uplift, then I question us labeling this a recursive scheme.
Cool project. A team at work was building something similar to internal use.
I'm curious how this compares to just using Claude Code directly and giving it a dump of the agent traces? It seems like Claude could probably do some of the same diagnostics / trace grouping to identify failure patterns. Why use a custom harness?
I'm curious how this compares to just using Claude Code directly and giving it a dump of the agent traces? It seems like Claude could probably do some of the same diagnostics / trace grouping to identify failure patterns. Why use a custom harness?
Yeah, fair question. For a small number of traces just dumping them into Claude Code can work well.
However, once you're at production scale the problem changes. You can't always fit 10,000+ traces in Claude Code and still have it be effective especially when the relevant pattern of agent failures may only become apparent when you pass that many in. That's where the RLM based methodology helps. HALO recursively decomposes the trace data into smaller investigations, analyzes those sub-pieces, and then synthesizes those up to determine the recurring harness-level failure modes better than Claude Code or Codex ever could at a large scale.
However, once you're at production scale the problem changes. You can't always fit 10,000+ traces in Claude Code and still have it be effective especially when the relevant pattern of agent failures may only become apparent when you pass that many in. That's where the RLM based methodology helps. HALO recursively decomposes the trace data into smaller investigations, analyzes those sub-pieces, and then synthesizes those up to determine the recurring harness-level failure modes better than Claude Code or Codex ever could at a large scale.
I'm very curious to see a benchmark for this - have toyed with the idea myself but haven't put in the hard work to test these hypothesis on extracting learning signal from deep-agent traces.
There's some benchmarks in the repo for AppWorld. Looks promising
> The engine decomposes the traces to understand common failure modes across harness executions and produces a report with its findings.
What are some examples of these common failure modes?
What are some examples of these common failure modes?
Nice work
What sort of systematic issues would teams typically uncover using HALO? I guess there's some sort of built-in checklist you include in the RLM prompt
What sort of systematic issues would teams typically uncover using HALO? I guess there's some sort of built-in checklist you include in the RLM prompt
neat. when you say production-scale traces don't fit in context — does halo cluster failures first or just stream-summarize each trace?
It's more-so recursive decomposition rather than clustering failures or stream summarizing. You can look at this repo if you want to learn more about how RLM's work: https://github.com/alexzhang13/rlm
[deleted]
It’s a loop. Run your agent, feed the traces to HALO, get the report, apply the fixes, then re-run your agent.
HALO takes in OTEL compliant traces from AI agents using tracing frameworks such as Langfuse, Arize/OpenInference, or even just plain JSONL. It uses an RLM (Recursive Language Model) to more efficiently break trace analysis into smaller subproblems in order to find recurring patterns across large amounts of data and fix systemic issues that regular LLMs might typically miss.
You can also optionally provide a path to where your agent code lives to give the engine more context so it can more concretely provide useful insights.
The repo also includes a desktop app that you can run locally without having to sign up for anything or configure anything complex.
Check out the readme in the repo for more in depth information on what HALO is and how you can use it to your benefit :)