Original author here. I have been working on setting up agents in collaborative swarms that are less formally orchestrated, and more self-organizing.
The context window meltdown problem becomes really painful then, you really want the agents to be able to build up experience and adapt behaviors over time.
The solution turned out to be pretty basic. Instead of summarizing the entire context window when it overflows, we run a background worker that basically “distills” related sequences in the history of the agent.
By distill, I mean extracting a “narrative” for continuity, then a series of observations and facts established in that sequence. 2000k tokens worth of back and forth compresses effectively down to a couple of 100 without losing useful context.
In one sense, the quality of the context window is refined over time, rather than deteriorating.
Initially inspired by the ideas behind Letta, the long term memory agent, but wanted to contribute a better way to manage “time”, if that makes sense.
The context window meltdown problem becomes really painful then, you really want the agents to be able to build up experience and adapt behaviors over time.
The solution turned out to be pretty basic. Instead of summarizing the entire context window when it overflows, we run a background worker that basically “distills” related sequences in the history of the agent.
By distill, I mean extracting a “narrative” for continuity, then a series of observations and facts established in that sequence. 2000k tokens worth of back and forth compresses effectively down to a couple of 100 without losing useful context.
In one sense, the quality of the context window is refined over time, rather than deteriorating.
Initially inspired by the ideas behind Letta, the long term memory agent, but wanted to contribute a better way to manage “time”, if that makes sense.