Original author here. I have been working on setting up agents in collaborative swarms that are less formally orchestrated, and more self-organizing.
The context window meltdown problem becomes really painful then, you really want the agents to be able to build up experience and adapt behaviors over time.
The solution turned out to be pretty basic. Instead of summarizing the entire context window when it overflows, we run a background worker that basically “distills” related sequences in the history of the agent.
By distill, I mean extracting a “narrative” for continuity, then a series of observations and facts established in that sequence. 2000k tokens worth of back and forth compresses effectively down to a couple of 100 without losing useful context.
In one sense, the quality of the context window is refined over time, rather than deteriorating.
Initially inspired by the ideas behind Letta, the long term memory agent, but wanted to contribute a better way to manage “time”, if that makes sense.
Sanity.io | Multiple JavaScript and Go developers positions | Oslo, Norway / San Francisco, USA | Full time | On-Site | Visa/Relocation support | https://sanity.io/careers
Sanity.io is a startup building world-class cloud-based content infrastructure at enterprise scale. Specifically we are building APIs, tools, and user interfaces for editing and managing content. Our product is loved by developers and content editors alike in settings ranging from solo developers to global multi-billion dollar corporations.
Building a product that both appears real-time across the planet, scales to a large numbers of documents and still manages to take care of all the hard stuff without burdening the end-developer is a really interesting challenge.
We are looking for:
- Lead Document Store Engineer – Lead the design and implementation of our planet spanning, real time document store
- Document Store Engineer – Take a central role in the team developing our document store
- Frontend Developer – Work on our user facing applications including the content studio
- Full-stack Developer – Work on our user facing applications including the content studio
The context window meltdown problem becomes really painful then, you really want the agents to be able to build up experience and adapt behaviors over time.
The solution turned out to be pretty basic. Instead of summarizing the entire context window when it overflows, we run a background worker that basically “distills” related sequences in the history of the agent.
By distill, I mean extracting a “narrative” for continuity, then a series of observations and facts established in that sequence. 2000k tokens worth of back and forth compresses effectively down to a couple of 100 without losing useful context.
In one sense, the quality of the context window is refined over time, rather than deteriorating.
Initially inspired by the ideas behind Letta, the long term memory agent, but wanted to contribute a better way to manage “time”, if that makes sense.