Something I’m finding while testing SWE-context-bench for the agent memory layer I’m building: evaluating memory is harder than checking whether the agent solved the next task with fewer tokens.
The setup:
An agent solves a coding task. Later, it gets a related task that should benefit from the earlier session.
That is the right shape for testing memory. But the details get messy.
Tool use:
Sometimes the agent can just web search, inspect the repo, or rediscover the answer.
The task passes, but did memory help?
You have to inspect the logs and ask where the answer came from: memory, current codebase, web search, or the model figuring it out again.
So the benchmark is not just measuring success. It is also measuring provenance.
Timeline issues:
The benchmark has an original task and a related task.
The related task is supposed to use context from the original task. But sometimes the ordering is weird. The “original” task is effectively from the future, and the “related” task is from the past.
So the repo can already contain the answer that memory was supposed to provide.
Dataset issue, completely changes what the score means.
Benchmark gaming:
There is also an easy bad strategy: after every task, write a very detailed summary of everything.
If you know the next task will be related, this works.
Now, lets say you solve all of the above problems. Will this still mean your system is good?
Creating a benchmark that actually mimics product performance looks like most of the battle here.
The setup: An agent solves a coding task. Later, it gets a related task that should benefit from the earlier session. That is the right shape for testing memory. But the details get messy.
Tool use: Sometimes the agent can just web search, inspect the repo, or rediscover the answer.
The task passes, but did memory help? You have to inspect the logs and ask where the answer came from: memory, current codebase, web search, or the model figuring it out again.
So the benchmark is not just measuring success. It is also measuring provenance.
Timeline issues: The benchmark has an original task and a related task.
The related task is supposed to use context from the original task. But sometimes the ordering is weird. The “original” task is effectively from the future, and the “related” task is from the past.
So the repo can already contain the answer that memory was supposed to provide. Dataset issue, completely changes what the score means.
Benchmark gaming: There is also an easy bad strategy: after every task, write a very detailed summary of everything.
If you know the next task will be related, this works.
Now, lets say you solve all of the above problems. Will this still mean your system is good?
Creating a benchmark that actually mimics product performance looks like most of the battle here.
Would love to know a good way to benchmark?