Ask HN: Views on Mastra's SOTA Memory?
1 comments
There are so many SOTA memory claims out there.
There are also very few memory benchmarks (if any) that measure "memory" realistically.
Not to mention the industry can't seem to decide on what "memory" even means.
We need more general-purpose long-term memory systems.
We need ways to measure those. Verify they fulfill the need and are actually SOTA.
From what I can see:
1. The implementation appears heavily tuned toward performing well on LongMemEval. That's a useful signal, but it doesn't necessarily translate to robust long-term memory behavior in production environments.
2. It feels closer to context compression/context management than a durable long-term agent memory system. This will perform really well for a single long-running task
3. Both the Observer and Reflector rewrite memory in compressed form. That's helpful for token control, but compression is inherently lossy and can drop smaller details that might become important later.
4. The Reflector seems to validate success primarily via token thresholds, rather than checking whether the rewritten memory remains semantically faithful to the original. Over time, this could allow memory drift.
5. The Observer prompt may introduce assumptions (e.g., inferring that a planned action happened if enough time has passed), which risks creating incorrect memories.
6. The design appears to emphasize recency when rewriting observations. While that keeps context fresh, it may bias the system toward recent information and gradually compress away older but still important details. Durable memory systems usually need mechanisms to preserve salient long-term facts, not just recent activity.
7. The full observations block is repeatedly injected into context. This may increase token cost and introduce irrelevant noise depending on the task.
8. There appears to be limited grounding back to raw message evidence at response time, which makes it harder to detect and correct incorrect compressed memories.
9. Finally, I think we should be cautious about claiming "SOTA" based on performance on a single benchmark. LongMemEval results may demonstrate strong performance on that setup, but production workloads are much messier. Robustness, drift, grounding, and cost behavior typically show up only under sustained real-world usage.
Overall, this looks like a strong benchmark-oriented context handling. I am just less convinced that it yet qualifies as a robust, general-purpose long-term memory system. Curious how the team is thinking about these trade-offs beyond benchmark performance.