MemPalace's benchmark claims have been picked apart and debunked in the project's GitHub issues:

github.com/milla-jovovich/mempalace/issues/27 github.com/milla-jovovich/mempalace/issues/29 github.com/milla-jovovich/mempalace/issues/39 github.com/milla-jovovich/mempalace/issues/125 github.com/milla-jovovich/mempalace/issues/242

(and others)

TL;DR: When independent third parties ran actual end-to-end QA instead of retrieval metrics, the scores dropped dramatically.

dial481

·4 mesi fa·discuss

That's encouraging to hear from someone with IR experience, thanks. Agree completely.

dial481

·4 mesi fa·discuss

We audited the LoCoMo benchmark (one of the most cited eval for LLM agent memory) and found 99 score-corrupting errors in 1,540 questions (6.4%). Separately, we tested the LLM judge with adversarially generated wrong answers, it accepted 62.81% of vague-but-topical wrong answers. Some published system scores barely clear that bar. Full audit with methodology, all 99 errors documented, and reproducible scripts.