MemPalace's benchmark claims have been picked apart and debunked in the project's GitHub issues:

github.com/milla-jovovich/mempalace/issues/27 github.com/milla-jovovich/mempalace/issues/29 github.com/milla-jovovich/mempalace/issues/39 github.com/milla-jovovich/mempalace/issues/125 github.com/milla-jovovich/mempalace/issues/242

(and others)

TL;DR: When independent third parties ran actual end-to-end QA instead of retrieval metrics, the scores dropped dramatically.

dial481

·il y a 4 mois·discuss

That's encouraging to hear from someone with IR experience, thanks. Agree completely.

dial481

·il y a 4 mois·discuss

We audited the LoCoMo benchmark (one of the most cited eval for LLM agent memory) and found 99 score-corrupting errors in 1,540 questions (6.4%). Separately, we tested the LLM judge with adversarially generated wrong answers, it accepted 62.81% of vague-but-topical wrong answers. Some published system scores barely clear that bar. Full audit with methodology, all 99 errors documented, and reproducible scripts.