HackerTrans
TopNewTrendsCommentsPastAskShowJobs

dial481

no profile record

Submissions

[untitled]

1 points·by dial481·13 दिन पहले·0 comments

Show HN: Allelix – Annotate your DNA against 7 public databases offline

github.com
3 points·by dial481·28 दिन पहले·0 comments

[untitled]

1 points·by dial481·3 माह पहले·0 comments

Show HN: Proposal for a real long-term AI memory benchmark

penfieldlabs.substack.com
4 points·by dial481·3 माह पहले·0 comments

Milla Jovovich's MemPalace Claims 100% on LoCoMo. Its Benchmarks.md Disagrees

penfieldlabs.substack.com
4 points·by dial481·3 माह पहले·0 comments

LoCoMo AI Benchmark: 6.4% of answer key wrong, judge accepts 63% of fake answers

github.com
3 points·by dial481·4 माह पहले·3 comments

comments

dial481
·27 दिन पहले·discuss
This is badly needed. Jira is horrendous.

Can your plugin system support custom workflow triggers yet?
dial481
·28 दिन पहले·discuss
[flagged]
dial481
·28 दिन पहले·discuss
[flagged]
dial481
·3 माह पहले·discuss
MemPalace's benchmark claims have been picked apart and debunked in the project's GitHub issues:

github.com/milla-jovovich/mempalace/issues/27 github.com/milla-jovovich/mempalace/issues/29 github.com/milla-jovovich/mempalace/issues/39 github.com/milla-jovovich/mempalace/issues/125 github.com/milla-jovovich/mempalace/issues/242

(and others)

TL;DR: When independent third parties ran actual end-to-end QA instead of retrieval metrics, the scores dropped dramatically.
dial481
·4 माह पहले·discuss
That's encouraging to hear from someone with IR experience, thanks. Agree completely.
dial481
·4 माह पहले·discuss
We audited the LoCoMo benchmark (one of the most cited eval for LLM agent memory) and found 99 score-corrupting errors in 1,540 questions (6.4%). Separately, we tested the LLM judge with adversarially generated wrong answers, it accepted 62.81% of vague-but-topical wrong answers. Some published system scores barely clear that bar. Full audit with methodology, all 99 errors documented, and reproducible scripts.