HackerLangs
TopNewTrendsCommentsPastAskShowJobs

zone411

no profile record

Submissions

LLM Position Bias Benchmark: Swapped-Order Pairwise Judging

github.com
1 points·by zone411·vor 3 Monaten·0 comments

Show HN: Buyout Game Benchmark: Multi-Agent Bargaining, Transfers, and Takeovers

github.com
6 points·by zone411·vor 3 Monaten·0 comments

LLM Persuasion Benchmark: Multi-Turn Persuasion Between Models

github.com
9 points·by zone411·vor 4 Monaten·0 comments

Show HN: LLM Debate Benchmark

github.com
9 points·by zone411·vor 4 Monaten·3 comments

Show HN: LLM Sycophancy Benchmark: Opposite-Narrator Contradictions

github.com
3 points·by zone411·vor 4 Monaten·0 comments

Show HN: LLM Round‑Trip Translation Benchmark

github.com
6 points·by zone411·vor 10 Monaten·0 comments

Show HN: LLM Creative Story‑Writing Benchmark V3

github.com
8 points·by zone411·vor 10 Monaten·0 comments

comments

zone411
·vor 25 Tagen·discuss
Yes, definitely not a new idea. I had a multi-turn composite model in 2024 that was outperforming the top models across benchmarks: https://x.com/LechMazur/status/1828804485033992514.
zone411
·letzten Monat·discuss
That's not proof. Emergent intelligence is not consciousness.
zone411
·letzten Monat·discuss
I’ve tested this model on four of my benchmarks:

https://github.com/lechmazur/buyout_game 10th out 36.

https://github.com/lechmazur/pact/ 14th out 25.

https://github.com/lechmazur/nyt-connections/ 60th out 81.

https://github.com/lechmazur/debate 16th out of 29.
zone411
·vor 2 Monaten·discuss
100%. It's sad to see that this attitude has spread to HN
zone411
·vor 2 Monaten·discuss
I actually tried using GPT-5.5 Pro on this problem recently. It thought it was making progress on one path, but it made so many mistakes that it didn't feel worth it pushing further. It'll be interesting to check whether it's the same route. I got partial results (proved in Lean) that improve on the best-known results for four Erdős problems with GPT-5.5 Pro
zone411
·vor 3 Monaten·discuss
[flagged]
zone411
·vor 3 Monaten·discuss
https://variety.com/2020/digital/news/twitter-unblocks-new-y...
zone411
·vor 3 Monaten·discuss
I built this benchmark this month: https://github.com/lechmazur/sycophancy. There are large differences between LLMs. There are large differences between LLMs. For example, Mistral Large 3 and GPT-4.1 will initially agree with the narrator, while Gemini will disagree. I swap sides, so this is not about possible viewpoint bias in the LLMs. But another benchmark shows that Gemini will then change its view very easily in a multi-turn conversation while Kimi K2.5 or Grok won't: https://github.com/lechmazur/persuasion.
zone411
·vor 3 Monaten·discuss
I built two related benchmarks this month: https://github.com/lechmazur/sycophancy and https://github.com/lechmazur/persuasion. There are large differences between LLMs. For example, good luck getting Grok to change its view, while Gemini 3.1 Pro will usually disagree with the narrator at first but then change its position very easily when pushed.
zone411
·vor 4 Monaten·discuss
Hmm, maybe in the next edition, Opus gets expensive. I should probably run GPT-5.4 xhigh too if I do that for fairness...
zone411
·vor 4 Monaten·discuss
Rationalists were right about everything that mattered: crypto, AI, COVID... HN commentators, by contrast, were wrong about everything that mattered.
zone411
·vor 4 Monaten·discuss
Results from my Extended NYT Connections benchmark:

GPT-5.4 extra high scores 94.0 (GPT-5.2 extra high scored 88.6).

GPT-5.4 medium scores 92.0 (GPT-5.2 medium scored 71.4).

GPT-5.4 no reasoning scores 32.8 (GPT-5.2 no reasoning scored 28.1).
zone411
·vor 5 Monaten·discuss
I've made top-10 lists of LLMs' favorite names to use in creative writing here: https://x.com/LechMazur/status/2020206185190945178. They often recur across different LLMs. For example, they love Elara and Elias.
zone411
·vor 5 Monaten·discuss
They're improved compared to 4.5 on my Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/).

Sonnet 4.6 Thinking 16K scores 57.6 on the Extended NYT Connections Benchmark. Sonnet 4.5 Thinking 16K scored 49.3.

Sonnet 4.6 No Reasoning scores 55.2. Sonnet 4.5 No Reasoning scored 47.4.
zone411
·vor 6 Monaten·discuss
For people interested in these kinds of benchmarks, I have two multiplayer, multi-round games:

- Elimination Game Benchmark: Social Reasoning, Strategy, and Deception in Multi-Agent LLM Dynamics at https://github.com/lechmazur/elimination_game/

- Step Race Benchmark: Assessing LLM Collaboration and Deception Under Pressure at https://github.com/lechmazur/step_game/
zone411
·vor 7 Monaten·discuss
Scores 92.0 on my Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/). Gemini 2.5 Flash scored 25.2, and Gemini 3 Pro scored 96.8.
zone411
·vor 7 Monaten·discuss
I've benchmarked it on the Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/):

The high-reasoning version of GPT-5.2 improves on GPT-5.1: 69.9 → 77.9.

The medium-reasoning version also improves: 62.7 → 72.1.

The no-reasoning version also improves: 22.1 → 27.5.

Gemini 3 Pro and Grok 4.1 Fast Reasoning still score higher.
zone411
·vor 7 Monaten·discuss
I haven't looked in the logs for this in this particular project, but I've seen this occur frequently in my multiplayer benchmarks.
zone411
·vor 7 Monaten·discuss
I did some searches when I posted this project, but I didn't find any at the time.
zone411
·vor 7 Monaten·discuss
Without monitoring, you can definitely end up with rule-breaking behavior.

I ran this experiment: https://github.com/lechmazur/emergent_collusion/. An agent running like this would break the law.

"In a simulated bidding environment, with no prompt or instruction to collude, models from every major developer repeatedly used an optional chat channel to form cartels, set price floors, and steer market outcomes for profit."