HackerLangs
TopNewTrendsCommentsPastAskShowJobs

zone411

no profile record

Submissions

LLM Position Bias Benchmark: Swapped-Order Pairwise Judging

github.com
1 points·by zone411·3 months ago·0 comments

Show HN: Buyout Game Benchmark: Multi-Agent Bargaining, Transfers, and Takeovers

github.com
6 points·by zone411·3 months ago·0 comments

LLM Persuasion Benchmark: Multi-Turn Persuasion Between Models

github.com
9 points·by zone411·4 months ago·0 comments

Show HN: LLM Debate Benchmark

github.com
9 points·by zone411·4 months ago·3 comments

Show HN: LLM Sycophancy Benchmark: Opposite-Narrator Contradictions

github.com
3 points·by zone411·4 months ago·0 comments

Show HN: LLM Round‑Trip Translation Benchmark

github.com
6 points·by zone411·10 months ago·0 comments

Show HN: LLM Creative Story‑Writing Benchmark V3

github.com
8 points·by zone411·10 months ago·0 comments

comments

zone411
·26 days ago·discuss
Yes, definitely not a new idea. I had a multi-turn composite model in 2024 that was outperforming the top models across benchmarks: https://x.com/LechMazur/status/1828804485033992514.
zone411
·last month·discuss
That's not proof. Emergent intelligence is not consciousness.
zone411
·last month·discuss
I’ve tested this model on four of my benchmarks:

https://github.com/lechmazur/buyout_game 10th out 36.

https://github.com/lechmazur/pact/ 14th out 25.

https://github.com/lechmazur/nyt-connections/ 60th out 81.

https://github.com/lechmazur/debate 16th out of 29.
zone411
·2 months ago·discuss
100%. It's sad to see that this attitude has spread to HN
zone411
·2 months ago·discuss
I actually tried using GPT-5.5 Pro on this problem recently. It thought it was making progress on one path, but it made so many mistakes that it didn't feel worth it pushing further. It'll be interesting to check whether it's the same route. I got partial results (proved in Lean) that improve on the best-known results for four Erdős problems with GPT-5.5 Pro
zone411
·3 months ago·discuss
[flagged]
zone411
·3 months ago·discuss
https://variety.com/2020/digital/news/twitter-unblocks-new-y...
zone411
·3 months ago·discuss
I built this benchmark this month: https://github.com/lechmazur/sycophancy. There are large differences between LLMs. There are large differences between LLMs. For example, Mistral Large 3 and GPT-4.1 will initially agree with the narrator, while Gemini will disagree. I swap sides, so this is not about possible viewpoint bias in the LLMs. But another benchmark shows that Gemini will then change its view very easily in a multi-turn conversation while Kimi K2.5 or Grok won't: https://github.com/lechmazur/persuasion.
zone411
·3 months ago·discuss
I built two related benchmarks this month: https://github.com/lechmazur/sycophancy and https://github.com/lechmazur/persuasion. There are large differences between LLMs. For example, good luck getting Grok to change its view, while Gemini 3.1 Pro will usually disagree with the narrator at first but then change its position very easily when pushed.
zone411
·4 months ago·discuss
Hmm, maybe in the next edition, Opus gets expensive. I should probably run GPT-5.4 xhigh too if I do that for fairness...
zone411
·4 months ago·discuss
Rationalists were right about everything that mattered: crypto, AI, COVID... HN commentators, by contrast, were wrong about everything that mattered.
zone411
·4 months ago·discuss
Results from my Extended NYT Connections benchmark:

GPT-5.4 extra high scores 94.0 (GPT-5.2 extra high scored 88.6).

GPT-5.4 medium scores 92.0 (GPT-5.2 medium scored 71.4).

GPT-5.4 no reasoning scores 32.8 (GPT-5.2 no reasoning scored 28.1).
zone411
·5 months ago·discuss
I've made top-10 lists of LLMs' favorite names to use in creative writing here: https://x.com/LechMazur/status/2020206185190945178. They often recur across different LLMs. For example, they love Elara and Elias.
zone411
·5 months ago·discuss
They're improved compared to 4.5 on my Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/).

Sonnet 4.6 Thinking 16K scores 57.6 on the Extended NYT Connections Benchmark. Sonnet 4.5 Thinking 16K scored 49.3.

Sonnet 4.6 No Reasoning scores 55.2. Sonnet 4.5 No Reasoning scored 47.4.
zone411
·6 months ago·discuss
For people interested in these kinds of benchmarks, I have two multiplayer, multi-round games:

- Elimination Game Benchmark: Social Reasoning, Strategy, and Deception in Multi-Agent LLM Dynamics at https://github.com/lechmazur/elimination_game/

- Step Race Benchmark: Assessing LLM Collaboration and Deception Under Pressure at https://github.com/lechmazur/step_game/
zone411
·7 months ago·discuss
Scores 92.0 on my Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/). Gemini 2.5 Flash scored 25.2, and Gemini 3 Pro scored 96.8.
zone411
·7 months ago·discuss
I've benchmarked it on the Extended NYT Connections benchmark (https://github.com/lechmazur/nyt-connections/):

The high-reasoning version of GPT-5.2 improves on GPT-5.1: 69.9 → 77.9.

The medium-reasoning version also improves: 62.7 → 72.1.

The no-reasoning version also improves: 22.1 → 27.5.

Gemini 3 Pro and Grok 4.1 Fast Reasoning still score higher.
zone411
·7 months ago·discuss
I haven't looked in the logs for this in this particular project, but I've seen this occur frequently in my multiplayer benchmarks.
zone411
·7 months ago·discuss
I did some searches when I posted this project, but I didn't find any at the time.
zone411
·7 months ago·discuss
Without monitoring, you can definitely end up with rule-breaking behavior.

I ran this experiment: https://github.com/lechmazur/emergent_collusion/. An agent running like this would break the law.

"In a simulated bidding environment, with no prompt or instruction to collude, models from every major developer repeatedly used an optional chat channel to form cartels, set price floors, and steer market outcomes for profit."