Evaluating 55 LLMs with GPT-4(benchmarks.llmonitor.com)
benchmarks.llmonitor.com
Evaluating 55 LLMs with GPT-4
https://benchmarks.llmonitor.com/leaderboard
8 comments
I always find evals of this flavor offputting given that 3.5 and 4 likely share preference models (or at least feedback data)
Should be evaluating each prompt multiple times to see how much variance in the scores there are. Even gpt-4 grading gpt-4 should probably be done several times
Why no multi-turn evaluation? A lot of these benchmarks fail to capture the strength of ghost attention used in Llama 2 chat models.
GPT-4-0314 is top of the league table (ie. Not the latest version, but the version released in March).
Is this our Concorde moment?
Is this our Concorde moment?
Really cool thanks
If I did the same sort of thing but used Claude to grade the tests, would I get similar results? Or would that be inherently biased towards Claude scoring high?