There were a few reasoning benchmarks that I noticed think they omitted a direct comparison since they weren't as competitive compared to GPT-4, and instead opted to just show the benchmarks comparing itself to other versions of PaLM or other language models
It's a good proxy for both. Both popularity and volatility have misleading statistical failure modes if measured purely by pull requests.
The author did narrow it down into what it mostly accurate represents though:
> Nevertheless, in my view, the number of pull requests is an important indicator of how much people are willing and capable of contributing to your software in the open source domain.
One of the worries I have is that the grand prize is so top weighted. It feels daunting to want to attempt to work on something like this if there's only one winner. If my team isn't the first to win it, but we contribute an interesting method, the most we can get is a 2000$ open source prize?
> making someone else's argument fit into your own system without misrepresenting it or making it unclear
I think this is where steelman is a superset of this, in that it includes the reconstruction definition but also includes making a whole new set of arguments that are entirely unrelated to your own argument or the other person's argument. i.e. Steelmanning can involve coming up with novel arguments for the other side.
It's not the same in that you can steelman a position and come up with brand new arguments that are better than what the other side is saying. "Reconstructing" doesn't necessitate the strongest form of the other argument.
HellaSwag: GPT-4: 95.3%, PaLM 2-L: 86.8%
MMLU: GPT-4: 86.4%, Flan-PaLM 2-L: 81.2%
ARC: GPT-4: 96.3%, PaLM 2-L: 89.7%
(from: GPT-4 paper: https://arxiv.org/pdf/2303.08774.pdf)