I disagree, 10 years ago AIs nailing these types of competition would have been seen as very impressive. The fact goal posts can move on this now shows how much AI has progressed.
(Also the term “approximate retrieval” is a bad one - reasoning is inherently a process of chaining together associations. What matters is whether the reasoning reaches the right conclusions. Still some way to go, but already very impressive in tasks traditionally considered harbours of human reasoning!)
Thanks for feedback! Yes, we’re looking to improve quality in the coming months. Couple of notes:
- The initial use of data is distillation so we’re less bound by question quality (anything that evinces output diversity is good).
- But moving onto RL, we’ll need stronger quality. We have much better things planned both on data filtering and verification!
- Surprisingly, a lot of ML datasets actually look like this when you look under hood. We’re hoping having more eyeballs on it will help improve quality in long run over less transparent status quo!
No, the primary reason is that the UK is a heavily service-based economy. Quarantine impacts the service-based subset of the economy the most. Therefore it makes sense that the UK would be hard hit. A “nation of shopkeepers” suffers when the shops are closed...
I'm from Suffolk - my school used to go here on school trips. The funny thing about the site is that it's basically just a hill in a field now, so needless to say it was a massive letdown for eight-year-old me who was expecting to see a massive Viking longboat. The tourist centre is nice though.
I agree with your meta-point that better benchmarks testing more types of task would be good!