There’s lots of work on distillation, smaller models, approximations, etc. People already have simpler forms of these running on smartphones. Models seem to be growing faster than we can make them small though :D
"$50,000 was divided among the human participants based on their performance to incentivize them to play their best. Each player was guaranteed a minimum of $0.40 per hand for participating, but this could increase to as much as $1.60 per hand based on performance."
So the humans weren't betting their own money, but they still made more money if they won.
I mean, that’s the thing though. Undergrad in state tuition plus R&B for NCSU and UTexas (programs I got in state tuition for) is at 9k and 10k per year with 5% interest for undergraduate, and more like 6-8% for MS students.
You can be employed in a high paying sector like tech after going to an in-state engineering school, and still l end up with $2k tacked on in your first year of employment. So then what is more rational: pay off the loan as fast as possible, or build up an emergency fund with a generous 2% ROI.
With the cost of education, healthcare, and housing all rising significantly faster than inflation, it's possibly just harder for many to save up that emergency fund. Indeed, it's particularly difficult to justify having an emergency fund if you're also facing five figure 8% interest student loans.
Yann LeCun did not, otherwise he’d be a coauthor. As it is, this was a collaboration between NYU and Facebook AI Research, with multiple authors working at both institutions.
I don’t know man, 20% of your annual income would be seen as a sizeable fine. That’s 20% of their yearly profits, and it wipes out most of their earnings for Q1.
It’s indeed a very high trials to success ratio. Again though, there’s enough papers preceding this one that you could have good confidence in the effort. Another thing that helps is orgs like OpenAI have their own servers, rather than renting ec2 instances.
You also don’t just launch that many things and them ignore it. You monitor it to make sure nothing is going terribly wrong.
But yeah there’s also the fact that if you’re Google, throwing $2m worth of compute at something becomes worth it for some reason (eg Starcraft)
There’s a natural way to parallelize these models so that using 128 GPUs is the same as a 128x batch size. You can similarly simulate 128x batch size by accumulated gradients before backpropping. So you can test on just one or a few GPUs before you run the full thing.
By that point you know it’s going to work, it’s just a matter of how well and whether you could’ve done nominally better with different tuning.
There’s been enough research leading up to this paper to suspect that just scaling larger would play out.
