HackerTrans
TopNewTrendsCommentsPastAskShowJobs

acegod

no profile record

Submissions

Problems in LLM Benchmarking and Evaluation

xent.tech
14 points·by acegod·11 miesięcy temu·4 comments

comments

acegod
·9 miesięcy temu·discuss
Serious question: why is watching YouTube shorts a worse way to spend your time than eg debugging CUDA problems?

The answer is "opportunity cost". But who really believes in that?

I call it the batman fallacy. Many people (young men in particular) say to themselves "if I was more disciplined, I could dedicate my whole life to martial arts (or programming, or art, or w/e) and become batman (or John Carmack, or Van Gogh)". But it's not true, of course.

And it's the same with many managers. "Instead of spending x% on task A and y% on task B, why dont you spend (x-z)% on A and (y+z)% on B?" It's absurd.

Brute attempts to capture opportunity costs are doomed to fail. You squeeze one end (block youtube shorts) and it comes out the other (eg you argue with coworker). It's really much easier to stop punishing yourself for lost time and find happiness in who and where you are.
acegod
·w zeszłym roku·discuss
Right, that's exactly what I'm saying.

>The point is to measure fluid intelligence in a way which supports comparisons between models and between models and humans. It's not the obligation of the test to be tailored to the form of model that's most popular now.

The problem is that the test may not be giving an accurate comparison because the test is problematic when used to assess LLMs, which are the kind of model that people are most interested in assessing for general capabilities.
acegod
·w zeszłym roku·discuss
[dead]
acegod
·w zeszłym roku·discuss
I agree with this article, however I think it misses the most valuable aspect of speed: compounded returns on experience.

Experience isn't 1:1 with time spent. It's easy to spend a lot of time on something but learn very little. Conversely, its possible to gain a large amount of experience in a short period of time.

By being able to develop faster, you become able to accumulate more experience in a smaller amount of time. This experience then enables you to develop faster, kicking off a virtuous cycle of growth.

Following this thought provides clarity on what action you should take immediately: do anything as long as its something. Through doing something, you will become more experienced and that experience will enable you to do something else even faster.

This is why you end up with so many aphorism in the industry promoting rapid action over inaction:

- Move fast and break things

- Worse is better

- Fail fast

- Hacker mentality

- "Action oriented"

See also: https://danluu.com/productivity-velocity/ and https://patrickcollison.com/fast