Same goes for money: It enables greedy jerks to be more greedy and more of a jerky, and it enables people who e.g. voluntarily donated already to do much more in that direction, too.
It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process).
“Good” benchmarks to gauge development skills at the moment seem to be:
DeepSWE [0] by Datacurve
FrontierCode [1] by Cognition
And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore.
But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again.
Thanks for the link (and underlying thoughts), I really hadn’t considered that.
So essentially, due to technological progress and other factors inducing price collapses (or at least cycles), you can’t start stockpiling insane amounts of finished-product semiconductor, which means you can’t scale production at current technology levels to infinity either?
Unless the raw materials have an inherent limit on mining/production due to the amount present on the planet, why should or would companies not ramp up to eventually meet demand?
Edit: Okay, this doesn’t mean that that’s actually possible in the short-term, so I think you’re right. But that means as the silver lining, in the medium term horizon there’ll be enough supply again? :’)
Genuinely thank you for linking to him and his case.
I live in Germany and am -principally- a massive advocate for and proponent of the free (or liberal) democratic basic order ("FDGO" [0]) we have had here for the last decades, and apart from Chat Control, I’m usually very highly pro-EU, too.
But reading this has genuinely left me in a bit of a shock now, and created some (for lack of a better word) FUD I haven’t felt before with regard to these two, eh, institutions governing us.
While that's true and pretty funny, it does make for a meaningful distinction IMO: Just like you wouldn't look at the "all Ds" student, but not the "straight A" student in order to get an impression of students' capabilities, you wouldn't ignore a newer, potentially SOTA-pushing model to then come out and say something along the lines of "all models suck"… right?
> Has this been just pure lack of funding and infra?
For the most part, yes.
France and Germany are the two biggest EU economies. France has well, Mistral, and we here have a government-funded VC entity that is way too proud [1] to be able to offer a whopping… €125 million (<$150 M USD) for helping European researchers achieve new SOTA in sovereign models. And that sum is not even going to a single challenge winner, it'll be split up among multiple recipients. Don't get me wrong, this is a cool first step, or rather, would have been one about three to four years ago.
> However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs.
Exactly. While I’m assuming this won’t be news for most here, for those that are still new and/or curious about some more explanation on e.g. the floating-point imprecisions, see this nice article: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
The weights they “etched” into the FPGA card that’s used for the ChatJimmy demo are that of a Llama 3-something 8b model.
The actually impressive and novel thing is that Taalas’ve managed to automate that process (clearly – nobody transforms 8 billion numbers into a physical representation by hand).
So now, they can work on scaling this process up, and with low enough lead times (I’ll be convinced they have inside connections to TSMC if they can actually deliver on the promised mere 3-4 months delay), will be able to offer 30-100b+ parameter models under half a year after they’re released, at thousands of tokens per second while probably drawing less wattage (per token, not sure about overall).
Seems anecdotal and/or a recent change. OpenRouter doesn't have my phone number either, but I do have an account. And I didn't sign up with them that long ago.
Correct, and/or in addition, most nowadays prepend something like `set -euo pipefail` to the scripts in the line immediately after the shebang which results in stopping on errors, including things such as syntax errors stemming from e.g. incomplete installer transmission over wire.
(At least for bash scripts, I’m not sure whether these are POSIX syntax to be frank.)
> it hallucinates like crazy but looks like its by design to boost benchmarks.
Wasn’t there a discussion around some new-ish benchmark _punishing_ hallucination answers (over not replying at all) recently?
Maybe in the not-so-distant future, this “spam replies until one’s correct” strategy won’t be able to game a benchmark much at all anymore.
Somewhat doubtful. (The first part of your statement, anyway. Of course the difference between abstract "value" and hard, spendable "money" is a thing.)
Like Mr. Hanson said in my sibling comment, some rulers are (or were!) bound to have amassed incredible amounts of resources. For historical/non-present-day examples, consider looking into figures like Jakob Fugger or Mansa Musa.