Signing this sounds like a good way to get fired. Executive in corporations gets to make the decisions. Employment is at will, if you don’t like it you get to leave otherwise you’re not fulfilling your contract
No, look a Composoer 2, it stands out starkly on its own in the pareto frontier on low cast and fast models.
Composer 2.5 was a huge leap with minimal compute from xAI.
They can compete with OpenAI and anthropic with xAI scale compute. They have a top notch model team and incredible training data and huge enterprise costumer contracts.
It’s a bit misleading to say nothing special, as they are doing more than just increasing parameter count. Progress has been steady in all the sub components of training from data filtering and weighting to sparse attention, optimizers to up and down the stack various efficiency in training computing.
They’re using more compute, a bigger model and tons of training quality improvements to get more out of an equivalent model.
This has been my thought for a long time. I think all that matters from attention is that there is crosswise comparison going on.
You need some amount of parallel compute and some amount of global comparison.
And the rest is basically a ways to parameters and scale.
(This is in theory, in practice you can get a lot of small % stability and efficiency improvements that really compound in algorithmic details of model architecture)
Confidently yes. OpenAI for sure has been training larger models internally and distilling.
Pre-training scaling laws all support larger models being more cost effeceint to train then smaller models. And distillation is comparably cheap. So you can get the most juice by training the biggest model you can and distilling it.
There is endless returns to frontier intelligence, just because most people can't make use of it doesn't mean someone can't make a ton of money off of it.
Most software engineers will just need cheap tokens.
But things like physics and drug discovery have no foreseeable upper bound.
There is endless returns to frontier intelligence, just because most people can't make use of it doesn't mean someone can't make a ton of money off of it.
Most software engineers will just need cheap tokens.
But things like physics and drug discovery have no forseeable upper bound.
There is step changes that actually merit this though. And a zero day machine IS one of those. It went from 4% zero day success rate to 85% on firefox.
Why does apple want to make this hardware hard to access?
What actual benefits do they get?
I guess they can have their own models run faster than the competition on their hardware? But they don't even really have anything that consumers use on the ANE as far as I can tell and local LLMs are taking off on macs and could really benefit from this
Multimodal models
Language models for art