Next Grok model training with 10T parameter model(twitter.com)
twitter.com
Next Grok model training with 10T parameter model
https://twitter.com/i/status/2041754402239975479
4 comments
Isn't what the leading labs are currently chasing after is not pretraining and massive parameters but enriched and deep fine tuning and post training for agentic tasks/coding? MoE with just new post training paradigms lets smaller models perform quite well, and much more pragmatic to scale inference with. Given that, this choice seems super odd, as the frontier labs seem to stay neck and neck, and I don't even see Grok being used in any benchmarks because of how poorly it performs
This is the best publically posted model size, ever since top AI labs started treating model size as a trade secret. This should also guide next generation of inference ASICs.
Sounds more them marketing number. Carolien eutrucking
Between moE, aggressive quantization, and synthetic data pipelines, it’s getting harder to tell whether bigger models are actually better, or just more expensive to train.
Would be more interesting to see -> capability per dollar or per watt, not parameter count...