You don't have to train from scratch but you can. Distillation ends up being somewhere in the ballpark of 1000x faster to train [1]. It also comes with the huge advantage of not needing to create RLHF datasets, since you can just copy the behavior of the teacher model. This saves an enormous amount of labeling money at the cost of making the model behave similarly to the teacher. If you are training from scratch, you can look at LLM scaling laws to figure out roughly the compute budget you need to optimally train a model [2].
Based on [2] a 30B model needs something like 2e+23 FLOPS to train from scratch whereas a 1.6T model needs something like 1e+27 FLOPs to train. So DeepSeek v4 Pro was roughly 5000x more expensive to train than this model. I'm not totally sure how MOE affects scaling laws, so these numbers might be different in reality, but it gives you a good ballpark estimate of the difference in training scale.
It depends on the facilities the language offers to you by default right?
C++ offers much higher level primitives out of the box compared to Zig, so I'd say its a higher level language. Of course you can ignore all the features of C++ and just write C, but that's not why people are picking the language.
I agree with you that package management has nothing to do with how low-level a language is.
That being said Rust is definitely a much higher level language than either C or Zig. The availability of `Arc` and `Box`, the existence and reliance on `drop`, and all of `async` are things that just wouldn't exist in Zig and allow Rust programmers to think at higher levels of abstraction when it comes to memory management.
> Having a rich standard library isn't just a pure positive. More code means more maintenance.
I would argue it's much worse to rely on packages that are not in the standard library since its harder to gain trust on maintenance and quality of the code you rely on. I do agree that more code is almost always just more of a burden though.
The languages trade complexity in different areas. Rust tries to prevent a class of problems that appear in almost all languages (i.e two threads mutating the same piece of data at the same time) via a strict type system and borrow checker. Zig won't do any of that but will force you to think about the allocator that you're using, when you need to free memory, the exact composition of your data structures, etc. Depending on the kind of programmer you are you may find one of these more difficult to work with than the other.
Rust is a world away from Zig as far as being low-level. Rust does not have manual memory management and revolves around RAII which hides a great deal of complexity from you. Moreover it is not unusual for a Rust project to have 300+ dependencies that deal with data structures, synchronization, threading etc. Zig has a rich std lib, but is otherwise very bare and expects you to implement the things you actually want.
This has not been my experience with Waymo. I drove a total of about ~3.5 hours in Waymos in LA when I was visiting and their robustness to very unusual situations absolutely floored me.
I am sure you can find truly out-of-distribution cases where the car will make a mistake, but the data shows that this is more rare than a human driver making a mistake.
GPT2 was definitely a risk, just not of the same magnitude. It would have (and did!) make social media bot farms way more convincing and widespread. There was specific worry about that being used to sway elections, which is why they held back the model.
I don't really understand what you mean by this. The claim is that the same prompt with the same question produces worse results when it's queried in a model that has more than 200k tokens in its context. That doesn't have to do much with the "skillfulness" of using a model.
I think that's a fair assessment. To your point `cbindgen` makes the mechanics of the whole thing painless & linking was trivial. That's worth a lot especially when compared to other languages.
I have to disagree here a little bit. Calling C functions from Rust is a very pleasant experience, but the other way around is not so nice. You usually have to manually create types that will unpack rust collections into C compatible structures (think decomposing `Vec` into ptr, len, capacity) & then ensure that memory passed between the two sides is free'd with the appropriate allocator. Even with `cbindgen` taking care of the mindless conversions for you, you still have to put a lot of thought into the API between the two languages.
I am currently working on a fairly involved C & Rust embedded systems project and getting the inter-language interface stable and memory-leak free took a good amount of effort. It probably didn't help that I don't have access to valgrind or gdb on this platform.
Likewise Ken Liu (the English translator for the Three Body Problem) has a really good short story "The Perfect Match" about the same concept, which you can read here: https://www.lightspeedmagazine.com/fiction/the-perfect-match... It was the first thing that came to mind when I read this announcement.
Based on [2] a 30B model needs something like 2e+23 FLOPS to train from scratch whereas a 1.6T model needs something like 1e+27 FLOPs to train. So DeepSeek v4 Pro was roughly 5000x more expensive to train than this model. I'm not totally sure how MOE affects scaling laws, so these numbers might be different in reality, but it gives you a good ballpark estimate of the difference in training scale.
[1] https://arxiv.org/abs/2505.12781 [2] https://arxiv.org/abs/2203.15556