It highly depends on the task. For math and coding, sure. But for knowledge tasks
GPT-4 is wayy better than even SOTA ~100B models. For my knowledge test cases the lines get blurry at >400B
I applaud that you recently started providing the KL divergence plots that really help understand how different quantizations compare. But how well does this correlate with closed loop performance? How difficult/expensive would it be to run the quantizations on e.g. some agentic coding benchmarks?
I would be really interested in a podcast with the CEO where he goes a bit into the trade-offs of backwards and forwards compatibility. I can not imagine that their planning was so immaculate that there aren't any regressions that a clean slate design could have cleaned up. Nevertheless, amazing job for putting this together it looks like a phenomenal product!
They are heavily post-trained on code and math these days. I don‘t think we can infer that much about their behavior from just the pre-training dataset anymore
Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype).
On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past?
I was recently trying Qwen 3 Coder Next and there are benchmark numbers in your article but they seem to be for the official checkpoint, not the quantized ones. But it is not even really clear (and chatbots confuse them for benchmarks of the quantized versions btw.)
I think systematic/automated benchmarks would really bring the whole effort to the next level. Basically something like the bar chart from the Dynamic Quantization 2.0 article but always updated with all kinds of recent models.
I find it hard to trust post training quantizations. Why don't they run benchmarks to see the degradation in performance? It sketches me out because it should be the easiest thing to automatically run a suite of benchmarks
Small feedback if any of the Antigravity people read here: "Fast" is not a great name for the "eager" option (vs. "Planning") because "Fast" is associated with "dumb" in LLMs (fast/flash/mini). Probably "Eager" would be a more descriptive name
Mechanically sure, but I still feel way safer when a Tesla (of any kind) is approaching me as a pedestrian or bicyclist than any other vehicle (except maybe Waymo) because I know they will alert the driver and brake if necessary. Any other car, especially older trucks, I'm quite afraid of, based on experience.
Makes sense! I like that you guys are more open about it. The other labs just drop stuff from the ivory tower. I think your style matches better with engineers who are used to datasheets etc. and usually don't like poking a black box
Why did you stop training shy of the frontier models? From the log plot it seems like you would only need ~50% more compute to reach frontier capability
I have a strong Tinnitus on one ear after an ear surgery for 8 years now. And I usually don‘t notice it for months at a time, even though it is there all the time (thanks for reminding me :p)
So it’s not as bad as it might feel in the beginning.
I‘m mostly bothered by my hearing being generally impaired by it. It sits at ~9kHz but it somehow still makes it significantly harder to comprehend voices.
Theoretically, when the market offers me an order book and I take offers on one or the other side that should be totally fair? I think until execution/fill the information should be totally between me and the exchange and no one else, right? I get that if I send a limit order that can not be filled, that that affects the market because new information is introduced (before the trade) but in the previously described case all the information going out should be after the trade already happened, right?