I agree. But I think you're missing that LLMs can internalise a lot of the thinking process in their layers without explicit CoT. That System 1-style reasoning is bounded depth computation but very, very broad. Yudkowsky called it "cached thoughts" and I think it's an incredibly important idea [1]. It's really stiking how the best LLMs don't even need to think where smaller LLMs do.
So as more thinking is cached in their weights through increased RL training, those weights are doing more useful work and the efficiency is increasing.
> whether these exams are testing knowledge that is still worth internalizing ... It is not clear from the article exactly how much of this course falls into that category
It's very clear from the (excellent) article linked by dang [1] what the exams required:
> This year, the economist decided that both the midterm and the final exams for his course would be of the take-home, closed-book type (there is a certain tradition of this at Ivy League schools). “It’s a very nice kind of exam, because as you’re giving students practically unlimited time to complete it, it lets you make it harder than normal, to see how far they can go.” In this case, Serrano changed some of the model assumptions they had seen in class, and asked students to demonstrate whether certain statements were true or false under the new assumptions.
> The majority of people with this mindset should not go to college
Seems obvious now that this is the solution. Unfortunately, universities would never agree to downsize, not due to politics or any US-specific problems.
Being serious. "Think about what this might mean..." Finding unexpected links between various ideas and background knowledge. What we call a "novel idea" is virtually always the repurposing of an idea/concept in a new context. A system that maps arbitrary inputs into abstraction spaces in which similarities are discoverable, such as say a deep learning system, is perfect for this.
Nice, I'm really interested in using this for simple semantic search in a native desktop application.
Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)".
But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 400/sec. Is there an issue with falling back to a non-SIMD path? I'll try a native Rust binary next.
Isn't it strange, how some incredibly crude game written in the 80's will probably live forever in an archive somewhere, while the sophisticated things we create today are amazing but too numerous to be preserved or remembered. Working on a rollback netcode system as well currently.
I'm pretty busy, so I only skimmed the article, but it's actually really interesting, and also informative as I'm not familiar with diffusion models. Maybe I'll some ask questions/write later. I do want to encourage you, but, honestly the websites are a bit over the top and there's no way to know how much human input actually went in to them.
Experimental science is very messy, as you've learnt. Agreed with the other commenter, there's value (for others and especially yourself) in writing down what went wrong, and the things in the "Small models cannot judge themselves" is so reminiscent of failure modes I've experienced myself. There are usually awful or subtle bugs, training just doesn't work, and even if the results are "interesting" rather than "bad", it can still be incredibly difficult to decide what to conclude from them. To distill knowledge from observations/experiments is the problem of science. You read papers about experiments seem neat and the results profound, but the truth is they're probably a mess too and the evidence for the conclusions is probably a lot weaker than it looks; ML experiments can be unreproducible too.
I suggest that you were running experiments at too large a scale given your resources: you should try to sort out these critical issues on a smaller scale. Yes, the painful problem with ML is that things change qualitatively with scale, you just don't know if a larger scale will fix your issues. But most of these bugs didn't need scale to discover. Think about how you could have more easily discovered them.
Sorry to tell you that your comment was dead (silently blocked, invisible to most users) until I vouched for it. Don't be discouraged from posting on HN. Clearly both you're a real person, and you wrote this with an LLM (quite understandably), but people are really put off by text that smells LLM generated, and it's really easy to tell. HN is flooded with LLM comments lately, they go dead. You can use an LLM to help write, but don't let it determine the content, be genuine, and make sure it doesn't read like one. They can write in any style.
Fable/Mythos are based on the same model. Not totally clear whether they have identical weights (just different external guardrails), or there's also some slight finetuning difference.
MiMo Code adds a lot of cool orchestration features to OpenCode! It definitely is NOT a quick find-replace job, it's genuinely someone's research project to create a better agent harness building on top of free software, and that's awesome. See https://mimo.xiaomi.com/blog/mimo-code-long-horizon
> Sometimes I'll have Claude implement a feature one way, then have GPT do it the other way, then have them both review each other's implementation. Then synthesize a final plan from the previous implementations+reviews.
I've done variants of this a number of times, but feel like it was a generally waste of my time to then have to compare them and write up which parts I liked or disliked: if the output is something substantial, each will have its pros and cons. Clear-cut wins aren't very common. Of course it could work well if we automated the whole thing with an orchestrator; you just need a model with actual good taste (according to your own preferences) ... so we'll have to compare all the models to find that one
Yes you are correct. That's one meaning of primary. I just think it's misleading by the more colloquial meaning to say "primarily a US company" when AFAIK basically all the engineering happened in the NZ subsidiary of what was initially more-or-less a USA shell company. Engine manufacture was the first thing they moved to the USA that I know of, after one or two launches IIRC.
Aside, Peter Beck has said (probably at a talk I was at in 2014) that they initially designed and built Electron in NZ so that they would be importing restricted technology into the USA, to minimise ITAR problems, which only covers exports.
That is false. They were a purely NZ operation launching sub-orbital rockets before they got into DARPA contracts. What you meant to say is "Before they ever launched Electron", and I'm pretty sure that is false too, they weren't "primarily" American, the majority of the workforce was in NZ until years after that.
Hi Scott! Was just considering signing up, NW looks great (fp8 GLM 5.2 is good!) Standard cached token pricing for GLM 5.2 is pretty high, I'm wondering whether the KV cache for that model actually is that expensive to serve on average, or if Neuralwatt's energy pricing for long-running GLM 5.2 agents is especially competitive? The live energy stats don't break down by token type, would love to see that. And 2/3 of the examples given in docs/energy-methodology are models you don't even host anymore. Uncertainty and selective stats puts people off signing up, they tend to assume the worst. Oh, and MiMo or DS4 please :)
I've also worked extensively on ARC AGI 1/2, and I mainly agree. Marketing and training. Performance of LLMs on ARC is most importantly a function of training on grid/table-like data. It doesn't have to be specifically synthetic ARC data though. Training an LLM to be better at perceiving grid-like arrangements of data in a spatial way like an image, rather than just tabular, is hugely useful for things outside of ARC benchmarks, though it's a narrow skill. Hence, I'm sure they do it. I want them to do that. I believe the labs when they say they didn't train specifically for ARC-AGI 1/2 (where did Google say otherwise? I don't see it). But it does not mean the models are getting better at general purpose reasoning. They were already plenty good enough at that. You can describe ARC images in words and reason about it using a level of intelligence LLMs have had for years: they're designed to be easy! LLMs just couldn't reason about image-like grids very well.
This explains a lot. But you merely need to look into the family of spice forks to realise, given the way that they're strangely limited to certain operating systems and embedded inside certain proprietary IDEs, that's there's something very wrong with the code architecture.
[email protected]