Solar-10B(twitter.com)
twitter.com
Solar-10B
https://twitter.com/_philschmid/status/1734992933764411788
11 comments
I agree for the most part, but there is also the other side which is how useful it can be. If you fine tune your model on anime girls, it becomes a very useful model for generating anime girls and perhaps it's now worse at drawing stick figures or cave paintings, but a lot of people don't need the model to draw stick figures or cave paintings and if they did, they could get a model fine tuned to do that.
On the LLM side, there's a similar phenomenon where a huge amount of training could be on SEO spam tags, or /r/counting or producing nonsense or generating obfuscated C or reciting excerpts of Shakespeare word for word, etc.
If what you're measuring is generally more useful then you can end up with a better model using these methods, likely at the cost worse performance on things that you aren't measuring.
I would gladly take a 10b parameter coding assistant that doesn't know how to write in iambic pentameter or recite digits of pi, or translate words from Swahili to Turkish etc. but is much better at code completion.
On the LLM side, there's a similar phenomenon where a huge amount of training could be on SEO spam tags, or /r/counting or producing nonsense or generating obfuscated C or reciting excerpts of Shakespeare word for word, etc.
If what you're measuring is generally more useful then you can end up with a better model using these methods, likely at the cost worse performance on things that you aren't measuring.
I would gladly take a 10b parameter coding assistant that doesn't know how to write in iambic pentameter or recite digits of pi, or translate words from Swahili to Turkish etc. but is much better at code completion.
Totally valid points! However I think a lot of the real "magic" comes from the sort of cross-domain "thinking" that the larger models are capable of and I think that axis hard to benchmark. (though because it seems so hard to quantify, there's a high chance i'm making shit up)
Valid, but how much can you distill a model while retaining (or refining) usefulness? You don't need to know how to paint Rembrandt to draw an anime girl, and you don't need to know about the biological taxa of North American Ducks to write a for loop. The cross-over might come to a point where you might need to explain what "duck typing" is, but even then, you only need to know that a duck is something which quacks (What does "quack" mean? Who knows..)
If a model forgets how to speak French but gets much better at generating unit tests, that might be perfectly fine for the type of work we want the model to perform.
The problem is we can't easily know what the model "forgets" when it gets better at doing something else. The best thing we can do is benchmark/measure their output and hope that those benchmarks cover what users care about.
I suspect high quality benchmarks will quickly become almost as important as the tuning process itself.
If a model forgets how to speak French but gets much better at generating unit tests, that might be perfectly fine for the type of work we want the model to perform.
The problem is we can't easily know what the model "forgets" when it gets better at doing something else. The best thing we can do is benchmark/measure their output and hope that those benchmarks cover what users care about.
I suspect high quality benchmarks will quickly become almost as important as the tuning process itself.
I know the hugging face leaderboard isn't wildly accurate.
But the top models right now are almost all under 70B. Most are 7B, and the top is 10B. If the benchmarks are even remotely accurate then this is rather wild.
Apparently multiple groups found different "secret sauces", names upstage and whatever UNA is?
But the top models right now are almost all under 70B. Most are 7B, and the top is 10B. If the benchmarks are even remotely accurate then this is rather wild.
Apparently multiple groups found different "secret sauces", names upstage and whatever UNA is?
I mean this isn’t too surprising that smaller models do better. I imagine transformers are as prone to overfitting as any statistical data model. Also there is probably some selection bias: bigger models are more expensive and there are just less people training and iterating with them
There are orders of magnitude fewer people playing with large (>40B) parameter models than the small ones, which means even fewer people finetuning those models.
I can’t imagine this is anything but selection bias.
I can’t imagine this is anything but selection bias.
> which means even fewer people finetuning those models.
Finetunes rarely led to "Top 5 performance" for the small ones. Previously the top 10+ were all 70B, with maybe a few 30B in there. There were nearly no 13B's, let alone 7B.
The Zephyr-7b-β was one of the best 7B mistral 0.1 finetunes the past month and a half, and that didn't beat most 70B's.
Even at 7B there are few foundational models as even those take a relatively large amount of money. The only decent one for months has been 7B mistral which again didn't come that close to 70B performance.
Finetunes rarely led to "Top 5 performance" for the small ones. Previously the top 10+ were all 70B, with maybe a few 30B in there. There were nearly no 13B's, let alone 7B.
The Zephyr-7b-β was one of the best 7B mistral 0.1 finetunes the past month and a half, and that didn't beat most 70B's.
Even at 7B there are few foundational models as even those take a relatively large amount of money. The only decent one for months has been 7B mistral which again didn't come that close to 70B performance.
Is there a reason all these language models rarely have any indication of the amount of GPU ram required to run them?
I would guess that a) they assume their users will have a lot of GPU ram, b) actual running costs will depend on what inference engine/framework you're using (for example, some GGUF quants are very cheap).
It's because you can quantize to different extents, at the expense of a small amount of accuracy. The original model probably needs much more vRAM than most people will actually end up needing in prod.
I have tried a lot of these LLMs locally and I have rarely ever found the smaller models to outperform the bigger models when used casually/generically. The same is generally true for 1.5 vs SDXL the moment the prompt strays away from some variant of beautiful woman.