Language models are Super Mario: Absorbing abilities from homologous models(arxiv.org)
arxiv.org
Language models are Super Mario: Absorbing abilities from homologous models
https://arxiv.org/abs/2311.03099
60 comments
I tend to think there's an explore/exploit trade-off in model scale. Animals, including humans, have lots of extra neurons when they are young, which are then shed once the learning phase settles down. And this makes sense: thinking is energy intensive, so it's more efficient to sparsify. And, of course, we see a similar dynamic in ML: you can train a big model then prune it and do much better than you would by just training the smaller model directly.
I've got some geometric handwaving for why this works, as well. It's easier to find a low-energy solution when you have more parameters... Sparse solutions are higher energy, and thus require longer walks (and more commitment) during training.
I've got some geometric handwaving for why this works, as well. It's easier to find a low-energy solution when you have more parameters... Sparse solutions are higher energy, and thus require longer walks (and more commitment) during training.
Sounds about right! If you take an analogy from poker, bankroll matters - some strategies you can only play well-capitalized.
Do you know if there is anybody who has made it their mission to shrink models to extremes? It feels like the sorta thing somebody would get really obsessed with doing, akin to those people who make executable out of a few bytes or shrink network payloads.
Some time ago, Andrej Karpathy posted a YouTube video about building GPT from scratch. Using the same dataset (a 1MB text file with all of Shakespeare’s works), the video goes from a very basic model to something like GPT2
What really stood out to me is that, while the typical wisdom is “more data -> better model”, in the video they use the exact same data for all the models, so all the gains are achieved only through improved algorithms and computation power
With that in mind, I wonder if at some point someone will figure out an algorithm for a model that can be trained on just an English dictionary and get a decent LLM-equivalent model that can have a basic conversation. Given the small size of the data (a dictionary), I assume the model would be pretty small and quite fast to run, even on older or smaller machines
What really stood out to me is that, while the typical wisdom is “more data -> better model”, in the video they use the exact same data for all the models, so all the gains are achieved only through improved algorithms and computation power
With that in mind, I wonder if at some point someone will figure out an algorithm for a model that can be trained on just an English dictionary and get a decent LLM-equivalent model that can have a basic conversation. Given the small size of the data (a dictionary), I assume the model would be pretty small and quite fast to run, even on older or smaller machines
This video? https://youtu.be/kCc8FmEb1nY?si=QR-ADr0_Y6_P2QxC
Yes, thank you, that’s the video
Not for DL models, but I was exploring doing this in a specific setting (I've posted about this earlier). For small-sized models (for some reasonable definition of size, e.g., depth of a decision tree, or # trees in a gradient boosting forest, or # non-zero coefficients in a linear model), I realized that you could make them even smaller while retaining their accuracy by selectively presenting training data to them, i.e., ignore some training data points, repeat certain others. See [1] - the x-axis shows the original model size and the y-axis shows the model size obtained by this process at the same or better accuracy.
Interestingly, this meant that the conventional wisdom that the test and train distributions have to be identical for optimal held-out performance, is not true at small model sizes. It is true as models grow larger - and I was explicitly able to show this. See [2] - where the x-axis is model size, and the y-axis measures (on a scale of 0-1) how close is the optimal training distribution to the test distribution. The different lines are for different datasets.
These images are from the paper here [3]. I have a library too [4]; that is in need of updates - it works today as-is though, but please use the latest minor release if this of interest [5]!
[1] https://imgur.com/a/NheK49Z
[2] https://imgur.com/a/N53GeNI
[3] https://arxiv.org/pdf/1906.06852.pdf
[4] https://compactem.readthedocs.io
[5] As of writing the install command should be `pip install compactem==0.9.9rc2`
Interestingly, this meant that the conventional wisdom that the test and train distributions have to be identical for optimal held-out performance, is not true at small model sizes. It is true as models grow larger - and I was explicitly able to show this. See [2] - where the x-axis is model size, and the y-axis measures (on a scale of 0-1) how close is the optimal training distribution to the test distribution. The different lines are for different datasets.
These images are from the paper here [3]. I have a library too [4]; that is in need of updates - it works today as-is though, but please use the latest minor release if this of interest [5]!
[1] https://imgur.com/a/NheK49Z
[2] https://imgur.com/a/N53GeNI
[3] https://arxiv.org/pdf/1906.06852.pdf
[4] https://compactem.readthedocs.io
[5] As of writing the install command should be `pip install compactem==0.9.9rc2`
Ooh, parameter golfing! maybe I will get into ML one day after all.
On a similar note, I once found a paper that explained artificial neural networks computability from a very tiny ground up method, showing how few pieces you need to build a NAND gate... and of course once you have that you have everything.
On a similar note, I once found a paper that explained artificial neural networks computability from a very tiny ground up method, showing how few pieces you need to build a NAND gate... and of course once you have that you have everything.
> For example instead of training an 8x7b mixture of experts then merging, just incorporate the sparsity constraint while pre-training a single 7b model (somehow).
I'm thinking it would help reduce the network demands for gradient updates if merge from time to time. That could unlock distributed training, like SETI@Home.
I'm thinking it would help reduce the network demands for gradient updates if merge from time to time. That could unlock distributed training, like SETI@Home.
Merging is still wild to me.
I naively merged a Dolphin fine-tune of Mistral 7B base 0.2 with Mistral 7b Instruct 0.2 and got a model that gets higher benchmark results:
https://huggingface.co/ichigoberry/pandafish-2-7b-32k
Took a few minutes in colaboratory.
I naively merged a Dolphin fine-tune of Mistral 7B base 0.2 with Mistral 7b Instruct 0.2 and got a model that gets higher benchmark results:
https://huggingface.co/ichigoberry/pandafish-2-7b-32k
Took a few minutes in colaboratory.
Yeah it's wild to me too, it feels like we are doing alchemy rather than programming, mixing intelligence out of a cauldron.
This whole field feels like magic, yet there are mathematical underpinnings behind all of it.
I don't know why everyone isn't in love with this field. It's so utterly fascinating.
I don't know why everyone isn't in love with this field. It's so utterly fascinating.
I'm not in love with it because it's magic. Not to say it isn't interesting, certainly it is, but I don't want my life ruled by magic.
Predictability and reproducibility and a provenance of logic are important for computing systems and society as a whole.
Predictability and reproducibility and a provenance of logic are important for computing systems and society as a whole.
I can see that. Unpredictability is the root of much anxiety. In academia, I'm the opposite. I'm thrilled by mystery and the unknown. It gives me this deep sense of profundity and gravitas bordering on religious experience. I'd love to learn more about that phenomenon. Anyway, it reminds me of that famous Newton quote:
"I do not know what I may appear to the world, but to myself I seem to have been only like a boy playing on the sea-shore, and diverting myself in now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me."
"I do not know what I may appear to the world, but to myself I seem to have been only like a boy playing on the sea-shore, and diverting myself in now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me."
In academia, I'm the opposite. I'm thrilled by mystery and the unknown.
Oh if I was in academia, I'd be positively giddy. So much to explore. Fascinating stuff.
But from my perspective, I don't like building houses on shaky ground. And I especially don't want to live in an economy based on it.
Oh if I was in academia, I'd be positively giddy. So much to explore. Fascinating stuff.
But from my perspective, I don't like building houses on shaky ground. And I especially don't want to live in an economy based on it.
That's certainly an interesting viewpoint on life, but I suspect it's a very rare one simply because most people would never have considered it and those who had would tend towards the curious.
You seem to be trapped in the interminable middle.
You seem to be trapped in the interminable middle.
I don't feel trapped. There is going to be (and has been in ways) a small reckoning when people realize magic (ai) and all its unpredictability is very difficult to manage.
There is no field of "probabilistic UX" for example. How do you provide a consistent user experience when the underlying engine of your application is inconsistent?
Same goes for QA, testing, root cause analysis.
Adding features can have exponential side effects that cannot be predicted, which can be deadly at scale. Both figuratively and literally depending on how the technology is adopted.
There is no field of "probabilistic UX" for example. How do you provide a consistent user experience when the underlying engine of your application is inconsistent?
Same goes for QA, testing, root cause analysis.
Adding features can have exponential side effects that cannot be predicted, which can be deadly at scale. Both figuratively and literally depending on how the technology is adopted.
> small reckoning when people realize magic (ai) and all its unpredictability is very difficult to manage.
Deploy it in the right places first. Most people don't realize it's the arts where this works the best. They're too focused on LLMs and reasoning, but the first verticals that work will all be image, video, audio, and games.
If you're imperfect, the human creator driving the creation can easily repair it. Nobody dies, no business is lost. Millions of hours are saved. Large capital intensive businesses get disrupted and democratized.
Self-driving cars will be last.
Deploy it in the right places first. Most people don't realize it's the arts where this works the best. They're too focused on LLMs and reasoning, but the first verticals that work will all be image, video, audio, and games.
If you're imperfect, the human creator driving the creation can easily repair it. Nobody dies, no business is lost. Millions of hours are saved. Large capital intensive businesses get disrupted and democratized.
Self-driving cars will be last.
> Deploy it in the right places first.
100% agreed. Unfortunately that’s not what we’re seeing.
100% agreed. Unfortunately that’s not what we’re seeing.
> There is no field of "probabilistic UX" for example. How do you provide a consistent user experience when the underlying engine of your application is inconsistent?
Sure there is. Every single in person store you go into and how staff are trained to interact with customers. It’s hard, and yet there are clear experts in it.
Sure there is. Every single in person store you go into and how staff are trained to interact with customers. It’s hard, and yet there are clear experts in it.
Is that a 1:1 comparison? There's a considerable difference between human-human interaction and human-computer interaction.
Also consider human-horse interaction. The horse was quite useful for most of human history, despite not being perfectly reliable.
Please explain yourself, otherwise this comes off as an unhelpful non-sequitur.
Mate, why would you need this explained in detail to you? You claimed that there is no field of probabilistic UX and that it would be difficult to "provide a consistent user experience when the underlying engine of your application is inconsistent" -- the other posters just gave you two highly analogous examples where humans have been dealing with inconsistent UX for thousands of years.
I guess I do need it explained. What is the horse in this situation?
> humans have been dealing with inconsistent UX for thousands of years.
Like I said: between humans.
Or animals. But do you believe the average consumer is going to put in the equivalent amount of effort it takes to break and ride a horse?
> humans have been dealing with inconsistent UX for thousands of years.
Like I said: between humans.
Or animals. But do you believe the average consumer is going to put in the equivalent amount of effort it takes to break and ride a horse?
He's saying all minds are probabilistic ux. We handle them fine, and built whole worlds on them.
The predictability and determinism of digital system is the detour that is perhaps the flash in the pan. My sense is that the information ecology is about to get MUCH more ecological and fuzzy...
(I appreciated the exchange in this thread fwiw.)
The predictability and determinism of digital system is the detour that is perhaps the flash in the pan. My sense is that the information ecology is about to get MUCH more ecological and fuzzy...
(I appreciated the exchange in this thread fwiw.)
Differential equations are mathematically quite simple, yet they are mostly intractable and the systems they describe chaotic and unpredictable. The mathematics in AI works similarly, being deceptively simple yet giving birth to incredible complexity
I think it's a bit like saying microcode in CPU underpins high-level programming languages - kind of, but good luck understanding what a program does based on its microcode translation. OTOH you can mess with microcode and maybe get better performance, but you won't really know before you try. It's similar with ML - much more like alchemy than science...
Lots of reasons I suppose. Outside script-kiddying with mergekit it takes some serious knowledge to properly train anything, and expensive amounts of compute to actually do it or even run it in the end. It's not the most accesible thing.
For classical methods once you got the algorithm nailed, it will work 100% of the time. For probabilistic methods, you do get better results most of the time but they can also screw up randomly for no reason so their deployability in production is hell on wheels. It's infuriating at times.
Still can't argue against it being very fascinating.
For classical methods once you got the algorithm nailed, it will work 100% of the time. For probabilistic methods, you do get better results most of the time but they can also screw up randomly for no reason so their deployability in production is hell on wheels. It's infuriating at times.
Still can't argue against it being very fascinating.
I couldn't agree more.
Summary: Fine tune a foundation model for a task. How did all weights change? That's called the "parameter delta." These changes are highly redundant. You can carefully (use DARE to) revert like 80% of them, yet maintain fine-tuned task accuracy! But only if the tuned weights didn't shift much. Otherwise DARE fails. Maybe you can make an LM polymath by melting together many fine-tunes of some base model. No GPU needed.
Image models support merging already. There exists thousands of StableDiffusion models for that reason only. Downside is that almost all models you see are now inbred. Community does talk about it and can see the effect this is having on the image quality. A prominent example is you will see the exact same japanese kind of girl face from almost all models out there when you generate image of a woman. Check out Civitai to see what I am talking about. It's not easy to train new models but super easy to merge.
We can expect an explosion of LLMs using this technique similarly. And later at some point the degradation of quality perhaps. Or may be all those LLMs will be just saying the same things at that point.
We can expect an explosion of LLMs using this technique similarly. And later at some point the degradation of quality perhaps. Or may be all those LLMs will be just saying the same things at that point.
Image models merging is kind of meh though. You can't really merge two SD models trained on different styles with different keywords and get a model that knows both independently.
Never done that myself but always thought that was the point of these merges. May be it doesn't understand keywords after merge but I think they do keep the styles in some form. What's the point of merges if that wasn't possible. People sometimes share how much of a model they merged by a factor.
Let's say i trained a model on the artworks of fiona staples that is invoked by typing "fiona staples style". Then i trained a model on the artworks of james daly that is invoked by "james daly style".
What i want when i merge such models is a model that can generate in the artsyle of either fiona and james independently or a mix of both if i specify both keywords in the same prompt.
Currently, if you merge these 2 models and generate "busy city street, fiona staples style", you will not get a model that can generate works in fiona's style, you will just get a model that will generate an odd mix of fiona and daly even if you only specify one of them.
It means you either need to train a million different models for a million different concepts with no chance of cross usage (e.g x person wearing y clothes in z style will not be possible) or train those concepts at the same time, which becomes very cumbersome requiring a retrain of n+1 concepts on a fresh model anytime you want to introduce a new concept.
Oh, and training on x then training on y doesn't work in practice either because the model will mostly forget x learning y.
What i want when i merge such models is a model that can generate in the artsyle of either fiona and james independently or a mix of both if i specify both keywords in the same prompt.
Currently, if you merge these 2 models and generate "busy city street, fiona staples style", you will not get a model that can generate works in fiona's style, you will just get a model that will generate an odd mix of fiona and daly even if you only specify one of them.
It means you either need to train a million different models for a million different concepts with no chance of cross usage (e.g x person wearing y clothes in z style will not be possible) or train those concepts at the same time, which becomes very cumbersome requiring a retrain of n+1 concepts on a fresh model anytime you want to introduce a new concept.
Oh, and training on x then training on y doesn't work in practice either because the model will mostly forget x learning y.
Intuitively looks like models should be close enough, or sparse enough for merge to work. I wonder if MoE experts can be merged(?)
Super Mario? Wouldn’t Mega Man be a better video game analogy?
For others like me who’d not heard of merging before, this seems to be one tool[0] (there may be others)
[0] https://github.com/arcee-ai/mergekit
[0] https://github.com/arcee-ai/mergekit
Ah yes. No Nintendo character is more characterized by consuming other characters and absorbing their abilities than Mario.
Didn't play Super Mario Odyssey I take it?
My first thought, Megaman would be my choice. Maybe Kirby, but he can only have one power at a time.
In Kirby 64 (and maybe others?) he could combine powers https://en.wikipedia.org/wiki/Kirby_64:_The_Crystal_Shards
I always imagined Megaman to be a Capcom character. But I also always imagine being unnecessarily pedantic about 80s video game IP.
Here to say : Kirby!!
Then why not start with a model that has all the weights zeroes out and start absorbing different models?
It would be cool if there was like a threshold in training from scratch, where after this actually works for adding higher level knowledge. So you would start training it like usual to get it to absorb generic langauge skills and reasoning but save all the domain knowledge absorption for merging later
The weights need to be connected to something interesting. Pre-training is how they get them all connected up, and fine-tuning is how they find which weights it would be useful to change.
Pretraining determines the weights (e.g. the connections), fine-tuning let's you change some subset of the weights (e.g. the final layers) with a smaller chunk of task-specific data.
Does this imply that some type of decentralized training mechanism is possible? Like an accumulation for ML models. I suspect in the limit
You will just have even more massive models which will require even more demand on the hardware. I also wonder if new capabilities emerge from the merging that are not present in any one model.
I think the reference should have been to Kirby
The drop and rescale method outlined in the paper makes the latent space increasingly sparse, which in turn allows weights to merged without much interference or degradation.
My instinct is that while merging models will have some use cases, ultimately these insights will lead to innovations in training and architecture that have the same result but with better computational efficiency.
For example instead of training an 8x7b mixture of experts then merging, just incorporate the sparsity constraint while pre-training a single 7b model (somehow).