DenseFormer: Enhancing Information Flow in Transformers(arxiv.org)
arxiv.org
DenseFormer: Enhancing Information Flow in Transformers
https://arxiv.org/abs/2402.02622
34 comments
This method has only been tested on tiny models (<1B) and tiny dataset (17B tokens). It’s not clear if it scales.
If a genie appeared and granted one wish, I would wish that we find an extremely powerful machine learning technique that doesn't scale. Imagine if an average desktop computer was almost as good as a billion dollar super computer.
In other words, I don't really care if it scales. I almost hope it doesn't.
In other words, I don't really care if it scales. I almost hope it doesn't.
Even pocket computers (smartphones) are already better than billion dollar supercomputers from decades past.
What is your point?
What is your point?
That no one has an advantage
Not sure I understand what you mean by “doesn’t scale”. Are you trying to say you would like to see a tiny model performing as well as a large model?
To be fair to the authors they are affiliated with a university and not a big industrial lab, so they may be working with significantly constrained resources. Not sure exactly what the best solution is for this case given that it affects most people outside of a very select few.
They could partner with big industrial labs.
Nah, nobody's begging for people to A) come use time on their GPUs B) come watch them train their biggest models. Nor does it make sense to spend $X00M training a big model using an experimental technique before you announce it, nor does it make sense to hold back breakthroughs as an academic until someone commercializes it at scale. Category error.
I do ML research at a small industrial lab. I’ll gladly provide some compute to people with a cool idea if that results in my company name listed on a paper in a top conference. Especially if the people are from a top university.
But it may scale -- that's science in progress
The architecture changes are very straight forward. Model merging has shown that pre-trained transformer layers are very robust. I’ll bet it’s possible to fine tune a pre-trained model like mistral to use this architecture. That would enable someone to test it with more parameters without training a whole new base model.
They try this in the appendix without success, unfortunately. It seems having this enabled early on in training is important.
We're still working on training the DWA weights on top of a pretained model. We're hopeful that this is feasible. The experiments you're mentioning in the appendix are not changing the learning rate scheduler. E.g., when starting to train the DWA weights after 20k iterations, the learning rate is already quite small. To some extent, this might explain the diminishing returns. Maybe this could work with a completely different learning rate scheduler.
Yeah, you can't change the model much with low LRs. That's the point! Same reason you don't get continual-learning if you just keep using low LRs: https://arxiv.org/abs/2403.08763 You need to really shake up the model if you want to learn some genuinely better (ie. different) internal representations that exploits the DenseNet (https://arxiv.org/abs/1608.06993)/LTG-BERT (https://arxiv.org/abs/2311.02265) arch you're using here.
I haven’t been able to make sense of model merging. Any insights?
Wouldn’t weights between models be completely different? And then there are architecture differences on top of that.
Wouldn’t weights between models be completely different? And then there are architecture differences on top of that.
Model merging is usually done with different fine-tunes of the same model. It doesn’t work if the base models are different.
One of the more surprising things is that you can actually repeat layers to improve model performance, ie 1-1-2-2 instead of 1-2. That’s how you get models with higher parameter counts than the original.
One of the more surprising things is that you can actually repeat layers to improve model performance, ie 1-1-2-2 instead of 1-2. That’s how you get models with higher parameter counts than the original.
C.f. also Universal Transformer: the same layer stacked a lot.
The sparse version of that is basically MoE with also a stick-breaking mechanism to prevent vanishing gradient while letting the model decide whether to terminate layer-count at a token early (ofc with training rewards to favor less layers, to represent the compute savings).
This is a very interesting idea, with DenseNets there are oftentimes some terrible memory gotchas that have gotten me over the past 7-8 years or so, so a part of me is sorta leaning back waiting for some memory usage shoe to drop not specified in the paper (even with the activation patterns!)
However, maybe this is not the case. I have a bit of a history of messing with residuals in neural networks, seeing more work on it is good. Fast training networks of course are a very slightly mild obsession of mine as well, and very useful to the field. Here's hoping it pans out as a motif, curious to see where it goes.
However, maybe this is not the case. I have a bit of a history of messing with residuals in neural networks, seeing more work on it is good. Fast training networks of course are a very slightly mild obsession of mine as well, and very useful to the field. Here's hoping it pans out as a motif, curious to see where it goes.
Even better is the result on page 7 that perplexity drops faster by wall-clock time. Even if you're getting fewer iterations per hour of rented GPU time, you're still coming out ahead in model performance.
Cool paper. Really interesting to see how even quite straightforward architectural modifications haven't yet all been exhausted yet, despite all the resources being poured into LLMs
The problem is that they have to be tested for 7B models at least to show promise for larger models. And that requires significant compute resources.
Due to some of my personal experiences over the years w/ model development, I believe that this is more due to a failure of the current mainline version of Transformers (the ++ version I believe) not scaling properly, vs an indicator of scale.
If that is the case, then it may well be possible to fix some of the scaling issues more apparent with smaller transformer models (maybe not, though). This is at least some of the reasoning that I've been applying when developing hlb-gpt, for example. It's partially also why I think changing how we use nonlinearities within the network might impact scaling, due to some of the activation spikes used in more linear regions of the network to control network behavior in a way not originally intended.
Agreed that it does require a ton of resources though. But I do think that the problem can be solved on a smaller scale. If we don't have a cleanly logarithmic curve, then I think that something is dearly wrong with our base architecture. (However, of course, I may entirely be missing something here).
If that is the case, then it may well be possible to fix some of the scaling issues more apparent with smaller transformer models (maybe not, though). This is at least some of the reasoning that I've been applying when developing hlb-gpt, for example. It's partially also why I think changing how we use nonlinearities within the network might impact scaling, due to some of the activation spikes used in more linear regions of the network to control network behavior in a way not originally intended.
Agreed that it does require a ton of resources though. But I do think that the problem can be solved on a smaller scale. If we don't have a cleanly logarithmic curve, then I think that something is dearly wrong with our base architecture. (However, of course, I may entirely be missing something here).
I wonder whether we're missing out on techniques that work well on large models but that don't show promise on small ones
More like we're missing out on techniques full stop. Proving things at scale is GPU expensive and gatekeeps publication and therefore accessibility.
Nice finding and makes a lot of sense! It is somewhat related to classification heads using their own weighted representation of all transformer layer outputs.
I only glanced the paper, but they don't seem to softmax ⍺_i for normalization?
I only glanced the paper, but they don't seem to softmax ⍺_i for normalization?
1. They compare with an older sort of standard implementation of a transformer Unsure whether the results would be equally significant compared to models with gated units or multiquery etc.
2. The difference seems to diminish with scale. Real life transformers obviously are much larger and train on many more tokens.
3. A very significant part of training transformer models are the throughoutput and memory optimizations. I wonder how their model would work with such fused kernels or specialized paged KV cache schemes. Or activation checkpointing, if run locally.
4. Indeed they claim no memory impact, but their code shows that their experiments are conducted with a special optimized version which requires all activations to reside in a single tensor at all times. Not sure this would work with 3d parallelism on multiple nodes etc.
2. The difference seems to diminish with scale. Real life transformers obviously are much larger and train on many more tokens.
3. A very significant part of training transformer models are the throughoutput and memory optimizations. I wonder how their model would work with such fused kernels or specialized paged KV cache schemes. Or activation checkpointing, if run locally.
4. Indeed they claim no memory impact, but their code shows that their experiments are conducted with a special optimized version which requires all activations to reside in a single tensor at all times. Not sure this would work with 3d parallelism on multiple nodes etc.
I'm one of the authors, happy to answer questions.
Is it possible to combine your approach with NATTEN? It seems that both approaches are optimizing from different directions and can be combined with significant throughput and small performance improvements?
Can't believe nobody thought of this yet
> Impact statement:
> This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
I found this particularly charming.
> This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
I found this particularly charming.
AFAIK this was the default, copy paste impact statement by ICML template.
[deleted]