No Train No Gain:Revisiting Efficient Training Algrthm for Transformer-BasedLM(arxiv.org)
arxiv.org
No Train No Gain:Revisiting Efficient Training Algrthm for Transformer-BasedLM
https://arxiv.org/abs/2307.06440
1 comments
Recent paper highlights the difficulty of creating a new optimizer as drop-in replacement. Sophia and Lion were recently proposed as superior alternatives to Adam, but appeared worse in an independent eval