Correct. UNA can align the MoE at multiple layers, experts, nearly any part of the neural network I would say.
Xaberius 34B v1 "BETA".. is the king, and its just that.. the beta. I'll be focusing on the Mixtral, its a christmas gift.. modular in that way, thanks for the lab @mistral!
UNA: Uniform Neural Alignment.
Haven't u noticed yet? Each model that I uniform, behaves like a pre-trained.. and you likely can fine-tune it again without damaging it.
If you chatted with them, you know .. that strange sensation, you know what is it.. Intelligence.
Xaberius-34B is the highest performer of the board, and is NOT contaminated.