Following what others have said, Josh W. Comeau's work is really impressive. Do you or your co-worker have a way to get in touch directly? I think I've got something they might be interested in.
- 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only used 16 different numbers instead of the thousands or millions that the model might usually use.
- Low Rank Adapters (LoRA): This is a way to change the model's knowledge without having to touch every piece of it. Imagine you have a huge, complicated Lego structure, and you want to change it. Instead of taking apart the whole thing, you just add or change a few pieces here and there. That's what LoRA does. It allows the researchers to fine-tune the model without having to use as much memory.
- Double Quantization: This is another trick to save memory. It's like if you realized that you could represent each of your 4 crayon colors with just 2 symbols, so you save even more space.
- Paged Optimizers: This is a way to handle moments when the model needs a lot of memory all at once. It's like if you have a small desk, but sometimes you need to work on a big project. Instead of getting a bigger desk, you just clear off and use the desk in small sections at a time.
By using these techniques, the researchers were able to train a very large model (Guanaco) on a single graphics card, which would normally not have enough memory for this task.