That's a fair point TBH. I said in my post that this LLM is first of all a learning project and I skipped an important step: the training loop. But on the other hand, how many data scientists are writing their own training loops? Is it even worth it? And how much learning do you want for one project, I mean, where do you stop? Why use "Huggingface Transformers" when you can write it from scratch, for learning? Why use Torch when you can write it from scratch, for learning? Why use Python when you can write in C, etc. It's cheating, right?
In my case, I decided to skip the training loop and focus on the data processing and the hyper params and the rest of the higher level steps that took a ton of time anyway, and I reduced the friction.
I do get your point tho. Now that I know how to train an LLM, maybe I'll write a training loop from scratch as a project, to learn how to do it.
That's exactly what I had in mind. When I started this, I was jumping back and forth between this thought: "Can this model size actually generate logical English text?" and I played with a few different models of the same size and I was really really depressed when seeing how bad they are.... but then I discovered more and more tiny models and LaMini-125M, LaMini-256M, and nanowhale-100m, and SmolLM2-135M-Instruct are very very decent. So I decided to give it a try.
I am creating my tiny Llama 340M base model from scratch. If you're curious about the steps, challenges and cost, read on. I am still working on the instruct model.
TwoFold is a small command line app that allows plain text files to behave like dynamic files. It is a hybrid between a text expande, a template engine and a mini programming language.
Tomb consists of a simple shell script (Zsh) using standard filesystem tools (GNU) and the cryptographic API of the Linux kernel (cryptsetup and LUKS).