Amazing work! Reminded me of LLM Visualization (https://bbycroft.net/llm) except this is a lot easier to wrap my head around and that I can actually run the training loops, which makes sense given the simplicity of the original microgpt.
To give a sense of what the loss value means, maybe you can add a small explainer section as a question and add this explanation from Karpathy’s blog:
> Over 1,000 steps the loss decreases from around 3.3 (random guessing among 27 tokens: −log(1/27)≈3.3) down to around 2.37.
to reiterate that the model is being trained to predict the next token out of 27 possible tokens and is now doing better than the baseline of random guess.
> Consequently, knowledge bottlenecks are a form of operational inefficiency in startups.
This resonates with my startup experience, especially when the organisation is growing rapidly (which is almost always the case for VC-funded startups)
Added an entry for my data visualisation tool here: https://github.com/stared/interactive-machine-learning-list/....
Edit: found an updated link for seeing theory so I fixed it in the PR above. Feel free to cherry-pick if #24 is not relevant.