Pure Rust implementation of a minimal GPT language model(github.com)

Pure Rust implementation of a minimal GPT language model

https://github.com/keyvank/femtoGPT

4 comments

mrkeen3 anni fa

> After hours of training on the Shakespeare database, on a 300k parameter model, this has been the output

I'm a bit out of the loop. What does it mean to train on Shakespeare? Does this imply that it's Shakespeare and a whole bunch of other English text?

I tried generating Shakespeare a few years back using something quite primitive by today's standards. (Maybe it was word2vec? I don't remember.). I was surprised that it emitted real Shakespeare-sounding stuff! But then someone explained that there is so little Shakespeare to train on, that you'll only ever reproduce parts of the actual text (which indeed I had).

So what are the expectations of this GPT model? Should it be bound by the same limitations?

gandalfff3 anni fa

Does this have potential for running on an SBC? Mine only has 2GB of RAM and I'm interested in running a toy language model on it.

keyvank3 anni fa

femtoGPT is a pure Rust implementation of a minimal Generative Pretrained Transformer.

Everything is implemented from scratch, including the tensor processing logic along with training/inference code of a minimal GPT architecture.

allisdust3 anni fa

Thanks for making this. Was searching for a pure rust GPT implementation just the other day!