Couple days ago, i used Claude to implement an improved version of gpt-1. I am no ML Engineer by no means. I am just a normal backend engineer. I ended up creating a hybrid between gpt-1 and modded-nanogpt (from KellerJordan).
I was able to reproduce the results of the original gpt-1 paper with my gaming PC. I don't even have alot of VRAM. My NVIDIA GeForce RTX 2060 SUPER was able to reproduce most of the results with just 1 hour of training. I would totally recommend to do the same, if you are interested in pre-training LLMs.
Thank you for your kind words. Yes, i hate subscriptions. But, also. Renovating is a one-off thing and i can't justify charging a monthly fee for a service that is only needed once.
This is one of the reasons people love using xona: no subscriptions :)
i was thinking the same. And i am sure it is in the training set, but i couldn't solve 10 hard problems. Also, most of the hard problems, the LLMs weren't able to solve in one shot. How do you explain this?
I was able to reproduce the results of the original gpt-1 paper with my gaming PC. I don't even have alot of VRAM. My NVIDIA GeForce RTX 2060 SUPER was able to reproduce most of the results with just 1 hour of training. I would totally recommend to do the same, if you are interested in pre-training LLMs.
The code is here: https://github.com/epoyraz/modded-gpt-1 But, you can also just ask Claude 4.8 or Codex 5.5