I built a GPT from scratch in PyTorch, trained on Kabir ke dohe. 🧘
Tokeniser, self-attention, multi-head attention, transformer blocks, inference, all hand-written, no transformer libraries.
Wrote up the full journey in blog below along with Github :