How to built a small Language Model from scratch

Here's exactly how it works 🧵
No APIs. No pre-trained weights. Just PyTorch and math.

Step 1 — Feed it text

I grabbed 50,000 children's stories from a dataset called TinyStories (huggingface). Then I used a tokenizer to convert every word into a number.
"Once upon a time" → [7406, 3504, 257, 640]
Computers don't read words. They read numbers.

Step 2 — Build the brain

I coded a GPT Transformer from scratch using claude. It has 3 key ingredients:

→ Embeddings — converts each number into a rich vector that captures meaning

→ Self-Attention — lets every word look at every other word to understand context. This is literally how "bank" knows if you mean a river or money.

→ 6 stacked Transformer Blocks — each one learns deeper patterns, from grammar all the way to storytelling
30 million parameters total. All built by hand.

Step 3 — Teach it to predict
The training loop is brutally simple:
Show the model: "Once upon a" Model guesses the next word. Gets it wrong. Calculate how wrong (the Loss). Adjust every single weight slightly in the right direction. Repeat 5,000 times.
That's it. Grammar, story structure, common sense — all of it emerges from this one loop.

Step 4 — Run it on a GPU
I connected the code to Modal.com and trained on an NVIDIA A10G cloud GPU.
30 minutes. Less than $1.

The result?
"𝙊𝙣𝙘𝙚 𝙪𝙥𝙤𝙣 𝙖 𝙩𝙞𝙢𝙚, 𝙩𝙝𝙚𝙧𝙚 𝙬𝙖𝙨 𝙖 𝙡𝙞𝙩𝙩𝙡𝙚 𝙗𝙤𝙮 𝙣𝙖𝙢𝙚𝙙 𝙏𝙞𝙢𝙢𝙮. 𝙃𝙚 𝙛𝙚𝙡𝙡 𝙙𝙤𝙬𝙣 𝙖𝙣𝙙 𝙝𝙪𝙧𝙩 𝙝𝙞𝙨 𝙠𝙣𝙚𝙚. 𝙃𝙞𝙨 𝙢𝙤𝙢 𝙜𝙖𝙫𝙚 𝙝𝙞𝙢 𝙖 𝙗𝙖𝙣𝙙-𝙖𝙞𝙙. 𝙏𝙞𝙢𝙢𝙮 𝙛𝙚𝙡𝙩 𝙗𝙚𝙩𝙩𝙚𝙧."
for codebase email to: amaanprogramming@gmail.com
A model I built myself is writing coherent stories.
If you want to truly understand AI — don't just use the tools. Build them.