open question
You work at OpenAI. How would you train GPT-4?
Not “explain transformers.” Build the thing: architecture, data, the training run, the alignment pass, and serving it to millions of people without going bankrupt.
Every explanation of GPT sits at one of two altitudes. Either it is a diagram of attention heads with no sense of scale, or it is a product announcement with no sense of mechanism. Neither one tells you what a person at OpenAI actually decided, or why.
This series takes the whole path. Each essay ends somewhere concrete: a component you could implement, a number you could budget against, a decision you could defend in a design review. The order matters — later essays assume the earlier ones, and the hard parts show up late, where they belong.
The series
- 01 Attention is a lookup table you learn Before any of the scale questions, the mechanism. Attention is not “the model focusing” — it is a differentiable dictionary, and once you see it that way every design choice downstream stops being arbitrary.
- 02 The residual stream is the model Every explanation of transformers spends itself on attention, which is a third of the model. The other two thirds are MLPs — and the thing they all write into is where the model actually lives.
Still to come
- Pretraining data — where a trillion tokens actually comes from, and what you throw away
- The training run — parallelism, precision, and what a loss spike costs you
- Supervised fine-tuning — why 15,000 good examples beat 15 million scraped ones
- RLHF — the reward model, PPO, and everything that goes wrong
- Evaluation — why benchmarks stop meaning anything above a certain capability
- Inference — KV caching, batching, and the economics of a served token