You work at OpenAI. How would you train GPT-4?
Not “explain transformers.” Build the thing: architecture, data, the training run, the alignment pass, and serving it to millions of people without going bankrupt.
The biggest questions in AI—rebuilt from first principles.
How would you train GPT-4, make inference 100 times cheaper, or bring $10 billion of datacenter capacity online in eighteen months? Between Tokens makes the assumptions, does the math, and follows every decision until the whole system comes into view.
Questions we’re working through
Latest essays
- The residual stream is the model Every explanation of transformers spends itself on attention, which is a third of the model. The other two thirds are MLPs — and the thing they all write into is where the model actually lives.
- Attention is a lookup table you learn Before any of the scale questions, the mechanism. Attention is not “the model focusing” — it is a differentiable dictionary, and once you see it that way every design choice downstream stops being arbitrary.