Transformer Math
How does a transformer actually work?
The biggest questions in AI—rebuilt from first principles.
How would you train GPT-4, make inference 100 times cheaper, or bring $10 billion of datacenter capacity online in eighteen months? Between Tokens makes the assumptions, does the math, and follows every decision until the whole system comes into view.
Questions we’re working through
Latest essays
- The transformer is a learned probability model A joint probability over sequences is impossible to store. The chain rule turns it into one function you can learn.
- From Bellman to Rainbow RL isn’t solvable — it can only be estimated. This essay collects everything you need to know about reinforcement learning.
- The residual stream is the model Every explanation of transformers spends itself on attention, which is a third of the model. The other two thirds are MLPs — and the thing they all write into is where the model actually lives.
- Attention is a lookup table you learn Before any of the scale questions, the mechanism. Attention is not “the model focusing” — it is a differentiable dictionary, and once you see it that way every design choice downstream stops being arbitrary.