open question
Transformer Math
How does a transformer actually work?
Every explanation of the transformer is either a diagram with no arithmetic or notation with nothing to say about what it is for. This series takes the mechanism apart in numbers, each essay ending at something you could count.
The series
- 01 The transformer is a learned probability model A joint probability over sequences is impossible to store. The chain rule turns it into one function you can learn.
- 02 Attention is a lookup table you learn Before any of the scale questions, the mechanism. Attention is not “the model focusing” — it is a differentiable dictionary, and once you see it that way every design choice downstream stops being arbitrary.
- 03 The residual stream is the model Every explanation of transformers spends itself on attention, which is a third of the model. The other two thirds are MLPs — and the thing they all write into is where the model actually lives.