Latest
Everything, newest first.
2 essays across 1 question.
- The residual stream is the model Every explanation of transformers spends itself on attention, which is a third of the model. The other two thirds are MLPs — and the thing they all write into is where the model actually lives.
- Attention is a lookup table you learn Before any of the scale questions, the mechanism. Attention is not “the model focusing” — it is a differentiable dictionary, and once you see it that way every design choice downstream stops being arbitrary.