The residual stream is the model

Part one treated attention as the interesting thing in a transformer. That was a useful simplification and it is time to drop it. By parameter count, attention is the minority partner:

Two thirds of the model is a stack of two-layer MLPs applied independently at every position. Any account of what a transformer does that only discusses attention is describing a third of the machine.

The block, written honestly§

def block(x, attn, mlp, ln1, ln2):
    x = x + attn(ln1(x))   # move information between positions
    x = x + mlp(ln2(x))    # transform information at each position
    return x

Note what the residual connections mean structurally. x is never replaced. Each sub-layer reads the current state, computes a correction, and adds it back. The tensor flowing down the model is not “the output of layer 12” — it is an accumulator that every layer has written into.

This is the residual stream, and reading it as a shared communication channel rather than a chain of transformations explains several things at once:

• Why depth works without degradation. The identity path is the default. A layer that has nothing useful to contribute can output near-zero and cost nothing, so adding layers is close to free in optimization terms.
• Why layer norm goes before the sub-layer, not after. Pre-norm keeps the residual path free of normalization, so gradients reach layer 1 from layer 96 without passing through 96 normalizations. Post-norm models above roughly 20 layers need learning-rate warmup and careful initialization to train at all; pre-norm models mostly just train.^[1]
• Why the same vector can be read differently by different heads. Different layers write into different subspaces of the stream, and later layers project out the parts they care about.

What the feed-forward layers are for§

The MLP is two linear layers with a nonlinearity, widening to $4d$ and back:

$$\mathrm{FFN}(x) = W_2 \, \sigma(W_1 x + b_1) + b_2, \qquad W_1 \in \mathbb{R}^{4d \times d}$$

Read it as key–value memory. $W_1$ produces $4d$ scores — how strongly does this position match each of $4d$ learned patterns. $\sigma$ gates them. $W_2$ maps each activated pattern to a vector written back into the stream. It is the same soft-lookup shape as attention, except the keys and values are model parameters rather than other positions.

That framing makes the division of labour clear. Attention moves information between positions. The MLP retrieves information from the weights. Factual recall lives predominantly in the second one, which is why the interpretability work on locating and editing facts targets MLP layers rather than attention.

The one number that matters§

The widening factor of 4 is conventional rather than derived, and it interacts with the nonlinearity. Gated variants (SwiGLU and relatives) use three matrices instead of two, so they set the hidden width to $\tfrac{8}{3}d$ to hold the parameter count fixed — and then win on quality at equal budget. That kind of substitution, equal parameters and better loss, is the whole game once you are compute-constrained.

Which is the next question: given a fixed number of FLOPs, how many parameters should you have and how many tokens should you train them on? Part three is about the scaling laws, and about how badly the field got that answer wrong the first time.