https://betweentokens.com Between Tokens rebuilds the biggest questions in AI from first principles, following every assumption, calculation, and decision to a working answer. en https://betweentokens.com/questions/train-gpt-4/the-block/ https://betweentokens.com/questions/train-gpt-4/the-block/ You work at OpenAI. How would you train GPT-4? Wed, 22 Jul 2026 00:00:00 GMT Every explanation of transformers spends itself on attention, which is a third of the model. The other two thirds are MLPs — and the thing they all write into is where the model actually lives. https://betweentokens.com/questions/train-gpt-4/attention/ https://betweentokens.com/questions/train-gpt-4/attention/ You work at OpenAI. How would you train GPT-4? Tue, 14 Jul 2026 00:00:00 GMT Before any of the scale questions, the mechanism. Attention is not "the model focusing" — it is a differentiable dictionary, and once you see it that way every design choice downstream stops being arbitrary.