Attention

Attention is how a transformer keeps track of meaning. For each token, the model computes a score for how much it should “pay attention” to every other token in the sequence — then blends their information in.

So when it reads “The bank by the river was muddy”, attention helps connect “bank” to “river”, not to “money”.

• It’s what lets models handle long, context-dependent sentences.
• “Attention is all you need” is the title of the 2017 paper that introduced the transformer.