I’ve heard that the "Attention is All You Need" paper changed everything. How exactly does the Self-Attention Mechanism allow an LLM to process long-range dependencies in text? In simple terms, how does a model like GPT-4 "attend" to a word at the beginning of a paragraph while generating a word at the end, and why is this superior to older RNN or LSTM architectures?
3 answers
Unlike older models that processed text word-by-word (sequentially), Transformers process entire sequences at once. The Self-Attention Mechanism assigns "weights" to every other word in a sentence to determine which are most relevant to the current word. For example, in the sentence "The bank was closed because it was a holiday," the model uses attention to link "it" directly to "bank," regardless of the distance between them.
This parallel processing is why GenAI scaled so rapidly. Because Transformers don't rely on a sequential "memory" like LSTMs, they can be trained on massive GPUs much faster. This allows them to learn the statistical relationships between Tokens across billions of pages of text, leading to the "emergent" reasoning capabilities we see today.
Think of Attention as a "searchable database" of the current prompt. Instead of forgetting the beginning of the sentence by the time it reaches the end, the model maintains a mathematical map of how every word relates to every other word.
This is also why they are prone to Hallucination. They are predicting the most statistically likely next token based on these weights, not necessarily "retrieving" facts from a database.
The "Context Window" is the literal limit of this attention. If a model has a 128k context window, it can "see" and weigh the importance of every single token within that range simultaneously.