I'm trying to understand why older RNNs failed at long paragraphs while Transformers excel. Can someone explain in simple terms how the "Self-Attention" mechanism allows a model like GPT-4 to remember a context from 10 pages ago without getting confused?
3 answers
The magic of Self-Attention is that it's "non-sequential." Older models like RNNs or LSTMs processed text word-by-word, like a person reading a book. By the time they reached the end of a chapter, they "forgot" the beginning because the mathematical signal decayed. Transformers look at every word in the context window simultaneously. For every new word generated, the model calculates a "score" for every previous word to see how relevant it is. This allows it to link a pronoun at the end of a page directly to a noun at the very beginning without any loss of information.
Isn't the quadratic complexity of attention still a bottleneck for really long documents? How do models like Claude handle 200k tokens?
Think of it like a spotlight. While the model is writing, it can shine that spotlight on any previous word to pull in the necessary context instantly.
That’s a perfect analogy, Samantha! It’s that ability to "look back" instantly that makes generative AI sound so much more human and coherent.
Great question, Jeffrey. Standard attention is indeed $O(n^2)$. To handle 200k tokens, researchers use "Sparse Attention" or "Flash Attention," which optimizes how the GPU handles these calculations. They also use "Sliding Window" techniques to focus on the most important bits of the conversation rather than calculating everything every single time.