I keep hearing that the "Attention is All You Need" paper changed everything. Can someone break down why the Transformer architecture is so much better than the old RNNs and LSTMs we used to use for Natural Language Processing tasks?
3 answers
Do you think the high computational cost of the self-attention mechanism will eventually lead us to a new, more efficient architecture?
Transformers aren't just for text anymore; "Vision Transformers" (ViT) are now beating traditional CNNs in many image recognition tasks too.
Very true, Nancy. It's becoming a "universal architecture" for almost all modalites, including audio and video.
The game-changer was "Parallelization." Old RNNs had to process words one by one in a sequence, which was incredibly slow and couldn't be easily trained on massive GPUs. Transformers use a "Self-Attention" mechanism that allows the model to look at every word in a sentence simultaneously and understand the relationships between them regardless of how far apart they are. This meant we could finally train on the entire internet's worth of data in a reasonable timeframe. It solved the "vanishing gradient" problem that plagued older LSTMs.
We are already seeing that, Brian! Researchers are looking into "State Space Models" (SSMs) like Mamba, which aim to provide Transformer-level performance but with much better scaling for extremely long sequences of data.