I’ve been reading that Vision Transformers are outperforming ResNets on ImageNet. As someone used to Convolutional Neural Networks, I’m struggling to understand how a model with no "inductive bias" for images can learn spatial relationships. Is it worth switching our production models to ViT?
3 answers
The main advantage of ViT is the "Global Receptive Field." A CNN looks at images through small windows (convolutions), so it takes many layers for the model to "see" how a pixel in the top-left corner relates to one in the bottom-right. A ViT uses self-attention to compare every patch of the image to every other patch simultaneously. This allows it to capture long-range dependencies much more effectively. However, ViTs require massive amounts of data (like ImageNet-21k) to overcome the lack of inductive bias. If you have a small dataset, a CNN like ConvNeXt or a hybrid model will likely still outperform a pure Transformer.
If ViTs require such large datasets, are they practical for niche medical imaging where we only have a few hundred labeled scans? Or are they strictly for big tech companies?
CNNs are still more efficient for mobile deployment. ViTs have a quadratic complexity with respect to the number of patches, which can be a battery killer on edge devices.
Spot on, Martha. For mobile apps, a MobileNet or a specialized "MobileViT" is a much more realistic choice than a standard large-scale Transformer.
For small datasets, Gregory, you shouldn't train a ViT from scratch. The standard approach is "Transfer Learning." You take a ViT pre-trained on millions of images and fine-tune only the last few layers on your medical data. We’ve seen that a pre-trained ViT-Base model can adapt to specific tasks like lung nodule detection with very high accuracy. Also, look into "Swin Transformers"—they use a hierarchical window approach that brings back some of that CNN-like local bias, making them much more efficient for smaller datasets and higher-resolution images.