I am working on a project involving action recognition and video classification. I have traditionally used 3D Convolutional Neural Networks, but I am hearing that Video Vision Transformers are the new standard in 2024. How do these models handle the temporal dimension differently, and is the computational cost justified for real-time edge applications?
3 answers
In 2024, Video Vision Transformers have indeed begun to outperform 3D CNNs by treating video as a sequence of patches across both space and time. Unlike CNNs, which use fixed kernels that might miss long-range temporal dependencies, the self-attention mechanism in Transformers allows the model to attend to any frame in a sequence simultaneously. This results in a much deeper understanding of complex actions. For edge applications, however, they can be quite heavy. We are currently seeing a shift toward "Tubelet Embedding" to reduce the number of tokens, which makes these deep learning models significantly faster without losing accuracy.
Have you looked into the specific latency differences when running these Transformer models on NVIDIA Jetson or similar edge hardware?
Transformers are definitely the future for video. Their ability to globalize context across frames is something CNNs just can't do natively without massive depth.
Martha is right. The global receptive field is the biggest selling point. Even if they are slower to train, the results in 2024 for complex video understanding are undeniable.
Patrick, that is a great question. On hardware like the Jetson Orin, we see that 3D CNNs still have a slight edge in raw frames-per-second because of their optimized CUDA kernels. However, by using TensorRT for model quantization to INT8, we’ve managed to get Video Transformers running at near real-time. The trade-off is that the Transformer provides much better accuracy for "fine-grained" actions that 3D CNNs often confuse. For deep learning developers, it really comes down to whether your use case prioritizes absolute speed or the precision of action detection.