I have a PyTorch model for real-time gesture recognition that is nearly 500MB. This is way too large for a mobile app download. I’ve heard about Pruning and Knowledge Distillation, but I’m worried about losing too much accuracy. What is the most "bang for your buck" method to shrink the model?
3 answers
Knowledge Distillation is usually the most effective method for maintaining accuracy. You take your "Teacher" model (the 500MB one) and use it to train a "Student" model (like MobileNetV3). The student learns to mimic the teacher’s output distribution, often achieving 95% of the performance at 1/10th the size. After distillation, apply Post-Training Quantization (PTQ). Converting weights from Float32 to INT8 can shrink the model size by another 75% with negligible accuracy loss. It’s the standard pipeline for production-grade mobile AI.
Have you looked at the architecture itself? Sometimes simply switching from a heavy backbone like ResNet50 to a ShuffleNet or EfficientNet-Lite can solve the size problem before you even start pruning.
Structural Pruning is another option. You can remove entire filters that aren't contributing much to the final decision. PyTorch has a built-in 'torch.nn.utils.prune' module for this.
I’ve used that module! It’s great, but be careful—unstructured pruning makes the model sparse but doesn't always reduce the file size unless you use specialized compression for the sparse matrices.
Charles, I'm currently using a custom CNN. Nancy, your point about Knowledge Distillation is great. I was worried it would take too long to retrain, but if it gets me down to 50MB, it's worth it. Does PTQ work well with gesture recognition where spatial precision is vital? I'm worried the rounding errors might mess up the finger-tip coordinates.