I’m struggling with the latency of my image classification model on older Android devices. I’ve converted my model to the .tflite format, but the inference speed is still too slow for a real-time app. Should I be looking into quantization or pruning? I’d love to hear some real-world experiences on balancing model accuracy with mobile performance.
3 answers
You should definitely start with Post-Training Quantization (PTQ). By converting your weights from float32 to int8, you can reduce the model size by 4x and significantly speed up inference on CPUs and NPUs. We did this for a real-time object detection app and saw a 60% reduction in latency with only a 1% drop in accuracy. If that isn't enough, try "Quantization Aware Training" (QAT), which handles the precision loss during the training phase itself. Also, make sure you are utilizing the GPU delegate in TFLite, as it can offload the heavy lifting from the mobile CPU.
Have you checked if your specific model architecture is supported by the NNAPI delegate for hardware acceleration on those older devices?
Quantization is your best friend here. It’s the fastest way to get a performance boost without having to completely redesign your neural network architecture from scratch.
I agree, Joshua. It's often the "low-hanging fruit" of optimization that people overlook before jumping into more complex pruning techniques.
Mark, I actually tried NNAPI but found it inconsistent on budget chipsets. I’ve found that sticking to the XNNPACK delegate for CPU optimization usually provides a more stable speed boost across a wider range of Android versions. It's often safer than relying on vendor-specific drivers that might be outdated on older phones.