We are training a large-scale image recognition model on AWS P3 instances, but the costs are becoming unsustainable. Each training run costs us hundreds of dollars, and we are still in the experimentation phase. Are there specific techniques like "Mixed Precision Training" or "Distributed Training" that can help speed up the process and reduce the bill? Also, is it worth looking into specialized AI chips like Google's TPUs or AWS Trainium for better price-to-performance?
3 answers
You can significantly cut costs by using "Spot Instances" for your training jobs. Since training is usually an asynchronous process, you can handle the occasional interruption by saving checkpoints to S3 every hour. Additionally, implementing Mixed Precision Training (using FP16 instead of FP32) can double your training speed on modern NVIDIA GPUs like the V100 or A100 with almost no loss in accuracy. We switched our pipeline to AWS SageMaker Managed Spot Training and saw a 65% reduction in our monthly Deep Learning compute spend while actually increasing the number of experiments we could run.
Have you tried profile-driven optimization to see if your data loading bottleneck is keeping your expensive GPUs idle for 40% of the training time?
Thomas, I hadn't even thought of the data bottleneck! Our training data is stored in standard EBS volumes. If the GPU is waiting for the disk to fetch the next batch of images, we are literally burning money. Do you think moving the dataset to an FSx for Lustre filesystem would provide enough throughput to keep the GPUs at 100% utilization, or is there a cheaper way to optimize the input pipeline?
You should definitely try Google Cloud TPUs if you are using TensorFlow or PyTorch. They are custom-built for tensor math and are often much cheaper than high-end GPUs.
William is right; for specific model architectures like Transformers, the price-to-performance ratio of TPUs is currently unbeatable in the cloud market.
test