We are trying to train a recommendation engine using Spark MLlib’s ALS algorithm, but the training time is increasing exponentially with our data size. Should we be switching to Spark-Deep-Learning (TensorFlow on Spark) or are there specific hyperparameter tuning tricks in MLlib to speed up the convergence?
3 answers
For the ALS (Alternating Least Squares) algorithm, the most common bottleneck is the "Rank" and the number of iterations. Before jumping to TensorFlow, try increasing the number of partitions for your input data to maximize parallelism across all your cores. Also, check your checkpointInterval; setting it too low slows down training due to frequent disk writes, but setting it too high risks a long re-computation if an executor fails. For true Deep Learning, MLlib is limited. I’d recommend Horovod on Spark if you want to use TensorFlow or PyTorch, as it allows for much better distributed GPU utilization than the standard MLlib pipelines.
Dorothy, that’s great advice. But how do you handle the "Unbalanced" data in MLlib? I’ve found that if one user has thousands of interactions and others only have two, the ALS model gets skewed. Does MLlib have a built-in way to handle this, or do we need to manually downsample before the training stage?
Use a smaller subset of data for hyperparameter tuning with CrossValidator. Once you find the best params, then run the full training on your entire dataset to save time.
Excellent point, Jennifer. Using a 10% sample for tuning can save days of compute time. Most people don't realize that the "Optimal" parameters for 10 million rows are usually very similar for 100 million rows in the ALS context.