I have been using Scikit-Learn for my models, but as my dataset has grown to several terabytes, my local machine is crashing. Is moving to Apache Spark Programming and its MLlib library the standard move? I want to know if Spark's distributed nature changes how I need to think about feature engineering and model training compared to a single-node environment.
3 answers
When you hit the "memory wall" with Scikit-Learn, Spark MLlib is the logical next step. It is designed specifically to scale out across a cluster of machines. The main shift in thinking is that your data is now partitioned. In a single-node environment, you have global access to all data, but in Spark, algorithms must be designed to work on distributed chunks of data and then aggregate the results. Most common algorithms like Linear Regression, Random Forests, and K-Means have been rewritten in MLlib to support this distributed execution. It’s also worth looking into "Pandas on Spark" which allows you to use your familiar Pandas syntax while Spark handles the heavy lifting in the background.
Does MLlib support deep learning frameworks like PyTorch or TensorFlow, or do I have to use a separate specialized cluster for those types of models?
The scalability is unmatched. I recently ran a recommendation engine on a 5TB dataset using Spark, and it finished in under an hour. That would have been impossible otherwise.
That is a perfect example of why Spark is a must-have skill. Once you go beyond 100GB, traditional tools just can't keep up with the processing requirements.
You can actually use projects like Horovod or Spark-Deep-Learning to integrate them. However, many people use Spark for the "heavy lifting" of data preprocessing and feature engineering, and then pipe the cleaned, smaller subset into a dedicated GPU cluster for the actual deep learning training.