I am running a large join operation between two datasets in Spark, but one or two tasks always take 10x longer than the rest, hanging at 99% completion. I suspect data skewness in our "Region_ID" key. What are the best techniques to redistribute this data evenly? Should I use salting, or is there a newer Spark 3.x feature that handles this automatically?
3 answers
Spark 3.0+ introduced Adaptive Query Execution (AQE), which has a specific feature for "Skew Join Optimization." You should ensure spark.sql.adaptive.skewJoin.enabled is set to true. It automatically detects skewed partitions and splits them into smaller sub-partitions. However, it isn't a silver bullet. If your skew is extreme, "Salting" is still the best manual fix. You add a random integer (a "salt") to your join key in the skewed dataset and then replicate the other dataset's keys to match. This forces Spark to distribute the "hot" key across multiple executors instead of jamming them all into one.
Are you sure it's a join skew and not a "Shuffle" issue? Sometimes a poorly timed groupBy can cause the same symptoms. Have you checked the Spark UI to see if the "Shuffle Read Size" is significantly higher for those specific tasks?
Salting is the way to go for truly massive datasets. Just make sure your salt range isn't too large, or you'll end up with too many small files and "metadata bloat."
Totally agree with Barbara. I usually start with a salt factor of 10 and only increase if the skew persists. James, definitely try AQE first though—it's much cleaner if it works for your case.
Daniel, I checked the UI and it’s definitely the Join. One executor is pulling in 4GB of data while the others have 50MB. James, another thing to try before salting is a "Broadcast Join." If one of your tables is small enough to fit in memory (usually <10MB by default, but you can increase it), use the broadcast() hint. This sends the small table to every executor, avoiding the shuffle entirely. It completely bypasses the skew problem because no data needs to be redistributed by the key.