Which practice minimizes data shuffling when joining a large fact dataset with a small dimension dataset in Spark?

Which practice minimizes data shuffling when joining a large fact dataset with a small dimension dataset in Spark?

• A. Shuffle the large dataset on the join key
• B. Broadcast the small dimension DataFrame to all workers during the join
• C. Collect the large DataFrame to the driver first and then join
• D. Persist both DataFrames to disk and perform the join

Answer: (B)

This question tests how to minimize shuffling by using a broadcast join when one side of the join is small. When the small dimension dataset is broadcast to all workers, each executor gets a local copy of it and can perform the join with its portion of the large fact dataset without reshuffling the large data across the cluster. Spark uses a BroadcastHashJoin under the hood in this scenario, which eliminates the need to shuffle the big dataset and greatly reduces network I/O. This is efficient only if the small dataset fits in memory on each executor; if it’s too large, broadcasting can cause memory pressure. The other approaches would still involve shuffling the large dataset, collecting data to the driver, or persisting data in a way that doesn’t inherently avoid the shuffle.

This question tests how to minimize shuffling by using a broadcast join when one side of the join is small. When the small dimension dataset is broadcast to all workers, each executor gets a local copy of it and can perform the join with its portion of the large fact dataset without reshuffling the large data across the cluster. Spark uses a BroadcastHashJoin under the hood in this scenario, which eliminates the need to shuffle the big dataset and greatly reduces network I/O. This is efficient only if the small dataset fits in memory on each executor; if it’s too large, broadcasting can cause memory pressure. The other approaches would still involve shuffling the large dataset, collecting data to the driver, or persisting data in a way that doesn’t inherently avoid the shuffle.