Which approach is recommended when joining a large transactions DataFrame with a small customers DataFrame to minimize shuffling in PySpark?

Which approach is recommended when joining a large transactions DataFrame with a small customers DataFrame to minimize shuffling in PySpark?

• A. Perform a standard join without broadcasting.
• B. Broadcast the small DataFrame to all workers.
• C. Repartition the large DataFrame to a single partition.
• D. Collect both DataFrames to the driver before joining.

Answer: (B)

Broadcasting the small DataFrame to all workers is the best approach here. When one side of a join is very small, sending that dataset to every executor allows each partition of the large DataFrame to be joined locally, without moving the large data across the network. This avoids the expensive shuffle of the big DataFrame and dramatically reduces network I/O and overall runtime. If you don’t broadcast, Spark would need to shuffle the large DataFrame (and possibly the small one) to align on the join key, which is costly for big data. Repartitioning the large DataFrame to a single partition would create a bottleneck on one node and doesn’t scale. Collecting both DataFrames to the driver before joining is impractical and risks driver memory overflow.

Broadcasting the small DataFrame to all workers is the best approach here. When one side of a join is very small, sending that dataset to every executor allows each partition of the large DataFrame to be joined locally, without moving the large data across the network. This avoids the expensive shuffle of the big DataFrame and dramatically reduces network I/O and overall runtime.

If you don’t broadcast, Spark would need to shuffle the large DataFrame (and possibly the small one) to align on the join key, which is costly for big data. Repartitioning the large DataFrame to a single partition would create a bottleneck on one node and doesn’t scale. Collecting both DataFrames to the driver before joining is impractical and risks driver memory overflow.