Which statement best describes a broadcast join in Spark?

Which statement best describes a broadcast join in Spark?

• A. It broadcasts the larger DataFrame to all workers to ensure complete local joins.
• B. It requires streaming the data.
• C. It uses a hash join without shuffle by default.
• D. It broadcasts the smaller DataFrame to all workers to perform local joins.

Answer: (D)

Broadcast join in Spark is an optimization that reduces shuffling by sending the smaller DataFrame to every worker. When one side is small enough to fit in memory, Spark broadcasts it to all executors, and each partition of the larger DataFrame is joined locally with that broadcasted data. This minimizes data movement across the cluster and speeds up the join. This is why describing it as broadcasting the smaller DataFrame to all workers to perform local joins is the best way to capture how broadcast joins operate. Broadcasting the larger DataFrame would be inefficient, and the concept isn’t tied to streaming requirements. Also, while a broadcast join can use a hash-based approach for the per-partition join, it’s not accurate to say it’s always without shuffle by default.

Broadcast join in Spark is an optimization that reduces shuffling by sending the smaller DataFrame to every worker. When one side is small enough to fit in memory, Spark broadcasts it to all executors, and each partition of the larger DataFrame is joined locally with that broadcasted data. This minimizes data movement across the cluster and speeds up the join.

This is why describing it as broadcasting the smaller DataFrame to all workers to perform local joins is the best way to capture how broadcast joins operate. Broadcasting the larger DataFrame would be inefficient, and the concept isn’t tied to streaming requirements. Also, while a broadcast join can use a hash-based approach for the per-partition join, it’s not accurate to say it’s always without shuffle by default.