To populate the results DataFrame by joining a large transactions DataFrame with a small customers DataFrame in PySpark while minimizing data shuffling, which code should you run?

To populate the results DataFrame by joining a large transactions DataFrame with a small customers DataFrame in PySpark while minimizing data shuffling, which code should you run?

• A. results = transactions.join(customers, transactions.customer_id == customers.customer_id)
• B. results = transactions.join(customers, transactions.customer_id == customers.customer_id).hint('broadcast')
• C. results = transactions.join(F.broadcast(customers), transactions.customer_id == customers.customer_id)
• D. results = transactions.join(F.broadcast(customers), transactions.customer_id == customers.customer_id).alias('joined')

Answer: (C)

Broadcasting the smaller DataFrame to all workers is the efficient approach when you’re joining a large transactions table with a small customers table. By wrapping the small customers DataFrame with a broadcast and using it in the join, Spark replicates that small dataset to every executor and performs the join locally against partitions of the large transactions DataFrame. This avoids shuffling the big table across the cluster, dramatically reducing network and shuffle cost. That’s exactly what the code in this option does: it uses the broadcasted small DataFrame in the join, so Spark can execute a broadcast hash join. The other options are less reliable for minimizing shuffle. A plain join shuffles the large data. A hint may guide the planner but isn’t as explicit or guaranteed as broadcasting the small table. Aliasing the joined result doesn’t affect the join strategy and isn’t necessary for the shuffle optimization.

Broadcasting the smaller DataFrame to all workers is the efficient approach when you’re joining a large transactions table with a small customers table. By wrapping the small customers DataFrame with a broadcast and using it in the join, Spark replicates that small dataset to every executor and performs the join locally against partitions of the large transactions DataFrame. This avoids shuffling the big table across the cluster, dramatically reducing network and shuffle cost.

That’s exactly what the code in this option does: it uses the broadcasted small DataFrame in the join, so Spark can execute a broadcast hash join. The other options are less reliable for minimizing shuffle. A plain join shuffles the large data. A hint may guide the planner but isn’t as explicit or guaranteed as broadcasting the small table. Aliasing the joined result doesn’t affect the join strategy and isn’t necessary for the shuffle optimization.