Which PySpark code should you run to load the Customers.parquet data into a pandas DataFrame?

Which PySpark code should you run to load the Customers.parquet data into a pandas DataFrame?

• A. import pandas as pd; df = pd.read_parquet("/lakehouse/default/Files/Customers.parquet")
• B. import pandas as pd; df = pd.read_parquet("/lakehouse/default/Files/Customers.parquet").head()
• C. import pyspark; df = spark.read.parquet("/lakehouse/default/Files/Customers.parquet")
• D. import pandas as pd; df = pd.read_csv("/lakehouse/default/Files/Customers.parquet")

Answer: (A)

Directly loading a Parquet file into a pandas DataFrame is done with pandas’ read_parquet. This reads the Parquet file and returns a pandas DataFrame in memory, which is exactly what you want when your goal is to work with pandas. The approach uses the file path and returns all the data in one DataFrame, provided the file is accessible to your Python environment. Adding .head() would return only the first few rows, not the entire dataset, so it’s not ideal if you need all the data. Using PySpark to read Parquet gives you a Spark DataFrame, not a pandas one, unless you explicitly convert with toPandas(). Reading with read_csv on a Parquet file is incorrect because Parquet and CSV are different formats. So, the direct and complete load into a pandas DataFrame is achieved with pandas.read_parquet.

Directly loading a Parquet file into a pandas DataFrame is done with pandas’ read_parquet. This reads the Parquet file and returns a pandas DataFrame in memory, which is exactly what you want when your goal is to work with pandas.

The approach uses the file path and returns all the data in one DataFrame, provided the file is accessible to your Python environment. Adding .head() would return only the first few rows, not the entire dataset, so it’s not ideal if you need all the data. Using PySpark to read Parquet gives you a Spark DataFrame, not a pandas one, unless you explicitly convert with toPandas(). Reading with read_csv on a Parquet file is incorrect because Parquet and CSV are different formats.

So, the direct and complete load into a pandas DataFrame is achieved with pandas.read_parquet.