To improve a Dataflow Gen2 ingestion performance, which ordering of steps is recommended when dealing with a DateTime column that must be filtered to the current year and then split by position?

To improve a Dataflow Gen2 ingestion performance, which ordering of steps is recommended when dealing with a DateTime column that must be filtered to the current year and then split by position?

• A. Apply the split by position transformation first, and then apply the filter transformation
• B. Skip filtering and proceed with the split transformation
• C. For step 3, first apply the filter transformation to the OrderDateTime column to current year, and then apply the split by position transformation
• D. Apply the transformations in parallel without ordering

Answer: (C)

Early filtering to reduce data before downstream transforms is the key performance pattern in Dataflow Gen2. Here, applying the filter to keep only the current year on the DateTime column first means only those limited records move on to the split by position step. That minimizes the amount of data that needs to be shuffled, stored, and further processed, lowering CPU, memory usage, and overall latency. If you split first, you would propagate all records through the split, and only then filter. That increases data volume on the split operation, driving up compute and shuffle costs and slowing ingestion. Skipping the filter altogether means processing every record, which ignores the requirement and wastes resources. Running transforms in parallel without ordering would break the dependency that the split should operate on filtered data, risking incorrect results and unnecessary processing.

Early filtering to reduce data before downstream transforms is the key performance pattern in Dataflow Gen2. Here, applying the filter to keep only the current year on the DateTime column first means only those limited records move on to the split by position step. That minimizes the amount of data that needs to be shuffled, stored, and further processed, lowering CPU, memory usage, and overall latency.

If you split first, you would propagate all records through the split, and only then filter. That increases data volume on the split operation, driving up compute and shuffle costs and slowing ingestion.

Skipping the filter altogether means processing every record, which ignores the requirement and wastes resources. Running transforms in parallel without ordering would break the dependency that the split should operate on filtered data, risking incorrect results and unnecessary processing.