Spark Shuffle

A Spark shuffle redistributes records across partitions to satisfy a new grouping or distribution. For a sum by country, contributions for the same country must meet at the appropriate downstream partition. Upstream tasks write buckets that downstream tasks read and combine.

Local partial aggregation can reduce the amount shuffled: a partition with many records for one country may emit one subtotal instead of every record. The final aggregation combines these subtotals. This is suitable for sums; averaging partial averages without counts generally gives a different answer.

Shuffle work can involve serialization, disk writes and reads, network transfer, and memory pressure. Some reads are local, so the presence of an Exchange is not itself a measured network byte count. A broadcast join transfers the broadcast side even when it avoids shuffling the large side.

An extra repartition before an aggregation can add redistribution without changing the result. It may still be justified by later work. Compare identical inputs and outputs, then examine task counts, shuffle metrics, spill, and duration; the number of exchanges alone does not predict elapsed time.

See Apache Spark Architecture and Execution for local execution examples.

Reference: Apache Spark documentation.