Repartition and Coalesce

In Spark DataFrames, repartition requests redistribution, optionally by specified columns. coalesce reduces the count through a narrow dependency without adding a full shuffle. They change execution partitions, not directly a table’s directory partitioning.

repartition(n, key) hash-partitions by the key, so a hot equal key still shares one destination. repartition(n) uses round-robin distribution in the tested Spark SQL path. Neither promises equal processing time for rows with unequal costs.

coalesce cannot split a single input partition into eight useful parallel inputs. A drastic reduction can also reduce upstream parallelism. It need not merge only partitions on one executor, and it does not preserve the exact skew ratio of the input.

Use the distribution required by downstream work; another immediate redistribution can make an earlier one redundant. Before writing, consider directory values and file rollover as well as task count. One writer task may produce several data files, so verify the committed layout and restored rows.

See Spark Partitioning, Shuffles, and Data Skew for worked examples.

Reference: Apache Spark documentation.