Data Skew

Data skew is an uneven distribution of data or work across processing units. A customer key holding most records can create a large hash partition, but row width and per-row computation also matter.

A long-running task is a symptom to investigate. Compare records, input and shuffle bytes, spill, garbage collection, and host behavior. A duration forty times the median does not prove forty times the data, and a zero median needs special care.

For ordinary hash partitioning, equal keys select the same bucket. More buckets may reduce collisions with other keys but do not divide one key. A round-robin distribution may spread rows only for a later key aggregation to regroup them.

Choose a remedy for the observed operator: suitable partial aggregation, a supported broadcast join, deliberate hot-key salting, or eligible adaptive skew handling. Preserve join and aggregation semantics and verify complete results before comparing performance.

See Spark Partitioning, Shuffles, and Data Skew for worked examples.

Reference: Apache Spark documentation.