Adaptive Query Execution

Adaptive query execution, or AQE, allows Spark SQL to revise selected physical-plan choices using statistics collected during execution. Examples include combining small shuffle partitions, adjusting eligible joins, and handling supported skewed-join cases.

An initial plan is a proposed route with estimates. After an exchange produces runtime statistics, Spark can adapt later work. For a small grouped sum, an initial eight-way shuffle can be read through fewer coalesced partitions. This does not mean that the original input always has fewer partitions or that earlier work was never performed.

Compare the initial and final executed plans, then inspect actual tasks and runtime metrics. An AQEShuffleRead coalesced marker describes an observed plan choice. It does not guarantee the same outcome for different data, settings, or Spark versions, and its query-stage labels are not scheduler stage IDs.

AQE optimizes supported SQL execution paths; it does not automatically optimize arbitrary Python RDD functions. It cannot determine whether a business metric is correct. Keep the input and answer fixed when assessing a plan change, and measure runtime rather than inferring a speedup from a smaller partition count.

See Apache Spark Architecture and Execution for local execution examples.

Reference: Apache Spark documentation.