Logical and Physical Plan

A logical plan describes what relational operations a query requests, such as filtering, joining, and grouping. A physical plan selects execution operators and algorithms for carrying out that work.

In Spark, analysis resolves column names and types. Catalyst applies valid rewrites before physical planning chooses operators using available information. Runtime adaptation can revise supported choices. An unresolved column is therefore a different problem from an expensive but valid join strategy.

Use explain(“extended”) to compare parsed, analyzed, optimized, and physical plans. Read from the sources toward the result, checking required columns, predicates, joins, aggregates, and exchanges. A SQL physical operator ID is not a scheduler stage ID.

A rewrite must preserve semantics, including nulls and outer joins. A right-side WHERE condition after a left join can remove unmatched rows, whereas moving it into ON can retain them. A plan is evidence of execution choices, not proof of business correctness or measured runtime savings.

See Spark DataFrames and Spark SQL for worked examples.

Reference: Apache Spark documentation.