Catalyst Optimizer
Catalyst is the query optimizer inside Spark SQL. It takes the query a user wrote — as SQL or as DataFrame operations — and rewrites it into an execution plan, applying transformations to a tree representation of the query at each step. The 2015 Spark SQL paper by Armbrust and colleagues introduces it as a “highly extensible optimizer” built with Scala language features so that composable rules, code generation, and extension points are easy to add.
Four phases
The paper describes rule sets for four phases of query execution, and Spark’s EXPLAIN output still mirrors them:
| Phase | What it settles |
|---|---|
| Analysis | Resolving names against the catalog — which table, which column, what type |
| Logical optimization | Rewrites that do not change the answer: pushing filters toward the scan, trimming unused columns, folding constants |
| Physical planning | Choosing operators that the execution engine can run, including which join implementation |
| Code generation | Compiling parts of the query to JVM bytecode rather than interpreting operators row by row |
Rules and costs
The paper states that “Catalyst supports both rule-based and cost-based optimization,” and is specific about how the two divide the labor. Rules carry the rewriting. For the cost side, several physical plans are generated and one is chosen with a cost model — but, as the paper puts it, “at the moment, cost-based optimization is only used to select join algorithms: for relations that are known to be small, Spark SQL uses a broadcast join.”
That sentence describes Spark as of the paper, and later versions added more statistics-driven optimization, so check the behavior of the version you run. The practical lesson has not changed: the join strategy depends on size estimates. When statistics are missing or stale, the optimizer is guessing, and the plan it produces will reflect the guess rather than the data.
Why it is worth knowing about
- The plan explains the runtime. Reading the optimized logical plan shows whether a filter reached the scan; reading the physical plan shows how many shuffles the query will do.
- The optimizer works on what it can see. Logic hidden in a user-defined function is opaque to the rules, which is one reason expressing work in built-in operations tends to run faster.
- Optimization is not execution. Modern Spark re-optimizes at runtime with adaptive query execution, so the plan that ran can differ from the plan that was printed.
How these plans become stages and tasks is worked through in How Spark Decides What to Run. For the same idea in a relational database, see query execution plan.
References: Michael Armbrust et al., Spark SQL: Relational Data Processing in Spark, SIGMOD 2015; Apache Spark documentation, EXPLAIN.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.