Catalyst Optimizer
Spark’s optimizer rewrites a query in stages before anything runs. Rules do most of the work; size statistics mainly decide which join implementation you get.
Spark’s optimizer rewrites a query in stages before anything runs. Rules do most of the work; size statistics mainly decide which join implementation you get.