Spark Driver and Executor

The Spark driver coordinates an application: it owns the context, builds plans, schedules work, and receives action results. Executors run tasks and hold cached data and shuffle state. A cluster manager allocates application resources; it does not replace the driver’s task scheduling role.

A machine can host multiple executors. In classic PySpark, Python and JVM processes participate in the application, and Python workers can run alongside the executor JVM. JVM heap size therefore is not the entire memory footprint. Spark Connect additionally separates the client from a remote driver.

Adding executors does not increase the memory of the process receiving collect or toPandas. A large result can overwhelm that receiver. Use a distributed write for large output, count for a count, or a bounded preview for inspection. A row limit still does not bound the bytes in one unusually large row.

Captured Python values are sent to worker tasks; ordinary variables are not shared writable driver memory. Updating a captured list in a task does not update the original driver list. Return and aggregate results through Spark, and design external writes to tolerate task retries.

See Apache Spark Architecture and Execution for local execution examples.

Reference: Apache Spark documentation.