Dynamic Allocation

Dynamic allocation changes the executors assigned to a Spark application as demand changes. Pending work can lead to requests for more executors, while idle executors may be removed. It is different from changing a query’s partition count.

Configure the minimum, initial, and maximum allocation and inspect backlog and idle behavior. Requests depend on cluster capacity, quotas, and startup delay; more requested executors do not mean immediately available task slots.

Removing executors must preserve the shuffle data needed by surviving work through a supported mechanism. Options include a configured external shuffle service or shuffle tracking, with deployment-specific requirements. Cached data and ongoing shuffle use can affect executor removal.

Check actual executor lifetimes, recomputation, latency, and costs. Removing an executor does not necessarily stop a billed worker node, and adding executors cannot divide one indivisible task. Validate recovery and shared-workload effects before broad deployment.

See Optimizing Spark Jobs in Production for the measurement and rollout workflow.

Further reading

• Apache Spark documentation