Dynamic Resource Allocation

Dynamic resource allocation lets a running application adjust how much compute it holds: it returns executors it is no longer using and asks for more when demand rises. Spark’s documentation describes it as dynamically adjusting the resources an application occupies based on the workload, and notes it is particularly useful when several applications share a cluster. Behavior below follows Spark 4.2.0 documentation, checked in September 2026.

How it behaves

• It is off by default. Enabling it takes more than one switch: along with spark.dynamicAllocation.enabled, the deployment must provide an external shuffle service, shuffle tracking, or decommissioning.
• Executors are removed after an idle period, governed by spark.dynamicAllocation.executorIdleTimeout, which defaults to 60 seconds.
• Executors holding cached data are not removed by default, to avoid losing that data. A separate timeout controls whether they ever are.

Why shuffle data is the complication

During a shuffle, each executor writes its map output to local disk and serves those files to other executors later in the query. If the executor is released before the data has been consumed, the documentation states the consequence directly: the shuffle files it wrote “must be recomputed unnecessarily.” That is the reason for the prerequisite.

How the prerequisite is satisfied differs, though, and the three options are not the same mechanism. Spark’s documentation states the shared goal — “to allow executors to be removed without deleting shuffle files written by them” — and then lists distinct ways to get there.

• An external shuffle service serves the files after the executor is gone. It is “a long-running process that runs on each node of your cluster independently of your Spark applications and their executors,” and with it enabled “any shuffle state written by an executor may continue to be served beyond the executor’s lifetime.” The distinguishing feature is where the data stays: the files remain on the original node, served by a process that outlives the executor.
• Shuffle tracking keeps the executor alive instead. Nothing is served after death here — the executor is simply not released while it holds shuffle data that may still be needed. The configuration reference makes the shape clear: the tracking timeout “controls the timeout for executors that are holding shuffle data,” and its default of infinity means Spark “will rely on the shuffles being garbage collected to be able to release executors.” So the cost of this option is executors held longer than an idle timeout alone would suggest.
• Decommissioning migrates the blocks first. With graceful decommission enabled, Spark “will try to migrate all the RDD blocks … and shuffle blocks … from the decommissioning executor to a remote executor,” and will decommission rather than kill an executor when dynamic allocation is on. Here too the executor goes away and the output stays available — but from whichever executor now holds it, not from the original node. The qualifier is in the documentation’s own wording: it tries, so a migration that does not complete before shutdown leaves the work to be recomputed.

So two of the three let the executor terminate with its output still reachable, and they differ in where it lives; the third keeps the executor instead. That is the distinction to carry, rather than “only the external service survives termination.”

It matters when reading a cluster that is not shrinking as expected. With an external service, executors going away while shuffle files remain readable is the intended behaviour. With tracking, executors that look idle but refuse to be released are also the intended behaviour, and the lever is the tracking timeout rather than the idle timeout. Diagnosing the second as a bug in dynamic allocation is the common mistake.

Enabling dynamic allocation without one of those arrangements produces the worst of both worlds: executors are reclaimed and the work they did is recomputed, so the cluster spends money re-deriving data it already had.

Where it pays, and where it does not

It is also worth separating this from cloud autoscaling, which adds and removes machines. Dynamic allocation moves executors between applications within whatever cluster exists; if the underlying nodes are fixed and reserved, returning executors reduces contention but not the bill. The two are usually worth configuring together, and each has its own delay before capacity arrives.

How this fits with skew, memory, and the rest of what decides a job’s time and cost is worked through in Why More Executors Stopped Helping.

References: Apache Spark documentation, Job Scheduling; Apache Spark Documentation, Configuration.