Spill
A spill is an engine writing intermediate data to local disk because it does not fit in the memory allotted to the operation, then reading it back to finish the work. Sorting, grouping, joining, and shuffling all build state that grows with the data, and when that state exceeds the budget, spilling is what happens instead of failing.
Where the budget comes from
In Spark, the memory available for execution and storage is a documented fraction of the heap: spark.memory.fraction defaults to 0.6 of heap space minus 300 MB, and the documentation notes that lowering it means “spills and cached data eviction occur” more frequently. Within that region, spark.memory.storageFraction protects cached blocks from eviction, and raising it means “less working memory may be available to execution and tasks may spill to disk more often.” Both defaults come with an explicit recommendation to leave them alone.
Spark’s shuffle behaves the same way on the map side: tasks organize records in memory and, when capacity is exceeded, sort and write them to disk. Other engines implement the same idea with their own knobs — DuckDB, for instance, spills to a temporary directory whose location is configurable.
What it costs, and what it saves
- It saves the query. Without spilling, an operation that exceeds memory fails outright. This is the same mechanism that lets a single machine process datasets larger than its RAM — see out-of-core processing.
- It changes the speed class. The step now runs at the speed of the temporary storage, writing data out and reading it back rather than keeping it in memory.
- It adds a capacity requirement. The temporary directory needs room. A job that spills tens of gigabytes per executor can fail on disk space rather than on memory, which reads as a confusing error at first.
Reading it as a signal
Spill volume per task is one of the more useful numbers in a job’s metrics, because it distinguishes two problems that look alike from the outside:
| Observation | Likely reading | First move |
|---|---|---|
| Every task spills a similar amount | Each task is handling more data than its memory allows | More partitions, so each task holds less; or more memory per executor |
| One task spills enormously, the rest not at all | Skew — the partitioning, not the memory, is the problem | Fix the key distribution rather than the memory setting |
| Spill appears only after a code change | A new wide operation, a cached dataset holding memory, or a lost broadcast | Compare the plans before and after |
The order that usually works: reduce the data per task, remove caching that is not paying for itself, and only then raise memory — because a larger container hides the cause without removing it. How spill fits with skew, file sizes, and resource allocation is worked through in Why More Executors Stopped Helping.
References: Apache Spark documentation, Configuration; Apache Spark documentation, RDD Programming Guide; DuckDB Documentation, How to tune workloads.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.