Spill
A spill is an engine writing intermediate data to local disk because it does not fit in the memory allotted to the operation, then reading it back to finish the work. Sorting, grouping, joining, and shuffling all build state that grows with the data, and for operators that support it, spilling is what happens when that state exceeds the budget instead of failing.
Where the budget comes from
In Spark, the memory available for execution and storage is a documented fraction of the heap: spark.memory.fraction defaults to 0.6 of heap space minus 300 MB, and the documentation notes that lowering it means “spills and cached data eviction occur” more frequently. Within that region, spark.memory.storageFraction protects cached blocks from eviction, and raising it means “less working memory may be available to execution and tasks may spill to disk more often.” Both defaults come with an explicit recommendation to leave them alone.
Spark’s shuffle behaves the same way on the map side: tasks organize records in memory and, when capacity is exceeded, sort and write them to disk. Other engines implement the same idea with their own knobs — DuckDB, for instance, spills to a temporary directory whose location is configurable.
What it costs, and what it saves
- It keeps the query alive — for the operators that can spill. Without spilling, an operation that exceeds memory fails outright, and an operation whose state cannot be spilled, such as some aggregate states, still can. This is the same mechanism that lets a single machine process datasets larger than its RAM, within limits — see out-of-core processing.
- It changes the speed class. The step now runs at the speed of the temporary storage, writing data out and reading it back rather than keeping it in memory.
- It adds a capacity requirement. The temporary directory needs room. A job that spills tens of gigabytes per executor can fail on disk space rather than on memory, which reads as a confusing error at first.
Reading it as a signal
Spill volume per task is one of the more useful numbers in a job’s metrics, because it distinguishes two problems that look alike from the outside:
| Observation | Likely reading | First move |
|---|---|---|
| Every task spills a similar amount | Each task is handling more data than its memory allows | More partitions, so each task holds less; or more memory per executor |
| One task spills enormously, the rest not at all | Skew is the leading hypothesis — confirm it by comparing that task’s input, output, and state size with the others, and whether it was competing with cache or concurrent tasks for memory | Fix the key distribution if the data is uneven; if the work is uneven at equal input, the operation is the problem |
| Spill appears only after a code change | A new wide operation, a cached dataset holding memory, or a lost broadcast | Compare the plans before and after |
A usual order is to reduce the data per task, remove caching that is not paying for itself, and then raise memory; which is right depends on the measurements — a larger container can hide a partitioning cause, and can equally be the correct fix when every task spills a similar amount. How spill fits with skew, file sizes, and resource allocation is worked through in Why More Executors Stopped Helping.
Further reading
- Apache Spark documentation, Configuration
- Apache Spark documentation, RDD Programming Guide
- DuckDB Documentation, How to tune workloads
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.