Spark Persistence and Lineage

Persistence selects how Spark may retain computed partitions for reuse. Lineage describes the dependencies used to compute an RDD. Calling persist selects a policy; it does not immediately fill the cache. An action materializes the partitions it needs.

For a persisted filtered RDD, a first count can compute all partitions and a second count can reuse them. Removing the cache with unpersist does not erase lineage, so another action can compute the RDD again. Partial actions may populate only part of the cache.

If an executor holds a partition’s only cached copy and disappears, Spark needs a surviving copy or recomputation from available dependencies. Lost shuffle output can require upstream work too. Available, unchanged inputs and deterministic transformations matter: lineage does not back up a deleted input or promise the same result from a changed source.

Reliable checkpointing stores a recovery boundary and can truncate lineage; local checkpointing sacrifices that durability. Neither executor lineage recovery nor a cache alone restores an entire application after driver loss. Retried writes to external systems still need their own retry-safe identity and commit design.

See Apache Spark Architecture and Execution for local execution examples.

Reference: Apache Spark documentation.