Spark RDD

An RDD, or resilient distributed dataset, represents a collection split into logical partitions that Spark can process in parallel. Its dependencies describe how to compute those partitions. It need not already exist entirely in memory.

A transformation such as map or filter creates a new RDD description. An action such as count requests a result. For eight source partitions followed by a filter, constructing the filter does not by itself evaluate every source record; an action determines which partitions are needed. Discovery and metadata operations may still do work before that action.

RDD transformations do not update the original collection in place. This immutability is not a snapshot guarantee for an external file or database. Recomputing a partition requires suitable available inputs, and changing inputs or nondeterministic functions can change the result.

An arbitrary Python function in RDD.map does not expose relational expressions to the Spark SQL optimizer. A DataFrame provides a schema and a relational plan. Choose the abstraction for the required operations rather than assuming that lazy execution makes every function optimizable.

See Apache Spark Architecture and Execution for local execution examples.

Reference: Apache Spark documentation.