Spark DataFrame

A Spark DataFrame represents rows organized into named columns with a schema and a relational computation. It can refer to distributed data; it is not a pandas table already held entirely in the Python process.

select chooses columns or derives expressions, filter retains rows where its condition is true, and groupBy with an aggregation changes the output grain. A Column expression describes a computation on rows; it is not a Python scalar value that can be tested with an ordinary if statement.

Transformations normally build plans. An action such as collect requests results, while schema inference and discovery can do work earlier. collect brings the result to the receiver, so a large distributed input does not make a large local result safe.

DataFrame expressions and SQL use Spark SQL planning. A temporary view gives the DataFrame a session-scoped SQL name without automatically materializing it. Check schema and result semantics first; equivalent results do not promise identical plan text.

See Spark DataFrames and Spark SQL for worked examples.

Further reading

• Apache Spark documentation