Spark DataFrames and Spark SQL
Understand Spark transformations through explicit schemas, join rules, numeric limits, reporting days, and Python UDF versus built-in expression semantics.
-
-
Predicate Pushdown
Understand when a reader can apply filters or skip data, why overlapping statistics are only candidates, and how to verify actual work.
-
Data Formats: Parquet, ORC, Avro, and Arrow
Learn what each format stores, how readers skip data, and which schema and layout choices change analytical work.