Repartition and Coalesce
Choose redistribution or narrow partition merging, preserve the needed downstream layout, and distinguish task counts from committed data files.
-
-
Broadcast Join
Understand the build-side copy in a broadcast join, preserve match multiplicity, and avoid treating file size as a safe executor memory budget.
-
Key Salting
Split a hot key with a stable row-based salt, merge valid partial states, and preserve duplicates and unmatched rows when salting a join.
-
Data Skew
Diagnose uneven Spark work using records, bytes, and runtime metrics, and distinguish a hot key from other causes of slow tasks.
-
Python UDF
Define a Python function’s input and return contract, handle nulls, inspect its execution boundary, and compare it with equivalent built-in expressions.
-
Column Pruning
Remove unneeded columns while retaining filter, join, and grouping inputs, and distinguish a smaller projection from measured storage savings.
-
Logical and Physical Plan
Separate a query’s relational intent from execution operators, read analyzed and optimized plans, and avoid confusing plan nodes with task stages.
-
Spark DataFrame
Understand a Spark DataFrame as rows with a schema and relational plan, build expressions, and distinguish transformations from collected results.
-
Spark DataFrames and Spark SQL
Understand Spark transformations through explicit schemas, join rules, numeric limits, reporting days, and Python UDF versus built-in expression semantics.