Python UDF

A Python user-defined function lets Spark SQL invoke Python logic with a declared return type. Catalyst generally cannot inspect the arbitrary Python body as it can inspect built-in relational expressions.

Scalar Python UDFs cross a Python worker boundary. Arrow-enabled scalar execution and pandas UDFs change data transfer and batching, but are not identical interfaces. Neither implies a fixed speedup. Inspect the execution plan and test the installed Spark version and configuration.

Handle null and exceptional input inside the function. Do not rely on a surrounding conditional to guarantee that unsafe function code is never evaluated. A UDF need not run last, and Spark can still optimize eligible surrounding expressions or discard an unused result.

Compare semantics before timing: Python -101 // 100 gives -2, while division followed by truncation toward zero gives -1. Test negative values, nulls, overflow, and the return type. Prefer a suitable built-in when it expresses the intended computation, then benchmark equivalent outputs rather than attributing every timing difference to Python.

See Spark DataFrames and Spark SQL for worked examples.

Reference: Apache Spark documentation.