Apache Airflow Fundamentals
Follow an Airflow workflow from definition to execution, choose interval and retry rules, and understand what a successful run actually confirms.
-
-
Single-Node Analytics: DuckDB, Polars, and Arrow
Compare DuckDB, Polars, and Arrow with Parquet queries, null-safe joins, and interchange checks. Separate result correctness from memory and speed claims.
-
Testing and Debugging Spark Data Pipelines
Build fixtures and independent expected results, test real storage boundaries, and reduce Spark failures into useful regression tests.
-
Building Incremental and Idempotent Pipelines
Define incremental bounds and retry identity, preserve newer updates and deletes, and coordinate publication, progress, and backfills.
-
Optimizing Spark Jobs in Production
Diagnose production Spark work, preserve query results, compare repeated runs, and decide whether a faster job also meets cost and delivery requirements.
-
Spark Partitioning, Shuffles, and Data Skew
Understand partition distributions, salted aggregates and joins, broadcast plans, coalesce, and Parquet file counts while preserving query semantics.
-
Spark DataFrames and Spark SQL
Understand Spark transformations through explicit schemas, join rules, numeric limits, reporting days, and Python UDF versus built-in expression semantics.
-
Apache Spark Architecture and Execution
Understand Spark drivers, executors, partitions, memory limits, caching, and the conditions for recomputation and recovery.