COST Metric
A system that scales beautifully may still be slower than one thread. COST names the hardware a platform needs before it beats a good single-threaded program.
-
-
When Distributed Processing Is Needed
A cluster is not a size upgrade; it is a different set of costs. What actually forces distribution, and what a single node handles that people assume it cannot.
-
Adaptive Query Execution
Read how Spark revises SQL physical plans with runtime statistics, compare initial and final plans, and avoid treating coalescing as guaranteed.
-
Spark Persistence and Lineage
Distinguish cached partitions from the recipe to recompute them, check what survives failure, and separate lineage from a durable input backup.
-
Spark Shuffle
Understand how Spark regroups records across partitions, why partial aggregation helps, and why an exchange alone does not measure network bytes.
-
Spark Job Stage and Task
Trace an action through Spark jobs, stages, and tasks, distinguish attempts from logical work, and avoid a universal shuffle-count formula.
-
Spark Driver and Executor
Separate Spark coordination from task execution, account for Python worker memory, and understand why collect puts pressure on the receiver.
-
Spark RDD
Understand an RDD as partitioned data with dependencies, distinguish transformations from actions, and keep immutability separate from source snapshots.
-
Apache Spark Architecture and Execution
Understand Spark drivers, executors, partitions, memory limits, caching, and the conditions for recomputation and recovery.