Why More Executors Stopped Helping: Skew, Memory, Files, and the Cost Curve
Doubling the cluster rarely halves the runtime. What actually sets the limit is the slowest task — and the fixes for it are usually not more machines.
Doubling the cluster rarely halves the runtime. What actually sets the limit is the slowest task — and the fixes for it are usually not more machines.