Latency Budget
Step percentiles do not add to an end-to-end percentile. A useful budget reserves fallback time and tests the complete request path under load.
-
-
Spill
Spilling is not an error; it is the engine refusing to crash. It also turns the fast part of a query into the slow part, which is why it shows up in metrics.
-
Spark Performance and Cost Factors
More executors help while tasks await slots. Skew, layout, memory, and serialization can set limits that extra machines do not remove.
-
When Distributed Processing Is Needed
A cluster is not a size upgrade; it is a different set of costs. What actually forces distribution, and what a single node handles that people assume it cannot.
-
Amdahl’s Law
Amdahl’s law shows how serial work limits parallel speedup. Added coordination costs can make more workers slower rather than merely less useful.
-
Out-of-Core Processing
Data bigger than RAM does not require a cluster. Engines stream what they can and spill the rest to disk — with limits worth knowing before you rely on it.