TPU Clusters
TPU clusters combine accelerator memory and compute through communication. Parallelism, batch size, input supply, and recovery determine useful scaling.
-
-
Distributed Training with PyTorch DDP
Multiple GPUs are not free speed. Here is what actually has to be synchronized.
-
Training Job Checkpoints and Distributed Failure
A distributed training job stops when any one worker fails, and what it loses depends on its checkpoints. This guide covers the job lifecycle, distributed failure modes, what a checkpoint must contain, atomic writes, verified resumption, and choosing the interval with a waste model.
-
Goodput
Throughput counts everything a system processes, including repeated and wasted work. Goodput counts only the part that advanced the goal, which makes it the right measure for jobs that fail, restart, and recompute.
-
Gang Scheduling
Gang scheduling admits a group of interdependent workers only when all of them can be placed together. It prevents jobs from holding resources while waiting for each other, at the price of waiting for a large enough block of free capacity.