GPU Fragmentation

GPU fragmentation is the state in which a cluster has enough idle GPUs in total to run a job but not in the shape the job needs. A job that requires eight GPUs on one node cannot use eight idle GPUs spread across four nodes. The free capacity exists and is useless to that job, and utilization dashboards, which count GPUs in use and not the shape of what is free, do not show it.

Where the shape comes from

Fragmentation follows from constraints on which GPUs may serve a request together.

• Node boundaries. Multi-GPU workers usually need their GPUs on one node, because the fast links between GPUs (and between GPUs and the local network adapters) stop at the node edge. A request for eight GPUs is a request for one whole eight-GPU node. Kubernetes expresses a GPU request as a whole number in a container’s limits, and the scheduler matches it against the free count on each node.
• Topology inside a node. Some GPUs on a node are connected more closely than others, and a job that needs the fast connectivity can be refused a set of GPUs that are free but poorly connected.
• Partitioned or shared GPUs. When a physical GPU is divided into slices, free slices on different devices do not combine into a larger slice.
• Memory within one GPU. Inside a process, repeated allocation and release of differently sized buffers can leave free memory in pieces too small for the next allocation. This is the same word for a related effect, but it is handled by the memory allocator and not by the cluster scheduler.

A measurement and an example

A useful number is relative to the request size. For a request of k GPUs on one node, the free GPUs that can serve it are k times the sum of floor(free / k) over the nodes. The stranded share is one minus that amount divided by total free GPUs. The script applies it to two situations: a snapshot of four 8-GPU nodes, and the result of placing eight single-GPU jobs on four empty nodes under two policies, pack (use the fullest node that still has room) and spread (use the emptiest node).

def stranded(free, k=8):                   # share of free GPUs that cannot serve a k-GPU single-node request
    return 1 - k * sum(f // k for f in free) / sum(free)

free = [3, 2, 5, 2]                        # free GPUs on four 8-GPU nodes
print("free total", sum(free), "| largest block", max(free), "| stranded(8) %.2f" % stranded(free),
      "| 8-GPU job fits:", max(free) >= 8)

def place_singles(policy, n_jobs=8):       # eight 1-GPU jobs on four empty 8-GPU nodes
    free = [8] * 4
    for _ in range(n_jobs):
        cand = [i for i in range(4) if free[i] > 0]
        i = min(cand, key=lambda i: free[i]) if policy == "pack" else max(cand, key=lambda i: free[i])
        free[i] -= 1
    return free

for p in ("pack", "spread"):
    f = place_singles(p)
    print(f"{p:6s} free per node {f} largest block {max(f)} stranded(8) {stranded(f):.2f}")

Output from Python 3.12.14 on macOS 26.7.1:

free total 12 | largest block 5 | stranded(8) 1.00 | 8-GPU job fits: False
pack   free per node [0, 8, 8, 8] largest block 8 stranded(8) 0.00
spread free per node [6, 6, 6, 6] largest block 6 stranded(8) 1.00

In the snapshot, 12 GPUs are free and the largest block is 5, so an 8-GPU job cannot start and the stranded share for that request is 1.0. In the placement comparison the same eight jobs and the same 24 free GPUs end very differently. Packing leaves three completely free nodes. Spreading leaves six free GPUs on every node and no node that can host an 8-GPU job, although spreading has its own reasons: it limits the damage from one node failing and spreads heat and load.

Reducing it

• Pack by default. Place small jobs on nodes that are already partly used so that other nodes stay whole, and leave spreading for services that need failure isolation.
• Reserve capacity for large jobs. Dedicate some nodes, or a share of the cluster, to jobs that need whole nodes, so small jobs cannot slowly occupy every node.
• Bound the wait of the oldest large request. Without a limit, a large job can wait indefinitely while smaller jobs keep arriving. Queueing policy has to decide when to hold capacity for it. See gang scheduling for the related all-or-nothing rule.
• Defragment by moving jobs. If a job can be preempted or restarted cheaply, for example because it checkpoints, the scheduler can move small jobs to consolidate free GPUs. For a job that cannot be interrupted without losing its progress this has a cost that must be weighed against the wait it removes.
• Track the stranded share. Measure free GPUs that no pending request can use, and the wait of the largest job class, in addition to utilization.

The simple model above counts only whole-node requests and ignores topology and sharing. Real fragmentation also depends on the mix of request sizes and on how long jobs run, so the measure should be computed for the request sizes the cluster actually sees.

References

• Kubernetes documentation, Schedule GPUs