GPU Scheduling and Resource Isolation

Consider a hypothetical GPU cluster with eight nodes of eight GPUs each. A dashboard shows twelve GPUs idle, and a team’s eight-GPU training job has been pending for an hour. Both facts are true: the idle GPUs are spread across six nodes, and the job needs eight on one node. On the same cluster, two distributed jobs have each been granted three of the four workers they need. Between them they hold twelve GPUs, neither can start, and neither will release what it holds. A third team’s quota says it may use 32 GPUs and it is using 12, but its job still waits because the other team borrowed the difference and has not finished.

None of these is a hardware fault. Each follows from how GPUs are described to a scheduler and from the policies layered on top. The goal of this article is to explain those mechanisms well enough to diagnose such situations and to choose policies deliberately. After reading it you should be able to distinguish GPU memory from GPU compute and say which one limits a given workload; describe how Kubernetes represents GPUs and what its quotas can express; compare whole-GPU allocation, Multi-Instance GPU (MIG), time-slicing, and the CUDA Multi-Process Service (MPS) by what each isolates; explain fragmentation and gang scheduling and why they waste capacity; reason about quotas with borrowing and preemption; and plan for node failure.

The article builds on Versioning Models, Data, Prompts, and Evaluation Artifacts, where a release or a training run is a combination of immutable artifacts, and on Operating Shared Clusters: Tenancy, Policy, and Upgrades, which covers namespaces, ResourceQuota, and admission in Kubernetes. It assumes you know what a container, a Pod, and a node are, and that a scheduler decides which node runs a Pod. No GPU programming knowledge is needed. No GPUs were available while writing, so the behavior of the scheduler itself is shown with simulations written for this article: Python 3.12.14 on macOS 26.7.1, standard library only. They model queues, nodes, and policies. They do not model GPU hardware, and their numbers say nothing about how fast any real device runs.

Memory and compute are different limits

A GPU offers two resources that behave differently under sharing. The first is device memory, the high-bandwidth memory attached to the GPU that holds a model’s weights, intermediate results, and anything else a program allocates. The second is compute, the capacity of the GPU’s processing units to execute work, which the hardware divides among whatever programs are submitting kernels. Memory is a space: a program either has the bytes it asked for or its allocation fails with an out-of-memory error, and no amount of waiting makes more bytes appear. Compute is a rate: running two programs on the same processors slows each of them but lets both proceed.

That difference tells you which resource limits a workload, and the answer is often the first thing to compute when sizing a job. For training, the dominant demand is the model state. The ZeRO paper (Rajbhandari et al., 2019) breaks it down for mixed-precision training with the Adam optimizer: 2 bytes per parameter for 16-bit weights, 2 for 16-bit gradients, and 12 for the 32-bit master copy of the weights plus Adam’s two moment estimates, 16 bytes per parameter in total. Activations and temporary buffers come on top. For serving a decoder-style language model, the demand is the weights plus a key-value cache for every sequence in flight, which per token holds a key and a value vector for each layer and each key-value head. The PagedAttention paper (Kwon et al., 2023) describes this cache as large and dynamically growing and shrinking, which is why managing it limits how many requests a GPU can serve at once.

The script below turns those two rules into numbers for a hypothetical 80 GiB device and a hypothetical decoder with 32 layers, 8 key-value heads, a head dimension of 128, and 16-bit values. The model sizes are round numbers chosen for illustration.

GiB = 2**30
def training_bytes(params):            # mixed-precision Adam: 2 + 2 + 12 bytes per parameter
    return 16 * params
def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes=2):
    return 2 * layers * kv_heads * head_dim * dtype_bytes     # keys and values
GPU = 80 * GiB                         # a hypothetical 80 GiB device
print("training state (weights, gradients, optimizer), excluding activations")
for b in (1, 3, 7, 13):
    need = training_bytes(b * 1e9)
    print(f"  {b:>3}B params: {need/GiB:7.1f} GiB  -> {need/GPU:4.2f} x one 80 GiB GPU")
layers, kvh, hd = 32, 8, 128           # hypothetical 7B-class decoder with grouped-query attention
per_tok = kv_bytes_per_token(layers, kvh, hd)
w = 7e9 * 2                            # fp16 weights
free = GPU - w
print(f"\ninference: weights {w/GiB:.1f} GiB, KV cache {per_tok/1024:.0f} KiB per token")
for ctx in (2048, 8192, 32768):
    seqs = int(free // (per_tok * ctx))
    print(f"  context {ctx:>6}: {per_tok*ctx/GiB:5.2f} GiB per sequence -> at most {seqs:>3} concurrent sequences (ignoring activations and fragmentation)")

The output:

training state (weights, gradients, optimizer), excluding activations
    1B params:    14.9 GiB  -> 0.19 x one 80 GiB GPU
    3B params:    44.7 GiB  -> 0.56 x one 80 GiB GPU
    7B params:   104.3 GiB  -> 1.30 x one 80 GiB GPU
   13B params:   193.7 GiB  -> 2.42 x one 80 GiB GPU

inference: weights 13.0 GiB, KV cache 128 KiB per token
  context   2048:  0.25 GiB per sequence -> at most 267 concurrent sequences (ignoring activations and fragmentation)
  context   8192:  1.00 GiB per sequence -> at most  66 concurrent sequences (ignoring activations and fragmentation)
  context  32768:  4.00 GiB per sequence -> at most  16 concurrent sequences (ignoring activations and fragmentation)

A 7-billion-parameter model needs 104 GiB for its training state alone, so it cannot be trained on one 80 GiB device however long it waits; its state must be split across devices, which is a placement constraint that the scheduler has to honor. A 1-billion-parameter model fits with room to spare, and the leftover memory is what makes sharing the device with other jobs possible. For serving, the same 7-billion-parameter model’s weights take 13 GiB, and what remains goes to the cache: 128 KiB per token means a single 8,192-token sequence holds 1 GiB, and the device can hold at most 66 such sequences, or 16 sequences at 32,768 tokens. The result is a concurrency limit set by memory, even when the processors are idle between requests. These are upper bounds, since real runtimes also need space for activations and lose some to fragmentation.

So memory decides whether a workload fits and how many copies of it share a device, while compute decides how fast each copy runs. Every sharing mechanism below makes some trade between the two, and it helps to ask of each one: who guarantees the memory, and who guarantees a share of the compute.

How Kubernetes sees a GPU

Kubernetes does not know what a GPU is. Accelerators are exposed through the device plugin framework: the administrator installs the vendor’s driver and a device plugin on each GPU node, and the node then advertises a custom resource such as nvidia.com/gpu with a count. The Kubernetes documentation on scheduling GPUs spells out how the resource behaves. It is specified under limits; you may give a limit alone (the request then defaults to the limit) or equal request and limit, but not a request without a limit. It is an integer count. Containers cannot share a device through this interface, because each GPU is assigned to a single container. A minimal pod that asks for one GPU looks like this configuration sketch (not applied to a cluster, since none was available):

apiVersion: v1
kind: Pod
metadata:
  name: trainer-worker-0
spec:
  restartPolicy: Never
  containers:
    - name: worker
      image: registry.example/trainer:v7        # placeholder image
      resources:
        limits:
          nvidia.com/gpu: 8                      # whole devices only

The scheduler treats nvidia.com/gpu: 8 like any other countable resource: the Pod fits on a node whose unallocated count is at least 8. This model has three consequences that the rest of the article keeps returning to. Placement is a counting problem on each node, so one node’s three free GPUs are no help to a Pod that needs eight. Two ways of saying “give me a GPU” cannot be distinguished by count alone, so a cluster with different GPU generations needs node labels or feature-discovery tooling and node selectors to steer Pods to the right ones, and the documentation describes this. And quotas can only count devices: a ResourceQuota names the extended resource with the requests. prefix, for instance requests.nvidia.com/gpu: 4, and for extended resources there is no overcommit, so the quota is a hard count of devices.

Kubernetes has a second, newer way to describe devices. Dynamic Resource Allocation (DRA), first available in Kubernetes v1.30, is documented as stable since v1.35. Where a device plugin advertises a count, DRA lets drivers publish descriptions of individual devices (ResourceSlice objects), lets administrators define classes of devices (DeviceClass), and lets workloads request devices through ResourceClaim and ResourceClaimTemplate objects, with selection by attributes expressed in CEL. The documentation lists the differences that matter here: DRA supports sharing a device between containers and Pods that reference the same claim, supports per-workload configuration instead of only per-node configuration, and filters devices by attribute rather than by count. It also lists a limitation: the scheduler does not preempt Pods that hold DRA resources, so a high-priority Pod that needs a held device stays pending until the holder ends. The practical reading is that DRA gives the scheduler a richer description to place against, and that this does not remove the policy questions below. A scheduler can only place what it can describe, and whichever description is used, fragmentation, gang admission, and quota policy remain your decisions.

Sharing a GPU, and what each mode isolates

A whole-GPU allocation gives one workload the entire device. It is the only mode with no sharing questions, and for training jobs that already use all of a device’s memory and compute, it is usually the right one. Sharing matters when workloads are small relative to the device: many small inference services, notebooks, and experiments. Three mechanisms for it come from NVIDIA’s own documentation.

• **Multi-Instance GPU (MIG).** On supported GPUs (Ampere architecture and later, per the user guide), MIG partitions one physical GPU into up to seven GPU instances. The partition covers the streaming multiprocessors, copy engines and decoders, L2 cache banks, memory controllers and the paths to memory, so each instance has its own isolated route through the memory system. The guide describes the resulting properties as fault isolation, meaning one client cannot disrupt another’s work, and predictable throughput and latency through a consistent cache allocation and memory bandwidth per instance.
• **Time-slicing.** The NVIDIA GPU Operator can advertise a GPU as several replicas, set by a replica count in a configuration map, and hand each replica to a different Pod. Its documentation states plainly that time-slicing provides no memory isolation and no fault isolation between replicas. The GPU divides time among the processes using it, and a Pod that requests two replicas does not thereby get twice the compute of a Pod that requests one.
• **CUDA Multi-Process Service (MPS).** MPS is a runtime service that lets several CUDA processes share a GPU cooperatively, with a control daemon, a server process that owns the GPU scheduling resources, and a client runtime inside the CUDA driver library. The documentation says it reduces context switching and improves utilization, and that it can limit each client: an active thread percentage bounds the share of the GPU’s threads a client may use, and a pinned device memory limit bounds the memory a client can allocate. Clients have isolated GPU address spaces, but error containment is limited: a fatal GPU fault from one client on Volta-class hardware is contained to the clients sharing that GPU, and is reported to all of them without saying which one caused it. They must exit and restart.

Mode	Memory isolation	Compute isolation	Fault isolation	Granularity and fit
Whole GPU	Complete (single tenant)	Complete	Complete	One Pod per device; training and large serving
MIG	Hardware partition per instance	Hardware partition per instance	Yes, per the NVIDIA guide	Up to seven slices on supported GPUs; several small services with predictable latency
Time-slicing	None	Time shared, no guaranteed share	None	Many light, trusting workloads; widest GPU support
MPS	Separate address spaces; optional memory limit	Optional thread-percentage limit	Limited: one fault reaches co-located clients	Cooperating processes that need concurrent kernels

The table sorts modes by what is guaranteed, and the choice follows from the trust boundary set in the platform contract. Teams that trust each other and run small notebooks may accept time-slicing, whose cost is that one Pod allocating too much memory can fail its neighbors, and whose benefit is that it oversubscribes freely. Workloads from tenants who must not interfere with each other belong on MIG instances or separate devices. And no GPU-level mode partitions the other resources a job touches: host CPU and memory, the network, and shared storage are governed by the Kubernetes mechanisms from the earlier tenancy article. A MIG slice isolates the device; it does not stop a neighbor from saturating the node’s network link. The workload isolation entry lists the separate jobs isolation can be asked to do: limiting contention, containing failure, controlling access, and attributing cost. Each GPU mode covers some of them and none covers all.

Placement and fragmentation

When every request is a count on a single node, scheduling GPUs is a bin-packing problem. A node with eight GPUs is a bin; a Pod asking for four, two, or one is an item; and the scheduler must also decide where to put each item. The decision matters because free capacity can be useless when scattered. The scenario in the introduction, twelve idle GPUs and a pending eight-GPU job, is fragmentation: the free total is large and the largest free block on any node is small.

The first simulation measures this. It models eight nodes with eight GPUs, a stream of single-node jobs needing 1, 2, 4, or 8 GPUs (with probabilities 0.35, 0.30, 0.20, and 0.15) and exponentially distributed run times with a mean of 120 minutes, arriving so that the offered load is 80 percent of capacity, over 20,000 simulated minutes. It compares three node-choice policies: first (the lowest-numbered node with enough free GPUs), best (the node with the fewest free GPUs that still fits, which packs tightly), and worst (the node with the most free GPUs, which spreads load evenly). It also compares two queue disciplines: skip-ahead, where a job that cannot be placed lets later jobs be tried, and strict first-in-first-out, where the job at the head of the line blocks everyone behind it. This is the core of the program (an excerpt):

def pick(free, need, policy):
    fits = [i for i, f in enumerate(free) if f >= need]
    if not fits: return None
    if policy == "first": return fits[0]
    if policy == "best":  return min(fits, key=lambda i: (free[i], i))      # tightest fit
    if policy == "worst": return max(fits, key=lambda i: (free[i], -i))     # most free (spreads load)


# inside run(), once per simulated minute:
        for j in list(queue):
            n = pick(free, j["gpus"], policy)
            if n is None:
                if strict_fifo: break           # head of line blocks everything behind it
                continue
            free[n] -= j["gpus"]; running.append((t + j["dur"], n, j["gpus"])); queue.remove(j)
            waits[j["gpus"]].append(t - j["arr"])
        busy += per * nodes - sum(free)
        if any(j["gpus"] == 8 for j in queue) and sum(free) >= 8 and max(free) < 8:
            stranded += 1

The metric strand counts minutes during which an eight-GPU job was waiting, at least eight GPUs were free in total, and no single node had eight free. The results:

2807 jobs, 8 nodes x 8 GPUs, offered load 0.80, 20000 simulated minutes
policy    queueing        util   wait1   wait2   wait4   wait8   max8  strand  queued8
first     skip-ahead     0.748     0.4     1.6     7.0    49.9    575    3046        0
best      skip-ahead     0.748     0.2     1.1     7.4    54.4    393    3297        0
worst     skip-ahead     0.741     1.5     6.2    13.2   171.6    902   10669        2
first     strict FIFO    0.741    37.8    39.1    45.7    57.0    406    4987        2
best      strict FIFO    0.744    33.3    34.8    42.1    51.2    343    4881        2
worst     strict FIFO    0.735    77.5    81.1    90.0   109.5    546   10671        5

The columns wait1 to wait8 are mean waiting times in minutes for jobs of each size, max8 is the longest any eight-GPU job waited, and queued8 is how many eight-GPU jobs were still waiting at the end.

Utilization hardly differs, between 0.735 and 0.748 in every row, which is the first lesson: a cluster-wide utilization number does not reveal fragmentation. The waiting time for the eight-GPU jobs does. Under skip-ahead queueing, packing policies (first and best) keep their mean wait near 50 minutes, while spreading (worst) raises it to 172 minutes and the stranded time from about 3,000 minutes to over 10,000. Spreading is attractive for single-node reliability and for heat, and it is the cause of this behavior: after a few hours every node holds a few small jobs and none has eight free. Tight packing leaves whole nodes empty for the large jobs.

The two queue disciplines trade differently. Skip-ahead lets small jobs flow around a blocked large one, and the mean waits of one- and two-GPU jobs stay under 7 minutes. The price is starvation risk for large jobs: the longest eight-GPU wait under worst is 902 minutes, and two eight-GPU jobs were still waiting at the end. Strict FIFO protects the head-of-line job, and the price is that everything waits behind it, with mean waits of 33 to 78 minutes even for one-GPU jobs. Kueue exposes the same choice as queueing strategies: StrictFIFO makes older workloads that cannot be admitted block newer ones, BestEffortFIFO lets newer workloads that fit proceed, and BestEffortFIFO is the default. Neither is correct in general. The first favors large jobs and the second favors throughput of small ones.

The model leaves out much: no preemption, no defragmentation by moving jobs, no topology (all nodes are equivalent), and no real run-time distribution. What it shows is the mechanism and its direction, not a size to expect in production. Practical remedies follow from it: pack by default, reserve some nodes for large jobs, bound the wait of the oldest large job, and, where jobs can be preempted or restarted cheaply, let the scheduler defragment by moving small jobs.

Gang scheduling

A distributed training job is a set of workers that must run at the same time, since each step synchronizes gradients across all of them. A job with four workers that has three running is not three-quarters done; it is making no progress at all. The default Kubernetes scheduler places Pods one at a time, with no notion that a group of Pods must be placed together or not at all, which is called gang scheduling. That is the second scenario in the introduction, and it can be reproduced exactly.

The second simulation uses a cluster of four nodes with four GPUs each (16 GPUs), and three jobs that each need four workers of two GPUs (8 GPUs per job). The workers’ Pods are created interleaved, as they would be if three controllers reacted at the same time. In pod-by-pod mode, each worker is placed independently whenever a node has room. In gang mode, a job is placed only if all four of its workers fit simultaneously. Each job runs for 60 minutes once all four workers are in place.

"""Three distributed jobs, each needing 4 workers x 2 GPUs, on 4 nodes x 4 GPUs.
Compare placing workers one at a time (as independent pods) with all-or-nothing admission."""

def simulate(mode, nodes=4, per=4, workers=4, gpw=2, njobs=3, duration=60, horizon=300):
    free = [per] * nodes
    placed = {j: [] for j in range(njobs)}          # job -> list of node indices holding its workers
    started = {}; finished = {}
    pending = [(j, w) for w in range(workers) for j in range(njobs)]      # interleaved creation order
    useful = held = 0
    for t in range(horizon):
        for j in list(started):
            if j not in finished and t >= started[j] + duration:
                for n in placed[j]: free[n] += gpw
                finished[j] = t
        if mode == "pod-by-pod":
            for item in list(pending):
                j, w = item
                n = next((i for i, f in enumerate(free) if f >= gpw), None)
                if n is not None:
                    free[n] -= gpw; placed[j].append(n); pending.remove(item)
        else:                                       # gang: admit a job only if all its workers fit now
            for j in range(njobs):
                todo = [x for x in pending if x[0] == j]
                if not todo or len(placed[j]) > 0: continue
                trial = free[:]; spots = []
                for _ in todo:
                    n = next((i for i, f in enumerate(trial) if f >= gpw), None)
                    if n is None: break
                    trial[n] -= gpw; spots.append(n)
                if len(spots) == len(todo):
                    for n in spots: free[n] -= gpw
                    placed[j] = spots; pending = [x for x in pending if x[0] != j]
        for j in range(njobs):
            if j not in started and len(placed[j]) == workers: started[j] = t
        running = [j for j in started if j not in finished]
        useful += gpw * workers * len(running)
        held += per * nodes - sum(free)
    return started, finished, useful, held

for mode in ("pod-by-pod", "gang"):
    s, f, useful, held = simulate(mode)
    print(f"{mode:11} started={dict(sorted(s.items()))} finished={dict(sorted(f.items()))}"
          f"  GPU-minutes held={held}  doing useful work={useful}")

The output over 300 simulated minutes:

pod-by-pod  started={} finished={}  GPU-minutes held=4800  doing useful work=0
gang        started={0: 0, 1: 0, 2: 60} finished={0: 60, 1: 60, 2: 120}  GPU-minutes held=1440  doing useful work=1440

Pod-by-pod placement fills all eight slots with three workers of the first job, three of the second, and two of the third, so every GPU is held (4,800 GPU-minutes over the run) and none does useful work. Nothing will ever change, because each job is waiting for a resource that another waiting job holds. This is a deadlock, and it does not resolve by waiting. Gang admission starts two jobs immediately with all eight of their workers, and the third as soon as GPUs free up at minute 60. All 1,440 GPU-minutes held are useful.

The remedy is all-or-nothing admission, and it is the reason batch systems for ML sit beside the base scheduler instead of relying on it. Kueue’s documentation describes the unit it admits as a Workload, an application that runs from start to finish, and states that admission is all-or-nothing for the workload. Quota is reserved for the whole set or not at all, and a Workload is admitted only when its quota reservation, node capacity (for topology-aware scheduling), and any admission checks are all satisfied. The simulation shows what happens without that property, and it also shows what a gang scheduler does not remove: a gang job needs its workers to be placeable together, and that depends on the fragmentation covered in the previous section. A job with four two-GPU workers on a cluster where no node has two free GPUs waits however much quota it has.

Quotas, queues, borrowing, and preemption

A shared cluster needs a way to divide scarce GPUs among teams, and quota is the usual one. The simplest design assigns each team a fixed number of GPUs, which the Kubernetes ResourceQuota expresses with requests.nvidia.com/gpu in each team’s namespace. Static quotas are easy to explain and have two costs: capacity a team does not use sits idle while another team waits, and the quota says nothing about order, so jobs inside the quota are admitted whenever they are created.

Kueue, a Kubernetes-native job queueing system, separates these concerns into objects the documentation defines. A ResourceFlavor describes a kind of capacity, for instance a GPU model or a node pool. A ClusterQueue is a cluster-wide pool with quotas per flavor and resource and a queueing strategy, and only batch administrators should create it. A LocalQueue is a namespaced entry point that a team submits to. A ClusterQueue gives each resource a nominalQuota, and ClusterQueues that share a cohort can borrow each other’s unused nominal quota, bounded by a borrowingLimit on the borrower and a lendingLimit on the lender. Preemption policies decide whether a workload that is within its own nominal quota may displace workloads that are borrowing. The following is a configuration sketch in the shape the documentation shows (the API group version kueue.x-k8s.io/v1beta2 is the one its current examples use); it was not applied:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: team-a
spec:
  cohort: gpu-pool
  namespaceSelector: {}
  preemption:
    reclaimWithinCohort: LowerPriority
    withinClusterQueue: LowerPriority
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: gpu-a100-80g                    # a ResourceFlavor defined separately
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 32
              borrowingLimit: 16

The third simulation compares three policies on a cluster of eight nodes with eight GPUs, two teams with a nominal quota of 32 GPUs each, and jobs that are gangs of 1, 2, or 4 whole nodes. In the first 600 minutes team A floods the cluster with about 56 GPUs of demand and team B is nearly idle, with about 8. In the next 600 minutes, both are busy, with demands of about 28 and 36 GPUs. The policies are static (no team may exceed its quota), borrow (a team may use idle GPUs beyond its quota, with no preemption), and reclaim (borrow, plus a team that is within quota may preempt the other team’s borrowed jobs). A preempted job loses all progress and rejoins the queue, since this model has no checkpoints. The reclaim decision, an excerpt:

            need = j["nodes"] * PER; team = j["team"]; other = "B" if team == "A" else "A"
            within = usage[team] + need <= QUOTA[team]
            if policy == "static" and not within: continue
            if free_nodes < j["nodes"] and policy == "reclaim" and within:
                victims = sorted([r for r in running if r["team"] == other and usage[other] > QUOTA[other]],
                                 key=lambda r: -r["start"])             # youngest borrowed jobs first
                got = free_nodes
                chosen = []
                left = usage[other]                          # the other team's usage if the chosen victims are removed
                for v in victims:
                    if got >= j["nodes"]: break
                    if left - v["nodes"] * PER < QUOTA[other]: continue      # only borrowed capacity may be reclaimed
                    chosen.append(v); got += v["nodes"]; left -= v["nodes"] * PER
                if got >= j["nodes"]:
                    for v in chosen:
                        running.remove(v); free_nodes += v["nodes"]; usage[other] -= v["nodes"] * PER
                        lost += (t - v["start"]) * v["nodes"] * PER; preempts += 1
                        queue.insert(0, {"team": v["team"], "arr": t, "nodes": v["nodes"], "dur": v["dur"], "first_arr": v["first_arr"]})

The table shows means over 20 random job streams; waits are in minutes, measured from a job’s first arrival to each start.

mean over 20 random job streams (8 nodes x 8 GPUs, quota 32 + 32 GPUs, 1200 simulated minutes of arrivals)
policy    util(0-1200)   A wait   B wait  B wait, 600+  preempts  lost GPU-h
static           0.612    259.4    103.0         117.9       0.0         0.0
borrow           0.791     75.7     83.6          85.9       0.0         0.0
reclaim          0.795    109.4     73.5          82.4       5.8        54.3

Static quotas leave the cluster 61 percent utilized over the 1,200 minutes of arrivals, because team A’s jobs queue while team B’s quota sits empty, and team A’s mean wait is 259 minutes. Borrowing lifts utilization to 79 percent and cuts A’s wait to 76 minutes, with no cost to B in this setting, since B’s mean wait also improves (84 minutes from 103). That B gains as well is a feature of this workload: B’s jobs, arriving into a cluster that is not clogged with A’s backlog, find room sooner. Reclaim shows the usual trade. It shortens the wait for B’s late jobs only a little (from 86 to 82 minutes), because B’s demand is modest, and it makes team A’s wait longer again (109 minutes) and destroys 54 GPU-hours of work on average through 5.8 preemptions per run. Whether that is worthwhile depends on what the reclaiming team is entitled to. If the quota is a promise of access when needed, reclaim is how the promise is kept, and the lost work is the cost of lending. The cost is small when jobs checkpoint, which is why the next article treats checkpointing as part of the scheduling story and not only as a recovery feature.

The setting also exposes a design question that the YAML cannot answer: what a quota is for. A quota can be a ceiling to protect others from a runaway job, a guarantee to give a team capacity on demand, or a budget to charge back. Static quotas serve the first well and the second badly. Borrowing with reclaim serves the second at the price of preemptions. Summing the nominal quotas to more than the real capacity, as some clusters do, promises more than can be delivered if everyone asks at once. The simulation can only compare the policies on one invented workload, so the numbers show direction, not a general ranking.

What sharing and scheduling do not isolate

A scheduler assigns resources; it does not enforce what the workload does with them. Several limits are worth stating, because they decide how far a tenancy promise can reach.

• **Memory pressure crosses Pods in modes without memory isolation.** Under time-slicing, one Pod that allocates device memory aggressively can cause allocation failures in others sharing the device. MPS memory limits and MIG partitions are the controls that stop this, and only where configured.
• **Faults propagate under shared contexts.** Time-slicing has no fault isolation, and the MPS documentation describes a fatal fault reaching the co-located clients. When a fault in one tenant’s code can restart another tenant’s process, the sharing boundary is a trust boundary, and it should match the tenancy decision taken in the platform design.
• **Quota counts devices, not what they cost.** A GPU of one generation and a GPU of another count the same in requests.nvidia.com/gpu. Separate flavors, node labels, or separate quota resources are how a platform keeps a team from consuming scarce devices under a generic quota.
• **Host resources are outside the GPU.** CPU, host memory, network, and storage are shared according to the Kubernetes mechanisms in the earlier article. A well-isolated GPU slice with a saturated data loader or network link still performs badly, and it can slow neighbors that share that link.
• **Preemption reaches only what the scheduler manages.** The documentation notes the DRA limitation above, and with any preemption the cost falls on the preempted job’s progress, which is only as small as its latest checkpoint.

Capacity and failure

Capacity planning for GPUs is dominated by the fact that devices are coarse, expensive, and sometimes unavailable. The last simulation reuses the first one’s job stream and removes a node: the same 80-percent offered load now meets seven nodes (an offered load of 0.91) and then six (1.07), using the best-fit policy.

nodes up  offered load   util   wait1   wait4   wait8  queued at end
8                 0.80  0.748     0.2     7.4    54.4              0
7                 0.91  0.842     2.7    18.9   172.6             34
6                 1.07  0.947    36.6   176.0   596.7             52

Losing one of eight nodes removes 12.5 percent of capacity and multiplies the mean wait of eight-GPU jobs by more than three (54 to 173 minutes) with 34 jobs unfinished at the end. Losing two makes the offered load exceed capacity: waits climb into the hundreds of minutes and 52 jobs are still queued at the end. The effect is nonlinear because queueing delay rises sharply as load approaches capacity, so a cluster run near full utilization has no slack for failure, and the same fragmentation effects get worse as capacity shrinks. The practical conclusions are to keep headroom for the failures you intend to ride out (a cluster that must tolerate one node loss at the large-job tier has to be sized for the load after that loss), and to measure waiting time of the largest job class, which degrades first.

What happens to the workload on a failed node depends on layers above the hardware. In Kubernetes the node controller taints a node that becomes not ready or unreachable, and the documentation states that Pods are given a default toleration of 300 seconds for those taints unless set otherwise, so they remain bound to the failing node for five minutes. For a gang job that period matters: one lost worker stalls all the others, which keep holding their GPUs and doing nothing until the job is restarted. A platform therefore needs the job controller or queueing layer, not only the node controller, to notice a lost worker and tear the whole gang down, and it needs the job to be able to restart from saved state, rather than from the beginning. How failures appear in a distributed job, and how checkpoints and restart limit the wasted work, is the subject of the next article.

Two operational habits follow from the simulations. First, watch the metrics that expose the problems here and not only utilization: the wait of the oldest large pending job, the count of GPUs free but unplaceable (stranded), GPU-hours lost to preemption, and the time gang jobs spend holding partial allocations. Second, treat scheduling policy as part of the platform contract: the queueing strategy, borrowing limits, the preemption rule, and the sharing mode for each tier are promises to consumers, and the contract from the first article in this module is where they belong.

From scheduling to surviving failure

This article followed a GPU from the memory arithmetic that decides whether a workload fits, through the Kubernetes description of devices and the sharing modes with their isolation limits, to the policies that decide who runs when: packing against fragmentation, all-or-nothing admission for gangs, and quotas with borrowing and preemption. It ended with the observation that a scheduler’s work is undone when a worker dies in the middle of a job. The next article, Training Jobs: Checkpoints, Resumption, and Distributed Failure, follows a training job through its life: how it fails, what a checkpoint must contain to make a restart equivalent to an uninterrupted run, and how to choose a checkpoint interval that balances saving cost against lost work.

References

• Kubernetes documentation, Schedule GPUs
• Kubernetes documentation, Dynamic Resource Allocation
• Kubernetes documentation, Resource Quotas
• Kubernetes documentation, Taints and Tolerations
• Kueue documentation, Concepts
• Kueue documentation, Cluster Queue
• NVIDIA Multi-Instance GPU User Guide
• NVIDIA GPU Operator documentation, Time-Slicing GPUs in Kubernetes
• NVIDIA Multi-Process Service documentation
• Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention