Goodput

Goodput is the amount of useful work completed per unit of resource or time, with the definition of “useful” stated explicitly. Throughput counts everything a system processes. Goodput keeps only what counted toward the goal. The term comes from networking, where it separates the bits a link carries from the bits that arrive correctly and are not retransmissions. It carries over to any system in which some work is wasted.

Throughput and goodput in a training job

A training job that crashes and resumes from a checkpoint recomputes the steps it had done since the last save. The step counter on a dashboard shows those steps twice. The simulation below runs 1,000 steps at one second per step, saves a checkpoint every 100 steps (10 seconds blocked per save), restarts in 60 seconds, and crashes just after steps 250 and 730.

STEP_S, CKPT_S, RESTART_S, CKPT_EVERY, TARGET = 1.0, 10.0, 60.0, 100, 1000
fail_at = [250, 730]                       # the job crashes just after completing these steps

t = 0.0; executed = 0; step = 0; last_ckpt = 0; blocked = restart = 0.0; fails = list(fail_at)
while step < TARGET:
    step += 1; executed += 1; t += STEP_S
    if fails and step == fails[0]:
        fails.pop(0); step = last_ckpt; t += RESTART_S; restart += RESTART_S; continue
    if step % CKPT_EVERY == 0:
        t += CKPT_S; blocked += CKPT_S; last_ckpt = step

print(f"wall time            {t:7.0f} s")
print(f"steps executed       {executed:7d}   -> throughput {executed/t:.3f} steps/s")
print(f"steps of progress    {TARGET:7d}   -> goodput     {TARGET/t:.3f} useful steps/s; runtime fraction {TARGET*STEP_S/t:.3f}")
print(f"rework               {(executed-TARGET)*STEP_S:7.0f} s | checkpoint blocking {blocked:.0f} s | restart {restart:.0f} s")

Output from Python 3.12.14 on macOS 26.7.1:

wall time               1300 s
steps executed          1080   -> throughput 0.831 steps/s
steps of progress       1000   -> goodput     0.769 useful steps/s; runtime fraction 0.769
rework                    80 s | checkpoint blocking 100 s | restart 120 s

The job executed 1,080 steps in 1,300 seconds, a throughput of 0.831 steps per second. It advanced the training run by only 1,000 steps, giving a goodput rate of 1,000 / 1,300 = 0.769 useful steps per second. Because each useful step takes one second in this model, the runtime-goodput fraction is also numerically 0.769: 1,000 useful compute seconds divided by 1,300 wall-clock seconds. The remaining 300 seconds break down into the 80 seconds of re-executed steps (rework), 100 seconds blocked on checkpoint writes, and 120 seconds of restart. Throughput is 0.831 executed steps per second versus 0.769 useful steps per second because it counts the rework as production.

Choosing the numerator and denominator

The ratio is only meaningful with both defined.

• Numerator. For training, the time or compute that advanced the run: steps that survive in a later checkpoint. Steps that are recomputed after a restart, spent in a hang before a timeout, or slowed by a straggler are excluded or discounted. For serving, a common version counts requests that completed within their latency objective, so a response that arrives too late is throughput but not goodput.
• Denominator. Resources held, such as allocated GPU-hours, or elapsed wall-clock time. Whether time spent waiting in a queue is included changes the answer, so say which. A denominator of allocated resources measures waste on the hardware, and one that begins at job submission measures the delay the user experiences.
• Window. A whole run or a recent period. A short window right after a failure looks much worse than the run-long figure.

Google Cloud’s description of ML productivity goodput shows how one number can be factored. It multiplies scheduling goodput (the fraction of time all required resources are available), runtime goodput (time making forward progress as a fraction of the time resources are available), and program goodput (the fraction of peak hardware performance the job extracts, which is also called model FLOPs utilization). The factors correspond to different owners and remedies: capacity and quotas, failure handling and checkpointing, and the efficiency of the training code.

A ledger for a cluster

For a longer job the same arithmetic is a ledger of GPU-hours. The numbers below are a hypothetical example, not measurements: 64 GPUs held for 24 hours is 1,536 GPU-hours, from which the loss categories are subtracted.

category               GPU-hours
startup and restart       38
checkpoint blocking       55
rework                    71
hang until abort          24
straggler slowdown        40
lost total               228
goodput = (1536 - 228) / 1536 = 0.852

The categories matter because each has a different lever: shorter checkpoint writes, a shorter timeout on hung collectives, a more suitable checkpoint interval, or replacing slow nodes. A single utilization figure cannot say which lever to pull, since a hung job and a healthy one both show allocated GPUs at full occupancy, and a job re-executing lost steps looks busy.

Limits

• Goodput says how much of the work was useful, not whether the work was worthwhile. A job can have high goodput training a model that does not improve.
• It depends on what the instrumentation can see. Rework and hang time need step timestamps, resume steps, and checkpoint events; without them the loss is invisible and goodput is overestimated.
• Comparisons across jobs need the same definitions. Two teams that count queue wait differently, or one that excludes straggler time, are reporting different quantities.

References

• Google Cloud blog, Goodput metric as a measure of ML productivity