Model Serving Latency, Batching, and Autoscaling
Consider a hypothetical team that serves a summarization model behind an internal API. In a test with one client, the service answers in about 450 ms. On launch day the same service, with a healthy GPU and an unchanged model, shows a 700 ms median and a 99th percentile of nine seconds. During the afternoon peak, requests begin to time out. The team adds replicas by hand, the new replicas need four minutes to load the model, and the peak is over before they help. The scenario is invented, but each step in it is ordinary, and none of them is a model-quality problem. The causes are queueing, a batching setting chosen without a latency target, a scaling signal that reacted too late, and no plan for what the service should do when demand exceeds capacity.
A model server is a queue in front of an expensive machine. Four decisions determine how that queue behaves: how requests are grouped into batches, how many replicas exist and how that number changes, what latency the service promises, and what the service does when it cannot keep the promise. This article takes them in that order, after the queueing arithmetic that governs all four. By the end you should be able to explain why tail latency rises much faster than average latency as load approaches capacity, choose and bound a batching policy, pick an autoscaling signal and predict how a slow model load affects it, write a latency target for a streaming language-model endpoint, define an overload policy, and design a benchmark that does not hide the tail.
The latency figures come from small discrete-event simulations written in Python for this article and listed in full. They model queues, batches, and scaling decisions under stated assumptions. They are not measurements of a GPU or of any serving product, and the parameters (a 40 ms fixed batch cost, a 20 ms decode step) are illustrative round numbers. Where a mechanism comes from a product, it is described from that product’s documentation, and the documentation is listed in the references.
What this article builds on
Serving a model means running replicas that have each loaded one specific version of an artifact. Versioning Models, Data, Prompts, and Evaluation Artifacts covers how those versions are identified, so “the model” below always means one immutable version. Scheduling GPU Workloads and Managing Resource Isolation covers how a replica obtains accelerator memory and compute from a shared cluster. Here a replica is a unit of capacity that takes time to start and costs money while it exists. General background in HTTP services and Kubernetes Deployments is assumed. No queueing theory is assumed; the needed pieces are introduced as they appear.
Latency is waiting plus service
A request’s latency is the time it waits for the server plus the time the server spends on it (plus network time outside the server, which this article leaves out). The service time is a property of the model and the hardware. The waiting time is a property of the load, and almost every technique below acts on the waiting time. Response Latency explains why the measurement boundary matters when comparing latency numbers from different places.
Two quantities describe load. The arrival rate \(\lambda\) is how many requests arrive per second. Utilization \(\rho\) is the fraction of the server’s capacity in use: for a single server, \(\rho = \lambda \times\) (mean service time), and a server with \(\rho = 1\) has no spare capacity at all. Concurrency and Request Rate separates request rate from the number of requests in flight, which is the distinction the next result formalizes.
Little’s law: concurrency, rate, and time
Little’s law relates three averages of a stable system:
\[ L = \lambda W \]
Here \(L\) is the average number of requests in the system (waiting plus in service), \(\lambda\) is the average arrival rate, and \(W\) is the average time a request spends in the system. The law does not depend on how arrivals are distributed or how service times vary, and it applies to any part of the system. Applied to the servers alone, it says the average number of busy slots equals the arrival rate times the mean service time.
Its most useful role is sizing. If a service receives 8 requests per second and each request takes 2 seconds on average, then on average \(8 \times 2 = 16\) requests are in flight. A replica that can run 4 requests at once is fully busy at 4 requests in flight, so 16 in flight needs 4 replicas at 100 percent utilization, or \(16 / (4 \times 0.6) = 6.67\), which rounds up to 7 replicas if you want to run at 60 percent. This arithmetic returns in the autoscaling section.
The following simulation draws Poisson arrivals at 8 requests per second into a single FIFO server whose service time averages 100 ms with lognormal variation, records the arrival and departure of every request, and measures the time-average number in the system by integrating over events.
import random
# Little's law check: time-average number in system L versus (arrival rate) x (mean time in system) W
rng = random.Random(7)
lam, m, n = 8.0, 0.100, 200_000
t, arr = 0.0, []
for _ in range(n):
t += rng.expovariate(lam); arr.append(t)
free, dep = 0.0, []
for x in arr:
start = max(free, x)
free = start + m * rng.lognormvariate(-0.5**2 / 2, 0.5)
dep.append(free)
events = [(x, +1) for x in arr] + [(d, -1) for d in dep]
events.sort()
area, cur, prev = 0.0, 0, events[0][0]
for tm, d in events:
area += cur * (tm - prev); cur += d; prev = tm
span = events[-1][0] - events[0][0]
L = area / span
W = sum(d - a for a, d in zip(arr, dep)) / n
print(f"observed lambda = {n/span:.3f}/s, W = {W*1000:.1f} ms, lambda*W = {n/span*W:.3f}, time-average L = {L:.3f}")observed lambda = 8.006/s, W = 356.7 ms, lambda*W = 2.856, time-average L = 2.856The two sides agree because both count the same request-seconds. On a finished simulation that is an accounting identity, and what the law adds is a way to relate three quantities you can measure independently in production: the number of requests in flight from a gauge, the request rate from a counter, and the time in the system from a histogram. If two of them are known, the third follows.
Why the tail grows faster than the average
For a single server with Poisson arrivals and general service times, the Pollaczek-Khinchine formula gives the mean time a request waits before service starts:
\[ W_q = \frac{\rho}{1-\rho} \cdot \frac{1 + C^2}{2} \cdot E[S] \]
\(E[S]\) is the mean service time, \(C\) is the coefficient of variation of the service time (its standard deviation divided by its mean), and \(\rho\) is utilization. Two factors matter. The factor \(\rho / (1 – \rho)\) grows slowly at low utilization and without bound as \(\rho\) approaches 1: it is 1 at 50 percent, 9 at 90 percent, and 19 at 95 percent. The factor \((1 + C^2)/2\) says that variable service times make everything worse. A deterministic server (\(C = 0\)) waits half as long as one with exponentially distributed service times (\(C = 1\)) at the same utilization.
The mean is only part of the story, because users and timeouts react to the tail. The next simulation sweeps utilization for a single FIFO server with a mean service time of 100 ms and lognormal variation (\(C^2 \approx 0.28\)). It compares the simulated mean wait with the formula and reports percentiles. The listed helper simulate_batch is the engine used for the batching experiments later. Here it runs with a batch size of 1.
import random, math
def pct(xs, q):
"""nearest-rank percentile of a list"""
s = sorted(xs)
k = max(1, math.ceil(q / 100 * len(s)))
return s[k - 1]
def simulate_batch(lam, B, T, a, c, n=200_000, sigma=0.3, seed=1):
"""Single server, Poisson arrivals at rate lam (req/s).
A batch of b requests takes (a + c*b) seconds times a lognormal jitter with mean 1.
The server starts a batch when B requests are waiting or the oldest has waited T seconds,
but never before it is free."""
rng = random.Random(seed)
t, arr = 0.0, []
for _ in range(n):
t += rng.expovariate(lam)
arr.append(t)
free, i, lat, wait, sizes, last = 0.0, 0, [], [], [], 0.0
while i < n:
lead = arr[i]
full_at = arr[i + B - 1] if i + B - 1 < n else float("inf")
start = max(free, lead, min(full_at, lead + T))
j = i
while j < n and j - i < B and arr[j] <= start:
j += 1
b = j - i
svc = (a + c * b) * rng.lognormvariate(-sigma**2 / 2, sigma)
done = start + svc
for k in range(i, j):
lat.append(done - arr[k]); wait.append(start - arr[k])
sizes.append(b); free = done; last = done; i = j
return dict(lat=lat, wait=wait, mean_batch=sum(sizes) / len(sizes), thr=n / (last - arr[0]))import math
from qsim import *
m, sigma = 0.100, 0.5 # mean service time 100 ms, lognormal jitter
print("rho mean_wait_sim mean_wait_PK p50_ms p95_ms p99_ms")
for rho in [0.30, 0.50, 0.70, 0.80, 0.90, 0.95]:
lam = rho / m
r = simulate_batch(lam, B=1, T=0, a=m, c=0, sigma=sigma, n=400_000)
es2 = m * m * math.exp(sigma**2) # E[S^2] of mean-m lognormal
pk = lam * es2 / (2 * (1 - rho)) # Pollaczek-Khinchine mean queueing delay
w = sum(r["wait"]) / len(r["wait"])
L = r["lat"]
print(f"{rho:.2f} {w*1000:9.1f} ms {pk*1000:9.1f} ms {pct(L,50)*1000:7.0f} {pct(L,95)*1000:7.0f} {pct(L,99)*1000:7.0f}")rho mean_wait_sim mean_wait_PK p50_ms p95_ms p99_ms
0.30 27.4 ms 27.5 ms 106 286 420
0.50 64.0 ms 64.2 ms 130 399 588
0.70 148.7 ms 149.8 ms 187 651 964
0.80 252.3 ms 256.8 ms 260 956 1425
0.90 558.8 ms 577.8 ms 477 1852 2733
0.95 1171.5 ms 1219.8 ms 891 3789 5479The simulated mean wait matches the formula within a few percent, which is a check on the simulator (the small shortfall at 0.95 is finite-sample noise in a heavy-tailed quantity). The percentiles show what the formula does not. Going from 50 to 90 percent utilization multiplies the mean wait by about 9 and the 99th percentile by about 4.6, from 588 ms to 2,733 ms. At 95 percent the 99th percentile is 5,479 ms, 55 times the mean service time. A request that needs 100 ms of work spends most of its life waiting for requests ahead of it. Percentile covers how percentiles are computed and why they cannot be averaged across servers.
Three consequences follow, and the rest of the article builds on them.
- Capacity planning should target the utilization at which the tail still meets its objective, not 100 percent. For this workload and a 1-second 99th percentile target, that would be near 70 percent.
- Variance is a lever. Anything that makes service times more uniform, such as limiting the maximum output length, reduces waiting at every utilization.
- The curve is optimistic. Poisson arrivals are smoother than real traffic, which arrives in bursts. Burstier arrivals make the same utilization worse. Dean and Barroso, in “The Tail at Scale”, note that latency episodes unimportant in a small system come to dominate overall performance as utilization and system size grow.
Batching and queues
Why servers batch
An accelerator runs a forward pass more efficiently when it processes several requests together. A pass has costs that do not depend on how many requests share it (launching work, reading the model’s weights from memory) and costs that grow with each request. Batching spreads the fixed part across the requests in the batch. A simple model is that a batch of \(b\) requests takes \(a + c\,b\) seconds, where \(a\) is the fixed cost and \(c\) is the per-request cost. When batches of size \(b\) are always available, the server completes
\[ \text{capacity}(b) = \frac{b}{a + c\,b} \]
requests per second. With the illustrative values \(a = 40\) ms and \(c = 5\) ms, a batch of 1 gives 22 requests per second, a batch of 8 gives 100, and a batch of 32 gives 160. As \(b\) grows, capacity approaches \(1/c = 200\) requests per second, so the per-request cost sets a ceiling. Batching also lengthens each pass, since a batch of 32 takes 200 ms, and every request in the batch waits for the whole pass. Larger batches buy capacity and cost latency per pass.
A dynamic batcher has two knobs
A server cannot wait for a fixed number of requests, since at low traffic it would wait forever. A dynamic batcher forms a batch from whatever is queued and is governed by two settings: the largest batch it may form, and the longest it may hold the oldest request waiting for company. NVIDIA Triton Inference Server documents this as max_batch_size together with the dynamic batcher’s max_queue_delay_microseconds: when a full batch cannot be formed, the batcher delays dispatch as long as no request has waited longer than the configured delay, and sends the batch as is when the delay expires. A request that arrives during the delay and completes a batch triggers an immediate send.
The simulator in qsim.py implements that rule. The batch starts at the latest of three times: when the server becomes free, when the oldest queued request has arrived, and the earlier of “the batch is full” and “the oldest request has waited T seconds”. With T = 0 the batcher is greedy: whenever the server is free it takes whatever has accumulated, up to B. The next experiment compares five policies on the cost model above, across arrival rates from light to near capacity. Cells show the median and 99th percentile latency in milliseconds and the mean batch size the policy formed.
from qsim import *
a, c = 0.040, 0.005 # a batch of b requests takes 40 ms + 5 ms per request
configs = [("B=1, no batching", 1, 0.0), ("B=8, wait 0", 8, 0.0), ("B=8, wait 20ms", 8, 0.020),
("B=32, wait 0", 32, 0.0), ("B=32, wait 50ms", 32, 0.050)]
rates = [10, 20, 60, 90, 140]
print("cells: p50 / p99 latency in ms, mean batch size\n")
print(f"{'config (capacity)':<26s}" + "".join(f"{r:>18d} req/s" for r in rates))
for name, B, T in configs:
cap = B / (a + c * B)
cells = []
for lam in rates:
if lam >= 0.98 * cap:
cells.append("over capacity".rjust(22)); continue
r = simulate_batch(lam, B, T, a, c, n=150_000)
cells.append(f"{pct(r['lat'],50)*1000:.0f} / {pct(r['lat'],99)*1000:.0f}, b={r['mean_batch']:.1f}".rjust(22))
print(f"{name+f' ({cap:.0f}/s)':<26s}" + "".join(cells))cells: p50 / p99 latency in ms, mean batch size
config (capacity) 10 req/s 20 req/s 60 req/s 90 req/s 140 req/s
B=1, no batching (22/s) 54 / 196, b=1.0 197 / 1191, b=1.0 over capacity over capacity over capacity
B=8, wait 0 (100/s) 52 / 125, b=1.1 63 / 141, b=1.3 92 / 211, b=3.5 152 / 578, b=6.6 over capacity
B=8, wait 20ms (100/s) 66 / 127, b=1.2 70 / 141, b=1.6 92 / 215, b=3.7 153 / 534, b=6.6 over capacity
B=32, wait 0 (160/s) 52 / 125, b=1.1 63 / 141, b=1.3 91 / 208, b=3.5 117 / 299, b=6.6 228 / 730, b=18.7
B=32, wait 50ms (160/s) 92 / 144, b=1.5 91 / 152, b=2.0 101 / 214, b=4.5 121 / 303, b=7.4 227 / 765, b=18.9Reading the table row by row and column by column gives the main lessons.
- Capacity is the first effect. Without batching the server is over capacity at 60 requests per second and already at the edge at 20 (99th percentile 1,191 ms). The batching policies keep the 99th percentile between 141 and 152 ms at that load.
- The ceiling determines the tail at a given load, as the previous section predicts. At 90 requests per second,
B=8runs at about 90 percent of its 100 requests per second ceiling and has a 99th percentile of 578 ms.B=32runs at about 56 percent of 160 and has 299 ms. The larger maximum batch does not help by batching more at that load (mean batch sizes are both 6.6). It helps by moving the server down the utilization curve. - Greedy batching adapts without being asked. At 10 requests per second the mean batch is 1.1, and at 90 it is 6.6, because requests pile up while the server is busy. The queue itself forms the batches.
- A wait timeout adds latency at low load and buys efficiency, not speed. At 10 requests per second, waiting up to 20 ms raises the median from 52 to 66 ms and waiting up to 50 ms raises it to 92 ms. The mean batch size rises from 1.1 to 1.2 and 1.5. With the cost model, server time per request drops from \(40/1.1 + 5 = 41\) ms to \(40/1.5 + 5 = 32\) ms, a saving in GPU time, which matters when the hardware is the expensive part and the latency budget has slack. Near capacity the timeout changes little, since batches fill on their own.
In practice B is bounded by accelerator memory and by the per-pass time that the latency target tolerates, and T is set to a small fraction of the latency budget. Triton also exposes queue controls on the same batcher: max_queue_size, and default_timeout_microseconds with a timeout_action that rejects or defers a request whose time in the queue exceeds a limit. Those controls matter again in the overload section.
Serving language models: continuous batching
Language-model generation breaks the assumption behind the batcher above. A batch is no longer one pass over the inputs. The model generates one output token per forward pass (called an iteration or decode step), each new token depends on the previous ones, and the number of tokens a request will produce is not known in advance. One request in a batch may stop after 8 tokens and another after 400.
If the server forms a batch and runs it until every member finishes (static batching), the early finishers idle while the longest request completes, and no new request can join until the whole batch ends. The Orca paper (Yu et al., OSDI 2022) names the problem and proposes iteration-level scheduling: the scheduler runs one iteration at a time, returns finished requests immediately, and admits waiting requests at the next iteration boundary. This is what serving engines now call continuous batching. The paper also introduces selective batching, because requests at different positions in their generation have different shapes, so only some operations can be batched across them.
Scheduling is only half of it. Each active sequence holds a key-value (KV) cache, the stored attention state for all tokens so far, and the cache grows by one entry per token. If memory for it is reserved contiguously for the longest possible output, much of it sits unused and fragments. The vLLM paper (Kwon et al., SOSP 2023) proposes PagedAttention, which stores the KV cache in fixed-size blocks, as an operating system pages memory, and reports near-zero KV cache waste and 2 to 4 times the throughput of earlier systems at the same level of latency. A larger effective batch is what that memory efficiency buys. vLLM’s documentation also describes what happens when memory still runs out: the engine preempts requests, puts them back in the waiting queue, and recomputes them later, which shows up as extra latency. It describes chunked prefill, which splits a long prompt into pieces so that decode steps of other requests are not stalled behind it, and a per-iteration token budget (max_num_batched_tokens) whose smaller values favor inter-token latency and larger values favor time to first token.
Simulating static and continuous batching
The following simulation isolates the scheduling difference. An iteration with \(b\) active sequences takes 20 ms plus 0.5 ms per sequence, the batch holds at most 16 sequences, and output lengths follow a skewed lognormal distribution (mean about 59 tokens, minimum 4, maximum 512). Static batching starts a batch of whatever is waiting, runs until the longest member finishes, and returns every result at that point. Continuous batching admits waiting requests at every iteration and returns each request when its last token is produced. Time to first token (TTFT) is the wait for a slot plus the first iteration.
import random, math
from qsim import pct
A, C, B = 0.020, 0.0005, 16 # one decode iteration with b active sequences takes 20 ms + 0.5 ms * b
def step(b): return A + C * b
def workload(lam, n, seed):
rng = random.Random(seed); t = 0.0; reqs = []
for _ in range(n):
t += rng.expovariate(lam)
out = min(512, max(4, int(rng.lognormvariate(math.log(40), 0.9)))) # output tokens
reqs.append((t, out))
return reqs
USEFUL = [0, 0] # useful slot-iterations, total slot-iterations (static batching)
def static_batching(reqs):
"""Take up to B waiting requests, run every iteration until the longest one finishes,
and return all results when the batch ends."""
now, i, n, res = 0.0, 0, len(reqs), []
while i < n:
now = max(now, reqs[i][0])
j = i
while j < n and j - i < B and reqs[j][0] <= now: j += 1
batch = reqs[i:j]; b = len(batch)
first = now + step(b) # first token of every request in the batch
longest = max(o for _, o in batch)
now += longest * step(b) # finished slots still occupy the batch
USEFUL[0] += sum(o for _, o in batch); USEFUL[1] += longest * b
for arr, out in batch: res.append((first - arr, now - arr, out))
i = j
return res
def continuous_batching(reqs):
"""Admit waiting requests at every iteration boundary; return a request when it finishes."""
now, i, n, active, res = 0.0, 0, len(reqs), [], []
while i < n or active:
if not active: now = max(now, reqs[i][0])
while i < n and len(active) < B and reqs[i][0] <= now:
active.append([reqs[i][0], reqs[i][1], reqs[i][1], None]); i += 1
now += step(len(active))
keep = []
for r in active:
if r[3] is None: r[3] = now - r[0] # time to first token
r[1] -= 1
if r[1] == 0: res.append((r[3], now - r[0], r[2]))
else: keep.append(r)
active = keep
return res
mean_out = sum(o for _, o in workload(1, 100_000, 5)) / 100_000
print(f"mean output length {mean_out:.1f} tokens; batch cap B={B}")
print(f"iteration-level capacity at full batch: {B/step(B)/mean_out:.1f} req/s ({B/step(B):.0f} tokens/s)\n")
print(f"{'rate':>5s} {'policy':<11s} {'TTFT p50':>9s} {'TTFT p99':>9s} {'e2e p50':>9s} {'e2e p99':>9s} (seconds)")
for lam in [1.0, 2.0, 4.0, 8.0]:
reqs = workload(lam, 8000, 11)
for name, fn in [("static", static_batching), ("continuous", continuous_batching)]:
USEFUL[:] = [0, 0]
r = fn(reqs); ttft = [x[0] for x in r]; e2e = [x[1] for x in r]
print(f"{lam:5.1f} {name:<11s} {pct(ttft,50):9.2f} {pct(ttft,99):9.2f} {pct(e2e,50):9.2f} {pct(e2e,99):9.2f}" + (f" useful slot-iterations {USEFUL[0]/USEFUL[1]:.0%}" if name == "static" else ""))mean output length 58.6 tokens; batch cap B=16
iteration-level capacity at full batch: 9.7 req/s (571 tokens/s)
rate policy TTFT p50 TTFT p99 e2e p50 e2e p99 (seconds)
1.0 static 0.84 8.93 2.95 15.31 useful slot-iterations 54%
1.0 continuous 0.03 0.04 0.83 7.40
2.0 static 3.07 25.48 8.12 35.65 useful slot-iterations 32%
2.0 continuous 0.03 0.04 0.86 7.75
4.0 static 626.50 1154.36 630.88 1160.27 useful slot-iterations 26%
4.0 continuous 0.03 0.05 0.92 8.23
8.0 static 1119.34 2120.89 1123.57 2127.28 useful slot-iterations 26%
8.0 continuous 0.05 2.16 1.24 9.57At full batches the continuous policy can produce 571 tokens per second, which is 9.7 requests per second at this output length. Static batching wastes most of that. The last column shows the share of occupied slot-iterations that produced a token anyone wanted: 54 percent at 1 request per second, where batches are small, and 26 percent once batches fill. That fraction multiplies the capacity, so static batching saturates at roughly 2.5 requests per second here, which matches the table: at 2 requests per second its TTFT 99th percentile is 25 seconds, and at 4 the queue grows for the whole run (the median TTFT of 626 seconds is a backlog that never drains). Continuous batching keeps the TTFT 99th percentile at 0.04 to 0.05 seconds up to 4 requests per second and reaches 2.16 seconds at 8, approaching its 9.7 per second limit. Its end-to-end 99th percentile of 7 to 10 seconds comes from the longest outputs, since a 512-token response needs 512 iterations regardless of the scheduler.
The model leaves out the prefill cost of long prompts, KV memory limits and preemption, and any difference between the first and later iterations. The comparison shows the scheduling effect only. Real engines add the memory mechanisms above to make the large batch feasible.
Latency targets
A latency target says which latency, at which percentile, over which window, under which load. “Fast” is not a target. Three percentiles carry different information. The median describes the typical request. The 95th and 99th describe what a user sees on a bad day, and for a user who makes many calls in a session, a high percentile is the likely experience. High percentiles also need enough samples. A one-minute window at 2 requests per second contains 120 requests, and with nearest-rank percentiles the 99th is the second-largest value, so a single stall moves it. Windows for tail objectives should contain many hundreds of requests, or the objective should be evaluated over a longer interval.
Language-model endpoints that stream tokens need more than one number, because the user perceives two different waits. vLLM’s metrics documentation defines the relevant ones.
| Metric | What it measures | What mostly degrades it | Who notices |
|---|---|---|---|
| Time to first token (TTFT) | Request arrival to the first generated token | Queueing, prompt length (prefill work), preemption | Interactive users waiting for a response to start |
| Inter-token latency (ITL) | Time between consecutive streamed outputs | Iteration time (batch size), prefill work scheduled into the same iteration, preemption | Users reading a stream; downstream parsers |
| Time per output token (TPOT) | Per-request average: (end-to-end latency minus TTFT) divided by (output tokens minus 1) | Same as ITL, averaged over a request | Throughput-oriented clients |
| End-to-end latency | Arrival to the last token | TTFT plus output length times TPOT | Non-streaming and batch clients |
Because end-to-end latency is roughly TTFT plus \((n – 1)\) times TPOT for \(n\) output tokens, a single end-to-end target mixes requests that differ in \(n\) by orders of magnitude. A workable target is written per class of request and per metric, in the form: for requests with prompts under 2,000 tokens, at the planned peak rate, the 99th percentile TTFT is at most 1 second and the 99th percentile ITL at most 100 ms, measured at the server, over 10-minute windows. The numbers in that sentence are examples of the form, not recommendations. The right values come from what the product needs and from the benchmark in the last section.
Replica scaling
The control loop
Batching sets how much one replica can do. The number of replicas sets how much the service can do in total, and an autoscaler changes that number as load changes. Kubernetes’ Horizontal Pod Autoscaler (HPA) is the baseline design, and KEDA and KServe’s HPA integration build on it. It is a control loop that runs periodically (every 15 seconds by default), reads a metric, and computes
\[ \text{desiredReplicas} = \left\lceil \text{currentReplicas} \times \frac{\text{currentMetricValue}}{\text{desiredMetricValue}} \right\rceil \]
With 6 replicas averaging 90 percent of some resource against a 60 percent target, the controller asks for \(\lceil 6 \times 1.5 \rceil = 9\) replicas. Several details of the documented behavior shape what happens in practice. The controller skips changes when the ratio is within a tolerance of 1.0, which is 0.1 by default. Pods that are not yet ready are set aside, and when a scale-up is being computed the controller assumes the not-yet-ready pods consume none of the metric, which damps the size of the step. Scale-down is slowed by a stabilization window, 300 seconds by default, during which the controller uses the highest replica count it computed. The default scale-up policy permits doubling or adding four pods per 15 seconds, whichever is larger. HPA can scale to zero replicas only when it is driven by an object or external metric (the HPAScaleToZero feature gate, enabled by default in the current documentation), because a resource metric such as CPU cannot be measured on pods that do not exist.
Choosing the signal
The loop is only as good as its metric. For model serving, the candidates differ in how directly they track unmet demand.
| Signal | What it tells you | Weakness for model serving |
|---|---|---|
| CPU utilization | Busy host cores, available to HPA with no extra setup | The accelerator is the bottleneck, and host CPU can be low while the GPU is saturated |
| GPU utilization | How busy the device is | Measures how busy, not how much more it can take; a device running small batches can look busy and still have headroom |
| Request rate | Arrivals per second | Cost per request varies widely for language models (prompt and output length), so the same rate means different load |
| Requests in flight per replica (concurrency) | Load in the unit Little’s law uses | Needs a per-replica target that depends on the batching configuration |
| Queue length or waiting requests | Demand that has no capacity yet; a leading indicator | Zero until capacity is exceeded, so it says nothing about how far below capacity the system is |
| KV cache usage | How close the engine is to preemption | Specific to engines that expose it |
KServe’s documentation shows two routes. In Knative deployment mode, the Knative Pod Autoscaler scales on concurrency or qps against a scaleTarget, and the docs state that the target is a soft limit that a burst can exceed. In raw deployment mode, KEDA drives the HPA from Prometheus queries, and the KServe example scales a vLLM service on a vLLM request-count metric. KEDA, according to its documentation, feeds metrics to the HPA and can scale a workload to zero when its event source is empty. vLLM exports histograms for TTFT, ITL, and end-to-end latency, and gauges for running and waiting requests and for KV cache usage, which is the data these scalers need. Concurrency is the signal closest to the arithmetic of the previous section: a target of \(k\) in-flight requests per replica at a planned utilization comes directly from Little’s law.
A subtler problem is that replicas are not interchangeable at the request level. If a load balancer spreads requests evenly by count, one replica can receive several long generations while another sits idle, because the cost of a language-model request depends on its output length, which is unknown at arrival. Average metrics hide this. Per-replica queue depth, or routing that sends each request to the replica with the least work in flight, addresses it. The simulation below uses one shared queue and so omits the effect.
Cold start: the delay the loop cannot see
A new replica contributes nothing until it has been scheduled onto an accelerator, started its container, fetched the model weights, loaded them into accelerator memory, and passed a readiness check. For a large model these steps take minutes. The control loop decides, the replica appears much later, and the demand that triggered the decision has been waiting the whole time. KServe’s autoscaling guide shows the effect at small scale: in its load test, five requests took about ten seconds because they arrived while pods were being created and a model downloaded, and the guide notes that pulling a serving image that is not cached on the node takes longer. For a multi-gigabyte model those ten seconds become minutes.
The next simulation reproduces the control loop with HPA’s documented defaults, which are a 15-second tick, a 10 percent tolerance, a 300-second scale-down stabilization window, and not-yet-ready replicas counted as zero load. The metric is the average number of requests in flight over the last 60 seconds, divided by the replicas’ slots. Each replica serves 4 requests at once and a request takes 2 seconds on average, so a replica handles 2 requests per second. Traffic is 3 requests per second, steps to 8 at 300 seconds, and returns to 3 at 900 seconds. The scaling target is 60 percent. The experiment varies only the delay between the decision and the replica becoming ready. The last case uses a pre-provisioned floor of 7 replicas, which Little’s law gave as the number needed at the peak.
import heapq, math, random
from collections import deque
from qsim import pct
SLOTS, MEAN_SVC = 4, 2.0 # each replica serves 4 requests at once; mean service time 2 s
TICK, TARGET, TOL, AVG_WINDOW = 15.0, 0.60, 0.10, 60
def rate_at(t): return 8.0 if 300 <= t < 900 else 3.0 # req/s: step up at 300 s, back down at 900 s
def run(load_delay, min_rep=3, max_rep=12, down_window=300.0, horizon=1500.0, seed=3):
rng = random.Random(seed)
ev, seq = [(0.0, 0, "arr", None), (TICK, 1, "tick", None), (1.0, 2, "sample", None)], 3
heapq.heapify(ev)
ready, starting, busy = min_rep, 0, 0
q, samples, desired_log, rows, out = deque(), deque(maxlen=AVG_WINDOW), [], [], []
def push(t, kind, p=None):
nonlocal seq; heapq.heappush(ev, (t, seq, kind, p)); seq += 1
def start_jobs(now):
nonlocal busy
while q and busy < ready * SLOTS:
arr = q.popleft(); busy += 1; out.append((arr, now - arr)) # (arrival time, queue wait)
push(now + rng.expovariate(1 / MEAN_SVC), "dep")
while ev:
now, _, kind, p = heapq.heappop(ev)
if now > horizon: break
if kind == "arr":
q.append(now); start_jobs(now); push(now + rng.expovariate(rate_at(now)), "arr")
elif kind == "dep":
busy -= 1; start_jobs(now)
elif kind == "up":
ready += 1; starting -= 1; start_jobs(now)
elif kind == "sample":
samples.append(busy + len(q)); push(now + 1.0, "sample") # requests in flight, queued ones included
elif kind == "tick":
cur = ready + starting
# average per-replica load; replicas still starting count as zero load, as the Kubernetes HPA assumes
metric = (sum(samples) / len(samples)) / (cur * SLOTS)
ratio = metric / TARGET
want = cur if abs(ratio - 1) <= TOL else math.ceil(cur * ratio)
want = max(min_rep, min(max_rep, want))
desired_log.append((now, want))
recent = [w for tt, w in desired_log if now - tt <= down_window]
new = want if want > cur else min(cur, max(recent)) # scale-down: highest desired over the window
if new > cur:
for _ in range(new - cur): push(now + load_delay, "up"); starting += 1
elif new < cur:
ready = max(min_rep, ready - (cur - new))
rows.append((now, ready, starting, len(q)))
push(now + TICK, "tick")
return out, rows
if __name__ == "__main__":
cases = [("start delay 5 s", dict(load_delay=5)), ("start delay 60 s", dict(load_delay=60)),
("start delay 180 s", dict(load_delay=180)), ("start delay 180 s, min replicas 7", dict(load_delay=180, min_rep=7))]
for name, kw in cases:
out, rows = run(**kw)
print(f"\n{name}")
print(" window (s) requests p99 queue wait (s) peak queue replicas ready/starting at end")
for lo in range(240, 1260, 120):
w = [l for a, l in out if lo <= a < lo + 120]
qs = [r[3] for r in rows if lo <= r[0] < lo + 120]
last = [r for r in rows if lo <= r[0] < lo + 120][-1]
print(f" {lo:4d}-{lo+120:<4d} {len(w):9d} {pct(w,99):18.1f} {max(qs):11d} {last[1]:9d}/{last[2]}")
waits = [l for a, l in out]
print(f" waited more than 5 s: {sum(1 for x in waits if x > 5)} of {len(waits)}; replica-seconds used: {sum(r[1]+r[2] for r in rows)*TICK:.0f}")
start delay 5 s
window (s) requests p99 queue wait (s) peak queue replicas ready/starting at end
240-360 656 4.9 26 9/0
360-480 955 0.0 0 11/0
480-600 956 0.0 0 11/0
600-720 878 0.0 0 7/0
720-840 956 0.0 0 7/0
840-960 653 0.0 0 7/0
960-1080 368 0.0 0 7/0
1080-1200 363 0.0 0 7/0
1200-1320 327 0.0 0 4/0
waited more than 5 s: 4 of 7372; replica-seconds used: 9630
start delay 60 s
window (s) requests p99 queue wait (s) peak queue replicas ready/starting at end
240-360 661 20.8 95 3/9
360-480 932 17.9 158 12/0
480-600 964 0.0 0 12/0
600-720 875 0.0 0 12/0
720-840 956 0.0 0 7/0
840-960 651 0.0 0 7/0
960-1080 372 0.0 0 7/0
1080-1200 370 0.0 0 7/0
1200-1320 321 0.0 0 4/0
waited more than 5 s: 585 of 7356; replica-seconds used: 10380
start delay 180 s
window (s) requests p99 queue wait (s) peak queue replicas ready/starting at end
240-360 661 21.1 95 3/9
360-480 970 53.2 347 3/9
480-600 993 34.9 390 12/0
600-720 866 0.0 0 12/0
720-840 960 0.0 0 12/0
840-960 649 0.0 0 8/0
960-1080 373 0.0 0 8/0
1080-1200 358 0.0 0 7/0
1200-1320 334 0.0 0 4/0
waited more than 5 s: 1746 of 7434; replica-seconds used: 11280
start delay 180 s, min replicas 7
window (s) requests p99 queue wait (s) peak queue replicas ready/starting at end
240-360 642 0.0 0 7/0
360-480 950 0.0 0 7/0
480-600 968 0.0 0 7/0
600-720 870 0.0 0 7/0
720-840 962 0.0 0 7/0
840-960 643 0.0 0 7/0
960-1080 379 0.0 0 7/0
1080-1200 370 0.0 0 7/0
1200-1320 319 0.0 0 7/0
waited more than 5 s: 0 of 7353; replica-seconds used: 10500With a five-second start delay the scaler keeps up. Only the window around the step shows a visible wait (99th percentile 4.9 seconds), and four requests in the run wait more than five seconds. With 60-second delays 585 requests wait more than five seconds, and with 180-second delays 1,746 do, with a 99th-percentile wait of 53 seconds in the worst window and a backlog of nearly 400 requests. The same decision logic, the same target, and the same traffic produce very different results, because the loop acts on the load it saw three minutes earlier. The slow case also overshoots. The queue that built up during the delay keeps the metric high, so the controller keeps requesting replicas that have not arrived yet, and it later holds 12 replicas for the stabilization window after the surge has passed.
The last case removes the wait entirely, at a cost. Seven replicas for the whole run used 10,500 replica-seconds, close to the 11,280 of the slow reactive case, and more than twice the 4,500 that 3 replicas would use over the same time. Here the surge lasts 600 of the 1,500 seconds. If surges were rare and short, the floor would cost more than the reactive scaler and the choice would favor accepting some slow requests. Whether to pay for headroom is a decision about the latency objective and the budget, and the simulation shows its terms.
Common ways to shrink the problem, each with a price:
- Keep a minimum number of replicas sized for the expected peak ramp, as in the last case. This costs idle accelerators.
- Lower the target utilization so that a fraction of the current capacity is always spare. This is the same cost expressed continuously.
- Scale on a leading signal such as queue length or the arrival-rate trend, and allow large scale-up steps.
- Shorten the start: keep images and model weights cached on nodes, use faster storage, and avoid work at startup that is not needed to serve. This lowers the delay in the table above, which is the most valuable lever.
- Schedule capacity for predictable surges, such as a daily peak, instead of waiting for the loop to see it.
- Avoid scale-to-zero for interactive endpoints with large models, because the first request pays the entire cold start.
New replicas also depend on the cluster having free accelerators. If the scheduler cannot place the pod, the start delay is unbounded, which is a reason to reserve capacity for latency-critical services as the scheduling article describes.
When demand exceeds capacity
Sometimes the load is simply higher than the capacity of the replicas that exist, either because the autoscaler has not caught up, the cluster is out of accelerators, or traffic exceeds the maximum the service was sized for. The policy for that situation determines whether the service degrades gracefully or collapses.
A queue with no bound accepts every request. Waiting time then grows with the backlog, and Little’s law explains why: the queue length equals the arrival rate times the waiting time, so a queue that holds 700 requests at a capacity of 20 per second means each new arrival waits 35 seconds. Clients give up long before that, typically after a timeout. The server still processes the abandoned requests, since it cannot know the client left, so its capacity is spent on responses no one will read. The result is that useful throughput falls toward zero while the server is fully busy, and recovery is slow because the backlog must drain even after load decreases.
The remedy is to limit how much is accepted. Google’s SRE book chapter on cascading failures describes the same approach: limiting queue length is a form of per-task throttling, and one effective approach is to return HTTP 503 (service unavailable) to any new request when the number of requests in flight exceeds a limit. It also describes switching from first-in-first-out to last-in-first-out or using controlled delay to drop requests unlikely to be worth processing, and it lists excessively long queues, which raise latency and use memory, among the causes of failure under overload. The chapter on handling overload adds client-side adaptive throttling: a client that sees many of its requests rejected starts rejecting some locally, because even a rejection costs the backend resources. Backpressure covers how a bounded queue passes the capacity limit upstream to the producer.
The next experiment sends the same arrival stream to five policies. The server has 8 slots and a mean service time of 0.4 seconds, so its capacity is 20 requests per second. For 120 seconds the offered load is 26 per second (130 percent of capacity), and then 14 per second. Clients consider a response useful only if it arrives within 2 seconds. “In time per second” is the goodput: responses delivered within that limit, averaged over the 300-second run. The deadline policy drops queued requests that have already waited more than one second, since a response started that late will probably miss the 2-second limit.
import heapq, random
from collections import deque
from qsim import pct
SLOTS, MEAN_SVC, SLO = 8, 0.4, 2.0 # capacity = 8 / 0.4 = 20 req/s; clients give up after 2 s
def rate_at(t): return 26.0 if t < 120 else 14.0 # 130% of capacity for 120 s, then 70%
def run(policy, qmax=40, max_wait=1.0, horizon=300.0, seed=9):
rng_arr, rng_svc = random.Random(seed), random.Random(seed + 1) # same arrivals for every policy
ev, seq = [(0.0, 0, "arr", None)], 1
q, busy = deque(), 0
stat = dict(offered=0, rejected=0, dropped=0, served_ok=0, served_late=0)
waits = []
def push(t, kind, p=None):
nonlocal seq; heapq.heappush(ev, (t, seq, kind, p)); seq += 1
def dispatch(now):
nonlocal busy
while q and busy < SLOTS:
arr = q.popleft()
if policy == "deadline" and now - arr > max_wait: # too late to be useful: skip the work
stat["dropped"] += 1; continue
busy += 1; waits.append(now - arr)
push(now + rng_svc.expovariate(1 / MEAN_SVC), "dep", arr)
while ev:
now, _, kind, p = heapq.heappop(ev)
if now > horizon: break
if kind == "arr":
stat["offered"] += 1
if policy == "bounded" and len(q) >= qmax: stat["rejected"] += 1 # fail fast, as 429 or 503
else: q.append(now)
dispatch(now); push(now + rng_arr.expovariate(rate_at(now)), "arr")
else:
busy -= 1
stat["served_ok" if now - p <= SLO else "served_late"] += 1 # latency seen by the client
dispatch(now)
return stat, waits
print(f"{'policy':<24s} offered in time/s late rejected dropped p99 queue wait (s)")
for name, pol, kw in [("unbounded FIFO", "fifo", {}), ("bounded queue, 40", "bounded", dict(qmax=40)),
("bounded queue, 20", "bounded", dict(qmax=20)), ("bounded queue, 10", "bounded", dict(qmax=10)),
("drop if waited > 1 s", "deadline", {})]:
s, w = run(pol, **kw)
print(f"{name:<24s} {s['offered']:7d} {s['served_ok']/300:10.1f} {s['served_late']:5d} {s['rejected']:9d} {s['dropped']:8d} {pct(w,99):14.1f}")policy offered in time/s late rejected dropped p99 queue wait (s)
unbounded FIFO 5741 3.5 4679 0 0 36.9
bounded queue, 40 5741 11.4 1583 718 0 2.5
bounded queue, 20 5741 16.1 172 738 0 1.4
bounded queue, 10 5741 16.3 66 785 0 0.8
drop if waited > 1 s 5741 16.2 150 0 733 1.0The unbounded queue accepts all 5,741 requests and answers 4,679 of them after the client’s 2-second limit. Its goodput is 3.5 per second out of a possible 20, and the 99th-percentile wait is 37 seconds. Bounding the queue changes the result. A queue of 40 is too long: the waiting time at 20 requests per second can reach 2 seconds, before the service time is added, so 1,583 responses are still late. A queue of 20 or 10 gives roughly 16 in-time responses per second, about as good as any policy here, at the price of about 740 to 790 explicit rejections. The deadline policy reaches the same goodput without a fixed bound. A sensible bound comes from the arithmetic above: queue limit is about capacity times the waiting time the latency target can afford.
Rejecting is useful only if the rejection is cheap and informative. HTTP defines 429 (Too Many Requests) for a client that has exceeded a rate limit, and the RFC allows a Retry-After header; a 503 signals that the server cannot handle the request now, which is the status the SRE chapter uses for overload. Retry-After and Backoff explains how clients should combine a server’s retry hint with backoff and jitter. Clients that retry immediately and without backoff turn a brief overload into a sustained one, since every rejection generates another request. Priorities help when some traffic matters more: interactive requests keep their slots while batch jobs are rejected first.
Benchmarking serving systems, and failure behavior
A benchmark produces the numbers that targets, queue bounds, and scaling parameters are set from, so its design determines whether those settings are sound.
Workload model
A benchmark needs a model of the traffic, and the model must match the properties that drive the results above.
- Arrival process: open loop, with requests issued at a schedule that does not depend on responses (Poisson at a chosen rate, or a replay of recorded arrival times). A closed loop, where each of N virtual users sends the next request after receiving the previous response, measures a different system, since a slow server automatically receives less load.
- Request size distributions: prompt lengths and output lengths, drawn from real traffic where possible. A benchmark with fixed lengths shows no variance effect (\(C \approx 0\)) and overstates both capacity and tail behavior.
- A warm-up period before measurement and a steady-state window long enough for the percentile being reported.
- A sweep over load, since the curves above are the product. A single run at one rate cannot show where the knee is.
Coordinated omission
A load generator that sends a request only after the previous one completes, and measures each request from its actual send time, undercounts latency exactly when the server is slow, an error known as coordinated omission. Gil Tene’s wrk2 documentation explains the effect: because each connection waits for a response, a slow response keeps the generator from sending the requests that would have waited behind it, so the high-latency period is missing from the data. The tool corrects this by issuing requests at a constant rate and measuring from the time each request was supposed to be sent.
The following experiment shows the difference with no randomness. The server takes 5 ms per request, except one request that stalls for one second. The intended load is 100 requests per second.
from qsim import pct
# Open-loop vs closed-loop measurement of the same server. The server takes 5 ms per request, except
# one request that hits a 1.0 s stall (a GC pause, a model reload, a noisy neighbor).
RATE, N, SVC, STALL_AT, STALL = 100, 1000, 0.005, 400, 1.0
sched = [i / RATE for i in range(N)] # intended send times: one request every 10 ms
def svc(i): return STALL if i == STALL_AT else SVC
# closed loop, one connection: send the next request only after the previous response arrives
t, closed = 0.0, []
for i in range(N):
send = max(t, sched[i]) # a closed-loop client sends when it is free (and not earlier than planned)
t = send + svc(i)
closed.append(t - send) # latency measured from the actual send time
# open loop: requests are issued on schedule whether or not earlier ones finished; the server is FIFO
free, opened = 0.0, []
for i in range(N):
start = max(free, sched[i]); free = start + svc(i)
opened.append(free - sched[i]) # latency measured from the intended send time
for name, L in [("closed loop", closed), ("open loop ", opened)]:
print(f"{name}: p50 {pct(L,50)*1000:6.0f} ms p99 {pct(L,99)*1000:6.0f} ms p99.9 {pct(L,99.9)*1000:6.0f} ms max {max(L)*1000:6.0f} ms requests over 100 ms: {sum(1 for x in L if x > 0.1)}")closed loop: p50 5 ms p99 5 ms p99.9 1000 ms max 1000 ms requests over 100 ms: 1
open loop : p50 5 ms p99 950 ms p99.9 1000 ms max 1000 ms requests over 100 ms: 180The closed-loop client reports a 99th percentile of 5 ms, because only one of 1,000 requests saw the stall. The open-loop measurement reports 950 ms and 180 requests slower than 100 ms, because 100 requests were scheduled during the stall and the server needed time to work off the backlog. In production those 100 requests would have belonged to real users. Any benchmark that reports a pleasant tail should be checked for this.
Reporting and sample size
Record latencies in a histogram with fine buckets, such as the HdrHistogram format that wrk2 uses, instead of storing averages or per-interval percentiles, since percentiles from different intervals or replicas cannot be combined. Report the load level, request size distribution, TTFT and ITL separately for language models, and the error and rejection counts, since a policy that rejects half the traffic has a good tail on the rest. The 99th percentile needs on the order of thousands of requests per load level and the 99.9th needs tens of thousands, as the earlier window example showed.
Failure behavior
Normal-load performance says nothing about behavior during failure. Add runs that are meant to fail and note what the service does.
- Overload: offer 130 percent of measured capacity for several minutes, then return to normal. Check that rejections begin, goodput holds, and latency returns to its baseline as soon as the load falls. A recovery that takes minutes means the queue was unbounded.
- Surge: step the load by a factor of 2 to 3 and watch how long the loop takes to restore the target latency. The time is dominated by the start delay measured above.
- Replica loss: terminate a replica under load, and check that in-flight requests fail cleanly (or are retried safely), that traffic stops reaching the dead replica, and that the survivors do not exceed their queue bounds.
- Cold start: start a new replica with an empty node cache and time each phase: scheduling, image pull, weight fetch, load into accelerator memory, first request.
- Slow dependency or stall: inject a one-second pause in the server or in something it calls, as in the coordinated omission example, and check that deadlines expire and the stalled capacity is not retried into collapse.
The results of these runs feed the settings in this article: the maximum batch size and wait, the queue bound, the scaling target and floor, and the latency targets for each request class.
What comes next
A serving endpoint is rarely called directly by every application. Callers need identities, quotas, routing between models, and a fallback when a provider fails, and those functions sit in front of the endpoints described here. Model Gateways: Routing, Quotas, and Provider Failover covers that layer, including how rate limits and budgets protect the capacity this article sizes.
References
- Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180)
- vLLM documentation, Metrics design
- vLLM documentation, Optimization and tuning
- NVIDIA Triton Inference Server documentation, Batchers
- Kubernetes documentation, Horizontal Pod Autoscaling
- KServe documentation, Autoscaling with Knative Pod Autoscaler
- KServe documentation, Autoscaling with KEDA
- KEDA documentation, Scaling Deployments, StatefulSets and Custom Resources
- Jeffrey Dean and Luiz André Barroso, The Tail at Scale (Communications of the ACM, 2013)
- Google SRE Book, Handling Overload
- Google SRE Book, Addressing Cascading Failures
- RFC 6585, Additional HTTP Status Codes
- wrk2, a constant-throughput HTTP benchmarking tool (README)
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.