Latency Guardrails

Latency guardrails are agreed limits used to decide whether a system's response times are acceptable and what action to take when they are not. A limit becomes useful when it names the measurement boundary, the requests being measured, the observation period, and the response to a breach. It does not make slow requests impossible.

Define where the clock starts and stops

For an online prediction, a server-side measurement might start when the service receives a request and end when it finishes sending the response. A user-side measurement can also include network travel and client processing. Model execution time covers only part of either path. Compare measurements taken at the same boundary.

For a streaming text response, time to first token and time to complete the answer describe different aspects of the wait. For a batch scoring job, the relevant limit may be total completion time. None of these measurements establishes data freshness: a response delivered in 50 ms may still use yesterday's features.

Why the mean can hide a slow group

Consider 100 illustrative requests: 94 take 50 ms and six take 1,000 ms. Their mean is (94 × 50 + 6 × 1,000) / 100 = 107 ms. Using the nearest-rank convention, p95 is the 95th sorted observation, which is 1,000 ms. The mean is relatively low even though six users wait a full second.

The p95 value describes a point in the observed distribution, not the maximum and not a guarantee about each future request. Quantile conventions and monitoring approximations can differ. Google SRE explains why distributions and percentiles help characterize service behavior. Google SRE: Service level objectives Prometheus also distinguishes the ways histograms and summaries estimate quantiles. Prometheus: Histograms and summaries

Do not add feature-service p95 to model-service p95 and call the result end-to-end p95. Their slow requests need not be the same requests. Measure complete request durations, then inspect components to find the cause of delays.

Make the decision rule specific

An illustrative rollout gate could require p95 at or below 200 ms over a 30-minute test at 100 requests per second, with a stated payload mix and separate limits on failures and timeouts. These numbers belong to this example, not a universal target. Include warm-up or cold-start behavior according to the traffic the system must support, and measure production after rollout too.

Fast errors must not make a deployment look healthy. If latency is calculated only for successful requests, report failures and timeouts alongside it. A timeout is not a measured fast success; its completed duration may be unknown. A percentile target is also different from a per-request deadline, which determines when a caller stops waiting.

A breach should lead to a defined action

Consider an illustrative tradeoff: model A has ROC-AUC 0.91 and p95 80 ms; model B has ROC-AUC 0.93 and p95 500 ms under comparable tests. With a 100 ms p95 gate, A passes this latency check and B fails it. A is not automatically approved: quality, access, cost, and other requirements still apply, and B might be optimized or used for a different workload.

Before release, a failed gate can block rollout. During operation, the response might be investigation, a rollback, capacity adjustment, or an approved fallback. Name the owner and trigger. An internal service objective can inform the guardrail without being a customer contract. Faster hardware or caching may help, but only if the measured bottleneck and correctness requirements support that choice.