Response Latency

Response latency is the elapsed time between a defined request start and response milestone. A client may measure from sending a request to receiving the complete response; a server may measure only its own handling interval. Values from different boundaries are not directly interchangeable.

Add the times only when the path is sequential

Suppose identity checks take 20 ms, feature lookup 30 ms, waiting for compute 40 ms, inference 60 ms, and response preparation 10 ms. If all five are sequential, the measured path totals 160 ms. These are illustrative values; network travel outside the boundary is excluded.

If steps overlap, summing their durations overstates the elapsed time. Trace the actual path. Model execution time covers only the inference portion, so a faster model may leave other delays unchanged.

A quick answer can use old data

A 160 ms response can use features prepared twelve hours ago. Freshness concerns the underlying data’s currency and is a separate objective from response speed. Throughput counts completed work per interval; latency measures elapsed time for a request, including waiting. With variable arrivals or service times, queueing can grow sharply near a bottleneck’s capacity. Additional capacity helps when it relieves that bottleneck; reducing work or fixing a slow dependency may help more in other cases. Interpret latency alongside workload, arrival rate, and measurements of waiting and execution time.

Report the request population, time window, failures, and a distribution rather than an average. An average is a poor summary here because the slow tail is where the complaints come from: if 95 percent of requests finish in 200 ms and the rest take 3 seconds, the mean sits near 340 ms, a value that describes nobody’s experience and hides the group having a bad time. In a sample of 100 requests with those exact durations, nearest-rank p50 and p95 are both 200 ms; p99 is 3,000 ms. The share exceeding 200 ms is 5 percent. Choose percentiles or threshold-exceedance measures that reveal the behavior at issue rather than assuming any percentile will show the tail.

A target stated on such a measure is an objective, and the demand it is measured under needs its own statement, as concurrency and request rate sets out. Google SRE discusses latency as a service indicator.

For streaming output, distinguish time to the first output from time to completion; a fast first token does not establish a short total wait.

References

• Google SRE: Service level objectives