Response Latency

Response latency is the elapsed time between a defined request start and response milestone. A client may measure from sending a request to receiving the complete response; a server may measure only its own handling interval. Values from different boundaries are not directly interchangeable.

Add the times only when the path is sequential

Suppose identity checks take 20 ms, feature lookup 30 ms, waiting for compute 40 ms, inference 60 ms, and response preparation 10 ms. If all five are sequential, the measured path totals 160 ms. These are illustrative values; network travel outside the boundary is excluded.

If steps overlap, summing their durations overstates the elapsed time. Trace the actual path. Model execution time covers only the inference portion, so a faster model may leave other delays unchanged.

A quick answer can use old data

A 160 ms response can use features prepared twelve hours ago. Freshness concerns the underlying data's currency. Throughput counts completed work per time interval, while latency measures the time for a request. They can influence one another under load but are different quantities.

Report the request population, time window, failures, and a distribution such as p50 and p95. Google SRE discusses latency as a service indicator. For streaming output, distinguish time to the first output from time to completion; a fast first token does not establish a short total wait.

Reference: Google SRE: Service level objectives.