Latency Budget

A latency budget allocates time across the path to a defined response milestone. For a 300 ms checkout target, account for authentication, feature retrieval, inference, pricing, queuing, and response delivery. Sequential allowances can reserve part of that total for fallback and variability; overlapping work requires the actual dependency path rather than a sum of every duration.

Google’s Machine Learning Crash Course identifies latency and computation as constraints on online model complexity. A budget makes those constraints explicit before deployment. The model is one possible bottleneck; feature retrieval, queueing, or a remote dependency can instead consume most of the available time.

A time allocation is not a percentile guarantee

An individual request’s sequential durations add, but step percentiles generally do not. Consider 100 requests through two sequential steps. Each step takes 100 ms on 99 requests and 1,000 ms on one, with the slow events on different requests. Each nearest-rank p99 is 100 ms, yet the path’s p99 is 1,100 ms. A 250 ms path target is missed on two percent of requests.

A conservative allocation can assign step limits whose sum, including all other sequential work, fits the path target, and assign exceedance probabilities whose sum is no greater than the permitted path exceedance probability. The union bound supplies that upper bound without assuming independence, provided the step guarantees concern the same request population and all relevant work is covered. It can be conservative; validate the actual end-to-end distribution under the intended workload.

Parallel calls complete at the slowest required branch, including shared work around the fan-out. Their joint behavior and correlation matter; independent-call calculations do not apply automatically. Response latency explains measurement boundaries and individual trace durations. Report successful latency together with timeouts, failures, and fallback use so quick errors do not make the service appear healthy.

From budget to timeouts

gRPC’s deadline guide distinguishes an absolute deadline from a call’s waiting duration and describes carrying remaining time into downstream calls. Bound a wait by its step limit and the remaining end-to-end time after reserving fallback and response work. Retries and retry delays consume that same budget. For example, with 70 ms left and 20 ms reserved after the call, a 40 ms step cap allows at most 40 ms of waiting; with only 45 ms left, it allows at most 25 ms.

Choose what happens when time runs out: a valid cached score, a simpler path, a deferred action, or a safe refusal may fit different decisions. Graceful degradation provides an acceptable reduced answer; load shedding rejects work to protect capacity. Neither is free, and a fallback that cannot finish in the remaining time is not a synchronous rescue.

Propagate cancellation where work is no longer useful. As gRPC explains, the application must stop work it has spawned; ending a client wait does not guarantee that the remote calculation stops. An accepted action may still need to finish or record its outcome, and cancellation does not undo committed effects.

When a step is over budget

Profile the bottleneck and compare interventions by engineering cost and effect on the decision. Batch scoring can remove model execution from the request path but introduces refresh and coverage constraints. Caching requires valid reuse; parallel lookups need freedom from ordering dependencies and enough backend headroom. A smaller model can be the best first change when inference dominates. Cold start describes initialization delays that may appear during scale-out.

Compare the model-plus-fallback system with the incumbent, not just offline accuracy. For an illustrative 100 equally weighted decisions, an incumbent correct on 90 can be outperformed by a candidate correct on 89 of 90 timely predictions plus a fallback correct on 5 of the remaining 10: 94 in total. Different error costs can reverse that comparison. This arithmetic is not permission to violate a mandatory deadline: evaluate latency objectives, coverage, and cost-weighted quality separately.

Owning it

The user-facing service needs an owner for the end-to-end objective and clear responsibility for dependency budgets. Record the request population, measurement boundary, allowed exceedance rate, failure handling, and workload assumptions. Test bursts and relevant failures, measure the full path rather than only component percentiles, and revisit allocations when the workload or design changes. The Score Was From Tuesday applies these decisions to model serving.

References

• Google for Developers, Machine Learning Crash Course: Static versus dynamic inference — latency constraints on model complexity (CC BY 4.0)
• gRPC documentation, Deadlines — remaining time, propagation, and application cancellation responsibilities