Hedged Request
A hedged request gives an unusually slow call a second chance on another replica. The gain depends on independent delays and capacity for the extra work.
-
-
Optimizing AI Applications for Cost and Latency
A cheaper model, a cache, and shorter context each trade quality, latency, and cost. Price failures, read tails, budget evaluation, and compare configurations on shared cases.
-
Model Serving Latency, Batching, and Autoscaling
A model server is a queue in front of an expensive machine. This guide explains how batching, replica scaling, latency targets, and overload policy determine tail latency, with runnable simulations and a benchmarking method that avoids coordinated omission.
-
Coordinated Omission
A load generator that sends its next request only after the previous response skips the requests a slow server should have received, so the measured tail looks better than users would experience.