Model Serving Latency, Batching, and Autoscaling
A model server is a queue in front of an expensive machine. This guide explains how batching, replica scaling, latency targets, and overload policy determine tail latency, with runnable simulations and a benchmarking method that avoids coordinated omission.