When a model can act, the runtime around it decides what actually happens. This guide covers multi-tenant isolation tiers, short-lived delegated tool credentials, a policy engine, approvals bound to one call, idempotent execution, run state and resume, and a hash-chained action audit, with runnable code.
A prompt or model change should reach users only with evidence. This guide builds a release gate service: versioned evaluation sets, non-inferiority thresholds, paired bootstrap intervals, slice checks, judge calibration, approvals, canary stages, and rollback, with runnable simulations.
A model gateway is the single entry point between applications and model providers. This guide covers caller identity, routing, token-based limits and budgets, provider differences, and failover with audit, with a runnable gateway and fake providers.
A model server is a queue in front of an expensive machine. This guide explains how batching, replica scaling, latency targets, and overload policy determine tail latency, with runnable simulations and a benchmarking method that avoids coordinated omission.