Model Fallback

Model fallback routes a request to an alternative when the primary model cannot serve it acceptably, whether because of unavailability, rate limits, or excessive delay. A gateway or client library can implement the mechanism, while each application specifies allowed alternatives and the conditions for using them. Routing elsewhere does not cancel an earlier attempt or guarantee that the alternative will succeed.

Three levels of fallback

One alternative is the same model served elsewhere. Google Cloud recommends its global endpoint to reach a larger shared-capacity pool, with exponential backoff for retryable contention. Use it only where data-handling rules permit the possible destinations. A fixed model identifier reduces alias changes but does not promise identical outputs or serving behavior across regions. Validate the endpoint’s features and settings, and check whether the intended capacity contract applies there; a reservation in one region does not automatically establish capacity in another.

Another alternative is a different model. A model at the same provider can help with a model-specific failure but may share authentication, region, or quota dependencies with the primary. A full provider outage needs a path independent of that outage, such as a suitably prepared second provider or a non-model path. Even with independent availability, the same prompt can produce different answers, refusals, lengths, and structured output. Evaluate the alternative’s task quality, schema and tool support, tokenizer limits, latency, and costs before admitting traffic.

The last level is a path without a model: a queue that processes the work later, a rule-based answer, a hand-off to a person, or an honest message that the feature is unavailable. The paths that preserve useful service at reduced quality or with delayed completion are graceful degradation. An unavailable message communicates failure without providing that service. For some tasks, either response is safer than a weaker model producing plausible mistakes.

Designing it

Each application should define switching and recovery conditions, including error thresholds, probes, and a period of stable recovery to avoid repeatedly moving traffic back and forth. Validate the prompt on each alternative; reuse it when compatible and create a variant when necessary. Approve the destination’s data handling and test its ready quota and capacity under the diverted load. If the alternative cannot absorb all traffic, prioritize, limit, queue, or reject requests explicitly. Quality tests and load/failure exercises answer different questions, and both need repeating after relevant changes.

Give all attempts one end-to-end deadline and retry budget. On timeout, the first model may still finish and incur usage; cancellation needs confirmation under that service’s contract. Track a logical request ID and separate attempt IDs, including late outcomes and unknown usage. Decide which result may reach the user, and define a restart or failure policy after partial streaming rather than mixing two responses. Where tool effects follow model output, those effects need their own duplicate-execution protection. These limits may make a queue or an explicit unavailable response preferable to starting another model call.

An illustrative case: a support assistant’s primary provider has a two-hour outage. The gateway switches to a second provider’s model within a minute. The team had evaluated that model the previous month, found its answers slightly less complete but its structured output reliable, and prepared a prompt variant for it; customers saw a small change in tone and no failures. A second application, which had listed the same model as its fallback without testing it, began returning malformed results and was switched to its queue path by hand. How fallback fits among a shared platform’s responsibilities is covered in Every Team Had Its Own Key.

References:

• Google Cloud, Standard pay-as-you-go (Creative Commons Attribution 4.0) — global endpoint and retry recommendation.
• Google Cloud, Error code 429 (Creative Commons Attribution 4.0).
• Google AI for Developers, Gemini API: Models.
• Google Cloud, Purchase Provisioned Throughput.
• AWS Builders Library, Timeouts, retries, and backoff with jitter.