Model API Integration and Failure Handling

Consider a hypothetical support-ticket triage service. Each incoming ticket goes to a hosted language model that must return a category, a priority from 1 to 4, a one-line summary, a short quote from the ticket as evidence, and a flag saying whether a person must look at it. The task framing and acceptance criteria that precede this article define what a good enough answer is and what the non-AI fallback does when the model cannot supply one. This article covers what sits between your code and the model: a network call to a service you do not operate, which returns text that may be late, truncated, malformed, refused, or well-formed and wrong.

Five things go wrong in such a service sooner or later. A bulk import of old tickets triggers bursts of 429 responses. One request hangs for five minutes and ties up a worker. A reply stops at the output limit in the middle of a JSON string and the parser throws. A model update starts returning the priority as the string “2” instead of the integer 2. And when the provider has an incident, three layers of your own code each retry three times, so a single ticket becomes up to 27 calls. All five are predictable. The work is to write down, before they happen, what each one should do to your application.

The sections follow the life of one call. A request contract fixes what is sent and what is expected back. Validation decides whether the reply may be used. Limits and timeouts bound the resources a call can consume, retries decide which failures are worth repeating, and tests keep the contract true when the provider changes underneath it.

A model call is a remote call with an uncertain result

A call to a model API has all the failure modes of any HTTP call, plus a class of failures where the call succeeds and the content is unusable. Separating the classes matters because each has a different detector and a different correct response.

Class	Example	Detected by	Usual response
Transport	Connection reset, DNS failure, no response within the timeout	Client library, timer	Retry within a budget, if repeating is safe
Service rejection	429 rate limit, 500, 503, 529 overloaded	Status code	Retry with backoff, or shed load
Request rejection	400 malformed request, 401 or 403 credentials, 413 request too large	Status code	Do not retry; fix the request or configuration
Unusable completion	Cut off at the output limit, refusal, not JSON, wrong schema	Response body and finish reason	Repair once or twice, then fall back
Well-formed but wrong	Valid JSON with the wrong category	Semantic checks, evaluation	Validators for what can be checked; evaluation for the rest

Two properties make a model call harder to wrap than a typical service call. First, the same request does not reliably produce the same output, and a provider-side model update can change behavior without any change in your code. Second, latency and cost depend on the amount of text. Output tokens are generated one after another, so a reply with a long output takes longer than a short one (the mechanics of prefill and decode are covered in LLM Inference Optimization). A timeout, a retry policy, and a rate limit that suit a 20-millisecond lookup do not suit a call that legitimately takes eight seconds.

Request contracts

A request contract is the agreement about what the application sends and what it may rely on in return. It has two sides. The inside is an interface owned by the application. The outside is the wire format of one particular provider.

What a request pins down

Every value that can change the result, the cost, or the failure behavior of a call belongs in the contract:

• The model identifier. Providers typically offer both dated snapshots and aliases that move to newer versions. An alias keeps working through upgrades but changes behavior without notice; a pinned snapshot keeps behavior stable until the provider retires it, at which point you must migrate on the provider’s schedule. Production services usually pin and treat the migration as a planned change with its own tests.
• The instructions and input, with a version label for the instruction text. Designing Context and Prompt Interfaces covers how to write and version them; here they only need an identifier that can appear in logs and tests.
• The output cap (the maximum number of tokens the model may generate). It bounds cost and latency, and it is also the point at which a reply is cut off.
• Sampling parameters such as temperature. They change how much replies vary, and some models restrict which parameters they accept.
• The response schema, or the tool definitions, if the provider supports constrained output.
• Operational values that do not change the content: the timeout, a trace identifier, and the provider’s own request identifier once the response arrives. Anthropic’s API documentation, for example, states that every response carries a request id header and that error bodies repeat it; recording it next to your own identifier shortens a support conversation.

The size of the request must also fit. The context window bounds the input and, on many models, the output as well, so the check is that prompt tokens plus the output cap fit inside the window with some margin. Failing that check locally is cheaper than waiting for the provider to reject the request, and it gives the application a chance to shorten the input or split it deliberately.

An interface the application owns

The application’s code should depend on its own request and result types, and one adapter module should translate between them and the provider’s SDK or wire format. Three reasons follow from the failure list above. Provider response shapes differ and change, so the translation belongs in one place. Tests can substitute a fake behind the same interface. And switching or adding a provider changes one module instead of every caller.

The following program defines such a request type. It is standard-library Python 3.12 and calls no service. The token count is a deliberately crude estimate of four characters per token; a real service should use the provider’s own counter. The fingerprint is a hash of the fields that can change the output, so it can label logs, cache entries, and replay fixtures.

"""A provider-neutral request contract. Standard library only (Python 3.12)."""
from dataclasses import dataclass, asdict
import hashlib, json

@dataclass(frozen=True)
class ModelRequest:
    model: str                    # a pinned identifier, not a moving alias
    system: str
    user: str
    max_output_tokens: int
    temperature: float = 0.0
    response_schema: dict | None = None
    timeout_s: float = 20.0       # per-attempt limit, not part of the fingerprint
    prompt_version: str = "triage-v3"

    def fingerprint(self) -> str:
        """Hash of everything that can change the output; used for logs, caches and replay."""
        d = asdict(self)
        d.pop("timeout_s")
        blob = json.dumps(d, sort_keys=True, separators=(",", ":"))
        return hashlib.sha256(blob.encode()).hexdigest()[:12]

def rough_tokens(text: str) -> int:
    """Crude estimate (about 4 characters per token). A real service should use the provider's counter."""
    return -(-len(text) // 4)

def check_budget(req: ModelRequest, context_limit: int, safety: float = 0.10) -> tuple[bool, str]:
    prompt = rough_tokens(req.system) + rough_tokens(req.user)
    need = prompt + req.max_output_tokens
    allowed = int(context_limit * (1 - safety))
    ok = need <= allowed
    return ok, f"prompt~{prompt} + output cap {req.max_output_tokens} = {need} vs allowed {allowed}"

The second part builds a short ticket, an oversized synthetic input, and two variants of the short request, and prints the budget check and the fingerprints.

SCHEMA = {"category": ["billing", "shipping", "technical", "other"],
          "priority": [1, 4], "needs_human": "bool"}
SYSTEM = "Classify the support ticket. Reply with JSON only."

short = ModelRequest("triage-model-2025-01-15", SYSTEM, "My card was charged twice for order 1182.",
                     max_output_tokens=300, response_schema=SCHEMA)
long_ticket = "Order log line. " * 1900           # synthetic long input
big = ModelRequest("triage-model-2025-01-15", SYSTEM, long_ticket,
                   max_output_tokens=300, response_schema=SCHEMA)
retimed = ModelRequest("triage-model-2025-01-15", SYSTEM, short.user, 300,
                       response_schema=SCHEMA, timeout_s=5.0)
newprompt = ModelRequest("triage-model-2025-01-15", SYSTEM, short.user, 300,
                         response_schema=SCHEMA, prompt_version="triage-v4")

CONTEXT = 8000   # illustrative limit for a hypothetical model
for name, r in [("short", short), ("big", big)]:
    print(name, check_budget(r, CONTEXT))
print("fingerprint short      :", short.fingerprint())
print("fingerprint new timeout:", retimed.fingerprint())
print("fingerprint new prompt :", newprompt.fingerprint())

short (True, 'prompt~24 + output cap 300 = 324 vs allowed 7200')
big (False, 'prompt~7613 + output cap 300 = 7913 vs allowed 7200')
fingerprint short      : a59ed2c8f791
fingerprint new timeout: a59ed2c8f791
fingerprint new prompt : 24331c7274da

The short request needs about 324 tokens against an allowed 7,200 and goes through. The oversized input needs 7,913 and is rejected before any network call; the limit of 8,000 tokens is an illustrative value for a hypothetical model, and the 10 percent margin is meant to absorb the error of the estimate. The fingerprints show the design choice: changing only the per-attempt timeout leaves the fingerprint unchanged, because the timeout does not affect what the model produces, while changing the prompt version changes it. A cache keyed on this fingerprint would keep serving a stored reply after a timeout change and miss correctly after a prompt change.

Structured output validation

Downstream code consumes fields, not prose, so the application asks the model for JSON. There are three levels of help from the provider. A prompt that says “reply with JSON only” gives no guarantee. A JSON mode guarantees syntactically valid JSON but not any particular fields. Schema-constrained output restricts generation (constrained decoding) so the reply matches a supplied JSON Schema. OpenAI’s structured outputs guide draws exactly this line between JSON mode, which only validates that output is JSON, and structured outputs, which ensure schema adherence. How decoding is constrained, and why a grammar can force valid structure without making the content right, is the subject of Decoding: Sampling, Constraints, and Structured Output.

Constrained output lowers the frequency of format failures, but it does not remove the need to validate, for four reasons:

• The supported schema is a subset of JSON Schema. Anthropic’s structured outputs documentation lists, among unsupported features, numeric bounds such as minimum and maximum and string length limits, so constraints like “priority between 1 and 4” or “summary at most 200 characters” must still be checked by your code on that provider.
• A reply that is cut off or refused may not match the schema. Both OpenAI and Anthropic document that a refusal, or a stop at the token limit, produces output that is not guaranteed to conform, and each exposes a signal for it (a refusal field or incomplete status in OpenAI’s API, a stop reason of refusal or max tokens in Anthropic’s).
• The guarantee covers structure, not truth. OpenAI’s guide states that schema adherence does not guarantee semantic accuracy, and Anthropic’s says the same about the content of fields.
• You may run another provider, an older model, or a stub in tests, and the validator is the one component that behaves the same for all of them.

So the application validates every reply at the boundary, whatever the provider promises. The validator is also the place where a change in model behavior first becomes visible, as a shift in the rate of one failure type.

Three layers of checks

It helps to order the checks by what they need to know. Each layer assumes the previous one passed.

Layer	Question	Example failure	Handling
0. Completion status	Did the model finish normally?	Cut off at the output cap; refusal	Do not parse; raise the cap once if truncation is the cause, otherwise fall back
1. Syntax	Is the text one JSON value?	Prose around the JSON; unclosed string	Reject, or use one explicit normalization
2. Schema	Right keys, types, enum members, ranges?	Missing field, “2” for 2, category outside the enum	Bounded repair, then fall back
3. Semantics	Does the content make sense for this input and these rules?	Evidence not found in the ticket; urgent item without review	Reject or route to a person

Layer 3 is where application knowledge enters. A schema cannot say that the evidence string must be a substring of the ticket, or that priority 1 requires human review; code that has the ticket and the business rules can. These checks are cheap and catch a real class of errors, because a model that invents a quote has produced something no schema validator would flag.

The following program implements the four layers for the triage schema. It is standard-library Python 3.12 with a hand-written validator so every rule is visible. One point is Python-specific: bool is a subclass of int, so a priority of true would pass a naive integer check and the code excludes it explicitly. The only tolerance built in is stripping a single Markdown fence around the JSON, chosen deliberately because that failure is common and unambiguous.

"""Three-layer validation of a model's structured reply. Standard library only (Python 3.12)."""
import json, re
from dataclasses import dataclass, field

CATEGORIES = {"billing", "shipping", "technical", "other"}
REQUIRED = {"category": str, "priority": int, "summary": str, "evidence": str, "needs_human": bool}

@dataclass
class Outcome:
    status: str                   # ok | truncated | refused | not_json | schema_error | semantic_error
    errors: list = field(default_factory=list)
    value: dict | None = None

def parse_and_validate(raw: str, finish: str, ticket: str) -> Outcome:
    # Layer 0: the response cannot be treated as a value at all.
    if finish == "refused":   return Outcome("refused")
    if finish == "truncated": return Outcome("truncated")
    # Layer 1: syntax. One deliberate normalization: strip a single Markdown fence.
    text = re.sub(r"^```(?:json)?\s*|\s*```$", "", raw.strip())
    try:
        obj = json.loads(text)
    except json.JSONDecodeError as e:
        return Outcome("not_json", [f"{e.msg} at char {e.pos}"])
    # Layer 2: schema. Exact keys and types (bool is a subclass of int in Python, so exclude it).
    if not isinstance(obj, dict): return Outcome("schema_error", ["top level is not an object"])
    errs = []
    for k, t in REQUIRED.items():
        if k not in obj: errs.append(f"missing '{k}'")
        elif not isinstance(obj[k], t) or (t is int and isinstance(obj[k], bool)):
            errs.append(f"'{k}' should be {t.__name__}")
    errs += [f"unexpected '{k}'" for k in obj if k not in REQUIRED]
    if not errs:
        if obj["category"] not in CATEGORIES: errs.append(f"category '{obj['category']}' not in enum")
        if not 1 <= obj["priority"] <= 4:      errs.append("priority outside 1..4")
        if len(obj["summary"]) > 200:          errs.append("summary longer than 200 chars")
    if errs: return Outcome("schema_error", errs)
    # Layer 3: semantics. Checks that need information outside the schema.
    sem = []
    if obj["evidence"] not in ticket:
        sem.append("evidence is not a quote from the ticket")
    if obj["priority"] == 1 and not obj["needs_human"]:
        sem.append("priority 1 requires needs_human")          # an illustrative business rule
    return Outcome("semantic_error", sem, obj) if sem else Outcome("ok", [], obj)

TICKET = "I was charged twice for order 1182. Please refund one charge."
GOOD = {"category": "billing", "priority": 2, "summary": "Duplicate charge",
        "evidence": "charged twice for order 1182", "needs_human": True}
def variant(**kw):
    d = {**GOOD, **kw}
    return json.dumps({k: v for k, v in d.items() if v is not Ellipsis})

The second part feeds twelve replies to the validator for one invented ticket. Each reply is built by changing one thing in a valid reply.

good = variant()
cases = [
  ("valid",                 good, "complete"),
  ("fenced valid",          "```json\n" + good + "\n```", "complete"),
  ("cut off by max tokens", good[:60], "truncated"),
  ("refusal",               "I can't help with that.", "refused"),
  ("prose around JSON",     "Sure! Here is the result: " + good, "complete"),
  ("missing field",         variant(needs_human=Ellipsis), "complete"),
  ("enum miss",             variant(category="payments"), "complete"),
  ("priority as string",    variant(priority="2"), "complete"),
  ("extra key",             variant(confidence=0.93), "complete"),
  ("evidence not in text",  variant(evidence="charged three times"), "complete"),
  ("urgent, no review",     variant(priority=1, needs_human=False), "complete"),
  ("valid, wrong category", variant(category="shipping"), "complete"),
]
print(f"{'case':24} {'status':15} detail")
for name, raw, finish in cases:
    o = parse_and_validate(raw, finish, TICKET)
    print(f"{name:24} {o.status:15} {'; '.join(o.errors)}")

case                     status          detail
valid                    ok              
fenced valid             ok              
cut off by max tokens    truncated       
refusal                  refused         
prose around JSON        not_json        Expecting value at char 0
missing field            schema_error    missing 'needs_human'
enum miss                schema_error    category 'payments' not in enum
priority as string       schema_error    'priority' should be int
extra key                schema_error    unexpected 'confidence'
evidence not in text     semantic_error  evidence is not a quote from the ticket
urgent, no review        semantic_error  priority 1 requires needs_human
valid, wrong category    ok              

Reading the output from the top, the fenced reply passes because the normalization is explicit. A reply wrapped in a sentence (“Sure! Here is the result: …”) fails at layer 1, and that is a choice: extracting the first JSON object from free text would accept it, but it would also accept the wrong object when a reply contains two, so the code rejects and counts. The next four rows are layer 2 failures, each with an error message specific enough to feed back to the model. The two semantic rows show what only layer 3 sees: a quote that is not in the ticket, and a priority-1 item that skipped review. The final row is the limit of the whole structure. A reply that says “shipping” for a duplicate-charge ticket is valid at every layer, because nothing in the validator knows the right answer. Detecting that requires labeled examples and measurement, which is the work of Evaluating AI Applications with Reproducible Test Sets, and its cost is what the error-cost analysis in the acceptance criteria exists to weigh.

Designing schemas that can be validated

A few design choices make the validation layers more useful.

• Prefer closed sets to free text where downstream code branches on the value, and include an explicit member such as other or a nullable field for the case where no option fits. Without an escape value, a model that has no good answer must still choose a wrong one, and the validator cannot distinguish that from a good answer.
• Keep required and nullable separate. A required field that may be null says “I looked and found nothing”; a missing field says the reply is broken.
• Reject unknown keys. The extra key in the table above is harmless in itself, but a model that starts adding fields is a signal worth surfacing, and silent ignoring hides it. Rejection is safe here because the keys come from the model’s output, not from a partner whose additions you must tolerate.
• Ask for quotes or identifiers that can be checked against the input, such as the evidence field, in preference to a free-form justification that cannot be checked.
• Put limits that the provider’s schema subset does not support in your own validator, and keep the provider-side schema small. Providers document complexity limits for schemas, and a large schema can fail to compile or add latency on first use.
• Treat the schema as a contract with downstream consumers. Adding an optional field is usually a compatible change for readers; renaming a field or changing its type is not. The glossary entry on schema evolution and compatibility covers the rules for evolving such a contract.

Repair, retry, or fall back

When validation fails, the application has three options: repeat the call, repeat it with the error attached, or stop using the model for this item. The right choice depends on the failure.

A truncated reply is a limit problem. Repeating the same request returns the same truncation, so the fix is a larger output cap or a shorter requested output, and the application should not loop. A refusal is a decision by the model or the provider’s policy, and the same input is likely to be refused again; route it to the fallback. A schema failure is the one worth a bounded repair: send the original request plus the previous reply and the list of errors, and ask for a corrected reply. Because the input changes, the second answer is not a blind repeat of the first. Each repair costs a full call and its latency, so the bound is small, commonly one or two repairs, and the failure after the bound goes to the fallback path from the acceptance criteria: the non-AI baseline, or a human queue.

The next program implements that loop over a deterministic stub that replays scripted replies, which is why it needs no service. It reuses the validator above.

# Bounded repair: re-ask with the validation errors, at most two repairs, then fall back.
def call_with_repair(model, ticket, max_repairs=2):
    feedback = None
    for attempt in range(1 + max_repairs):
        raw, finish = model(ticket, feedback)
        o = parse_and_validate(raw, finish, ticket)
        print(f"  attempt {attempt + 1}: {o.status}")
        if o.status == "ok":
            return o
        if o.status in ("refused", "truncated"):
            break                       # re-asking does not fix these
        feedback = o.errors
    return Outcome("fallback", ["route to the non-AI path"])

def stub_model_factory(script):          # deterministic stand-in for a model
    it = iter(script)
    return lambda ticket, feedback: next(it)

print("repair succeeds on second attempt:")
r = call_with_repair(stub_model_factory([(variant(category="payments"), "complete"), (good, "complete")]), TICKET)
print("  ->", r.status)
print("model keeps returning the wrong enum:")
r = call_with_repair(stub_model_factory([(variant(category="payments"), "complete")] * 3), TICKET)
print("  ->", r.status, r.errors)
print("truncated reply is not repaired by re-asking:")
r = call_with_repair(stub_model_factory([(good[:60], "truncated")]), TICKET)
print("  ->", r.status)

repair succeeds on second attempt:
  attempt 1: schema_error
  attempt 2: ok
  -> ok
model keeps returning the wrong enum:
  attempt 1: schema_error
  attempt 2: schema_error
  attempt 3: schema_error
  -> fallback ['route to the non-AI path']
truncated reply is not repaired by re-asking:
  attempt 1: truncated
  -> fallback

In the first run the model fixes its enum on the second attempt. In the second run it repeats the same mistake and the loop stops after three calls in total, one original and two repairs, handing the item to the fallback with the last error list attached. In the third, a truncated reply ends the loop immediately because re-asking cannot help. The status of each attempt should also be counted as a metric: a rising share of schema errors is often the first visible sign that a provider-side model change has shifted the output format.

Limits

A call is constrained by several different limits, and they fail in different ways. Providers publish the numbers per model and per account tier, and the numbers change, so the application should read them from configuration rather than hard-code them.

Limit	Measured in	When it is exceeded	Where to handle it
Request rate	Requests per minute	429 response, often with a retry delay	Client-side pacing, retry with backoff
Token throughput	Tokens per minute	429 response	Pacing by estimated tokens, smaller prompts
Concurrency	Simultaneous requests	Queueing or 429	A semaphore in the client
Context window	Tokens of input plus output	Rejection, or truncation of the output	Local budget check, trimming the input
Output cap	Tokens generated	Reply cut off	Set the cap deliberately; check the finish reason
Request size	Bytes	413 or similar	Check before sending
Spend or quota	Currency or tokens per period	Rejection until the period resets or the limit is raised	Alerts before the limit; no retry

Two of these deserve a closer look because they look alike and need opposite handling. A 429 caused by a rate limit clears after a short wait, so retrying is appropriate. A 429 caused by a spend cap or an exhausted quota does not clear until a person acts. Anthropic’s error documentation says that a spend-cap 429 has no retry-after header and keeps failing until access resumes, and OpenAI’s error guide states that quota and spend limits cannot be restored by retrying. A retry policy that cannot tell the two apart will spend its attempts and its deadline on a call that cannot succeed. The distinction is in the error type or message of the provider’s response, so the adapter has to expose it.

A worked budget

Suppose a hypothetical account has a limit of 600 requests per minute and 90,000 tokens per minute, and a typical ticket costs 900 input tokens and 150 output tokens, so about 1,050 tokens per call. How a provider counts tokens against the limit (input only, input plus output, or something else) is stated in its documentation, so assume here that both directions count. The sustainable rate is the smaller of what each limit permits:

\[ \text{rate} = \min\left(R,\ \frac{T}{\bar{n}}\right) = \min\left(600,\ \frac{90{,}000}{1{,}050}\right) \approx \min(600,\ 85.7) \text{ requests per minute} \]

Here \(R\) is the request limit, \(T\) the token limit, and \(\bar{n}\) the average tokens per call. The token limit binds, at about 1.43 requests per second. Two consequences follow. A batch of 10,000 old tickets cannot finish in less than 10,000 / 85.7, about 117 minutes, and no retry policy changes that; it only decides how much of the wait is spent usefully. And the number of workers needed is given by Little’s law, concurrency equals arrival rate times time in the system: if a call takes four seconds on average, 1.43 × 4 is about 5.7, so six workers keep the limit saturated and more workers only add queueing and 429 responses.

Pacing and priorities

Staying under the limit on the client side is better than discovering it through 429 responses, because every rejected request still costs a round trip and counts as load. A client-side limiter, typically a token bucket, that releases requests at, for example, 80 percent of the published rate, plus a cap on concurrent requests, turns most 429s into brief local waits.

The limit is shared by everything that calls with the same credentials, which creates a priority question. If a batch backfill and an interactive agent screen share the 85.7 requests per minute, the backfill will consume all of it unless the two are separated into queues with a reserved share for the interactive path. Queues should be bounded. An unbounded queue turns an overload into very long waits, so that by the time a request reaches the model its caller has already given up. When the expected wait exceeds the request’s deadline, rejecting at once and taking the fallback path serves the user better. Capacity, Load Shedding, and Resilience Testing develops the queueing and overload-control side of this in general; the model API adds only that the capacity being protected belongs to someone else.

The output cap is a limit you set yourself, and it couples cost, latency, and truncation risk. A lower cap shortens the slowest replies and the cost per call, and raises the chance that a long reply is cut off. Choose it from the longest legitimate reply the schema can produce, not from a round number.

Timeouts

A timeout answers the question “how long is this call worth waiting for?” For a model API there are several waits, and a single number covers them poorly.

• Connect timeout: how long to wait for the connection. Seconds are enough; a connection that takes longer is unhealthy.
• Time to first token or first byte: relevant when streaming, where it measures queueing and prompt processing before generation starts.
• Idle timeout: the longest silence between chunks of a streamed reply. A stalled stream is detected here long before any total limit.
• Total timeout: the longest the whole call may take, from request to complete reply.

Without streaming, only the total applies, and it must cover the whole generation. Generation time grows with the number of output tokens, so the timeout and the output cap have to be chosen together. Suppose measurements of your own traffic show a first-token wait of about one second and a slow-end generation speed of about 40 tokens per second; the numbers are assumed for the example. A 300-token cap then needs about 1 + 300 / 40 = 8.5 seconds, and a total timeout of 10 seconds leaves a margin of about 15 percent. Raise the cap to 1,000 tokens and the same arithmetic gives 26 seconds, which is a different timeout and probably a different design. The slow end of the distribution is what matters, so the values should come from high percentiles of recorded latencies, grouped by output length, not from the mean. Instrumenting Services with Metrics, Logs, and Traces covers how to record them as histograms.

Streaming helps in two ways: the idle timeout detects stalls early, and the user sees text sooner. It also changes error handling. Anthropic’s documentation notes that when a response arrives over server-sent events, an error can occur after the API has already returned a 200 status, so a successful status line does not mean a complete reply. A streaming client needs its own completion check, and a structured reply can only be validated after the final chunk, so streaming improves perceived latency for text but not time to a usable parsed value. The same documentation recommends streaming or batch processing for long-running requests because some networks drop idle connections.

A timeout on the client is not a cancellation on the server. Closing the connection may or may not stop generation at the provider; the provider’s documentation says which, and until you know, assume that a timed-out call may still have done work. Two consequences follow in the next section: a timed-out call is not proof that nothing happened, and a retry may duplicate work.

Deadlines instead of independent timeouts

The user-facing latency requirement from the acceptance criteria is an end-to-end deadline. Every call, retry, and repair inside that request must fit within it. If each layer applies its own timeout independently, the sum can exceed the deadline by a wide margin. The remedy is deadline propagation: each step receives the remaining time and sets its own timeout to the smaller of its usual value and what remains. Google’s SRE book makes the same point about cascading failures, recommending that remaining time be passed down the stack so that work already past its client’s deadline is not started. The retry loop in the next section applies it with one line: an attempt is never allowed to outlive the deadline.

Retries

A retry converts a transient failure into a success at the cost of more load and more latency. Whether that trade is good depends on three questions: is the failure transient, is repeating the call safe, and can the system afford the extra load.

Which failures to retry

Condition	Retry?	Reason
Connection reset or timeout before a complete reply	Yes, within the deadline	Often transient; see the idempotency note below
429 from a rate limit	Yes, after the server’s delay	The limit clears with time
500, 502, 503, 504	Yes, with backoff	Server-side and often transient; OpenAI’s guide lists 503 for temporary lack of capacity
529 overloaded	Yes, with backoff	Anthropic documents it as a temporary overload
400, 401, 403, 404, 413	No	The request or credentials are wrong; repeating cannot fix them
429 from a spend cap or quota	No	Does not clear without human action
Refusal	No	A policy decision; use the fallback
Schema or semantic failure	Bounded repair, not a blind retry	See the previous section on repair
Error after a stream began	Restart the whole request	Discard the partial output; a partial structured reply cannot be validated

Is it safe to repeat?

HTTP defines a method as idempotent when repeating it has the same effect as sending it once. RFC 9110 lists GET, PUT, and DELETE among them and does not make POST idempotent, and model APIs are typically called with POST. Whether repeating a model call is harmful depends on what the call does in your system. The call itself changes nothing of yours; it only costs tokens and may return a different answer the second time. The danger is in what happens with the reply. If a timeout occurs after the provider finished, and your code retries, you pay twice, which is a cost problem. If the reply then triggers an action, such as closing a ticket or sending an email, and the first reply’s action also happened because the response was merely slow, the action runs twice, which is a correctness problem.

The remedy is to make the effects idempotent and the work deduplicated, not to avoid retries. Derive a key from the business identity of the work, here the ticket id and the prompt version, and record the result under that key before acting on it. A repeated task then finds the stored result instead of calling the model again, and a repeated write overwrites the same record. The idempotency glossary entry explains why a key that identifies the logical operation is what makes repetition safe. Where a model’s reply selects tools that change the outside world, the controls are stricter, and Tool Interfaces, Authorization, and Argument Validation and Durable Agent Workflows cover them.

Backoff, jitter, and the server’s delay

If a call fails because a service is overloaded, the worst response is to send the same request again immediately. Exponential backoff waits longer after each failure, and jitter randomizes the wait so that many clients do not retry in lockstep. A common form, called full jitter, draws the wait before attempt \(n+1\) uniformly at random:

\[ d_n \sim \mathrm{Uniform}\bigl(0,\ \min(c,\ b \cdot 2^{n})\bigr) \]

Here \(n\) counts the failures so far starting at 0, \(b\) is the base delay, and \(c\) is a cap on the delay. The bound doubles with each failure until it reaches the cap, and the actual wait is anywhere below it. The AWS Architecture Blog compares full jitter with other variants and finds it does the least total work among the variants it compares. When the response carries a retry delay, the server’s value is a floor: wait the larger of \(d_n\) and the server’s delay. The glossary entry on Retry-After and backoff covers the two forms of the header and how it combines with a computed backoff.

Retries also have a statistical cost model. If each attempt fails independently with probability \(p\) and the policy allows at most \(N\) attempts, a request succeeds with probability \(1 – p^{N}\), and the expected number of attempts per request is

\[ E[\text{attempts}] = \sum_{k=0}^{N-1} p^{k} = \frac{1 – p^{N}}{1 – p} \]

With \(p = 0.05\) and \(N = 3\), success probability rises from 0.95 to 0.999875 and the average load rises by only about 5 percent, to 1.0525 calls per request. That is the case retries are good at. The assumption of independence is where the calculation breaks. During an incident, failures are strongly correlated and \(p\) approaches 1, so every request makes all \(N\) attempts and load becomes \(N\) times normal at the moment the provider can least afford it. The same multiplication happens across layers: if each of three layers makes up to three attempts, one user action produces up to \(3^3 = 27\) calls to the provider, and Google’s SRE book gives the same warning with a 4^3 = 64 example for three retries per layer. The defenses are to retry in one layer only, usually the one closest to the provider call, and to cap retries with a retry budget, such as a limit on retries per minute per process after which calls fail without retrying, an approach the SRE book also describes.

A retry policy in code

The next program implements a retry loop with the rules above on a fake clock, so it runs instantly and needs no service. Responses are scripted: each scenario returns a status, a latency, and optionally a retry delay or a flag marking a spend-cap 429. The loop retries only the statuses in the table, honors the server’s delay as a floor, bounds attempts, and shortens each attempt so it cannot outlive the deadline. It is standard-library Python 3.12; the random seed makes the output reproducible.

"""Retry policy with a deadline, simulated on a fake clock. Standard library only (Python 3.12)."""
import random
from dataclasses import dataclass

@dataclass
class Resp:
    status: int | str             # HTTP status, or "timeout"
    retry_after: float | None = None
    kind: str = ""                # "quota" marks a 429 that retrying cannot fix
    latency: float = 1.0

RETRYABLE = {408, 429, 500, 502, 503, 504, 529, "timeout"}

def backoff(n, base=1.0, cap=30.0, rng=random):
    """Full jitter: uniform between 0 and min(cap, base * 2**n)."""
    return rng.uniform(0, min(cap, base * 2 ** n))

def call_with_retries(send, deadline_s, max_attempts=4, attempt_timeout=10.0, rng=None, log=print):
    rng = rng or random.Random(0)
    t = 0.0
    for n in range(max_attempts):
        limit = min(attempt_timeout, deadline_s - t)      # an attempt never outlives the deadline
        r = send(n)
        timed_out = r.status == "timeout" or r.latency > limit
        t += limit if timed_out else r.latency
        status = "timeout" if timed_out else r.status
        if status == 200:
            log(f"  attempt {n+1}: 200 at t={t:.1f}s"); return ("ok", n + 1, t)
        retryable = status in RETRYABLE and r.kind != "quota"
        log(f"  attempt {n+1}: {status}{' (quota)' if r.kind=='quota' else ''} at t={t:.1f}s retryable={retryable}")
        if not retryable or n == max_attempts - 1:
            return ("fail", n + 1, t)
        wait = backoff(n, rng=rng)
        if r.retry_after is not None:
            wait = max(wait, r.retry_after)        # the server's value is a floor
        if t + wait + 1.0 > deadline_s:            # less than 1 s would remain for another attempt
            log(f"  next wait {wait:.1f}s leaves no room before the {deadline_s:.0f}s deadline; stop")
            return ("deadline", n + 1, t)
        t += wait
    return ("fail", max_attempts, t)

scenarios = {
  "two 503s then success":      lambda n: Resp(503, latency=0.4) if n < 2 else Resp(200, latency=2.0),
  "429 with Retry-After: 7":    lambda n: Resp(429, retry_after=7, latency=0.2) if n < 1 else Resp(200, latency=2.0),
  "400 invalid request":        lambda n: Resp(400, latency=0.2),
  "429 that is a spend cap":    lambda n: Resp(429, kind="quota", latency=0.2),
  "timeouts until the deadline":lambda n: Resp("timeout"),
}
for name, send in scenarios.items():
    print(name)
    call_with_retries(send, deadline_s=30, rng=random.Random(3))

two 503s then success
  attempt 1: 503 at t=0.4s retryable=True
  attempt 2: 503 at t=1.0s retryable=True
  attempt 3: 200 at t=4.1s
429 with Retry-After: 7
  attempt 1: 429 at t=0.2s retryable=True
  attempt 2: 200 at t=9.2s
400 invalid request
  attempt 1: 400 at t=0.2s retryable=False
429 that is a spend cap
  attempt 1: 429 (quota) at t=0.2s retryable=False
timeouts until the deadline
  attempt 1: timeout at t=10.0s retryable=True
  attempt 2: timeout at t=20.2s retryable=True
  attempt 3: timeout at t=30.0s retryable=True
  next wait 1.5s leaves no room before the 30s deadline; stop

Each scenario exercises one row of the table. After two 503 responses the third attempt succeeds at 4.1 seconds, after waiting 0.2 and 1.1 seconds between attempts. The rate-limited call waits 7 seconds rather than the computed backoff, because the server’s delay is a floor; the computed backoff after a first failure is at most one second. The 400 and the spend-cap 429 stop after one attempt, which saves the whole deadline for the fallback path. In the last scenario every attempt times out. The first two attempts run to the 10-second attempt limit; the third is shortened to the time left in the 30-second deadline and ends exactly at it, and the loop stops because the next wait would leave no time. The caller gets its failure at 30 seconds, with the fallback still possible, instead of overshooting the deadline by an attempt.

The next experiment shows why jitter matters. It is a toy model: 1,000 clients all fail at the same instant, the service then accepts at most 150 requests in each wall-clock second, and rejected clients retry according to a policy. It does not model cost of rejected requests, which is why it is a demonstration of synchronization and not a ranking of policies.

# --- Synchronized retries against a recovering service -------------------------------------------
TICK = 0.1                                         # time step in seconds

def herd(policy, clients=1000, capacity_per_s=150, seed=7):
    """All clients fail at t=0. The service accepts at most capacity_per_s requests in each wall-clock
    second and rejects the rest; a rejected client retries after policy(n) seconds (at least one tick)."""
    rng = random.Random(seed)
    due = {i: 0 for i in range(clients)}           # client -> tick of its next attempt
    tries = {i: 0 for i in range(clients)}
    sent, served, done_tick = {}, {}, 0            # per-second request count and accepted count
    while due:
        tick = min(due.values())
        sec = int(tick * TICK)
        arrivals = 
        rng.shuffle(arrivals)                      # order within a tick is arbitrary
        for c in arrivals:
            sent[sec] = sent.get(sec, 0) + 1
            if served.get(sec, 0) < capacity_per_s:
                served[sec] = served.get(sec, 0) + 1
                del due; done_tick = max(done_tick, tick + 1)
            else:
                tries += 1
                due = tick + max(1, round(policy(tries - 1, rng) / TICK))
    after_burst = max(v for s, v in sent.items() if s >= 1)
    return sum(sent.values()), after_burst, done_tick * TICK

policies = {
  "fixed 1 s, no jitter":        lambda n, rng: 1.0,
  "exponential, no jitter":      lambda n, rng: min(30.0, 1.0 * 2 ** n),
  "exponential, full jitter":    lambda n, rng: rng.uniform(0, min(30.0, 1.0 * 2 ** n)),
}
print("1000 clients fail at t=0; the service accepts 150 requests per second")
print(f"{'policy':26} {'total requests':>14} {'peak req/s after t=1':>21} {'all served by (s)':>18}")
for name, p in policies.items():
    total, peak, done = herd(p)
    print(f"{name:26} {total:14d} {peak:21d} {done:18.1f}")

1000 clients fail at t=0; the service accepts 150 requests per second
policy                     total requests  peak req/s after t=1  all served by (s)
fixed 1 s, no jitter                 3850                   850                6.1
exponential, no jitter               3850                   850               61.1
exponential, full jitter             3505                   517               18.3

# --- Retry amplification across layers -----------------------------------------------------------
print("\nworst-case calls to the provider from one user action")
for layers in (1, 2, 3):
    print(f"  {layers} layer(s), 3 attempts each: {3 ** layers}")

worst-case calls to the provider from one user action
  1 layer(s), 3 attempts each: 3
  2 layer(s), 3 attempts each: 9
  3 layer(s), 3 attempts each: 27

The two policies without jitter behave identically for the first waves, because every client in a wave waits the same time: 850 requests return together after the first rejection, then 700, and so on, 3,850 requests in total, with a peak of 850 requests in a second that the service rejects mostly again. The fixed 1-second policy clears the backlog in about 6 seconds and exponential backoff without jitter takes 61, because its later waves sit out waits of 16 and 30 seconds. Full jitter sends fewer requests (3,505) and a lower peak (517 per second), but drains in 18.3 seconds, slower than the fixed policy, because some clients draw long waits. So jitter lowers the pressure on the recovering service; it does not by itself make the whole backlog finish sooner, and the choice of base and cap shapes that trade. The last block shows the multiplication across layers.

Failure handling beyond retries

Some failures remain after the retry budget is spent. A circuit breaker watches the recent failure rate and, once it passes a threshold, rejects calls immediately for a cooling period so the system stops spending its deadline on a service that is down; after the period it lets a few probe calls through. The breaker and the fallback together define the degraded mode: for the triage service, tickets go to the non-AI rules or a human queue and the queue length is shown on a dashboard. Each call should log the fingerprint, the provider’s request id, the attempt count, the finish reason, the validation status, and the latency, so that a bad hour can be reconstructed. These are the quantities that make the failure rates in the earlier sections observable.

Provider changes and tests

The contract has a counterpart that you do not control. Providers add fields, rename or retire models, change defaults, introduce new finish reasons, adjust limits, and update models behind aliases. Some of these changes are announced and some arrive as new behavior. The application needs tests that fail when the contract stops being true, and a design that fails safely when it cannot recognize something new.

Four kinds of change cause most of the trouble:

• Retirement of a pinned model. The provider publishes a schedule; the application must migrate and re-run its acceptance checks on the replacement before the deadline.
• Behavior change behind an alias or in a new snapshot. The format can shift (for example, a field’s type) or the content quality can shift without any change in format.
• New response elements. A new finish reason, a new content block type, or a new error type. Anthropic’s error documentation states that error type values may expand over time, so code that matches on an exhaustive list will meet values it has never seen.
• Changed limits and parameter rules. A parameter may be rejected on a newer model, or a limit may differ by tier.

The adapter pattern makes two defenses concrete. First, translate every provider value into the application’s own small vocabulary, and map anything unrecognized to an explicit unknown that fails closed: raise an alert and use the fallback rather than treating an unknown finish reason as success. Second, test the adapter against stored response bodies, so a provider change shows up as a failing test or a new unknown count, not as a mysterious production incident.

The next program shows both. It defines two invented provider response shapes, a normalized result type, the application’s outcome rule, and unit tests. The fixtures are small hand-written bodies in the style of a real response, not recordings of a real service, and the provider names, field names, and values are made up for illustration. Python 3.12 and the standard library’s unittest module were used.

"""An adapter that normalizes two invented provider response shapes, plus contract tests against
recorded-style fixtures. The providers and fixtures are made up for illustration. Python 3.12, stdlib."""
import unittest
from dataclasses import dataclass

@dataclass(frozen=True)
class ModelResult:
    text: str
    finish: str            # complete | truncated | refused | unknown
    input_tokens: int
    output_tokens: int

FINISH_A = {"stop": "complete", "length": "truncated", "content_filter": "refused"}
FINISH_B = {"end_turn": "complete", "max_tokens": "truncated", "refusal": "refused"}

def adapt_a(body: dict) -> ModelResult:        # provider A: choices[0].message.content
    ch = body["choices"][0]
    return ModelResult(ch["message"]["content"] or "", FINISH_A.get(ch["finish_reason"], "unknown"),
                       body["usage"]["prompt_tokens"], body["usage"]["completion_tokens"])

def adapt_b(body: dict) -> ModelResult:        # provider B: content blocks plus stop_reason
    text = "".join(b["text"] for b in body["content"] if b["type"] == "text")
    return ModelResult(text, FINISH_B.get(body["stop_reason"], "unknown"),
                       body["usage"]["input_tokens"], body["usage"]["output_tokens"])

def to_outcome(res: ModelResult) -> str:
    """The application's rule: only 'complete' may be parsed; anything else is handled explicitly."""
    return {"complete": "parse", "truncated": "raise_limit_or_fail", "refused": "fallback"}.get(res.finish, "alert_and_fail_closed")

def body_a(content, reason, p, c):
    return {"choices": [{"message": {"content": content}, "finish_reason": reason}],
            "usage": {"prompt_tokens": p, "completion_tokens": c}}

FIXTURES_A = {
  "ok":         body_a('{"a":1}', "stop", 40, 9),
  "truncated":  body_a('{"a":', "length", 40, 300),
  "new_reason": body_a('{"a":1}', "tool_calls", 40, 9),     # a finish reason the adapter has never seen
}
FIXTURES_B = {
  "ok":        {"content": [{"type": "text", "text": '{"a":1}'}], "stop_reason": "end_turn", "usage": {"input_tokens": 40, "output_tokens": 9}},
  "refusal":   {"content": [], "stop_reason": "refusal", "usage": {"input_tokens": 40, "output_tokens": 0}},
  "extra_block":{"content": [{"type": "thinking", "text": "ignored"}, {"type": "text", "text": '{"a":1}'}], "stop_reason": "end_turn", "usage": {"input_tokens": 40, "output_tokens": 30}},
}

class AdapterContract(unittest.TestCase):
    def test_a_complete(self):        self.assertEqual(to_outcome(adapt_a(FIXTURES_A["ok"])), "parse")
    def test_a_truncated(self):       self.assertEqual(to_outcome(adapt_a(FIXTURES_A["truncated"])), "raise_limit_or_fail")
    def test_a_unknown_fails_closed(self):
        res = adapt_a(FIXTURES_A["new_reason"])
        self.assertEqual(res.finish, "unknown")
        self.assertEqual(to_outcome(res), "alert_and_fail_closed")
    def test_b_complete(self):        self.assertEqual(to_outcome(adapt_b(FIXTURES_B["ok"])), "parse")
    def test_b_refusal_empty_content(self):
        res = adapt_b(FIXTURES_B["refusal"]); self.assertEqual((res.text, to_outcome(res)), ("", "fallback"))
    def test_b_ignores_non_text_blocks(self):
        self.assertEqual(adapt_b(FIXTURES_B["extra_block"]).text, '{"a":1}')
    def test_both_providers_yield_same_shape(self):
        a, b = adapt_a(FIXTURES_A["ok"]), adapt_b(FIXTURES_B["ok"])
        self.assertEqual((a.text, a.finish, a.input_tokens), (b.text, b.finish, b.input_tokens))

if __name__ == "__main__":
    unittest.main(verbosity=2)

test_a_complete (__main__.AdapterContract.test_a_complete) ... ok
test_a_truncated (__main__.AdapterContract.test_a_truncated) ... ok
test_a_unknown_fails_closed (__main__.AdapterContract.test_a_unknown_fails_closed) ... ok
test_b_complete (__main__.AdapterContract.test_b_complete) ... ok
test_b_ignores_non_text_blocks (__main__.AdapterContract.test_b_ignores_non_text_blocks) ... ok
test_b_refusal_empty_content (__main__.AdapterContract.test_b_refusal_empty_content) ... ok
test_both_providers_yield_same_shape (__main__.AdapterContract.test_both_providers_yield_same_shape) ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK

The seven tests pass. The test that matters most for change management is test_a_unknown_fails_closed: it feeds a finish reason the adapter has never seen and requires the outcome alert_and_fail_closed. When a real provider introduces a value like that, the test suite will not have predicted it, but the production code path it protects already refuses to guess. The block-filtering test records another provider-specific habit, a reply containing a content block that is not text, handled by selecting text blocks only.

Contract tests like these are fast and run on every change. They answer whether the adapter still translates known shapes correctly. They cannot tell whether the model still does the task well, and a model update can pass every shape test while shifting accuracy. A second layer addresses that: a fixed set of labeled tickets run against the pinned model before and after any change to the model, prompt, or schema, with the validator’s status counts and the task metrics compared against the thresholds in the acceptance criteria. Building that set and judging the comparison is covered in Evaluating AI Applications with Reproducible Test Sets. A third layer is a small number of live probe calls on a schedule, which detect provider-side changes behind an alias or a deprecation that took effect early, and whose failures are alerts, not test failures.

Rollouts of a changed model or prompt should be gradual, with a comparison of status counts and fallback rates between old and new, and a switch that returns to the previous pinned configuration. The fingerprint from the request contract identifies which configuration produced each logged call, which is what makes that comparison possible.

References

• Anthropic, “Claude API errors” (Claude API documentation): HTTP error types, request ids, SDK retry behavior, streaming errors, long requests.
• Anthropic, “Structured outputs” (Claude API documentation): schema-constrained output, supported JSON Schema subset, refusal and token-limit behavior.
• Anthropic, “Handling stop reasons” (Claude API documentation): finish reasons and how to treat each.
• OpenAI, “Structured Outputs” and “Error codes” guides (OpenAI API documentation): JSON mode versus schema adherence, refusals, incomplete responses, rate limits versus quotas, retry guidance.
• Fielding, Nottingham, and Reschke, “HTTP Semantics,” RFC 9110, sections on idempotent methods and Retry-After.
• Marc Brooker, “Exponential Backoff and Jitter,” AWS Architecture Blog.
• “Addressing Cascading Failures,” in Beyer, Jones, Petoff, and Murphy (eds.), Site Reliability Engineering, O’Reilly Media.
• “Understanding JSON Schema” (json-schema.org): keywords for required properties, enums, and additional properties.