Shared AI Platform for Language Models

Consider a fictional company where nine teams have adopted language models within a year. Each team has its own API key, but the keys belong to the same provider project and some applications share a model’s quota. On a Tuesday afternoon, analytics starts summarizing two years of support tickets. The batch of ordinary API calls consumes the shared allowance, and the customer-facing assistant returns rate-limit errors for forty minutes. Later, a regional outage leaves applications without a route they can use. The quarterly bill triples without a useful breakdown by team, three teams discover a model retirement through production errors, and a security review finds customer names and account numbers in debug logs.

These incidents involve both application choices and shared responsibilities: capacity allocation, dependency tracking, cost ownership, and data handling. Once several teams use models, those shared decisions need an owner. This article covers what a platform should provide and what application teams still need to decide and test.

What to centralize, and what not to

A shared AI platform is a specialized internal developer platform: a set of integrated capabilities for model access, such as approved models, credentials, quotas, and routing, that teams obtain without assembling them themselves. The general question of which responsibilities a platform should take on, and which it must refuse, is covered in The Load Has to Land Somewhere. For language models the dividing line is fairly clear.

The platform owns	Application teams own
Access to approved models, credentials, and data-handling rules	Which approved model fits their task
Quotas, priorities, and isolation between teams	Their demand forecasts and traffic patterns
A catalog of model versions and migration notices	Testing and migrating their own applications
Routing, retries, and fallback when a provider fails	Choosing and testing their fallback behavior
Call and attempt records, usage reconciliation, and cost allocation by team	Their prompts, retrieval, and tools
A harness for running evaluations on every change	The evaluation sets and pass criteria for their task

The mechanism is usually a gateway: a service every model call passes through, in the same way an API gateway handles cross-cutting concerns for ordinary services. The gateway holds the provider credentials, applies quotas, routes requests, and records each call. Teams call the gateway with credentials the platform issues. A shared client library can add convenience, but the controls belong in the gateway, where they cannot be bypassed by an old library version. As with any platform, adoption is evidence of value; a gateway that is slower and harder to use than calling the provider directly will be routed around, which is why it should be run as a product, with the easy path also being the golden path.

Access, credentials, and data rules

Provider keys should exist only in the gateway, stored and rotated under secrets management, so that no application’s configuration or repository holds them. Teams receive their own credentials to the gateway, which identifies them for quotas and costs and can be revoked without touching anyone else.

Each approved model should carry a record of what data it may receive. Providers and deployment options differ in where data is processed, whether it is retained, and under what terms. The rule that customer account data may go only to models processed in a particular region, under a no-retention agreement, is a lookup against the same data classification the data platform uses, and the gateway is the place to enforce it. Prompts and responses are data too. If they are logged, the logs need log redaction for sensitive values and a retention period, which is exactly what the debug logs in the opening lacked.

Quotas, priorities, and isolation

Provider limits have both a scope and a unit. The Gemini Developer API applies its rate limits per project, with limits varying by model and usage tier. Its usual dimensions include requests per minute, input tokens per minute, and requests per day; exceeding any applicable limit can reject a request. Several keys in that project do not create separate allowances. A shared billing account does not, by itself, establish that different projects share one quota. Identify the actual shared boundary before trying to isolate a noisy neighbor.

The platform gives teams spending budgets and separately enforces the provider’s applicable request, token, and concurrency limits through throttling. A thousand one-token requests can stay below a token budget while exceeding a request limit. Gateway replicas must coordinate admission against the same project, model, and location limits where those scopes apply, using atomic counters or capacity reservations rather than independent local counters. Include retries and fallbacks in the appropriate destination’s allowance. When final usage is not known at admission, reserve a conservative amount under the provider’s accounting rules and reconcile it with reported usage. Interactive traffic gets priority; batch work uses explicitly allocated headroom or a separately governed Batch API. The same distinction between cost budgets and capacity controls matters for tenant isolation.

Google Cloud’s Standard PayGo is a separate service policy from the Developer API’s project quotas. For supported usage tiers, it provides an organization-level baseline and serves excess traffic on a best-effort basis. A baseline is not protection against every burst or outage. Provisioned throughput reserves processing capacity under a model- and location-specific contract. Size it using the product’s capacity units, input/output mix, cache behavior, and short-term demand, then test the latency target. Eligible online overages may use PayGo under the request policy; asynchronous batch work has separate support, prices, and quotas. The forecasting and headroom decisions belong to capacity planning.

Versions and the retirement calendar

The platform keeps an inventory of each application’s product, model identifier, release stage, support period, and retirement notice policy. Google Cloud’s lifecycle page says listed retirement dates will not be brought forward; its minimum 45-day migration notice is stated for the shorter-availability model group. Do not apply that clause to every model or to the Developer API’s separate preview policy. Plan evaluation, capacity procurement, rollout, and recovery within the actual notice window. Embedding migrations also need a compatibility check: query and document spaces, dimensions, normalization, preprocessing, and retrieval quality must agree. Incompatible spaces require re-embedding the affected corpus and rebuilding the relevant index; compatible stored vectors may be reusable after validation. Keep query encoders and index versions aligned during transition.

Use model version pinning to reduce unintended alias migrations under the provider’s version contract. A fixed identifier does not freeze sampling, serving infrastructure, safety layers, retrieval data, or every aspect of behavior. The gateway can reject moving aliases where production policy requires an exact version. Record the model, prompts, schemas, and settings as one release configuration, as described in LLMOps; changing one component does not require changing all the others.

Evaluation and regression checks as a service

Model, prompt, and retrieval changes need evaluation as well as production observation. Teams own task examples and acceptance criteria, while privacy, security, and other shared policies need their designated owners. The platform supplies a harness that versions configurations and results, compares candidates with the accepted release, and applies a fixed evaluation protocol. Keep stable regression fixtures for known failures separate from independent confirmation data used after candidate selection. Repeated tuning to one set can overfit it. Prespecify acceptable deterioration, report important segments and paired uncertainty, and account for stochastic output variation where it affects the decision. A threshold pass supports release only within those conditions. For a forced migration, schedule evaluation early enough to allow correction, capacity checks, staged rollout, and a recovery path that remains available before retirement.

When the provider fails

A 429 response needs diagnosis. Temporary contention may justify bounded retries; a daily quota, sustained excess demand, or an account restriction may require waiting for reset, reducing traffic, or changing the allowance. For retryable failures, use exponential backoff with jitter, honor the provider’s Retry-After contract when supplied, and impose a total deadline or attempt budget. A cap on the wait interval does not stop retries. Assign one retry owner so client and gateway loops do not multiply attempts. Google Cloud recommends its global endpoint to reach a larger shared-capacity pool where data-handling rules permit it. That routing recommendation is not a promise that a regional reservation automatically follows traffic elsewhere.

Model fallback needs a path independent of the failure being handled. Another model at the same provider may share the failed authentication, region, or quota service. Even another provider’s tested model needs approved data handling and enough ready capacity for the diverted load. Validate prompts, schemas, tool support, and quality for each path; reuse a prompt when compatible and adapt it when needed. If available capacity cannot meet demand, prioritize requests, limit load, queue work, hand it to a person, or declare the feature unavailable. The choices that preserve useful partial or delayed service are graceful degradation. Declaring the feature unavailable communicates failure when no acceptable service remains.

Allocate the application’s end-to-end deadline across attempts. A gateway timeout only ends its wait; the primary may still run and consume capacity or incur charges. Request cancellation where supported, but track whether it was confirmed. Define which result can be accepted and what happens after part of a stream has reached the user: do not silently append a second model’s answer to the first. Keep late completions and unknown outcomes visible. If a model call leads to tool effects, protect those effects separately against duplicate execution.

A record of every call

Record the logical application request separately from each provider attempt: team, application, destination, model identifier, timing, status, retry/fallback decisions, and usage reported by the provider. A lost response or interrupted stream may leave usage unknown; do not record it as zero or silently treat a local estimate as measured consumption. Reconcile attempts when later evidence arrives. Link them to application spans through request tracing so investigation can follow retrieval, model calls, and tools. Alert on shared error rates, quota exhaustion, and fallback activation, with an owner who can act; Only a Few Times a Day covers alert design. Full prompt and response retention remains a separate data-handling decision.

Cost by team, and the levers to reduce it

Per-attempt records are evidence for cost attribution, not a complete invoice. Separate measured usage, estimated charges, and confirmed billing, and reconcile them with the provider’s billing export. Allocate reservation commitments, unused headroom, shared gateway costs, and applicable discounts under an agreed policy; show unknown or unallocated balances rather than hiding them in token rates. Teams can then use showback to see costs or chargeback to assign them to budgets. Daily estimates and budget alerts can guide action before the invoice arrives, provided their uncertainty and later adjustments remain visible.

The platform can also offer the levers. Smaller models for simpler tasks, measured on the team’s own evaluation set. Context caching for requests that share long prefixes. Batch processing at lower prices for work nobody waits for. Limits on output length, which drives both cost and latency. The arithmetic behind each is worked through in Valid JSON, Wrong Date.

What it shares with the data platform

Much of an AI platform is not new. Identity and access, data classification, lineage of what data flowed where, observability, cost management, and workflow scheduling already exist in the data platform, and the AI platform should reuse them rather than build a second set. Search indexes and embedding stores behind retrieval are data products in their own right, with owners, freshness targets, and access rules. Evaluation sets are datasets that need versioning. For model calls, ensure that the existing services can support the model gateway’s routing and controls, model catalog and retirement tracking, token-based quotas, and the evaluation harness. Extend or add components where those responsibilities are missing; an organization that already supplies them can reuse them.

What a reviewer should ask

• Do all model calls pass through one controlled path, and where are the provider keys?
• Which data may go to which models, and where is that enforced?
• How is the provider’s shared limit divided, and what happens to interactive traffic when a batch job starts?
• Which applications use which model versions, and when is each retired?
• What happens, application by application, when the provider is unavailable for two hours?
• Has each fallback model been evaluated on the application’s own set?
• Can last month’s spending be broken down by team and application today?
• Are prompts and responses logged, and under what redaction and retention rules?

A separate API key per team identifies access, but does not by itself isolate a shared provider quota or establish a complete cost allocation. A second provider under contract is not a working fallback until applications, permissions, capacity, and failure transitions have been tested. A monthly bill alone arrives too late to steer ongoing usage; it needs timely usage estimates and controls as well as reconciliation.

For the fictional company, the useful next step is a controlled rollout with evidence. Move credentials into the gateway and test simultaneous short requests, long responses, retries, and a batch spike against each applicable limit. Size the assistant’s reservation from measured demand and test the intended alternative endpoint with its own capacity and data permissions. Run regression checks and independent confirmation for migrations, and exercise both model and queue-based fallback paths within the deadline. Choose log retention from approved purposes and policy rather than a universal number of days. Finally, reconcile team usage with the bill. Lower cost and fewer visible failures are outcomes to measure after rollout, not benefits a gateway alone guarantees.

References:

• Google AI for Developers, Gemini API: Rate limits (Creative Commons Attribution 4.0).
• Google Cloud, Standard pay-as-you-go (Creative Commons Attribution 4.0).
• Google Cloud, Provisioned Throughput overview (Creative Commons Attribution 4.0).
• Google Cloud, Error code 429 (Creative Commons Attribution 4.0).
• Google Cloud, Model versions and lifecycle (Creative Commons Attribution 4.0).
• Google Cloud, Provisioned Throughput: Supported models.
• Google AI for Developers, Gemini API: Models.
• Google AI for Developers, Gemini API: Embeddings.
• AWS Builders Library, Timeouts, retries, and backoff with jitter.
• FinOps Foundation, Allocation.