Platform Engineering Leadership for AI Adoption
A platform leader has 84 engineer-months and seven AI requests. Decide scope, standards, developer tooling, shared services, and reliability and cost investment.
-
-
Model Fallback
A useful fallback needs tested quality, independent availability, ready capacity, and a clear policy for timeouts and partial responses.
-
AI Agent Tool Execution and Reliability
A fictional support agent repeats a refund after a timeout. The case shows why model-driven actions need stable operation IDs and enforced execution rules.
-
Latency Budget
Step percentiles do not add to an end-to-end percentile. A useful budget reserves fallback time and tests the complete request path under load.
-
Model Serving Architecture
A fictional checkout failure shows why stored scores, online features, fallback capacity, and recovery compatibility need explicit contracts.
-
Recovery Drill
In 2011 Google restored Gmail from tape within hours of its own estimate. Being able to produce that estimate was itself the product of rehearsal.
-
Capacity Planning
Capacity planning connects demand, service targets, and failure tolerance. Test usable capacity and spare headroom before turning a forecast into clusters.
-
Graceful Degradation
Exercise degraded modes and their transitions under representative conditions. Consider bounded production rehearsals for uncertainty that other tests cannot address.
-
Risk Tolerance
Choose reliability targets with product owners and engineers. Compare failure impact and concrete design costs, then revisit the commitment as needs change.
-
Availability
A 99.99% target can limit failed requests or downtime. The population, success rule, and measurement window determine what that percentage promises.
-
Error Budget
Calculate the allowed failures, then agree what to do when the allowance runs low. Measurement, release restrictions, and recovery are separate decisions.
-
SLIs, SLOs, and Error Budgets
A reliability target needs a measurement boundary, a reason, and a response to failure. Learn how SLIs, SLOs, and error budgets support that decision.
-
Blast Radius
Ten cells do not guarantee 90% unaffected traffic. Estimate impact from the affected workload share and the shared dependencies that can spread a failure.
-
Two-Phase Commit
A prepared transaction keeps its locks until it is told the answer. A recovered coordinator replays its decision; until then the rows stay held.
-
Compensating Transaction
Cancel the reservation, do not put the number back. Between the booking and the cancellation, other people booked too.
-
Idempotency Key
Reuse a durable operation key within the provider’s retention window. A recovered response may still leave the business outcome unresolved.
-
Throttling
Throttling slows or refuses requests once a consumer passes its limit, so one burst does not take down everyone. Server-side limits protect capacity by themselves; callers that are told and back off keep retries from adding to the load.
-
Fault Injection
Fault injection exercises recovery paths before an incident. A refund example shows why an apparently successful reply can hide a repeated side effect.
-
Retry Budget
A retry budget limits the share of traffic spent on retries, preventing widespread failures from multiplying calls to an already struggling dependency.
-
Write-Ahead Step Record
A workflow step that changes another system cannot be made atomic with its own record. Writing the intent and a baseline first lets a restart check what happened instead of guessing.
-
Young-Daly Checkpoint Interval
Young’s first-order approximation balances the cost of writing checkpoints against the work lost to failures. Daly’s refinement shows when the simple square-root rule is accurate and when it overshoots.
-
Test Oracle
Define how a test decides correctness, preserve relevant duplicates and tolerances, and avoid deriving expected results from the same flawed logic.
-
Test Fixture
Create controlled input and environment for a test, include boundary cases, and clean up resources without losing the evidence needed to debug.
-
Fencing Token
Reject writes from obsolete worker generations at the protected resource, and distinguish an enforced fencing token from a scheduler lock or lease.
-
Incremental Processing
Update results from bounded changes, identify affected outputs, and distinguish incremental work from complete capture and safe retries.