Deciding When an AI Pilot Is Ready to Scale

A fictional electricity retailer ran a twelve-week pilot of an assistant that drafts replies to billing cases for its contact-center agents. Sixty agents volunteered. The vendor’s dashboard reports that average handling time fell from 20.0 minutes to 11.9 minutes, a 40.5 percent reduction, and the executive sponsor proposes extending the assistant to all 600 agents next quarter. The retailer, its agents, and every number in this article are invented. The VP who must approve the budget asks what the dashboard cannot answer: if the assistant reaches everyone, what will quality, risk, and cost be, and what would make the company stop?

Answering takes more than reading a favorable result. A pilot is a small, favorable, closely watched version of the future service, and scaling changes who uses it, what it is asked to do, how much capacity it needs, and what each unit of work costs. The decision rests on evidence about quality, safety, reliability, adoption, and unit economics, each judged against a limit agreed before the results arrived. Then it needs a way to extend the service in stages that can be reversed, and a clear choice among scaling, redesigning, and stopping.

What a pilot can and cannot show

The retailer’s pilot differed from the planned service in ways that are typical. Agents volunteered, so they were more willing to change how they work. The project team sat nearby and fixed problems within hours. Volunteers tended to use the assistant on the cases it handled best: 70 percent of pilot cases were simple billing questions, against 50 percent across the company in the previous quarter. Peak load was one tenth of what the full service would see. None of this makes the pilot useless. It means each pilot number answers a narrower question than the one the sponsor is asking, and the review must say which question each number answers.

Pilot exit criteria are the agreed rules that turn the evidence into a decision: what would justify advancing to a defined next stage, what would justify stopping, and what would justify one more bounded test. Written after the results, they tend to describe whatever happened. Written before, they force the team to name the measures, the limits, and the person who decides. For the retailer, the criteria cover the five kinds of evidence below, and they are tied to the next stage, not to company-wide use. Passing them authorizes a wider trial with its own limits.

The public NIST AI Risk Management Framework describes the same structure from a risk perspective. Its Measure and Manage functions call for testing before deployment and regularly during operation, for monitoring plans that capture user input after deployment, for a determination of whether the system meets its intended purpose and should proceed, and for mechanisms to supersede, disengage, or deactivate a system whose performance is inconsistent with its intended use. The pilot review is the point where the first of those determinations is made, and the last is a design requirement for the rollout that follows.

Quality and safety: counting the errors that matter

Quality evidence has to come from an examination that does not depend on the people who benefit from the assistant. The retailer defined a material error as a statement in a filed reply that is wrong about a charge, a deadline, or a customer’s rights. A harmful error is a material error that reached the customer uncorrected. A quality team, not the agents, audited a random sample of 400 filed replies from assisted cases and 400 from unassisted cases handled under the same procedures. The audit sample is drawn after the work is done. A fixed evaluation set of representative cases, kept separate, is used later to check that a new prompt or model version has not regressed.

The audit found 14 material errors among 400 assisted replies (3.5 percent) and 10 among 400 unassisted replies (2.5 percent). A 95 percent Wilson interval, which is a standard way to bound a proportion from a sample, runs from 2.1 to 5.8 percent for the assisted replies and from 1.4 to 4.5 percent for the unassisted ones. The intervals overlap heavily. The pilot gives no basis to say the assistant is worse than agents working alone, and none to say it is as good. A sponsor who reads 3.5 against 2.5 as a verdict, in either direction, is reading noise.

Harmful errors matter more, and they are rarer. Two of the 400 assisted replies were harmful, a rate of 0.5 percent, with an interval from 0.14 to 1.8 percent. Zero harmful errors in 400 would not have settled the question either: the interval for a zero count still reaches 0.95 percent. At the planned volume of 2.88 million cases a year, with the assistant used on the 61 percent of cases observed in the pilot, the same interval implies somewhere between roughly 2,400 and 31,700 harmful replies a year, with 8,800 at the observed rate. The pilot has not measured the harm rate to within a factor of ten, and the decision has to be made with that width in view.

Both harmful errors were on complex cases, which made up 30 percent of pilot cases. A pooled rate hides that concentration, so the audit reports each case type separately and the rollout gates below inherit the split. The consequences of a wrong statement also belong to the company. In Moffatt v. Air Canada (2024), a British Columbia tribunal ruled that an airline was responsible for incorrect refund information its website chatbot gave a customer, and rejected the argument that the chatbot was a separate entity. The cost of an error includes who answers for it, not only what it takes to correct.

Human review is the usual answer to this risk, and its strength is an empirical question. Automation bias is the tendency to accept automated output even when evidence points the other way, and it grows with volume, time pressure, and a record of correct drafts. The retailer tested its reviewers directly. The quality team planted 20 known errors among 200 drafts shown to agents, and the agents caught 11 of them, 55 percent. Real-world detection may be higher or lower, since planted errors are not natural ones, but the result shows that an agent’s approval is not a control by itself. Human oversight works when the person can see the evidence, has time, and has authority to reject, and it has to be checked with realistic cases and workload.

Reliability: what happens when the assistant is not there

Reliability evidence has two parts: how the service behaved, and what the work does when it fails. Over the 480 working hours of the pilot (12 weeks, five days, eight hours), the assistant was unavailable for three episodes totaling six hours, which is 98.75 percent availability during working hours. That is a statement about 60 users, one set of volumes, and one vendor quota. The pilot’s peak was 45 requests a minute. Ten times the users means about 450 requests a minute at peak, and the vendor contract allows 300. A test that only repeats the pilot cannot reveal that the service will be throttled in the first busy hour, so the load test for the next stage is designed from the scale arithmetic, not from the pilot’s dashboard.

The more important question is the fallback. When the assistant is down, agents return to the manual procedure, and the review should check that this works for the people who will be using it. The 60 volunteers remembered the old process; agents who start with the assistant will not. A provider-level outage may call for model fallback, which routes work to an alternative model that has been evaluated for the task, and for a non-model path when no alternative is acceptable. Offering a reduced but useful service, such as sending agents a library of approved reply templates, is graceful degradation. Google’s SRE guidance describes the same idea under overload: reduce the amount of work the system does rather than fail the request. Each fallback path is rehearsed, because an untested one is a hope.

Adoption and the size of the effect

The headline number needs two corrections before it can be compared with anything. The first is case mix. Last quarter’s baseline averaged 20.0 minutes across all cases: 10 minutes for simple cases and 30 for complex ones, half and half. In the pilot, simple cases took 8 minutes and complex cases 21, and with 70 percent simple cases the average was 0.7 × 8 + 0.3 × 21 = 11.9 minutes. Weighting the pilot’s own times by the baseline’s 50/50 mix gives (8 + 21) / 2 = 14.5 minutes. The like-for-like reduction is 5.5 minutes, or 27.5 percent, not 40.5. The reduction is real, since both case types improved, but a third of the headline came from who chose which cases. The baseline is a defined reference, and it has to be built on the same population and definitions as the result it is compared with.

The second correction is adoption, which is a measurement and not a launch date. Fifty-eight of the 60 agents used the assistant in week 2; 41 did in week 12, 68 percent. Across the twelve weeks it was used on 61 percent of cases. Those who dropped away may have found the drafts unhelpful, and that is information. Volunteers also differ from the agents who will be assigned the tool, so the sustained use rate in the next stage, among agents who did not ask for it, is the number that matters. Live for a Year, Used by a Third follows a program whose benefits depended on adoption that never came.

Keeping a primary metric and guardrail metrics apart prevents a good number from hiding a bad one. Handling time is the primary measure here. The harmful-error rate, the escalation rate, and customer re-contact within seven days are guardrails that must stay inside their limits, whatever happens to time. Measuring time saved at all requires first defining the work, the baseline, and the comparison method.

Unit economics: where the evidence meets the budget

Unit economics relates cost to a defined unit of work. FinOps guidance for AI starts from cost per token and advises weighing it against the business outcomes achieved, and the outcome side is the one a scaling decision needs. A cost per request says what a call costs. Cost per successful task charges the money spent on failures, retries, and review to the tasks that succeeded, and it is sensitive to the quality evidence above. The retailer’s per-case calculation uses the corrected figures and is hypothetical throughout.

Per case where the assistant is used	Amount	Basis
Agent time saved	\$3.30	5.5 minutes × \$0.60 loaded cost per minute
Model and platform usage	−\$0.30	Assumed contract price at the planned volume
Quality audit	−\$0.48	10 percent of cases audited × 8 minutes × \$0.60
Expected cost of harmful errors	−\$2.00	0.5 percent × \$400 average remediation per harmful error
Net variable value	\$0.52	3.30 − 0.30 − 0.48 − 2.00

Fixed costs are \$1,040,000 a year for the platform integration, evaluation work, agent support, and vendor minimum. Spread over 2.88 million cases (600 agents × 20 cases a day × 240 days), that is \$0.361 per case, whether or not the assistant is used on the case. If the assistant is used on a share a of cases, the net value per case is 0.52a − 0.361. It breaks even at a = 0.361 / 0.52 = 69.4 percent. The pilot’s 61 percent gives −\$0.044 per case, about −\$126,000 a year. At 80 percent use the result is +\$0.055 per case, about +\$158,000 a year. Whether this is worth doing turns on adoption, which no one has yet measured among non-volunteers.

The result is more sensitive to safety than to adoption. The net variable value is zero when the harmful-error rate is (3.30 − 0.30 − 0.48) / 400 = 0.63 percent. The observed 0.5 percent sits just inside that rate, and the upper end of the interval, 1.8 percent, produces a variable value of −\$4.70 per case, which no amount of adoption repairs. A switching value is the input value at which a comparison changes, and here the harmful-error rate has one close to the observation. The safety estimate and the economic case are the same question.

Two cautions apply. Minutes saved are capacity, not cash. They become a saving only if the retailer reduces overtime, avoids hiring, or redeploys the time to work with measured value, and benefits realization needs an owner who can decide that. Second, the money already spent on the pilot cannot change what comes next. A sunk cost neither argues for scaling nor for stopping, and the comparison should include only the future costs and benefits that differ between the options.

A staged rollout with gates and a way back

The next stage is a wave of 120 agents who did not volunteer, about 20 percent of the 600. The wave exists to produce the missing evidence at lower exposure, and it can be preceded by shadow testing for any new case type: the assistant drafts replies on live cases while agents use only their own, so output can be compared without customers seeing it. Shadow testing cannot show how agents would change their behavior in response to the drafts. A canary, or limited live release, can. A traffic percentage is an exposure setting, not a sample size, so the question of how many cases the wave must audit comes before the question of how many agents it includes.

Google’s SRE guidance on canarying adds two practical points: compare the canary against a concurrent control instead of against an earlier period, since time itself changes metrics, and choose a small number of measures that clearly belong to the change. The retailer therefore audits unassisted cases from agents outside the pilot and the wave during the same weeks. Auditing 10 percent of cases from 120 agents using the assistant on 61 percent of their 2,400 daily cases produces about 146 audits a day, so 1,200 audits take roughly two working weeks.

The gate is set before the wave starts. Leadership has set a harm ceiling of 1 percent: the upper bound of the 95 percent interval must be below it. With 1,200 audits, five harmful errors gives an upper bound of 0.97 percent and passes; six gives 1.09 percent and fails. The probability of five or fewer depends on the true rate. At a true 0.3 percent it is 84 percent, at 0.5 percent 45 percent, and at 0.63 percent 23 percent. That has a consequence for how the result is read. If the true harmful rate is the pilot’s 0.5 percent, the gate fails more often than it passes, because 1,200 audits are too few to show, with 95 percent confidence, that a rate near 0.5 percent is below 1 percent. A failure then does not mean the idea is worse than it looked; it means the design has to improve, or the sample has to grow, before more people are exposed. Passing also has to satisfy the economic limit: a point estimate at or under 0.63 percent, which five harmful errors in 1,200 (0.42 percent) does.

Reaching the gate does not end the safety work. A kill switch turns a function off in a running system without a deployment, and it works only if the check sits where the effect happens: if the switch is read when a session starts, sessions already open continue to send drafts. For the retailer the check is at the send step. Rollback is more than a flag, because agents who moved to the assistant must still be able to do the work without it. Triggers for pausing a wave are written in advance and include a harmful error found in audit, a guardrail breach, quota exhaustion, and an unexplained fall in sustained use. This design supplies the mechanism to disengage the system that the NIST framework calls for, with the person who may pull the switch named.

Scale, redesign, or stop

The decision is among three choices, and the evidence supports different ones depending on what failed. Scale is justified when each kind of evidence meets its limit in the stage that has just been run, the next stage’s controls exist, and someone owns the benefit. Redesign is justified when the pilot shows the idea works but the current design produces an avoidable problem. Stop is justified when the safety or economic limit cannot be met by any design the organization can afford, when the learning value of another test is low, or when the need has changed. Each is a decision about the current attempt, and none discards the underlying business need.

The retailer’s evidence points to redesign. Time savings are real after correction, at 27.5 percent. Quality cannot yet be distinguished from the unassisted baseline. The two harmful errors cluster in complex cases, where the approval screen shows the draft but not the tariff clause behind it. Agents, as the planted-error test showed, are not reliably catching what they cannot see. The redesign shows the cited clause beside each statement of a charge or right and requires the agent to open it before sending on complex cases. That change targets the failure that was found, and it is tested in the 120-agent wave and not assumed. The wave also tests whether use among non-volunteers reaches the 69.4 percent break-even.

The decision record states the stop conditions in the same terms. The wave stops if a harmful error appears that the audit attributes to missing source information, or if the sample’s upper bound exceeds 1 percent once 1,200 audits are complete and the sample cannot be extended at reasonable cost. The program stops if sustained use settles below 69.4 percent after support and workflow changes, because the fixed costs are then not recovered, or if the vendor cannot raise the quota before the following wave. If it stops, the company keeps what has value: the audited evaluation set, the measurement method, the baseline, and a short written account of why. Pilot exit criteria allow a fourth outcome, one more bounded test, only for a specific evidence gap with a budget and a date, which here is the harmful-error rate among non-volunteers.

A decision to scale is therefore a decision to enter the next stage under controls, with a review date, and a decision not to scale yet records what would change it. The sponsor’s original request for 600 agents next quarter becomes a staged plan: 120 agents now, a gate in about two weeks of audits, and a further wave only if the gate and the economic limit are both met.

References:

• NIST, AI Risk Management Framework: Core (Measure and Manage functions).
• Google, Site Reliability Engineering Workbook: Canarying Releases.
• Google, Site Reliability Engineering: Addressing Cascading Failures (load shedding and graceful degradation).
• FinOps Foundation, FinOps for AI Overview (unit economics for AI services).
• Manatt, AI Gone Wild: Airline Has to Honor a Refund Policy Its Chatbot Fabricated (summary of Moffatt v. Air Canada, 2024 BCCRT 149).