Building an AI Investment Case That Can Be Tested

A fictional equipment-leasing company, Harborline, approved a contract-review assistant on the strength of a vendor demonstration and one sentence in the proposal: reviewers would spend 40 percent less time on each lease. Twenty-four reviewers handle about 18,000 standard lease reviews a year. Six months after approval, the CFO asked three questions nobody could answer. How long did a review take before the assistant? What did the assistant cost once integration, support, and monitoring were counted? And how would anyone know that a change in review time had been caused by the assistant and not by a quieter quarter? Harborline and every figure in this article are invented for illustration.

The CFO’s questions describe what a testable AI investment case contains: a baseline that was measured, a cost that includes everything the assistant needs to keep working, a way of attributing benefit that survives a skeptical reader, a pilot built to produce that attribution, and thresholds written before the results arrive. The step before this one, deciding which candidates deserve a funded test at all, is the subject of Choosing an AI Transformation Portfolio. Here the candidate has been chosen, and the work is to find out whether it earns its money.

Turn the claim into something evidence can contradict

“Reviewers will spend 40 percent less time” cannot lose to any evidence as written. Less time doing what, measured how, on which contracts, compared with what, and at what cost? Harborline’s sponsor rewrote it as a claim with a denominator, a comparison, and a price: for standard lease reviews, mean active reviewer minutes per completed review will fall from 50 to 30, the share of reviews that miss a material clause will be no higher than today’s, and the three-year cost of the assistant will be recovered by the value of the time released. Each part can now fail separately. The time can fall by less than promised, quality can slip, or the cost can exceed the value even when the time falls as promised.

The 40 percent came from a vendor demonstration, which is a best-case input. The UK Treasury’s Green Book, its guidance on appraising public proposals, requires an explicit adjustment for optimism bias: costs are adjusted upward and benefits downward because forecasts made before delivery systematically lean the other way. A private company is not bound by that guidance, but the reasoning carries over. A pilot is the way to replace a forecast with a measurement, and it is worth building only if the measurement can change the decision.

Baseline performance: measure the work before anything changes

A baseline is a defined reference used to interpret change, and “about 45 minutes, I would say” from a manager is an assumption, not a baseline. Harborline defined its measure before collecting anything. The population is completed standard lease reviews, excluding amendments and non-standard clauses. The measure is active reviewer minutes, taken from work-session records, which exclude time a contract spends waiting in a queue or for a customer reply. Over the eight weeks before the pilot, 600 completed reviews consumed 30,000 active minutes, a mean of 30,000 / 600 = 50 minutes.

Time alone makes a poor baseline for a tool whose risk is that it makes people faster at being wrong. Harborline’s quality measure was the share of reviews in which an independent senior reviewer found a missed material clause. An audit of 120 baseline reviews found 9, or 7.5 percent. That figure is imprecise: an exact 95 percent interval for 9 of 120 runs from about 3.5 to 13.8 percent. The imprecision matters later, because a guardrail can only be defended against a baseline that is itself known well enough to compare with.

A baseline measured in the past also has a limit: conditions move. Contract mix, volumes, staffing, and templates all change between the baseline period and the pilot, which is why Harborline’s design, described below, keeps a concurrent comparison group instead of relying on the old number alone. The baseline still has two jobs. It defines the measures, so the pilot and the later steady-state reporting use the same definition, and it is the reference for the size of the claim: a 20-minute saving on a 50-minute task is a very different statement from a 20-minute saving on a 35-minute task.

Total implementation and operating costs

The proposal priced the assistant at its usage fee. A total cost of ownership estimate covers the stated period and workload: acquiring, integrating, operating, supporting, and eventually replacing the solution. The Case Said Four Million and the Cost Was Nine shows how much a partial estimate can leave out. For AI there are three lines that a traditional software estimate tends to miss: the usage fee scales with volume rather than seats, the evaluation set and quarterly re-evaluation are recurring work because model behavior and contract content both change, and someone must own the system after the project team leaves.

Harborline’s three-year estimate assumes flat volume of 18,000 reviews a year, no inflation, no discounting, and a loaded reviewer cost of 90 per hour, all in one currency. Salary lines use that hourly rate.

Cost line	Basis	One-time	Per year
Integration with the contract system	Build and test	240,000
Security and legal review	Before first production use	45,000
Reviewer training	24 reviewers × 8 hours × 90	17,280
Model usage	18,000 reviews × 1.20 per review		21,600
Document-platform add-on license	Annual contract		60,000
Support and operations	0.5 engineer × 150,000		75,000
Quarterly re-evaluation and audit	4 quarters × 80 hours × 90		28,800
Total		302,280	185,400

Three years of operation cost 302,280 + 3 × 185,400 = 858,480. Over the 54,000 reviews expected in that period, that is 15.90 per review, of which the usage fee that the proposal had in mind is 1.20. Reviewer time spent checking the assistant’s drafts is not a separate cost line, because it sits inside the active minutes that the pilot measures. The cost of discarded drafts is a different matter. If reviewers reject one draft in five, the fee per draft actually used is 1.20 / 0.8 = 1.50. This is the logic of cost per successful task: the denominator is the outcome the business wanted, not the number of calls.

Two things the table does not contain are worth stating. The pilot’s own cost, including the audit hours described below, is spent before the decision and does not belong in the comparison of going ahead against stopping. And the cost of leaving, if the supplier’s pricing or the model’s behavior later forces a change, is real but depends on contract terms that Harborline has not yet negotiated.

Benefits and what can be attributed to the assistant

During the pilot, reviewers using the assistant averaged 41 minutes per review. A before-and-after reading against the 50-minute baseline gives a 9-minute saving, or 18 percent. Reviewers handling standard reviews without the assistant during the same weeks averaged 48 minutes, because the mix of contracts had shifted toward simpler renewals. Against that concurrent comparison the saving is 48 − 41 = 7 minutes, about 14 percent of the baseline. Two minutes of the apparent improvement belonged to the calendar, not to the assistant. The Treasury’s term for outcomes that would have happened anyway is deadweight, and its term for the part that the intervention genuinely adds is additionality.

Two published studies show why attribution cannot rest on impressions or on average effects alone. METR ran a randomized trial in which 16 experienced open-source developers worked on 246 real issues, each randomly assigned to allow or disallow AI tools. The developers had expected AI to speed them up by 24 percent; the tasks took 19 percent longer when AI was allowed, and afterward the developers still believed it had sped them up by about 20 percent. That setting, with experienced developers on large familiar codebases and the tools of early 2025, is not Harborline’s, but it shows that people’s own sense of whether a tool saves time can differ from the measurement. In a study by Brynjolfsson, Li, and Raymond of 5,179 customer-support agents, access to a generative AI assistant raised issues resolved per hour by 14 percent on average, with a 34 percent gain for newer and less skilled agents and minimal impact for experienced, highly skilled workers. An average hid very different effects by worker group.

The second finding changes how Harborline should treat its reviewers. If gains concentrate among newer reviewers, then a benefit calculated on the whole team of 24 overstates what the experienced half will deliver, and a scaling decision should be tested on the population that will actually use the tool.

Measured minutes still have to be turned into value, and here the case is most easily inflated. Released reviewer time is capacity. It becomes money only if payroll, overtime, or contractor spend falls, or if the freed hours are put to work that the company values and can point to. Harborline spends 60,000 a year on contract reviewers for peak weeks, and that is the only line that clearly shrinks in cash. Benefits realization keeps the forecast, the observed change, and the realized financial effect apart for this reason, and Live for a Year, Used by a Third shows what happens to a budget when a forecast saving is booked before anyone checks that work has changed. The same hour cannot be counted both as redeployed work and as a payroll saving.

Valuation of the benefit	Hours released per year	Value per year	Three-year value	Three-year value minus 858,480 cost
Proposal: 20 minutes saved per review (40 percent)	6,000	540,000	1,620,000	761,520
Pilot: 7 minutes saved per review, all released time valued at 90 per hour	2,100	189,000	567,000	−291,480
Pilot, cash only: contractor spend removed	2,100	60,000	180,000	−678,480

The hours come from 18,000 reviews × minutes saved / 60. The proposal’s 6,000 hours equal 3.3 full-time equivalents at 1,800 productive hours each; the pilot’s 2,100 hours equal 1.2. The same assistant looks like a 761,520 gain or a 291,480 loss depending on a measurement and a valuation choice, and neither choice is hidden in a footnote when the table is laid out this way. Valuing every released hour at the full loaded rate is the generous reading. Even that reading loses.

Pilot design: produce the attribution before the money is committed

The pilot was built backward from the table above. It needed an estimate of the time difference that could be defended, a quality comparison, and a result that would not depend on who looked at it.

Twelve reviewers took part, six with under two years of experience and six with more, so that the result could be read by experience level. Each standard review entering their queue was randomly assigned, at entry, to assisted or unassisted handling, which gave every reviewer both kinds of work and removed differences between people from the comparison. Random assignment is also what protects against the calendar problem: contract mix and volume affect both arms alike. Within-reviewer assignment has a cost. Reviewers learn from the assistant and may carry habits into unassisted reviews, which would shrink the measured difference. That bias runs against the proposal, so it is one reason to treat a clear negative result with care, and it is a risk the pilot report should name.

The unit of analysis is the reviewer, not the review. Fifty assisted and fifty unassisted reviews from the same person are not 100 independent observations, because that person’s pace, habits, and judgment run through all of them. The sample size rule is to name the unit of randomization or dependence before counting, so Harborline computed one assisted-minus-unassisted difference per reviewer and analyzed 12 differences. That is far fewer than the 1,200 reviews in the pilot, and it is the count the data supports.

The quality guardrail needs more data than the time comparison, and this is the part of pilot design most often underestimated. To show with 95 percent one-sided confidence that the assistant does not raise the missed-clause rate by more than 3 percentage points, starting from 7.5 percent and with 80 percent power if there is truly no difference, the standard normal approximation needs n = (1.645 + 0.842)² × 2 × 0.075 × 0.925 / 0.03² ≈ 953 audited reviews per arm. At 20 minutes per senior audit, auditing 1,906 reviews takes about 635 hours. Harborline’s pilot budget covered 150 audits per arm. That sample can describe quality but cannot establish that quality held, and the pilot design said so in advance: the guardrail would be reported with its interval, and the full audit would be funded only if the time result justified it. A pilot whose measurement is too small for its question has not saved the cost of measuring; it has hidden it.

The rest of the design is discipline about change. Reviewers were trained and given two weeks of unmeasured practice, so that the pilot measures steady use and not the first awkward days, although a longer window could show further learning. The model version, prompt, and review procedure were frozen for the measured weeks, because a mid-pilot update turns two arms into three. The primary measure, the comparison, and the exclusion rules were written down once. Segment analyses by contract type or reviewer experience were allowed to be reported but only as leads for a later test: look at enough subgroups and one will always look good, which is the problem that a pre-registered decision gate exists to prevent.

Acceptance and scaling thresholds

The threshold that matters most is the break-even saving, the switching value at which the decision changes. With a three-year cost of 858,480, 54,000 reviews, and time valued at 90 per hour (1.50 per minute), the assistant pays back when the saving is at least 858,480 / (54,000 × 1.50) = 10.6 minutes per review, which is about 21 percent of the 50-minute baseline. That is roughly half the 20 minutes the proposal promised. It is also the figure that makes the pilot’s job concrete: the pilot is not trying to prove that the assistant saves some time, but to determine whether it saves at least 10.6 minutes on the work that will actually be given to it.

Harborline wrote its pilot exit criteria before the pilot began, with the owner named and the review date set. The pilot would be analyzed with a 90 percent confidence interval around the mean per-reviewer saving.

• Expand when the lower end of the interval is at least 10.6 minutes and the guardrail does not exceed its margin. The case then pays even at the pessimistic end of the estimate.
• Stop, for this case as costed, when the upper end is below 10.6 minutes. No plausible value of the measured saving covers the cost.
• Extend or redesign when the interval contains 10.6, with a budget for the added sample or the changed scope and a review date.

The outcome category exists to protect an inconclusive result from being forced into a yes or a no by whoever presents it. Separate from this statistical threshold are the conditions for scaling: a named operating owner, a monthly cost-per-review report, a re-evaluation schedule that triggers on model changes, and a documented way to turn the assistant off. Those conditions apply only if the evidence clears the first gate, and meeting them does not substitute for it.

The twelve reviewer-level savings, in minutes, were 2.4, 3.1, 3.9, 4.6, 5.5, 6.3, 7.2, 8.0, 8.9, 10.1, 11.0, and 13.0. Their mean is 7.0 and their standard deviation is 3.3, so the standard error is 3.3 / √12 = 0.96. The t value for a 90 percent two-sided interval with 11 degrees of freedom is 1.80, and the interval is 7.0 ± 1.7, or 5.3 to 8.7 minutes. The upper end is 8.7, below the 10.6 threshold, so the rule returns stop for this case as costed. Split by the design’s experience strata, the six newer reviewers averaged 9.7 minutes (13.0, 11.0, 10.1, 8.9, 8.0, 7.2) and the six experienced reviewers averaged 4.3 (6.3, 5.5, 4.6, 3.9, 3.1, 2.4). That pattern resembles the one in the customer-support study, and it is a lead for a new test, not a result: six people per group cannot carry a decision, and even the newer group’s mean is below 10.6.

The quality guardrail did not resolve. Assisted reviews had 9 missed clauses in 150 audits (6.0 percent) and unassisted reviews had 11 in 150 (7.3 percent), a difference of −1.3 points. The one-sided 95 percent upper bound for the difference, in which the assistant is worse, is +3.4 points, beyond the 3-point margin. The pilot therefore cannot show that quality held, and it cannot show that quality fell. Had the time gate passed, this result would have required the full audit before expansion.

The conclusion for Harborline’s leaders is narrower than “AI did not work.” Reviewers saved real time, about 7 minutes per standard review. The investment case, at its cost and at this volume, does not pay for that saving. The options that remain are specific. A cost reduction would work, but it has to be large: at a 7-minute saving the three-year cost would have to fall from 858,480 to 7 × 1.50 × 54,000 = 567,000, a cut of 291,480 or about 34 percent, and the largest lines are the integration build and the support engineer. A narrower scope could work if a segment has a larger saving, but the segment results from this pilot are leads, and the claim would need its own pre-registered test. Higher volume could work, because the fixed lines would then be spread over more reviews and the saving per review would count more times, but only if the contracts to be added are the same kind of work the pilot measured. Or the company can stop and spend the audit budget elsewhere, with its baseline measures intact for the next candidate. In each case the sponsor, not the vendor, owns the decision, and the case record keeps the original 40 percent claim beside the measured 14 percent so that the next proposal is read against how this one fared.

References:

• HM Treasury, The Green Book (2026), on business-as-usual baselines, deadweight and additionality, optimism bias, and switching values (Open Government Licence v3.0).
• METR, randomized trial of early-2025 AI tools with experienced open-source developers.
• Erik Brynjolfsson, Danielle Li, and Lindsey Raymond, Generative AI at Work, NBER Working Paper 31161.