Choosing an AI Transformation Portfolio

A fictional property insurer asked each business unit for AI ideas and received forty-one. A steering group rated every idea from one to five on value, feasibility, and risk (five is best), averaged the three ratings, and ranked the list. The top entry was a tool that summarizes adjuster notes on simple claims. Three of the next nine needed a document-intake service that no budget included. Five pilots from the previous year were still running and used 2.5 of the six machine-learning engineers’ time. When the chief operating officer asked which proposals would change how many claims were paid each day, which depended on one another, and which a process change could deliver without a model, nobody on the committee could answer. The ranking had been built from the ideas, not from the work.

The insurer, its claims data, and every figure below are invented for this case, and dollar amounts are hypothetical. Choosing AI initiatives follows the discipline of any investment portfolio, described in Ninety-Four Projects, Every One Approved, with four differences. Value reaches the business through a workflow, and a faster step does not necessarily make a faster workflow. Cost continues after launch, because models, data feeds, and evaluations need owners. Initiatives share data, platforms, and the same few specialists. And a change that involves no AI often competes for the same result. The strategic setting and the readiness of a single use case are covered in Fourteen Pilots and Nothing in Production; the question here is which of many candidates to fund, in what order, and which to stop.

Begin with the workflow, not the idea list

An idea box collects solutions: a summarizer, a chatbot, an agent. A use case is different. It describes who performs which task, in what situation, to obtain what result, and a tool name leaves all of that out. Of the forty-one proposals, seven named a tool but no workflow problem and went back to their sponsors, and six duplicated others. The remaining twenty-eight could be attached to a workflow, and the claims workflow alone carried eleven of them, aimed at four different steps.

Workflow discovery turns the list around. For each workflow with meaningful volume, cost, delay, error, or customer complaints, the team writes down the trigger, the steps, the handoffs, the decisions and who makes them, and the exception paths, in the form of a process map checked against real cases rather than the procedure manual. Queue data, a sample of closed files, rework codes, and conversations with the people who do the work each reveal something the others miss, and the documented process is often not the practiced one. The map also records what kind of task each step contains: finding information, extracting fields, classifying, summarizing, drafting, predicting, deciding, or acting. The label matters later, because the acceptable error rate and the authority required differ sharply between summarizing a note and approving a payment.

Every step needs two clocks, hands-on time and elapsed time. The insurer measured both for a simple claim and added the number of files per day that each stage can complete with its current staff and tools.

Stage	Hands-on hours	Elapsed days	Capacity (files per day)
First notice and intake	0.3	0.5	400
Coverage check	0.5	1.0	300
Document chase	0.4	9.0	180
Adjuster review	1.5	2.0	260
Payment	0.2	1.0	500
Total	2.9	13.5

Adjuster review holds 1.5 of the 2.9 hands-on hours (52 percent), which is why the note summarizer looked valuable: it targets the largest block of work. But 9 of the 13.5 elapsed days (67 percent) are the document chase, where a file waits for repair estimates and missing forms. Hands-on time is where staff cost sits, and elapsed time is what the customer experiences. A proposal has to say which of the two it improves, and by how much.

Value follows the constraint

Files arrive at 280 per day. A workflow cannot finish more files per day than its slowest stage can handle, so today it completes 180 and the open backlog grows by 100 files every day. The summarizer is estimated to raise adjuster capacity from 260 to 340 files per day. Throughput stays at 180, because the adjusters were never the limit; they would simply wait faster. A change with no AI in it, an estimate-request portal and an intake checklist that asks for the missing items on the first day, is estimated to lift the document chase to 300 files per day. Adjuster review at 260 then becomes the limit, and throughput rises to 260, an increase of 80 files per day, with the backlog growing by 20 a day. Only with both changes does throughput reach 280 and the backlog stop growing. The summarizer’s contribution is the last 20 files per day, and it exists only because the portal came first.

An AI initiative’s value therefore cannot be read from its own step. Take hypothetical first-year costs of \$250,000 for the portal and \$600,000 for the summarizer. The portal adds 80 files per day of capacity at \$3,125 per file (\$250,000 divided by 80); the summarizer adds 20 at \$30,000 per file (\$600,000 divided by 20). Those last 20 files also have a staffing alternative. Twenty files at 1.5 hours each is 30 hours of adjuster work a day, or four adjusters at 7.5 productive hours each; at a hypothetical loaded cost of \$90,000, that is \$360,000 a year. On first-year costs, hiring is cheaper. The summarizer’s case then rests on benefits outside throughput, on lower costs after the first year, and on arrivals growing past 280, and each of those has to be measured instead of assumed.

The DORA 2025 report on AI-assisted software development describes AI as an amplifier of an organization’s existing strengths and weaknesses. The finding concerns software teams, but the insurer’s workflow shows the same mechanism: its weakness was the wait for documents, and a faster adjuster step would have amplified nothing.

Judge value, feasibility, and risk separately

Value is a hypothesis that traces a capability to a measured result through the constraint: what changes in the work, for how many cases, at what error rate, and what the organization does with the time released. State it as a range, ramp it for adoption, and compare it with the baseline, as in Live for a Year, Used by a Third. Each effect must be counted once across the whole portfolio, which matters most when an AI tool and a process change claim the same shorter wait.

Feasibility asks whether the change can be carried out under real constraints: whether the needed data exists in usable form, whether using it for this purpose is permitted, whether the output can be placed in the system where the work happens, whether the team can evaluate quality on representative cases, and who runs the result afterward. The last cost is the easiest to underestimate. In “Hidden Technical Debt in Machine Learning Systems,” Sculley and colleagues argue that quick wins in machine learning are not free: deployed systems accumulate ongoing maintenance through data dependencies and glue code, and a change to one input can change the behavior of the whole system. That cost belongs in the initiative’s budget, with an owner assigned to it. Readiness applies to a defined next step, so a tool that is ready for a staff-only pilot on one product line is not thereby ready for every line.

Risk comes from what an error costs and what the system is allowed to do about it, and classification comes first (see risk class and regulation). The EU AI Act’s Annex III lists AI used for risk assessment and pricing in life and health insurance among its high-risk uses, so a candidate that scores applicants carries obligations that a note summarizer does not. For systems that call tools, OWASP’s guidance on excessive agency traces damaging actions to unnecessary functions, overly broad permissions, and too little human approval of high-impact actions. A design that drafts a payment recommendation for an adjuster and one that issues the payment can share a model and still sit in different risk classes. Human review lowers risk only if the reviewer has evidence, time, and authority to disagree, and reviewers tend to accept confident output as volume rises, so the cost of adequate review belongs in the initiative as well. The UK AI Playbook advises against fully automated decisions in high-stakes settings, and the insurer’s four proposals that paid or denied claims automatically were redesigned as assistive tools before anyone scored them.

Averaging the three ratings hides these differences. A proposal rated 5 for value, 5 for feasibility, and 1 for risk averages 3.67 and outranks a proposal rated 3, 3, and 4, which averages 3.33, even though the 1 may mean the design cannot be approved at all. Treat an unacceptable risk or a missing permission as one of the gates that removes a design from the list, and compare the survivors on value and cost with the three assessments kept apart.

Dependencies change the arithmetic

Three surviving proposals needed the same capability: a document-intake service that receives files, reads them, and applies access controls. Proposal A checks claim submissions for completeness the day they arrive. Proposal B answers service agents’ questions from policy documents. Proposal C extracts fields from underwriting submissions. Using hypothetical three-year figures in thousands of dollars, the values are 500, 400, and 300, their own costs are 220, 180, and 150, and the shared service costs 400. If each proposal is charged the full cost of the service, each loses. Only the combinations show the picture.

Funded together	Value	Own costs plus the service	Net
A alone	500	220 + 400 = 620	−120
B alone	400	180 + 400 = 580	−180
C alone	300	150 + 400 = 550	−250
A and B	900	220 + 180 + 400 = 800	+100
A and C	800	220 + 150 + 400 = 770	+30
B and C	700	180 + 150 + 400 = 730	−30
A, B, and C	1,200	220 + 180 + 150 + 400 = 950	+250

The service is a decision of its own. It is justified when the value of its consumers, after their own costs, exceeds its cost. A contributes 280 (500 − 220), B 220, and C 150, a total of 650 against 400, leaving 250. A and B together clear the service by 100, which is 11 percent of their combined value of 900, so an overstatement of 100 in either estimate erases the margin. Once the service exists, C is judged on its own numbers alone, 300 against 150. The logic also runs in reverse: if A and B are later stopped, C cannot carry the service, and a stop decision has to ask what depended on the thing being stopped. Sculley and colleagues call the hidden version of this problem undeclared consumers, systems that start using a model’s output without the owner knowing; recording who consumes each shared component is how dependencies stay visible.

Dependencies of value need the same care. Proposal A’s estimate assumes the portal exists. Its completeness check shortens the chase only for packets the portal does not already complete, and if both the portal and A claimed the full reduction in waiting, the same benefit would be paid for twice.

Put the non-AI alternative on the list

Every candidate carries its best alternative without AI, estimated with the same care. Google’s Rules of Machine Learning begin with “Don’t be afraid to launch a product without machine learning.” NIST’s AI Risk Management Framework Playbook asks organizations to identify non-AI or non-technology alternatives that could lead to more trustworthy outcomes, and the UK AI Playbook asks teams to stay open to the conclusion that AI is not the best tool. The alternatives form a short ladder, ordered by how much they change. Remove the step, or the rule that created it. Fix the cause upstream, as the portal does by asking for the missing item on day one. Configure software the company already owns. Write explicit rules for a pattern that is stable. Add or move capacity. Only then build a model. A rule also gives the model something to beat, and the comparison belongs in the decision the model is meant to serve.

Of the twenty-eight workflow-linked proposals, nine were resolved mainly by a form change, a routing rule, or a report from existing systems, and five were blocked because the data was not in usable form or its use for that purpose was not permitted. Fourteen went on to estimation. A non-AI result is not a defeat for the AI program. The portal’s labeled uploads give a later completeness check clean inputs, and a blocked proposal comes back with a named condition for reopening it.

Sequence by what unblocks what

Of the fourteen estimated candidates, four entered the portfolio: A, B, C, and the summarizer. The other ten were deferred, each with the evidence that would reopen it. The order follows the constraint and the dependencies. The portal comes first, because it is cheap, it acts on the binding stage, and its cost of delay is concrete: every month of delay forgoes about 1,600 completed files (80 files per day over 20 working days). The intake service and Proposal A follow at limited scope, with exit criteria written before the pilot starts: the sample on which reviewers must agree with the completeness check, the share of packets requested complete on the first day, and the review date. Proposal B starts when the service is operating, and C waits for usage evidence from the first two. The summarizer waits for a trigger, either arrivals above 280 files per day or measured benefits outside throughput that change the comparison with hiring.

Sequencing also respects people. Initiatives that look independent on a roadmap compete for the same engineers, data stewards, security reviewers, and business experts, so each stage is scheduled against named capacity rather than the department’s total headcount.

Stop what the evidence no longer supports

The five pilots from the previous year used 0.8, 0.5, 0.5, 0.4, and 0.3 of an engineer, a total of 2.5 of six, or 42 percent of the team. Each was reviewed against its original value path and its exit criteria. Two were stopped: one improved a step that was not the constraint, and the other needed data the compliance group had not permitted. One was paused until the intake service exists. Stopping the 0.5 and 0.4 allocations and pausing the 0.3 released 1.2 engineers, leaving 1.3 on the two continuing pilots, and the released capacity went to the intake service and Proposal A. The release is real only after handover, retained records, and the operating owner for anything already in use are settled.

Several signals mark a weak initiative: a value path that does not pass through the constraint, dependence on adoption by a team that did not ask for the tool, a permission or dataset that has not been obtained, an error cost larger than the benefit at realistic accuracy, no owner after launch, or a benefit already claimed by another initiative. The money already spent is sunk cost and is not a reason to continue; the comparison is the future cost and benefit of what remains against the alternatives, including the capacity the work would otherwise release. Record whether the decision is to stop the attempt, pause it, or abandon the underlying need, because those are different decisions, and write down the condition that would reopen it.

In the insurer’s revised plan, the portal starts first, followed by the intake service with Proposal A, then B, with C and the summarizer held against stated triggers and the ten deferred candidates on a reopening list. The plan is reviewed each quarter by rerunning the same numbers: backlog growth against the current 100 files per day, the dependency margins, the capacity committed by name, and the stop and reopen conditions. Whether the plan delivers is a matter for those measurements and not for the ranking that preceded it.

References:

• Google, Rules of Machine Learning: Best Practices for ML Engineering
• D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015
• UK Government Digital Service, Artificial Intelligence Playbook for the UK Government (Open Government Licence v3.0)
• NIST, AI Risk Management Framework Playbook: Map function
• EU Artificial Intelligence Act, Annex III: High-risk AI systems
• OWASP GenAI Security Project, LLM06:2025 Excessive Agency
• DORA, State of AI-assisted Software Development 2025