Leading AI-Assisted Engineering Adoption

A fictional regional insurer turned on an AI coding assistant for its claims-platform engineers ten weeks ago. The adoption dashboard looks excellent: almost everyone has used the tool, and the average engineer now opens 30% more pull requests each week. The vice president of engineering is asked to approve expansion to every team next quarter. Then a team lead mentions that her reviewers are working through the evening, and that changes sit in the queue for days. Nobody can say whether the rollout is working. The dashboard measures how much the tool is used and how much code is written, and the leader’s decision depends on what happens after the code is written.

The company and its numbers in this case are invented for illustration; the research cited along the way is public. The decision touches five things an AI coding tool changes: how changes are written, tested, and reviewed, how the tool is introduced, how its output is checked, what happens to engineers’ skills, and what the delivery results show.

What the tool makes cheap and what it leaves expensive

A coding assistant lowers the cost of producing a draft of code, a test, or an explanation. It does not lower the cost of deciding whether that draft is correct, secure, maintainable, and the change that was wanted. Someone still has to read it, run it, and take responsibility for it after it ships. A change that was cheap to write is not cheap to verify, and its author may understand it less well than code they typed themselves.

Public evidence on net effect is mixed, and the disagreement follows the conditions of each study. In field experiments at Microsoft, Accenture, and a Fortune 100 electronics manufacturer, developers given a coding assistant completed about 26% more tasks per week on average. Newer and junior developers gained the most, and the researchers had no data on code quality. In a 2025 randomized trial by METR, 16 experienced open-source developers working on large repositories they knew well took 19% longer on issues when AI tools were allowed, even though they had predicted a 24% speedup and afterward still believed they had been about 20% faster. The authors say the result does not extend to developers in general. DORA’s 2024 survey-based research associated higher AI adoption with higher individual productivity, flow, and job satisfaction, and also with lower delivery stability and throughput. For a 25% increase in adoption, it estimated a 7.2% drop in stability and a 1.5% drop in throughput, and it pointed to the basics of small batches and robust testing as what still governs delivery.

These findings do not contradict each other. One measured task completions for mostly less experienced developers, one measured elapsed time on difficult work in familiar code, and one related adoption to organization-level delivery. A leader reading them should conclude that the result depends on who uses the tool, on what work, and on what the surrounding system does with the extra output. DORA’s 2025 research puts the same idea in one sentence: “AI’s primary role is as an amplifier, magnifying an organization’s existing strengths and weaknesses.” The rest of this post is about the organizational conditions that decide which way the amplification goes. The METR result also carries a warning for measurement: the developers’ sense of their own speed was wrong in a measurable direction, so a survey that says people feel faster is not evidence that delivery got faster.

Where the extra output goes: a review-capacity example

Take one hypothetical team of six engineers at the insurer. Before the rollout, each engineer opens four pull requests a week, so the team produces 24. A reviewer spends about one hour on an average pull request, counting the reading, the comments, and the second look after changes. Engineers set aside roughly six hours each per week for reviewing other people’s work, which gives the team 36 hours of review capacity. The team therefore uses 24 of 36 hours, or 66.7% of its review capacity.

After the rollout, pull requests rise by 30%, from 24 to 31.2 per week. If review time per pull request stays at one hour, demand is 31.2 hours against the same 36, which is 86.7% utilization. That sounds comfortable, but waiting time does not grow in proportion to utilization. In the simplest queueing model, with random arrivals and a single pool of service, the average wait before service begins is utilization divided by one minus utilization, multiplied by the average service time. At 66.7% that factor is 2.0, so a pull request waits about two hours on average before someone starts reviewing it. At 86.7% the factor is 6.5, so the wait is about six and a half hours. A 30% increase in volume produced a 3.25 times longer wait. Real review is not a single queue, since reviewers work in bursts and some changes need a particular person, so the figures are illustrative, but the shape is general: the closer a queue runs to capacity, the faster delay grows.

Now make the more realistic assumption that generated changes are somewhat larger and less familiar to reviewers, so review time per pull request rises by 20%, to 1.2 hours. Demand becomes 31.2 times 1.2, which is 37.44 hours against 36 available. Utilization is 104%. The queue no longer settles at a long wait; it grows by about 1.4 hours every week, and the backlog compounds until someone changes the arrangement. To bring utilization back to 66.7% with these larger reviews, the team would need 56.2 hours of review capacity, about 9.4 hours per engineer instead of six, a 56% increase in time spent on reviewing.

The same numbers show why a lead-time chart can mislead in both directions. Suppose authoring a change took six hours before and four hours after. DORA’s change lead time begins when the change is committed, so the two hours saved in authoring are invisible to it. What it does see is the wait and review that follow the commit: three hours before (two waiting, one reviewing) and 7.5 hours after (6.5 waiting, one reviewing). The engineers write faster, the team’s measured lead time more than doubles, and neither observation is wrong. A leader who sees only authoring effort will approve the expansion, and one who sees only lead time will conclude the tool is harmful. Both need the queue.

The response is not simply to add reviewers, which only works if the organization has senior engineers to spare. There are three levers. Keep changes small, because review effort and risk both rise with size, which is the batch-size point in DORA’s findings. Move checking that a machine can do out of the reviewer’s hour, so human attention goes to design and intent. And decide deliberately how much engineering time review should receive, rather than letting it be whatever remains after authoring speeds up.

Checking the output: tests, review, and the limits of approval

The first principle for verifying generated code is that the evidence of correctness should not come from the same source as the code. If the assistant writes a function and then writes the tests for it, both may encode the same misunderstanding, and a green build proves little. A test oracle is the independent basis for deciding what correct behavior is: a calculated expected result, a trusted reference, or a property taken from the specification. Leaders do not write oracles, but they decide whether the teams have time and standards for them. The engineering article on testing and reviewing AI-generated code in CI shows, with running code, how a plausible generated function passes its example tests while failing on inputs nobody chose, and how independent property checks, dependency review, and security scans catch what a skim of the diff does not.

Human review remains necessary for the questions a machine cannot settle: whether this is the change that was asked for, whether it fits the design, whether the tradeoff it makes is acceptable. But review quality is fragile. Automation bias, excessive reliance on automated output, grows with volume, time pressure, and a good track record, which are exactly the conditions an assistant creates: more changes, a deadline for each, and mostly correct output. DORA’s 2024 survey found that 39% of respondents trusted AI-generated code little or not at all, yet an approval button records the same click from a careful reviewer and a tired one. Human oversight is only as good as what the reviewer can see, understand, and change, and how much time the reviewer has. A leader can test this rather than assume it: seed a few known defects into real review queues and see how many are caught at the current workload.

A practical policy sorts changes by what could go wrong. Documentation, test scaffolding, and internal tooling can take lighter review with automated checks as the backstop. Changes to authentication, payments, claims calculations, data migrations, and anything touching regulated data need a named human who understands the code well enough to defend it, regardless of who or what drafted it. The rule that the author remains accountable for what they submit, whether or not they typed it, is simple to state and is the one that most changes behavior. Who may approve which class of change is a question of delegated authority, and it should be written down before the volume of changes makes the answer implicit.

Introducing the tool in stages

A controlled rollout does two jobs. It limits the damage from what the organization has not yet learned, and it produces evidence that can be trusted. The usual approach, letting enthusiasts adopt first, serves neither well: volunteers are often the teams that were already fastest, so their results describe a different population than the one the decision covers, and comparing a team before and after credits the tool with everything else that changed that quarter. Assigning teams to start dates, even in a staggered order the teams all eventually receive, keeps a comparison in place while access spreads. The engineering article on measuring an AI development tool rollout covers unit of assignment, power, and gates in depth.

The scope of what the tool may do should expand separately from how many people use it, because those are different risks. A completion suggestion that a human accepts line by line is one thing. An agent that edits many files, runs commands, and opens pull requests on its own acts with the permissions it was given, and the reasonable boundary is narrow, short-lived credentials, a sandbox, and approval for consequential actions. The article on operating coding agents with restricted permissions describes those controls, and the article on integrating AI coding tools with enterprise repositories covers what the tool can read, where code and prompts go, and how sessions are audited. Those are conditions to satisfy before the first stage below, not tasks to schedule after it.

Stage	Who and what	Evidence needed to advance
Foundations	Platform and security teams only; approved repositories and data boundaries; audit on	Access controls tested; audit records attributable to a person; a documented way to switch the tool off
Assigned pilot	A few teams chosen for variety, not enthusiasm, using suggestions and test drafting; start dates staggered; agents excluded	Review queue time and rework no worse than the agreed margin; guardrail measures stable; usage holding after the novelty fades
Wider use with agents	More teams; agent-driven changes only in low-risk repositories, with restricted permissions and human approval	Defect escapes and incidents from agent changes within the margin; reviewers’ load acceptable
General availability	All teams, with high-risk areas still under the stricter review rule	Ongoing monitoring; the ability to narrow or pause access by team through a feature flag

The table is a sequence with a different gate at each step. The gates should be written before anyone sees results, and each should say what happens if it is not met: fix, hold, or stop. Stopping needs to be a real option. If the tool can be switched off for a team within a day and the engineers can still work without it, the cost of a wrong expansion is bounded. If the organization has quietly reorganized around the tool’s output, it is not.

Skills: what the tool teaches and what it replaces

Using an assistant well is itself a skill, and so is the skill it can erode. The review problem above is partly a skills problem. Reviewing code one did not write, judging whether a generated test is meaningful, and tracing a failure through unfamiliar logic are the abilities that determine whether verification works, and engineers build them mostly by writing and debugging code themselves.

Anthropic ran a randomized trial with 52 junior software engineers who learned an unfamiliar Python library while completing two features. The group using an AI assistant scored 50% on a follow-up quiz and the hand-coding group scored 67%, a 17-point gap, with the largest gap on debugging questions. The result applies to one library, a short session, and an immediate quiz. Within the AI group, though, people who asked for explanations, or who generated code and then questioned it, kept more of what they learned than people who delegated the work and accepted the result. How the tool is used, not just whether it is used, appears to determine what the user learns.

The field experiments add a second point. Junior and newer developers adopted the tool more and gained more measured output, while experienced developers gained less. For a leader, that pattern means the people most helped in the short run are those still forming the judgment that review and debugging require, and the people whose judgment the organization depends on to verify output are those who gain least from the tool. If juniors produce more and seniors spend more time reviewing, the review-capacity example turns into a talent problem: the most experienced engineers become a verification bottleneck, and the apprenticeship that creates the next group of them thins out.

Several practices follow, none of which requires restricting the tool. Teach reviewing as a skill, with examples of generated defects and time allotted for practice. Keep some junior work as write-it-first tasks, where the engineer attempts the problem before consulting the assistant and then compares. Ask engineers to explain generated code in review, not only to submit it. Rotate review duties so the load does not settle on two or three seniors. Include debugging without the tool in onboarding so the skill is exercised. DORA’s 2024 recommendations point the same way: establish clear guidelines, and give people dedicated time to explore the tools and build trust through hands-on experience. Learning time that is promised but not scheduled does not exist.

Quality and delivery outcomes: deciding what counts as success

The decision to expand needs one primary metric that states what the organization is trying to improve and a small set of guardrails that must stay acceptable while it does. For this rollout the primary metric might be the time from starting a change to having it running in production, which includes authoring and review wait, since lead time alone misses the upstream savings. The guardrails would come from the instability side of the DORA measures: the share of deployments needing immediate intervention such as a rollback or hotfix, the share of unplanned deployments caused by production incidents, and the time to recover from a failed deployment. Review queue time and reviewer hours belong beside them, because they show where cost moved. Activity counts such as pull requests opened or suggestions accepted describe use. A high acceptance rate is consistent with helpful suggestions and with careless approval, so it cannot be the success measure.

Two cautions apply to how the numbers are used. DORA advises applying its measures to a service or application and improving against one’s own history, not ranking teams, and not turning a metric into a target, since a target invites optimizing the number. A rollout creates pressure to show a number, and teams measured on pull requests per week will produce more pull requests. The second caution concerns what a pilot can show. A small pilot that finds no increase in failures has not shown safety; for guardrails the requirement is that the evidence rules out harm larger than the agreed margin, which usually takes more data than detecting a benefit.

The outcome that matters is also bounded by what the tool does not touch. If the delivery bottleneck is a weekly release window, a cross-team approval, or a slow test environment, faster code writing moves work to the front of a queue that was already full. The amplifier finding applies directly: teams with small batches, strong automated tests, and fast feedback turn extra code into extra delivery, and teams without them turn it into a larger backlog of unreviewed and untested change. Investing in those foundations before or alongside the tool usually pays more than a larger tool license.

What the insurer’s vice president decides

Back at the fictional insurer, the vice president does not approve company-wide expansion. She approves the next stage for a defined set of teams, chosen for variety, and she attaches conditions. Each team receives a limit on pull request size and a review-capacity plan before it starts, with reviewers’ hours protected rather than assumed. Machine checks for tests, dependencies, and security patterns run before a human is asked to look. Changes to claims calculations and payment logic stay under a named-approver rule. The primary metric is time from starting a change to production, the guardrails are the three instability measures plus review queue time, and the gates, with the margins of harm the business will tolerate, are written down before the staggered start dates begin. Engineers get scheduled time to learn the tool and to practice reviewing generated code, and junior engineers keep a share of write-first work.

The decision differs from the one she was asked to make. The request was whether the tool works. The question she answers is whether the organization can absorb what the tool produces, and what it will do if it cannot: fix the queue, narrow the scope, or pause. That question is answerable with a week of review-queue data and a pencil, and it is the one on which the outcome of the rollout depends.

References

• Google Cloud, Announcing the 2024 DORA report.
• DORA, State of AI-assisted Software Development 2025.
• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.
• MIT Sloan, How generative AI affects highly skilled workers, summarizing the field experiments by Cui, Demirer, Jaffe, Musolff, Peng, and Salz.
• Anthropic, How AI assistance impacts the formation of coding skills.