Measuring an AI Development Tool Rollout

Consider a hypothetical dashboard, three months after an organization turned on an AI coding assistant. Teams that use the tool show a median lead time for changes that is far lower than teams that do not. The usage chart rises every week. A survey says developers feel faster. A leader asks the obvious question: should the rollout expand to every team?

Each number on that dashboard can be accurate and the conclusion can still be wrong. If the teams that adopted early were already the fastest, the gap predates the tool. If everything improved over the quarter for unrelated reasons, a before-and-after chart credits the tool for the quarter. If review queues grew while authoring got quicker, lead time may have moved without total effort falling. And a feeling of speed is a measurement of feeling. This article is about designing the measurement so that the decision to expand, hold, fix, or stop rests on evidence that can support it.

It builds on Operating Coding Agents with Restricted Permissions. That article describes how to run agents with scoped identities, tool permissions, approval, audit records, and run limits. Here we use two of its products. The audit records and agent identities supply trustworthy usage and attribution data, which most adoption measures depend on. And the access tiers define what a pilot cohort may do, which determines which outcomes can change at all. The statistics here assume general programming knowledge and an acquaintance with means, variance, and hypothesis tests. The simulations in the article are small Python programs written for it, using synthetic data with stated assumptions (Python 3.12, numpy 2.2.3, scipy 1.18.1, fixed random seeds). Each output shown comes from running that code. None of the synthetic numbers is a measurement of any real organization or tool.

Start from the decision, not the dashboard

A rollout measurement exists to support a decision, so the first step is to write the decision question and what would change it. A usable question has the form: should this tool move from the current cohort to the next tier of teams, given what we observe on a stated outcome and on stated limits of harm? Writing it down forces three choices. A primary metric names the improvement the decision chiefly seeks. Guardrail metrics name outcomes that must stay acceptable while pursuing it. And a baseline defines what the primary metric is compared against. Define each metric’s population, calculation, observation window, and limit before looking at results, and do not replace the baseline after seeing them to make the change look better.

Why a local measurement is needed at all

Published evidence about AI coding tools exists, but it does not settle the local question, and it points in different directions depending on what was measured, in whom, and with which tool. The table summarizes four sources read while preparing this article, with what each measured and the conditions that limit generalization.

Source	Design and measure	Reported finding	Why it does not transfer automatically
Peng et al., 2023 (controlled experiment; abstract read)	Developers with and without an AI pair programmer (GitHub Copilot) implemented an HTTP server in JavaScript	The group with the assistant completed the task 55.8% faster	One self-contained task chosen for the study; the abstract also reports effects that differed across developers
Cui et al., 2025 (three field experiments)	Randomized access to a coding assistant for 4,867 developers at three companies; weekly counts of completed tasks (pull requests), commits, and builds	A 26.08% increase in completed tasks (standard error 10.3%) for developers using the tool; less experienced developers adopted more and gained more	Output counts, not quality or stability; noisy; take-up was incomplete; the estimate applies to users, derived with an instrumental-variable method
METR, 2025 (randomized trial)	16 experienced open-source developers; 246 real issues from their own repositories randomly assigned to allow or disallow AI tools; self-reported implementation time	Tasks took 19% longer with AI allowed, although developers had forecast a 24% speedup and, afterward, still believed AI had sped them up by about 20%	Authors state the result applies to experienced developers on their own large repositories and does not show AI fails to help in general
DORA, 2024 report (survey-based research)	Survey-based research relating AI adoption to individual outcomes and to delivery performance	AI adoption is associated with higher individual productivity, flow, and job satisfaction, and also with lower delivery stability and throughput	Associational research; the report itself stresses that fundamentals such as small batches and testing still govern delivery

Four sources with four designs produce a slowdown, a large speedup, a moderate speedup, and a mixed picture. The disagreement is informative, not embarrassing. The outcome (task time, task counts, delivery metrics), the population (experienced contributors on mature repositories, novices on a new task, enterprise developers), the tool generation, and the way effects were estimated all differ. The METR result also shows something specific to this domain: developers’ impressions of their own speed diverged from measured time, which is a direct warning against using perceived speedup as the primary metric. It does not mean perception is useless, a point the survey dimension below returns to. Because the answer depends on conditions, an organization needs evidence from its own repositories, at its own review standards, with its own mix of tasks.

Pilot cohorts

A pilot cohort is the set of developers or teams given the tool first. How the cohort is chosen and what it is compared with determine what a result can mean, more than any analysis done afterward.

Choose the unit of assignment

The unit is whatever you assign to receive or not receive the tool: an individual, a team, a repository, or an issue. Individual assignment gives the most statistical power, but it works only if one developer’s use does not affect a colleague’s outcomes. In software that assumption often fails. Developers on a team share a codebase, a review queue, and conventions, so a teammate’s faster output changes the reviewer’s workload. When such interference is plausible, assign at the team level. The Microsoft experiment analyzed by Cui and colleagues did both, randomizing 616 developers individually and 1,130 at the team level with an average team size of 6.2, and it accounted for the structure when computing standard errors. METR went the other way and randomized individual issues within each developer, which controls for who the developer is but relies on the allow/disallow condition not changing how the same developer handles other work.

Two further design features are worth borrowing. Randomizing the start date of access, as in the third experiment of that study where invitations went out weekly in random order, gives every unit access eventually, which makes the design easier to accept organizationally while preserving a comparison during the staggered period. And the study documents the cost of leaving a control group open: in the Microsoft experiment, control developers sought access as awareness grew, and the experiment ended at that point. Plan the comparison window with the expectation that the control group’s patience is limited.

Why volunteers mislead, and why before-and-after is not enough

The easiest cohort to form is volunteers: teams that ask for the tool. They are also the least informative comparison. Selection bias arises when the way units enter a group is related to the outcome, so the group describes a different population than the one the question is about. Teams that volunteer may be better resourced, more experimental, or more attentive to process, any of which can improve delivery metrics with or without the tool. A second tempting design, comparing the cohort’s metrics before and after rollout, attributes everything that changed over the period to the tool: a new build system, a seasonal slowdown, a staffing change.

A simulation makes the size of these errors concrete. The assumptions are stated in the code and are illustrative. The true effect of the tool is set to zero, so any measured effect is error. Teams have a baseline lead time, and faster teams are more likely to volunteer. Separately, all teams improve by 10% over the period because of an organization-wide change. We then compute four estimates from the same synthetic world, averaged over 2,000 repetitions.

import numpy as np
from scipy import stats

rng = np.random.default_rng(7)

def make_world(n, true_effect, trend=0.0, sd_team=0.5, sd_noise=0.15):
    """All numbers are assumptions for illustration, not measurements.
    Outcome: log of median lead time (hours) for a team, before and after the pilot period."""
    base = rng.normal(np.log(40), sd_team, n)            # team-level baseline (log hours)
    pre  = base + rng.normal(0, sd_noise, n)
    return base, pre, trend, true_effect, sd_noise

def post_outcome(base, treated, trend, true_effect, sd_noise):
    return base + trend + true_effect * treated + rng.normal(0, sd_noise, len(base))

def pct(x): return 100 * (np.exp(x) - 1)

# --- Scenario A: true effect is ZERO; faster teams volunteer more; org-wide trend of -10%
n = 400
reps = 2000
naive_adopter, pre_post_only, did_vol, did_rand = [], [], [], []
for _ in range(reps):
    base, pre, trend, te, sd = make_world(n, true_effect=0.0, trend=np.log(0.90))
    # volunteering depends on baseline speed: lower baseline lead time -> more likely to volunteer
    p_vol = 1 / (1 + np.exp(3.0 * (base - np.log(40))))
    vol = rng.random(n) < p_vol
    post = post_outcome(base, vol, trend, te, sd)
    naive_adopter.append(pct(post[vol].mean() - post[~vol].mean()))
    pre_post_only.append(pct((post[vol] - pre[vol]).mean()))
    did_vol.append(pct((post[vol] - pre[vol]).mean() - (post[~vol] - pre[~vol]).mean()))
    rnd = rng.random(n) < 0.5
    post_r = post_outcome(base, rnd, trend, te, sd)
    did_rand.append(pct((post_r[rnd] - pre[rnd]).mean() - (post_r[~rnd] - pre[~rnd]).mean()))
print("Scenario A: true effect = 0%, org-wide trend = -10%, volunteers are faster teams")
print(f"  adopters vs non-adopters, post only      : mean {np.mean(naive_adopter):7.1f}%")
print(f"  adopters, before vs after (no control)   : mean {np.mean(pre_post_only):7.1f}%")
print(f"  before/after difference, volunteer groups: mean {np.mean(did_vol):7.1f}%")
print(f"  before/after difference, random groups   : mean {np.mean(did_rand):7.1f}%")

Scenario A: true effect = 0%, org-wide trend = -10%, volunteers are faster teams
  adopters vs non-adopters, post only      : mean   -41.0%
  adopters, before vs after (no control)   : mean   -10.0%
  before/after difference, volunteer groups: mean     0.1%
  before/after difference, random groups   : mean     0.1%

Read the four lines as four designs. Comparing adopters with non-adopters after rollout reports a 41% reduction when the true effect is nothing, because volunteers were faster before they began. Watching the adopters before and after reports 10%, which is exactly the organization-wide trend. Taking the before-and-after change in the volunteer group and subtracting the same change in the non-volunteers, a difference-in-differences estimate, returns approximately zero, and so does the same calculation on randomly assigned groups. The difference-in-differences design removes any permanent difference between the groups and any trend that affects both. Its assumption is that, absent the tool, the two groups would have moved in parallel.

That assumption can fail, and the next simulation shows how. Here each team has its own improvement trend, and teams that were already improving are more likely to volunteer. The true effect is again zero.

import numpy as np
rng = np.random.default_rng(11)
def pct(x): return 100 * (np.exp(x) - 1)

# Scenario C: true effect = 0%. Each team already has its own improvement trend
# (log-hours per period, sd = 0.10). Teams that are improving are more likely to volunteer.
n, reps = 400, 2000
did_vol, did_rand = [], []
for _ in range(reps):
    base = rng.normal(np.log(40), 0.5, n)
    own_trend = rng.normal(0, 0.10, n)                 # team-specific drift, assumed
    pre  = base + rng.normal(0, 0.15, n)
    p_vol = 1 / (1 + np.exp(20 * own_trend))           # improving (negative trend) -> volunteers
    vol = rng.random(n) < p_vol
    post = base + own_trend + rng.normal(0, 0.15, n)   # no tool effect
    did_vol.append(pct((post[vol]-pre[vol]).mean() - (post[~vol]-pre[~vol]).mean()))
    r = rng.random(n) < 0.5
    did_rand.append(pct((post[r]-pre[r]).mean() - (post[~r]-pre[~r]).mean()))
print("Scenario C: true effect = 0%, volunteers are teams that were already improving")
print(f"  before/after difference, volunteer groups: mean {np.mean(did_vol):6.1f}%")
print(f"  before/after difference, random groups   : mean {np.mean(did_rand):6.1f}%")

Scenario C: true effect = 0%, volunteers are teams that were already improving
  before/after difference, volunteer groups: mean  -11.4%
  before/after difference, random groups   : mean   -0.0%

With volunteer groups, the difference-in-differences estimate is a 11.4% reduction in lead time that does not exist. With random assignment it is zero. Randomization protects against selection on trends that nobody knew about, which is why it is the strongest design when feasible. When it is not feasible, check parallel pre-trends by comparing the groups’ metrics over several periods before the rollout, and state plainly that a design which cannot be randomized supports a weaker claim. Because the simulation’s parameters were chosen to illustrate the mechanism, the magnitudes are not predictions. The qualitative lesson is that the size of the bias depends on how strongly selection relates to the outcome, and you rarely know that in advance.

Cohort design	What it can support	Main threat	Use when
Volunteers vs. everyone else, after rollout	A description of who adopted and how they fare	Selection: the groups differ before the tool	Never as evidence of effect; fine for learning what adopters need
Cohort before vs. after	That something changed in the period	Everything else that changed in the period	As a sanity check beside a comparison group
Volunteers with a comparison group, difference-in-differences	An effect estimate if pre-trends were parallel	Selection on trend; unmeasured events hitting one group	Randomization is impossible; report the pre-trend check
Randomized teams (or developers)	The effect of being offered the tool	Interference, contamination, low power	A decision justifies the cost; the default for gating expansion
Randomized staggered start	The same, with eventual access for everyone	Short comparison window; time effects must be modeled	The organization will not accept a permanent control group

Staged exposure mechanisms from delivery engineering help here, with one caution. A feature flag can switch the tool on for assigned units, and the canary deployment entry notes that a canary is a release-safety technique and not automatically a randomized experiment. Choosing who gets the flag is what makes it a design. The article on progressive delivery addresses staged exposure of service changes. Here the units are teams and the aim is evidence about people and process.

Quality and delivery measures

A rollout can raise one measure while damaging another, so the measurement set must pair a speed or throughput measure with measures of quality and stability. Two frameworks give a vocabulary that has been argued over in public, which is reason to prefer them to inventing one.

Delivery metrics

The DORA metrics as currently defined by the DORA program are five. Change lead time is the time from a change being committed to version control to its deployment in production. Deployment frequency is the number of deployments over a period. Failed deployment recovery time is the time to recover from a deployment that fails and requires immediate intervention. Change fail rate is the ratio of deployments that require immediate intervention, typically a rollback or hotfix. Deployment rework rate is the ratio of deployments that are unplanned and result from a production incident. The first three describe throughput and the last two describe instability. The DORA guide gives cautions that apply directly to a rollout: apply the metrics at the application or service level, because comparing very different systems is misleading; improve against your own history instead of ranking teams; and do not turn a metric into a target, since a target invites optimizing the number. Its Goodhart’s-law warning matters here because a rollout creates pressure to show a number.

Two details change what the numbers mean for an AI rollout. Change lead time starts at commit, so time spent writing the change is outside it. A tool that speeds up writing can leave lead time unchanged while it changes everything upstream of the commit, and a lead-time chart alone cannot see that benefit. Conversely, a tool that produces more changes per developer can raise deployment frequency without any gain in value, because frequency counts deployments and not their usefulness. These are reasons to read the throughput measures together with an upstream measure of effort and with the instability measures.

A multi-dimensional set

The SPACE framework (Forsgren and colleagues, 2021) organizes developer productivity into five dimensions: satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. Its guidance on use is specific. Capture several metrics across multiple dimensions, at least three, and include at least one perceptual measure such as survey data. The authors note that adding more activity metrics, such as counts of pull requests or coding time, rounds out only the activity dimension, and that a fuller picture needs at least one metric from two other dimensions. They also note that metrics from different dimensions are often in tension by design, so that a balanced view supports better trade-offs. The table maps the dimensions to candidate measures for an AI rollout. These are examples to adapt, not a fixed list.

Dimension	Example measure	Role in the rollout	Caution
Satisfaction and well-being	Survey of perceived usefulness, trust, and workload	Perceptual evidence; guardrail against burnout or frustration	Perception and measured effect can diverge, as the METR forecasts did
Performance	Change fail rate; defects found after merge; time to restore after failed deployments	Guardrail on quality and stability	Outcome measures lag and are noisy at small team counts
Activity	Pull requests per developer-week; share of changes with AI assistance	Context and adoption; not a result	Easy to inflate; a proxy for output, not value
Communication and collaboration	Time to first review; review rounds per change; reviewer load distribution	Detects cost shifted from authors to reviewers	Reviewer load can concentrate on a few seniors
Efficiency and flow	Change lead time; work in progress; time blocked	Primary throughput measure	Starts at commit; does not see authoring time

Be explicit about which measures are proxies. Acceptance rate of suggestions, lines generated, and prompts per day are all measurable and all describe activity or use. The proxy-metric entry makes the general point: a proxy is useful only to the extent that it moves with the outcome, and that link is an assumption to state and check. Tickets closed per hour is the familiar false proxy for resolution. For a rollout, a high suggestion-acceptance rate shows developers press accept, which is consistent with both helpful suggestions and low scrutiny.

Measurement cards

Every metric on the plan needs a written definition that two analysts would implement identically. A measurement card is a short record containing the following fields, and it is checked before data collection starts.

• Name and decision role: primary, guardrail, or context.
• Unit and population: change, team-week, or developer-week; which repositories and teams; what is excluded, for example automated dependency updates and reverts.
• Calculation: numerator, denominator, aggregation (mean, median, percentile), and what counts as an event, such as which commit starts the clock.
• Window and baseline: observation period, pre-period used as baseline, and the rule for a late-arriving record.
• Source and attribution: where the data comes from and how a change is marked as AI-assisted or not.
• Limit and consequence: for guardrails, the value that triggers a stop or fix, set before results are seen.

Attribution deserves care. The access tiers from the permissions article give a reliable source for agent-authored changes: changes opened by a dedicated agent identity can be counted directly, and the audit log records which tasks ran. Inline assistance inside an editor leaves no such trace, so any measure of AI-assisted share depends on telemetry from the tool or on developer self-labeling, and either is incomplete. Also note that a comparison of AI-assisted and unassisted changes within a team is confounded by task selection, since developers choose where to use the tool. That is the reason METR randomized at the issue level. Without randomization, treat within-team comparisons as descriptive. For the verification side of quality, the article on testing and reviewing AI-generated code in CI describes the checks whose outcomes become quality measures here.

Rework and review effort

Speed measured at one stage can hide cost at another. Faster authoring is useful only if it does not move the cost to review, to rework, or to later incidents, and this is the claim most worth testing in a rollout. The DORA 2024 report’s pairing of individual gains with lower delivery stability and throughput is consistent with cost moving downstream, but that report is associational and does not establish the mechanism. Treat it as a hypothesis to check with your own data.

Defining rework and review effort

Rework needs a definition tied to code, because the DORA deployment rework rate counts unplanned deployments caused by production incidents, which is a different and narrower quantity. A change-level definition might be: a change is reworked if, within a fixed window after merge, a later change modifies a substantial share of the same lines or reverts it. The window, the threshold, and the exclusion of planned follow-ups all change the number, so they belong on the measurement card. A revert is unambiguous, a fix-forward needs a rule, and a refactor that happens to touch the same lines is a false positive you should estimate by sampling.

Review effort has several measurable parts: time from opening a change to the first review, the number of review rounds, reviewer-minutes if your tooling can estimate them, change size, and how review load is distributed among reviewers. Report distributions and medians, not only means, because a few very large changes dominate means. Separate the effect of change size from the effect of authorship, since AI assistance may produce larger changes that take longer to review for reasons unrelated to quality. Whether any of this is true in your organization is an empirical question.

A net-effort model, with assumed numbers

A simple model keeps the pieces together. For one change, net effort change equals authoring time saved (negative), plus added review time, plus the change in expected rework, which is the change in rework probability times the cost of one rework. The script below applies this to hypothetical values chosen for illustration. They are assumptions and do not describe any real tool or team.

# Hypothetical per-change effort model. Every number is an assumption, not a measurement.
def net_minutes(author_saved, review_added, p_rework_before, p_rework_after, rework_cost):
    rework_delta = (p_rework_after - p_rework_before) * rework_cost
    return -author_saved + review_added + rework_delta

base_total = 120 + 45 + 0.12 * 150          # authoring + review + expected rework, per change
cases = {
    "assumed values":              dict(author_saved=30, review_added=15, p_rework_before=0.12, p_rework_after=0.16, rework_cost=150),
    "rework costs twice as much":  dict(author_saved=30, review_added=15, p_rework_before=0.12, p_rework_after=0.16, rework_cost=300),
    "review adds 25 min":          dict(author_saved=30, review_added=25, p_rework_before=0.12, p_rework_after=0.16, rework_cost=300),
}
print(f"baseline effort per change: {base_total:.0f} min")
for name, kw in cases.items():
    d = net_minutes(**kw)
    print(f"{name:28} net change {d:+6.1f} min  ({100*d/base_total:+.1f}%)")

baseline effort per change: 183 min
assumed values               net change   -9.0 min  (-4.9%)
rework costs twice as much   net change   -3.0 min  (-1.6%)
review adds 25 min           net change   +7.0 min  (+3.8%)

The point of the calculation is the sign flip. With the assumed values the tool saves about 5% of total effort per change. Doubling the assumed cost of a rework cuts the saving to under 2%. Adding ten more minutes of review per change reverses it into a small net cost. A 30-minute authoring saving, which sounds like a clear win, can be offset by modest shifts in two other quantities that a lead-time dashboard does not show. This is why the plan measures review and rework directly. It also shows what to estimate: the model’s three uncertain inputs are each measurable from a pilot, and the conclusion is sensitive to all three.

Adoption

Adoption is the measure most often reported and most often misread. It is necessary context, since an unused tool has no effect, but it does not establish value.

Define the funnel with explicit denominators

Several different quantities share the word adoption. A usable funnel separates them: eligible developers (those who may use the tool under your access tiers), activated developers (used it at least once), active developers (used it in a given week or in a trailing window), and depth (the share of a developer’s changes that involved the tool, where measurable). Each ratio needs its own denominator. Active share among activated developers measures retention. Active share among eligible developers measures reach. Mixing them produces an apparent contradiction, such as a rising count of active users alongside falling reach, that disappears once the denominators are written down. Segment by team, role, and tenure, since averages hide the distribution, and report cohorts by activation date so that you can see whether later activators stay as long as earlier ones. Low adoption among eligible developers may reflect friction in getting started, trust problems, or a poor fit with the work, and the article on measuring platform adoption and developer friction addresses the same distinction between not using something because of friction and not using it because it does not fit the work.

Take-up is not assignment: intention to treat

In a randomized pilot, only some offered developers use the tool. Cui and colleagues report that in the Microsoft experiment only 8.5% of the treated group had signed up two weeks in, which rose to 42.5% after reminders. The comparison that preserves randomization is by assignment, treating everyone offered the tool as treated whether or not they used it. This is called intention to treat, and it estimates the effect of the offer. Comparing the users within the offered group against the control group breaks the randomization because users chose themselves. A third estimate, sometimes called a Wald or instrumental-variable estimate, divides the effect of assignment by the difference in take-up between the two groups. Under assumptions (assignment affects the outcome only through use, and the offer does not discourage anyone from using the tool), it recovers the average effect on those who use the tool because of the offer. The simulation below shows all three on synthetic data. The true effect of using the tool is set at a 10% reduction, take-up is about 30%, and faster developers are more likely to take it up.

import numpy as np
rng = np.random.default_rng(3)
def pct(x): return 100 * (np.exp(x) - 1)

# Assumptions (illustrative): true effect of USING the tool is -10% on log lead time.
# Developers are assigned access at random, but only some take it up, and faster
# developers (lower baseline) are more likely to take it up when offered.
n, reps, true_use_effect = 2000, 1000, np.log(0.90)
itt, per_protocol, wald, takeups = [], [], [], []
for _ in range(reps):
    base = rng.normal(np.log(40), 0.5, n)
    offered = rng.random(n) < 0.5
    p_take = 0.6 / (1 + np.exp(3.0 * (base - np.log(40))))     # about 30% overall take-up
    uses = offered & (rng.random(n) < p_take)                    # nobody uses it without an offer
    y = base + true_use_effect * uses + rng.normal(0, 0.15, n)
    itt.append(pct(y[offered].mean() - y[~offered].mean()))
    per_protocol.append(pct(y[uses].mean() - y[~offered].mean()))
    tk = uses[offered].mean(); takeups.append(tk)
    wald.append(pct((y[offered].mean() - y[~offered].mean()) / tk))
print(f"take-up among offered   : {np.mean(takeups):.2f}")
print(f"true effect of use      : {pct(true_use_effect):6.1f}%")
print(f"compare by assignment   : {np.mean(itt):6.1f}%   (effect of the offer)")
print(f"compare users vs control: {np.mean(per_protocol):6.1f}%   (biased by who takes it up)")
print(f"assignment effect / take-up: {np.mean(wald):6.1f}%   (recovers the effect on users)")

take-up among offered   : 0.30
true effect of use      :  -10.0%
compare by assignment   :   -3.1%   (effect of the offer)
compare users vs control:  -30.9%   (biased by who takes it up)
assignment effect / take-up:   -9.7%   (recovers the effect on users)

The assignment-based comparison reports a 3.1% reduction. That is correct as the effect of offering the tool to everyone at this take-up rate, and it is diluted because most offered developers never used it. The comparison of users with the control group reports about 31%, which is a large overstatement caused by who chooses to use the tool. Dividing the assignment effect by the take-up recovers a figure close to the assumed 10%. The three numbers answer different questions, and the rollout decision needs the right one. If the organization will offer the tool and let developers opt in, the effect of the offer at realistic take-up is the number that predicts the benefit. If the question is whether the tool helps a developer who uses it, the scaled estimate answers it, with its assumptions stated.

Causal limits and rollout gates

What a pilot can and cannot establish

A randomized comparison supports a causal reading of the difference between groups. A randomized experiment makes the groups comparable in expectation, and the causal effect it estimates is an average over the units studied, not a property of any individual developer. Randomization does not make the difference large enough to see, does not make the metric measure what you care about, and does not extend the result beyond the population studied. METR’s authors are explicit about the last point, and Cui and colleagues find that effects differ with experience. An effect estimated on volunteers-turned-randomized teams in one business unit may not carry to another unit with different code, different review norms, and different seniority.

Power, intervals, and the honest inconclusive

Statistical power is the probability that a test detects an effect of a specified size if it is real, and it depends on that size, the noise, and the number of units. With teams as the unit, the number of units is small. The final simulation estimates power for a design that randomizes teams, measures each team’s change in median lead time, and tests the difference between arms. The assumed noise is a standard deviation of 15% in each period’s measurement, in log terms, and the table reports the fraction of 1,500 simulated pilots in which a two-sided test at the 5% level rejects no effect.

import numpy as np
from scipy import stats
rng = np.random.default_rng(7)

def make_world(n, true_effect, trend=0.0, sd_team=0.5, sd_noise=0.15):
    """Assumptions for illustration. Outcome: log of a team's median lead time (hours)."""
    base = rng.normal(np.log(40), sd_team, n)            # team-level baseline
    pre  = base + rng.normal(0, sd_noise, n)
    return base, pre, trend, true_effect, sd_noise

def post_outcome(base, treated, trend, true_effect, sd_noise):
    return base + trend + true_effect * treated + rng.normal(0, sd_noise, len(base))

def power(n_per_arm, effect_pct, reps=1500, sd_noise=0.15):
    te = np.log(1 + effect_pct / 100)
    hits = 0
    for _ in range(reps):
        n = 2 * n_per_arm
        base, pre, trend, _, sd = make_world(n, te, 0.0, sd_noise=sd_noise)
        t = np.zeros(n, bool); t[rng.permutation(n)[:n_per_arm]] = True   # random assignment
        post = post_outcome(base, t, trend, te, sd)
        delta = post - pre                                  # per-team change
        _, p = stats.ttest_ind(delta[t], delta[~t], equal_var=False)
        hits += p < 0.05
    return hits / reps

print("Scenario B: true effect = -8% lead time, per-team before/after change, two-sided t-test, alpha=0.05")
for n_arm in (5, 10, 20, 40, 80):
    print(f"  {n_arm:3d} teams per arm -> power {power(n_arm, -8):.2f}")
print("Same design, true effect = -3%:")
for n_arm in (20, 80, 200):
    print(f"  {n_arm:3d} teams per arm -> power {power(n_arm, -3):.2f}")

Scenario B: true effect = -8% lead time, per-team before/after change, two-sided t-test, alpha=0.05
    5 teams per arm -> power 0.08
   10 teams per arm -> power 0.13
   20 teams per arm -> power 0.23
   40 teams per arm -> power 0.42
   80 teams per arm -> power 0.68
Same design, true effect = -3%:
   20 teams per arm -> power 0.07
   80 teams per arm -> power 0.14
  200 teams per arm -> power 0.27

The power function above produced the table. For each design it simulates 1,500 pilots, randomly assigns half the teams to the tool, computes each team’s before-to-after change, and applies a Welch t-test. If the true effect is an 8% reduction in lead time, a pilot with 20 teams per arm detects it a little under a quarter of the time, and even 80 per arm detects it only about two thirds of the time. If the true effect is 3%, 200 teams per arm still detects it about a quarter of the time. Two consequences follow. A pilot with a few teams per arm cannot distinguish a worthwhile effect from none, so a nonsignificant result from it is not evidence of no effect. And the organization should decide in advance which effect size it cares about, since an improvement too small to matter is not a reason to build a bigger study. The confidence interval for the estimated difference conveys both the estimate and its uncertainty, and a confidence interval that spans both a useful gain and a harmful loss should be reported as exactly that. These are properties of the synthetic design with its assumed noise. Real lead-time data is heavy-tailed and has different variance, so estimate the noise from your own pre-period before planning the pilot.

Other pitfalls are cheaper to avoid than to correct. Looking at results repeatedly and stopping when they look good inflates false positives, so fix the analysis window in advance. Testing many metrics will produce a few apparent effects by chance, which is why one primary metric is named before the pilot. And watch for novelty and learning curves: an effect measured in the first weeks can differ from the steady state, so the observation window should be long enough to include the period in which usage stabilizes.

Gates

A gate is a pre-agreed rule that connects results to the next decision. Writing the rules before the pilot prevents the common failure of choosing the interpretation after the data arrives. The table gives a structure; thresholds are for the organization to set from its tolerance for risk, cost, and the effect size it cares about.

Gate	Question	Evidence required	If not met
Safety and control	Do the isolation, credential, and audit controls hold for this cohort?	Acceptance tests from the permissions article pass; audit records are complete and attributable	Fix controls before any measurement; do not expand
Guardrails	Is harm within the pre-set tolerance?	Change fail rate, post-merge defects, and recovery time show no deterioration beyond the margin, judged with intervals, not point estimates	Stop or restrict scope; investigate the mechanism
Effect	Is the primary metric better by at least the stated amount?	The interval for the difference excludes the minimum worthwhile effect, or the result is reported as inconclusive	Extend or enlarge the pilot, or accept that no worthwhile effect was found
Cost of work	Does review and rework effort offset the authoring gain?	Net-effort estimate from measured review and rework, with its uncertainty	Address review load or change task mix; reassess
Adoption and fit	Do developers who start keep using it, and in which segments?	Retention by activation cohort; reach among eligible developers; survey results	Investigate friction and fit by segment before expanding
Economics	Is the cost per active developer justified?	Licensing, compute, and support costs against the measured effect; see the article on platform unit economics	Hold or renegotiate; do not infer value from usage

Notice that the guardrail gate is about harm, which changes the statistical question. Showing that quality did not get worse is not the same as failing to find that it got worse. A small pilot that detects nothing has not shown safety. Specify a margin of harm you can tolerate and require the interval for the difference to lie inside it, which usually needs more data than detecting a benefit. The cost of an expansion that turns out to be harmful is bounded by how reversible it is, which ties the gates back to rollout mechanics: expand through flags by team, keep the ability to switch the tool off, and keep the rollback path tested. The platform unit economics article addresses cost per unit of platform usage.

A plan to write down before the pilot

1. State the decision question and the people who will make the decision at each gate.
2. Choose the unit of assignment, the assignment method, and the length of the comparison window, and list known sources of interference and contamination.
3. Pick one primary metric, three to five guardrails, and context measures across at least three dimensions, one of them perceptual. Write a measurement card for each.
4. Estimate noise from the pre-period and compute what effect size the design can detect. If it cannot detect an effect worth acting on, change the design or the decision.
5. Record baselines and check parallel pre-trends if the design is not randomized.
6. Fix the analysis: intention to treat as the primary comparison, the interval to report, how late data and exclusions are handled, and the stopping rule.
7. Set the gate thresholds before results are visible, including the harm margin for guardrails.
8. After the pilot, report the estimate with its interval, the take-up, what was inconclusive, and which populations the result does not cover.

The measurement plan will not make a rollout succeed. It can make the decision defensible: expand when worthwhile effects and tolerable guardrails are both supported, hold when the evidence is inconclusive, and stop when harm appears, with each state defined before anyone has an interest in the outcome.

References

• Peng et al., The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (arXiv:2302.06590)
• Cui, Demirer, Jaffe, Musolff, Peng, and Salz, The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers (February 2025 draft)
• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
• DORA, 2024 DORA report (research page)
• DORA, Guide to the software delivery performance metrics
• Forsgren, Storey, Maddila, Zimmermann, Houck, and Butler, The SPACE of Developer Productivity (Communications of the ACM, June 2021; copy of the article)