Measuring Platform Adoption and Developer Friction

A platform team ships a capability, and a dashboard reports that 140 services now use it. The number rises every month, and the question that matters is still open: did those teams get their work done more easily, or did they adopt it because a deadline, a default, or a mandate left them no choice? A count of users cannot separate those cases, and neither can a count of tickets or a pipeline success rate taken alone.

This article builds a measurement practice for that question. It treats adoption as one input and developer friction, meaning the effort and delay between a developer’s intent and a working outcome, as the quantity the platform exists to reduce. You will define workflow events that can be computed from systems the platform already owns, choose onboarding, task-success, and support-load measures with explicit denominators, add qualitative research that explains the numbers, and set a baseline whose limits you can state. The last part is a plain account of what these methods cannot prove about cause.

The article assumes the previous one, Designing Platform APIs and Extension Boundaries, which covered the lifecycle, tenancy, and compatibility rules of a platform’s interfaces. That gives you the surfaces where usage can be observed and the versions that consumers depend on. Everything below about how to measure those surfaces is explained here. Statistical background is limited to means, medians, percentiles, and resampling, which general computer science and programming knowledge covers.

Start from decisions, not from the dashboard

Measurement is expensive to maintain and easy to misuse, so each measure should be tied to a decision someone could actually make. A platform team typically faces four recurring decisions: which capability to improve next, whether a recent change helped, whether a capability is ready to move from a pilot to the default path, and whether it is time to retire something. The same raw events support all four, but the questions differ. “Which step loses the most teams?” is a diagnostic question. “Did the new scaffolder shorten time to first deploy?” is a causal question and needs much more care. “Is the old capability still in use?” is an inventory question, covered in the next article on retiring capabilities.

Treat adoption and value as separate quantities. Adoption is the share of an eligible population that uses a capability, so it needs a definition of eligible (every service? every service of a certain kind?) and of use (registered, tried once, used in the last 30 days?). Value is the improvement in the consumer’s work attributable to the capability. Platform as a product work makes the same point from the management side: a mandate removes the signal that usage normally gives about preference, so a mandated platform has to look harder at task success, time to complete common work, how often teams leave the supported path, and what users say directly.

Two research frameworks are useful for deciding what to look at. The SPACE framework argues that developer productivity cannot be captured by a single metric or dimension, and names five: satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. The DevEx framework narrows the focus to three dimensions of developer experience: feedback loops, cognitive load, and flow state, and recommends capturing both perceptual measures (what developers report) and workflow measures (what systems record) for each. The glossary entry on developer experience covers how those dimensions interact. The descriptions here rest on the SPACE summary page and a published summary of the DevEx paper, not on the full ACM Queue articles.

DORA’s delivery metrics are a third reference point. DORA lists change lead time, deployment frequency, failed deployment recovery time, change fail rate, and deployment rework rate, and its guidance warns against setting those metrics as goals and against comparing teams with unlike applications. DORA metrics describe delivery outcomes for a team; they do not tell you which platform capability influenced them, which is why the platform needs its own workflow measures as well.

Workflow measurement: events, units, and denominators

Define a workflow as a path with observable steps

A workflow here is a sequence of steps a consumer takes to reach a useful outcome with a platform capability. For a service-creation capability the path might be: request access, scaffold a service from a template, run the first pipeline, deploy, and receive first production traffic. The steps are examples and will differ by platform. What matters is that each step corresponds to an event some system already records, so measurement does not depend on asking people to self-report timestamps.

A minimal event record needs enough fields to answer the questions in this article and no more:

• consumer unit: the team or service that owns the work, not an individual. Individual-level productivity numbers invite misuse and, in small teams, identify people;
• capability and version: which platform interface was used, including the version, so a change can be compared with what came before;
• step and outcome: the step name and whether it succeeded, failed, or was abandoned, with an error category if it failed;
• attempt identifier: so a retry is distinguishable from a new attempt;
• timestamp: event time, recorded where the event happens, with the clock source stated.

Platform-side sources usually include the API gateway or control plane, the CLI, the portal, the pipeline system, and the provisioning workflows. Each sees only part of the path. A portal log shows clicks but not whether the resulting deployment worked; a pipeline log shows runs but not the abandoned attempts before the first run. Joining sources on the consumer unit and a shared correlation identifier is part of the work. Where a join is impossible, say which part of the path is invisible rather than treating the visible part as the whole.

Funnels, time to first success, and the completers-only trap

With step events, the first analysis is a funnel: of the teams that started, how many reached each step, and what fraction of those present at one step reached the next. The second is time to first success, the elapsed time from the start event to the outcome event. Both are easy to compute and both are easy to get wrong in the same way: they are computed over teams that finished, and the teams that stalled are exactly the ones whose friction you most need to understand.

The lightly condensed excerpts below, taken from a Python script, generate a synthetic onboarding log and compute these quantities. The probabilities and step durations are assumptions chosen for illustration; no real platform’s data is used. The log has 300 teams that started onboarding.

import random, statistics as st, math
random.seed(42)
STEPS=["request_access","scaffold_service","first_pipeline_run","first_deploy","first_prod_traffic"]
def onboard(team):
    t=0; reached=0; times={}
    p_ok=[0.97,0.93,0.85,0.90,0.95]; med=[2,1,6,4,20]  # hours, assumed
    for i,s in enumerate(STEPS):
        if random.random()>p_ok[i]: break
        t+=random.lognormvariate(math.log(med[i]),0.8); times[s]=t; reached+=1
    return reached,times
N=300; res=[onboard(i) for i in range(N)]
def boot_ci(xs,f,B=2000):
    xs=list(xs); vals=sorted(f(random.choices(xs,k=len(xs))) for _ in range(B))
    return vals[int(.025*B)],vals[int(.975*B)]

The output of the full script, executed with Python 3.12 and the fixed seed shown, was:

Part 1: funnel (synthetic, N=300 started onboarding)
  request_access       reached 295  step conversion 0.98
  scaffold_service     reached 273  step conversion 0.93
  first_pipeline_run   reached 237  step conversion 0.87
  first_deploy         reached 214  step conversion 0.90
  first_prod_traffic   reached 203  step conversion 0.95
  completed: 203 (68%); median time to first prod traffic 39.0 h, 95% bootstrap CI (36.4, 43.9)
  share of ALL 300 started teams reaching first prod traffic within 72 h: 56.0%
  note: median is over completers only; the 97 teams that stalled are absent from it

Read the output in three layers. First, the funnel shows where teams leave. The weakest step conversion is the first pipeline run (0.87), so a team looking for the largest drop to investigate would start there. This is a diagnostic ranking, not a measure of cause: it says where to look, not why teams leave. Second, 203 of 300 teams (68%) reached first production traffic, and their median time was 39.0 hours with a bootstrap 95% interval of 36.4 to 43.9 hours. Resampling the completers many times and taking the 2.5th and 97.5th percentiles of the medians gives that interval; the bootstrap confidence interval entry explains the procedure and its assumptions. Third, the median describes only the 203 completers. The 97 teams that stalled have no time to report, so a platform that makes onboarding impossible for a third of teams could still report an attractive median.

A safer headline combines both. Reporting that 56.0% of all started teams reached first production traffic within 72 hours uses the full starting population as the denominator and sets a fixed time window. It cannot be improved by losing the slow teams. The cost is that it requires choosing a window; choose it from how long the work is expected to take, state it, and do not change it after seeing results. The underlying issue is survivorship bias: an analysis that includes only the cases that lasted. A related technical issue is censoring. Teams still in progress when you take the snapshot have an unknown final duration, so recent starters should be excluded from a completion-within-window figure until they have been observed for the full window, or the number will look worse than it is. The cohort entry describes the same fix: compare groups at the same age since entry.

Choose the unit of analysis and keep it fixed

A service, a team, and a developer are three different denominators. Counting services undercounts the effort of a team that owns thirty small ones; counting developers inflates numbers for large teams. Pick the unit that matches the decision: services for a service-creation capability, teams for an access or onboarding process, and keep it consistent across periods. When a definition must change, run the old and new versions in parallel for a period and publish the difference, as described in metric change management, so a shift in the number is not mistaken for a shift in behavior.

Onboarding, task success, and support load

These three families answer different questions. Onboarding measures describe a consumer’s first encounter. Task-success measures describe repeated work after onboarding. Support load describes the human effort the platform consumes when self-service fails. Each has a denominator that must be stated, and each has a failure mode in which the number improves while the experience does not.

Measure	Numerator and denominator	Window	How it can mislead
Time to first success	Elapsed time from start event to first successful outcome, for teams that started; report the share finishing within a fixed window as well as percentiles of those that finished	Per starting cohort, observed for the full window	A median over completers hides teams that stalled; teams that stall silently are absent
Step conversion	Teams reaching step k divided by teams reaching step k-1	Per starting cohort	A fix that moves the drop to a later step can look like no improvement; a conversion near 1 on a trivial step says little
First-attempt success	Teams whose first attempt at a task succeeds divided by teams attempting it	Per task and version	Depends on what counts as success; an API returning 200 is not the same as a working deployment
Task success rate (repeat work)	Successful completions divided by attempts of a defined task, by consumer unit	Rolling period, by version	Averages across units; a few heavy users dominate, so report per-unit distributions
Retry and abandonment	Attempts that are retried, and attempts with no later success, divided by attempts	Per task	Abandonment is hard to observe without a timeout rule; the rule changes the number
Support load	Support requests, grouped by cause, divided by active consumer units (not by raw adoption count)	Per month, same ticket taxonomy each time	Falls when docs improve and also when people give up, bypass the platform, or ask in side channels

Define success by outcome, not by response code

A task is a unit of work a consumer wants done, such as creating a service, adding a database, or rotating a credential. Task success means the outcome exists and works, which is a stronger condition than a successful API response. For a provisioning request, an outcome check might confirm that the resource is reachable and the owner can use it. If the check is expensive to automate, sample outcomes and verify them by hand, and say that the rate is a sample estimate. A platform that counts every accepted request as a success will measure a number that rises with request volume and may be unrelated to whether work completed.

Error taxonomies matter here. Separate failures caused by the consumer’s input (invalid configuration), by missing permission, by the platform (capacity, bugs, timeouts), and by dependencies outside the platform. The categories point at different owners and different fixes, and a single failure rate mixes them. Keep the taxonomy small enough that two people classify the same failure the same way; if they do not, the categories are measuring the classifier.

Support load as a signal with two directions

Support requests are a direct record of friction that someone found worth asking about. Normalize them by active consumer units rather than by adoption count, and group them by cause: how-to questions suggest missing documentation or unclear defaults, access problems suggest permission design, bug reports suggest defects, and feature requests suggest missing capability. Track the share that were resolved without a person from the platform team, since that share measures how self-service the capability really is.

The limit is that support volume moves for reasons other than friction. It falls when documentation improves. It also falls when consumers stop asking: when they bypass the platform, copy a colleague’s configuration, or ask in a team chat the platform team does not read. It rises when a capability gains users, when a release changes behavior, and when a new support channel makes asking easier. For that reason, read support load next to the success measures above. Falling tickets with falling success rates or rising bypass suggests people stopped asking; falling tickets with rising success suggests friction decreased. Neither inference is proof, and the interviews in the next section are how you test it.

Measure the escape hatches

Because mandates and defaults can hold adoption up, some of the most informative workflow events are departures: a team that scaffolds from the template and then edits away its pipeline stages, that requests an exception to the golden path, or that provisions the same resource by hand outside the self-service workflow. Count exceptions and out-of-path usage per consumer unit and record the stated reason. A rising exception rate on one step is a candidate friction point even when adoption is stable. Treat the counts as leads; some exceptions are legitimate needs the path was never meant to cover.

Guard against metrics becoming targets

Once a number is what people are judged on, behaviors that move the number without moving the outcome become more likely. A proxy metric is useful only while it moves with the outcome it stands for, and that link is an assumption to check. A team measured on tickets closed can close tickets faster by asking users to refile; a team measured on adoption can obtain it by making the old path harder. Pair each primary measure with guardrails chosen before looking at results, as in primary and guardrail metrics: if time to first success is the primary measure, first-attempt success and support requests per active unit are guardrails. DORA’s guidance gives the same warning at the delivery level, advising against setting metrics as goals.

Qualitative research: finding out why

Workflow events tell you where and how often. They do not tell you why a team stopped at the first pipeline run, what they expected the template to do, or what they did instead. Qualitative research fills that gap, and it needs the same discipline as quantitative work: a clear question, a defined group of people, and a record of what was said rather than what the researcher expected.

Methods that fit platform questions

• Observed task walkthroughs. Ask a consumer who has not used the capability to perform a defined task while narrating, and observe without helping. Record the points where they pause, search, guess, or ask. This exposes cognitive load directly, since confusion is visible, and it produces a list of specific friction points with a count of how many participants hit each.
• Interviews about recent real events. Ask about the last time the person did the task: what they tried, what went wrong, what they did next. Questions about specific past events are easier to answer accurately than questions about general opinion. Prepare the same core questions for each interview and keep notes detailed enough that another person could check how you reached a conclusion.
• Support-ticket and chat coding. Read a sample of requests and assign each to a small set of causes. This reuses data you already have and ties the qualitative picture to the support-load measure. Have a second person code a subset independently and compare, to see whether the categories are used consistently.
• Surveys. Short, task-specific questions work better than broad satisfaction scores. The DevEx authors recommend designing questions from well-defined concepts, breaking results down by team and persona rather than relying on aggregates, running short transactional surveys right after a task alongside periodic ones, and following up visibly on findings to avoid survey fatigue (per the summary cited in References). A transactional survey after a failed provisioning attempt reaches the consumer while the event is fresh.

Who you talk to determines what you learn

The most reliable way to produce a flattering result is to interview the teams that adopted happily. Include the teams that tried and stopped, the teams that never started, and the teams that adopted but bypass parts of the capability. Each group shows a different part of the problem. Surveys have the parallel issue: respondents differ from non-respondents, and a survey with a low response rate describes the people who answered. Selection and nonresponse bias are systematic and are not reduced by a larger sample, so report the response rate, say who is absent, and compare respondents and non-respondents on anything known about both, such as team size or tenure.

Anonymity needs care in small populations. If a survey cell contains three people, a breakdown by team and role identifies them. Set a minimum group size below which results are not reported, state it in advance, and accept that some segments will not be separable.

Connecting qualitative findings to measures

Findings are most useful when they become something you can count. A friction log records each observed or reported problem with the step it occurred in, a severity (blocked, slowed, annoyed), and how many consumers reported it. Ranking by severity times frequency gives a backlog; tying each item to a workflow measure lets you check afterward whether fixing it moved the measure.

A hypothetical illustration: suppose the funnel above shows the lowest conversion at the first pipeline run. Task walkthroughs might reveal that new teams do not know which of several pipeline templates to start from. That is a hypothesis generated by observation, and it predicts something testable, namely that step conversion improves and support requests tagged “which template” fall after the choice is made clearer. It does not predict that time to first success falls by any particular amount, and nothing in the observation justifies a number. Writing the prediction down before the change is what lets the later measurement count as evidence.

Baselines and causal limits

What a baseline is

A baseline is a defined reference used to interpret change. It can describe the existing process before a change, or an unchanged comparison process observed alongside the change. State which meaning you use, with the population, dates, event definitions, and included work. A baseline taken from a manager’s estimate is an assumption, not a measurement. Do not replace the reference after seeing results to make the change look favorable. If the workflow or the definition changes, version it and re-establish a comparable reference.

Why adopters and non-adopters cannot be compared directly

The comparison everyone reaches for first is between teams that use the capability and teams that do not. It is also the comparison most likely to mislead, because teams choose, or are chosen, for adoption for reasons that also affect the outcome. A causal effect is the change in an outcome produced by changing the treatment; observing that two groups differ reports only an association. The simulation below builds that problem in on purpose. Small teams adopt more often and also have shorter lead times for reasons unrelated to the platform, and the true platform effect is set to a 15% reduction. All numbers are generated, not measured.

random.seed(5)
teams=[]
for i in range(2000):
    big=random.random()<0.4
    adopt=random.random()<(0.2 if big else 0.6)      # small teams adopt more (assumed)
    base=random.lognormvariate(math.log(60 if big else 30),0.4)   # lead time hours; big teams slower
    lt=base*(0.85 if adopt else 1.0)                  # true effect: -15% (assumed)
    teams.append((big,adopt,lt))

Part 2: adopters vs non-adopters (synthetic; true effect built in: -15%)
  naive median lead-time ratio adopters/non-adopters: -37.3%
  within small teams: -13.8%  (adopters 706, non 497)
  within large teams: -11.2%  (adopters 181, non 616)

The naive comparison reports a 37.3% shorter median lead time for adopters, more than double the built-in effect. The difference is composition: adopters are disproportionately small teams, who would have been faster anyway. Comparing within team-size groups gives 13.8% and 11.2% reductions. These are close to, but not equal to, the 15% built in. They are medians from finite samples with noisy lead times, so they scatter around the truth; this is a single run, and the spread across repeated runs was not measured. The lesson is the direction of the error, which stratifying on a known confounder reduced. It would not have reduced a confounder you did not measure, such as team motivation or the technical difficulty of the service, and that is the standing limit of this approach.

Before-and-after comparisons absorb every other change

A second common design measures the same teams before and after the platform change. It controls for team identity but not for time. If lead times are improving across the organization for other reasons, the before-and-after difference includes that trend. The simulation gives the treated group a 10% platform effect starting at month 6 and gives every group an independent 2% per month improvement.

def month_lt(m, treated, post):
    trend=1-0.02*m           # org-wide improvement unrelated to the platform (assumed)
    eff=0.9 if (treated and post) else 1.0
    return st.mean([random.lognormvariate(math.log(40*trend*eff),0.3) for _ in range(60)])
# months 0-5 are "pre", months 6-11 are "post"; the treated group gets the platform change at month 6

Part 3: before/after (synthetic; true platform effect -10%, plus a -2%/month org-wide trend)
  treated: pre 39.3 h, post 32.1 h -> naive before/after -18.3%
  comparison group: pre 38.4 h, post 35.1 h -> -8.6%
  ratio-of-ratios (difference-in-differences on log scale): -10.6%

The treated group’s naive before-and-after change is 18.3%, almost double the built-in 10%. The comparison group, which did not receive the change, improved 8.6% over the same months because of the trend. Dividing the treated group’s ratio by the comparison group’s ratio, a difference-in-differences calculation on the log scale, gives 10.6%, close to the built-in effect. The method works here because the simulation satisfies its key assumption: absent the change, the treated and comparison groups would have followed the same trend. In practice that assumption must be argued, and it fails when the groups are on different trajectories to begin with, when the comparison group is affected by the change, or when something else happens to only one group at the time of the rollout.

What you can do about it

• Plan the comparison before the rollout. If a change can be phased, choose which teams receive it first by a method that does not depend on their characteristics, such as random assignment within a stratum, and keep the later group as the comparison for the period. A phased rollout driven by a feature flag or release channel makes this practical for many platform changes. Randomization is not always possible, for example when a security mandate applies to everyone, and then the honest report is that cause is not identified.
• Measure the confounders you can name (team size, service type, prior lead time) and compare like with like, accepting that unmeasured ones remain.
• Compare cohorts at the same age. A team that adopted last week has a different history from one that adopted a year ago.
• Report intervals, not single numbers, and state the sample size behind each group.
• Check for spillover. If adopting teams share knowledge with others, or the platform team spends less time on the comparison group, the comparison group is not untouched.
• Say what the claim is. “Teams using the capability had shorter lead times” is an observation. “The capability shortened lead time by about X” is a causal claim and needs the design to support it.

Mandates deserve a separate note. When use is required, adoption stops measuring preference, and comparing adopters with non-adopters may be impossible because there are none. The remaining evidence is within-team change over time, the success and friction measures above, direct feedback, and out-of-path usage. Those support statements about whether the platform works for its users, not statements about whether they chose it.

Turning measures into a report people can act on

A useful report for one capability fits on a page and answers the same questions every period. It lists the eligible population and the number that adopted, the number active in the period, the onboarding funnel with step conversions, the share finishing within the fixed window, task success per consumer unit with the distribution rather than only the mean, support requests per active unit by cause, the latest survey result with its response rate, and the top items in the friction log with their status. Each figure carries its definition version, the date range, and the denominator.

Decide in advance what each pattern leads you to check. The table lists patterns and the first hypotheses to test; these are starting points, not diagnoses.

Pattern in the data	Hypotheses to test first	Evidence that would distinguish them
Adoption flat, success rate high among adopters	Capability does not fit most eligible teams, or awareness is low	Interviews with non-adopters; check eligibility definition and discovery path
Adoption rising, support rising faster	New users hit gaps the early adopters did not; documentation or defaults are wrong for the new segment	Ticket causes by cohort; walkthrough with a recent adopter
Adoption high, exception and bypass rates rising	Mandate or default holding usage up while the path does not fit real work	Reasons recorded on exceptions; interviews with teams that bypass
Funnel drop at one step, no tickets	Teams abandon silently; the step is unclear	Observed walkthrough of that step; survey sent after abandonment
Median time falling, share finishing in window flat	Slow teams drop out, so the median is computed on a shrinking, faster group	Compare funnel entry and exit counts across periods

Operating the measurements takes ownership. Name who maintains the event definitions, who reviews the report and decides, and who is responsible for the follow-up when the report reveals a problem. Review the definitions on a schedule, since platform interfaces change and events silently stop being emitted. Check that data collection respects the privacy expectations of the organization: aggregate by team, keep individual-level data out of performance evaluation, and apply retention limits. DevEx research recommends breaking results down by team and persona; where that would identify individuals, the minimum group size rule takes priority.

Failure modes and limits

• Instrumentation gaps. A step that stops emitting events looks like a step nobody reaches. Monitor event volume per source, and alert on a sudden fall to zero before interpreting it as a behavior change.
• Definition drift. A changed success definition, ticket taxonomy, or eligible population changes the number without changing behavior. Version definitions and annotate the charts.
• Metric gaming. Measures that become targets attract behavior that improves them without improving the work. Use guardrails and look at the distribution, not only the average.
• Small samples. A capability used by eight teams yields intervals so wide that most comparisons are inconclusive. Say so, and lean on qualitative evidence instead of dressing small counts as statistics.
• Confounding and trend. The simulations above show errors of more than a factor of two from naive comparisons. They show what can happen under the assumptions built into them; they do not predict the size of the error in any real organization.
• Perception versus workflow disagreement. Developers may report dissatisfaction when workflow measures look healthy, or the reverse. The disagreement is information: the DevEx authors recommend both kinds of measure because neither alone tells the complete story.
• Treating the platform as responsible for everything. Some friction comes from unclear priorities or deadlines that a platform team cannot change. Report it to the people who can, and keep measures focused on what the platform controls.

What these methods cannot establish is the value of the platform to the organization as a whole. Faster onboarding and fewer tickets support the claim that the platform reduced friction for its users, within the populations and time windows measured. Whether that translated into better products, lower cost, or retained engineers depends on evidence this article does not cover, and the next article, on retiring platform capabilities, uses adoption and usage data for a decision where the stakes of getting the denominator wrong are much clearer.

References

• SPACE of Developer Productivity: There’s More to It Than You Think (Microsoft Research publication page)
• InfoQ, DevEx metrics framework (summary of the DevEx framework by Noda, Storey, Forsgren, and Greiler)
• DORA, Software delivery performance metrics