The conventional wisdom about AI tool adoption says that familiar users lead adoption. Microsoft's first field study of CLI coding agents says the opposite: the engineers most likely to try a CLI agent are also the least likely to keep using one.
Published July 1st on arXiv, the study tracked tens of thousands of Microsoft engineers through the early-2026 rollout of both GitHub Copilot CLI and Claude Code. It is the first peer-reviewed field study to use direct developer telemetry — not surveys, not self-reports — to measure both adoption and output for agentic command-line tools at enterprise scale. The distinction matters. Surveys measure what developers think happened. Telemetry measures what actually did.
Trial and Retention Are Different Phenomena
The study separated two questions most adoption analyses treat as one: who tries a new tool, and who continues using it. The answers come from different variables.
For trial, prior Copilot IDE experience was strongly predictive. Engineers with 60 or more days of IDE Copilot usage were 83% more likely to try a CLI agent than engineers without that background. That tracks: familiarity with AI-assisted coding makes a new AI coding tool feel like a natural extension worth testing.
Retention ran the opposite direction. Engineers who had used Copilot's IDE features most heavily were 12-15% less likely to stick with the CLI agent. The heaviest prior users tried the CLI agent and went back to their existing flow.
The researchers describe this as "experienced AI users defaulting to established tools." The simpler interpretation is that there's a real switching cost that IDE completion experience doesn't reduce. Inline completions and agentic terminal sessions require different mental models. Trying the new thing is cheap. Rebuilding a working flow around it isn't, and the existing flow already works.
The variable that predicted retention was different: current output volume. Engineers already shipping two or more pull requests weekly retained CLI agents at 31% higher rates than lower-volume engineers. Career stage and tenure barely mattered. Active shipping velocity was the signal, not seniority or prior AI familiarity.
The reading is that developers in a high-output phase added the tool and kept it because it fit an already-active rhythm. Developers with more selective AI usage applied the same selectivity to the CLI agent. The tool has to fit into how you're already working, not reshape how you work.
How Adoption Actually Spreads
The social network findings are the part most likely to change how companies think about AI rollouts.
First use spread through peer observation more than through management direction. Engineers whose skip-level manager — two levels up — was already using Copilot CLI showed 216% higher odds of adopting it themselves. Direct manager usage increased odds by 82%. Having reviewer peers at 25%+ adoption added 54%.
The hierarchy here matters. Skip-level visibility beats direct-manager signal by more than 2.5x. The most plausible explanation is that skip-level peers represent something aspirational rather than managerial. Engineers observe them doing real work, notice the tools they're running, and experiment. A direct manager's tool use is observable but also interpretable as policy compliance — using the thing you're expected to use. A principal engineer running a CLI agent to handle a migration signals genuine utility in serious work, not just organizational alignment.
Career stage and tenure showed minimal predictive power in the adoption model once peer network effects were accounted for. You cannot predict whether an engineer will retain a CLI agent from how long they've been at the company or their seniority band. You can predict it from whether they're in a high-output phase and whether the engineers nearby are using the tool where it's visible.
This has concrete implications for how rollouts get designed. Seeding access to senior technical leads — the skip-level layer — before a broad rollout creates a peer visibility effect that mandate-based distribution doesn't. The study gives this a number: 216% higher adoption odds from skip-level observation. That's not a soft cultural effect. It's a large enough signal that rollout sequencing should probably be designed around it.
The Copilot CLI vs Claude Code Finding
The study's most counterintuitive result: within Microsoft's engineering organization, GitHub Copilot CLI outperformed Claude Code by 2.2x on measured PR lift. Copilot CLI adopters merged 24.9% more PRs than their pre-adoption baseline. Claude Code adopters merged 11.4% more.
This conflicts with 2026 developer surveys, which consistently show Claude Code as the preferred agentic tool for serious coding work. Developers rate it higher on quality, reasoning, and handling complex multi-file tasks. The preference gap in surveys is real and consistent.
The Microsoft data doesn't contradict those preferences. It shows something different: within a specific organizational context, with specific infrastructure, the tool that integrated more natively with the existing workflow produced better measurable outcomes. Microsoft's development environment runs on Azure DevOps and GitHub. Copilot CLI was built to fit into that stack. Claude Code operates more autonomously but more independently of that infrastructure.
The authors call this "organizational alignment advantages" rather than raw capability differences, which is careful but accurate. Tool effectiveness is a product of capability and integration, and integration is organization-specific. The best tool in an abstract benchmark ranking is not always the most effective tool in a particular environment.
The lesson is specific. If you're making tooling decisions based on benchmark leaderboards or developer preference surveys, you're missing a variable. The study produced a number from telemetry that surveys had predicted wrong by a factor of two. The tool your developers prefer in a survey may not be the tool producing the most merged code in your actual workflow. Knowing which is true for your team requires measuring it in your environment.
What Telemetry Makes Visible
The thing that makes this study different from most developer productivity research is the data source.
Surveys dominate the literature because they're cheap and fast. They are also systematically biased in a direction that now has multiple data points behind it. Developers overestimate the productivity benefit of AI tools and underestimate switching costs. METR's randomized trial established this at the task level — developers felt 20% faster while running 19% slower. The Microsoft study establishes it at the tool-preference level: developers preferred Claude Code in surveys, and Copilot CLI outperformed it in measured output.
Telemetry doesn't carry this bias. It records what happened — file edits triggered, PRs opened, sessions started — not what the developer estimated happened afterward. At the scale of tens of thousands of engineers over months, it produces estimates with confidence intervals the survey literature can't approach. The 24% PR lift carries a 95% CI of 14.5-33.7%, tight enough to be useful. Individual-level survey data at those sample sizes would have error bars wide enough to make the result uninterpretable.
The practical problem is that this kind of telemetry requires an organization the size of Microsoft to collect it. The fixed-effects Poisson regression the study runs — using each engineer as their own control across weeks of varying tool usage — isn't possible from a quarterly survey.
What individual developers and small teams do have is their own behavioral history. Commit frequency, PR cycle time, application-level time allocation across weeks where AI workflows ran well versus weeks they didn't — these are the signals telemetry captures and surveys miss. The Microsoft study establishes what that data can reveal when collected carefully. The smaller version of the same question is whether you're collecting any of it at all.
The Adoption Curve You're Not Tracking
The study's dose-response relationship is worth extracting as a standalone fact: engineers using CLI agents five or more days per week saw 50.1% more merged PRs than their zero-usage weeks. At three days per week, the gain reached approximately 15-30%. The marginal return to consistent daily use is significant.
That number is useful, but it also points at a measurement gap. Most teams know who has a Copilot seat. They don't know how many days per week that seat is actually in use, how that usage varies across engineers, or whether there's a usage level below which the tool is producing no measurable output gain. The seat-purchased-versus-tool-adopted question is already common. The daily-usage-versus-PR-output question is rarer and apparently more diagnostic.
At xeve, we track at the application level — every tool in use, every session, every switch — because the gap between what developers think their tools are doing and what they're actually doing is empirically large. The Microsoft study quantified that gap at enterprise scale with a methodology careful enough to give you real confidence intervals. The individual-developer version of the same problem is smaller but the bias runs the same direction: the tool that feels most useful in the moment may not be the one moving your output numbers.
The finding that the engineers who try CLI agents aren't the engineers who keep them is useful on its own. But the methodology that produced it is the more durable contribution. Stop asking developers how they feel about their tools. Watch what they do with them.