A research team from Stanford and MIT released SWE-chat in April — the first large-scale dataset of real coding agent sessions collected in the wild, not in a lab. Six thousand sessions, 63,000 user prompts, 355,000 agent tool calls, all from open-source developers using tools like Claude Code on real repositories. The dataset is designed to update continuously as more sessions accumulate.
Two numbers in the paper are worth sitting with. Agents ask users for clarification in 1.4% of turns. Users push back against agent output — corrections, failure reports, redirections — in 44% of turns.
That 43-point gap is not a calibration quirk that will get tuned away. It's structural.
What the Dataset Actually Captures
SWE-chat is different from previous research on AI coding because it measures what developers and agents actually do together, not what they do in controlled conditions. The sessions are messy in exactly the ways real sessions are messy: vague initial prompts, mid-task redirections, long stretches of agent autonomy interrupted by user course corrections, and plenty of agent output that never makes it into a commit.
The 44% user correction rate means nearly every other turn involves a developer telling the agent that something was wrong, missed, or not what they asked for. That's not a sign of bad agents — the models being used are the same frontier models that top the coding benchmarks. It's a sign that the gap between what a developer intends and what an agent can infer from a prompt is large and persistent, regardless of model capability.
The 1.4% clarification rate means agents almost never pause to resolve that gap before acting on it.
Agents Are Trained to Ship, Not to Ask
This asymmetry exists by design. Coding agents are trained and evaluated on task completion — did the feature get built, did the test pass, did the build not break? Asking a clarifying question interrupts that loop. It delays the response. It feels to the user, in the moment, like the agent is being less useful rather than more careful.
So agents learned to guess. They fill in the gaps in a prompt with plausible assumptions, generate code against those assumptions, and return the output. When the assumptions were wrong — which is apparently 44% of the time — the user corrects them.
This is not an argument that agents should ask more questions. Users don't want to be peppered with clarifications on every prompt; that's its own failure mode. The point is that the current equilibrium — nearly zero asking, nearly half of outputs requiring correction — is not working the way the productivity numbers suggest.
What 44% of Turns Actually Costs
When a user pushes back on agent output, they're not just hitting a backspace key. They're loading the generated code into working memory, evaluating it against the actual requirements, identifying the gap between intent and result, and then formulating a correction prompt clear enough that the agent won't repeat the same mistake.
That's a cognitive cycle. It takes time. The SWE-chat data shows that agent sessions are getting longer — the 99.9th-percentile turn duration now exceeds 100 minutes of autonomous agent operation before the user steps in. A session that runs 100 minutes and then requires a correction cycle is a very different cost structure than a session that checks in at 15 minutes and catches a wrong assumption before 85 minutes of work pile up on top of it.
The longer agents run without asking, the more correction debt accumulates.
The Bimodal Split and What It Means
SWE-chat found a usage pattern that's sharply bimodal. In 41% of sessions, the agent authors virtually all of the committed code — the developer writes the initial prompt, reviews the output, and commits. In 23% of sessions, the human writes all the code and the agent's contribution is negligible. The middle third — genuine collaborative authorship — is actually the minority pattern.
This split suggests that developers have implicitly figured out something the productivity discourse hasn't caught up to: either a task is appropriate for full delegation or it isn't. The overlap region — tasks where productive back-and-forth between developer and agent generates the best outcome — is smaller in practice than the marketing suggests.
The problem is that developers decide which mode they're in at the start of the session, based on how the task looks before they've tried it. Some tasks that look like clean delegation turn into the 44% pushback sessions. Some tasks that look like they'll need deep human involvement get handled cleanly by the agent. The sorting happens after the fact, and the miscategorizations are where the correction cycles live.
The 44% of Code That Didn't Make It
The code survival number from SWE-chat is the one that connects most directly to productivity accounting: only 44% of agent-produced code survives into user commits.
That's across all sessions. In some sessions, almost everything the agent wrote was committed. In others, most of it was discarded. The average across 6,000 sessions is that the agent wrote roughly twice the code that ended up in the repository.
This is distinct from what previous deletion-rate research measured. JetBrains found that AI users delete code at 13 times the rate of non-users — that's in-session review work, the write-evaluate-discard loop that happens before anything gets committed. SWE-chat's 44% survival rate measures what happens to the output that passes that initial review and gets generated as a candidate commit. Nearly half of it still doesn't make it.
The denominator for agent productivity calculations should be all agent output, not just the code that merged. If you measure throughput only on what survived, you're measuring the output of a process that discarded half its intermediate work. That's real developer time, real review effort, real sessions that ran for potentially hours and produced code that didn't ship.
Where the Security Number Comes From
SWE-chat also found that the sessions where agents write virtually all the committed code — the 41% full-delegation sessions — introduce 9 times more security vulnerabilities per thousand committed lines than sessions where humans wrote the code themselves.
This isn't about frontier models being bad at security. It's about the oversight pattern. When a developer authors code, they're running a continuous review process as they write. When they delegate fully to an agent and review the output as a batch at the end, they're reviewing code they didn't write, at a rate that grows with the size of the agent's output, often after a long session where the last thing they want to do is line-by-line security analysis.
The agents that ask for clarification 1.4% of the time also aren't flagging "I'm about to write 200 lines of authentication logic — do you want to review this in sections?" They ship it all at once. The developer reviews it all at once, or doesn't review it carefully enough, and the vulnerability rate reflects that.
What the Calibration Gap Looks Like on Your Own Data
The 1.4% and 44% numbers are population averages across 6,000 sessions. Your personal numbers depend on what you're building, how precisely you prompt, and how well your agent understands your codebase.
The interesting measurement question is what your pushback rate actually is, and whether it's stable or growing. If you're spending an increasing fraction of each agent session on correction cycles — reformulating prompts, explaining what went wrong, asking the agent to redo work that was close but not right — that's a signal about calibration, not just model quality.
It's also not a number that shows up in any standard coding metrics. Commit history captures what merged. Time-tracking tools capture that you were in a terminal. Neither captures the ratio of your prompts that resulted in usable output versus prompts that resulted in correction cycles.
At xeve, we track coding sessions at the OS level — when you're in your terminal, your editor, your agent interface. That tells you session duration and tool context. It doesn't yet tell you what fraction of those sessions involved significant correction overhead versus clean runs where the first output was close enough to use. That ratio — something like a personal calibration efficiency metric — is the thing the SWE-chat data is implicitly pointing at.
Agents ask in 1.4% of turns. The rest of the calibration burden falls on you. Knowing how much of your session time lives in that burden is the first step toward improving it.