← back to blog
developer productivity

When You Review AI Agent PRs, 58% of the Work Isn't Code Review

6 min read

AI agent pull requests require human intervention less often than PRs written by people. Researchers at the International Conference on Mining Software Repositories presented data on this in April: developers intervene in 52.17% of AI agent PRs versus 83.59% of human-authored PRs. Agents close more PRs without needing human involvement. If your mental model of the review bottleneck is "AI makes more PRs for people to review," that number should complicate it.

Here is the problem. When humans do intervene in agent PRs, the interventions are more expensive and they are not doing what most people think of as code review. The same study — "Behind Agentic Pull Requests," an empirical analysis of how developers interact with AI agent-authored code — built a taxonomy of intervention types. The result: 58.02% of human effort falls into "guidance-level" work. That means restricting what the agent is doing, enforcing project conventions, correcting behavior that conflicts with how this specific team works. It is not checking whether the logic is correct. It is something else.

What the Split Looks Like

The taxonomy from the MSR 2026 paper breaks human interventions in AI agent PRs into four categories.

Guidance-level: 58.02%. Restricting agent scope, enforcing conventions, correcting project-specific norms the agent violated. The agent did something reasonable in the abstract and wrong in context.

Decision-level: 21.16%. Choosing among options the agent surfaced but could not resolve. Deciding whether to proceed, what direction to take, how to handle a situation with multiple valid approaches.

Direct code changes: 17.05%. Actually editing the generated code. The traditional code review action — the one that all the vocabulary of "reviewing" implies.

Operational: 3.69%. Mechanical process steps: triggering reruns, updating PR state, handling CI configuration.

Only 17% of human intervention time is what you'd call code review in the traditional sense. The majority is something the field does not have a clean name for yet: a combination of constraint enforcement, scope management, and what amounts to real-time behavioral specification for a system that learned general patterns but not your team's specific ones.

Why Agent PRs Are Different to Interact With

Human-authored PRs carry tacit context. A developer who has worked on a codebase for eighteen months understands the architectural constraints, the unwritten conventions, the decisions made two years ago that you can't change without breaking things downstream. When they open a PR, that context shaped every decision they made. The reviewer can disagree with choices, but the choices were made with the full context in mind.

An AI agent starts without it. It has documentation, prior code, maybe a CLAUDE.md or a AGENTS.md — whatever was explicitly written down. What was never written down, because it was obvious to everyone on the team, is invisible to the agent. The agent then makes decisions based on general training and whatever explicit specification it received. The result can be technically correct, functionally sound, and still wrong for this project in ways that require human intervention to correct.

A companion paper presented at the same conference analyzed 9,799 human-reviewed agentic PRs to understand why they get merged or rejected. Only 35.7% of rejected PRs showed clear failures attributable to the agent itself. 31.2% were rejected because of workflow constraints — not because the code was wrong, but because the submission didn't fit how the team worked. Another 33.1% had no observable rationale: the rejection happened, and the interaction artifacts didn't capture why.

That's a substantial fraction of rejected agent work that wasn't rejected for correctness reasons. It was rejected for fit reasons. The agent didn't know what it didn't know.

The Work That Gets Called "Code Review"

There is a practical problem with calling all of this code review.

Code review is a category most developers track in some form. It shows up in retrospective estimates of how time was spent, in calendar blocks, in the overhead column of sprint planning. The tacit understanding of what "code review" means is pretty stable: you read what someone wrote, check the logic, catch errors, assess the architecture decision, leave comments.

That is approximately 17% of what you are doing when you interact with an AI agent PR. The other 83% is split between teaching the agent your conventions, making decisions it could not make, and handling process mechanics. None of that vocabulary fits cleanly into "code review."

The mismatch matters because developers and teams are trying to understand where their time goes. If AI tools increase the amount of convention enforcement and agent-guidance work, that shift should be visible in time data — but it won't be, because it disappears into the "code review" bucket alongside the shrinking fraction that is actually correctness checking.

Sonar's 2026 State of Code survey found that reviewing overtook writing as the single largest AI-assisted time sink among heavy agentic-tool users, with review hours climbing to 14 to 16 per week for that group. The survey did not break down what those review hours consisted of. The MSR paper suggests that at least half of it is not review in the usual sense.

The Out-of-Hours Signal

Multitudes analyzed over 500 developers and found that while engineers merged 27.2% more pull requests with AI tools, they also saw a 19.6% rise in out-of-hours commits. Multitudes CEO Lauren Peate flagged this directly: "If that out-of-hours work is going up, it's not good for the person."

Out-of-hours commits are a behavioral signal. They say: work that was supposed to fit inside regular hours is not fitting. Some of that is the familiar workload creep pattern — AI enables more scope, more scope fills the time, and then some. But there is a more specific mechanism in play.

Agent supervision is responsive work. When an agent opens a PR and needs guidance, the correction opportunity exists in a window before the agent produces more work on top of the wrong direction. Developers who have internalized that the agent needs steering tend to check in more frequently — including outside working hours — because the cost of letting an agent run uncorrected for eight hours is concrete. Out-of-hours commits may partly be developers catching up on agent guidance they didn't complete during the day, or responding to PRs that arrived after hours because the agent doesn't have working hours.

This is different from deadline-driven overtime. It is the supervision overhead of a system that generates work continuously and expects prompt feedback.

No One Is Measuring Convention Enforcement

There is no metric for "how long did I spend enforcing project conventions on an AI agent today."

There is no standard bucket in engineering analytics that captures the difference between reviewing a human's code and correcting an agent's convention violations. When you add a review comment like "we don't name files that way" or "this should use the internal auth helper, not the generic SDK method," that work records itself the same way as a comment about a logic error. Both are one review comment. The character of the work is completely different.

The agents will continue not knowing your conventions until someone writes them down explicitly enough for the agent to use — or until the feedback loop from guidance-level interventions becomes a documented norm. The Karpathy CLAUDE.md model and its equivalents (AGENTS.md, project rules files in Cursor) are attempts to systematize this: take the things you keep having to correct, write them down once, let the agent consult them. The MSR paper's 58% number suggests there is still a large gap between what teams have specified explicitly and what agents need to know to operate without correction.

The teams that close that gap fastest will spend less time on guidance-level interventions and more time on the 17% that is actually about code correctness. The data for knowing where you are on that journey — how much of your agent PR interaction is convention enforcement versus actual review — requires a level of session-level granularity that most productivity tools do not currently provide.

What the 52% vs 84% Number Actually Means

AI agent PRs need human intervention less often. That is a real productivity signal and not a trivial one. Agents are closing work that humans don't need to touch.

But the 52% number does not mean the review that does happen is lighter. The study is explicit: when interventions occur in agent PRs, they require higher review effort, with larger code churn and longer durations than equivalent human PR interventions. Fewer touchpoints, but each touchpoint is more expensive.

The picture that emerges is not a simple bottleneck story. It is a structural shift in what developer review work consists of. Developers who are good at the traditional skills — spotting logic errors, understanding architecture, catching security assumptions — are now also spending the majority of their agent-PR time on a different skill set: constraint specification, scope management, convention articulation. Some of that work translates back to code quality; most of it is overhead that only reduces as your explicit project specification improves.

The measurement problem is the same as it always is. You can see that review takes time. You cannot easily see what kind of time it is, which makes it hard to improve the right thing. If the guidance-level fraction of your agent PR work is high, the lever is not faster code review skills — it is better project specification. If you are measuring all of it as "review time," you will keep pulling the wrong lever.

Written by Kevin — builder of xeve

Track your apps, coding, music, and health — all in one place.

try xeve free