Developers using AI coding assistants estimated they were 20% more productive. When researchers actually measured their output, they were 19% slower.
That's a 39-point gap between what developers believed and what actually happened.
The finding comes from METR's study of experienced open-source developers, published in July 2025. Sixteen developers worked on 246 real issues across repositories averaging 22,000+ stars and over a million lines of code. These weren't toy problems or contrived benchmarks; they were actual tickets from established codebases. The perception gap persisted even after developers completed their tasks. They still believed the AI had helped, despite measurable evidence to the contrary.
This matters because vendor productivity claims are almost entirely based on developer surveys, not controlled measurements.
The Quality Picture Gets Sharper
Multiple independent analyses point to structural issues in AI-generated code, though they're measuring different symptoms of the same underlying problem.
GitClear analyzed 211 million lines of code and found copy/paste code rose from 8.3% to 12.3% of changed lines between 2021 and 2024. Refactoring dropped from 25% of changed lines to under 10%. The pattern is clear: AI tools generate code that works, but they don't encourage the kind of cleanup that keeps codebases healthy over time.
CodeRabbit's pull request analysis tells a complementary story:
AI-generated code produces 1.7x more issues overall: 10.83 issues per PR versus 6.45 for human-written code. Logic and correctness errors were 75% more common. Readability issues were 3x higher. Performance problems (particularly excessive I/O) were 8x more common.
The security picture looks similarly concerning. Academic research on Copilot-generated code found vulnerability rates of 29% in Python and 24% in JavaScript. CodeRabbit's data shows security issues up to 2.74x higher in AI-assisted PRs.
Our read: these aren't contradictory findings. They're measuring different facets of the same phenomenon. AI tools produce more code faster, but that code carries structural and quality debt that shows up downstream.
The Perception Problem
Why don't developers notice? The gap isn't developers being stupid. It's a genuine experience of feeling more productive while producing less.
METR identified five factors explaining the slowdown: context switching between the AI and the codebase, time spent understanding suggestions, debugging AI-generated code, integration difficulty, and quality requirements that exceed what AI can reliably produce. Qodo's survey of developer experiences found that 65% cite context gaps during refactoring as their primary pain point with AI tools. This beats hallucinations as the top complaint. The AI doesn't understand your codebase's history, conventions, or the subtle reasons behind existing patterns.
Stack Overflow's 2025 developer survey captured the frustration quantitatively: 45% of developers cite "almost right but not quite" as their top AI frustration. 66% report spending more time fixing AI code than they save. Trust in AI accuracy dropped from 40% to 29% year-over-year.
The "almost right" problem is insidious. Code that's 90% correct feels like a gift until you realize you still need to understand it completely to find the 10% that's wrong. For experienced developers on familiar codebases, writing the code themselves may actually be faster than auditing someone else's nearly-correct attempt.
What Actually Ships?
There's a 41% figure floating around about how much code is now AI-generated. The primary sourcing on this is weak, but the directional claim is plausible given adoption rates. Qodo found 82% of developers use AI coding tools daily or weekly.
But "AI-generated" doesn't mean "AI-shipped."
GitHub's own data suggests about 46% of code suggestions get surfaced to developers, roughly 30% of those get accepted, and retention over time is unknown. That's a lot of filtering between generation and production. The more interesting question isn't how much code AI writes, but how much survives. We don't have good longitudinal data on this yet, and vendors aren't eager to provide it.
Qodo's data reveals a telling pattern: teams using AI for code review, rather than just generation, see 2x quality gains compared to those who don't. This suggests the tool matters less than the workflow. AI is genuinely good at catching certain classes of bugs, enforcing style consistency, and flagging potential issues. It's less good at understanding why code exists or whether a change makes architectural sense.
The pattern emerging from the data: AI as a generator requires human oversight that may cost more than it saves. AI as a reviewer amplifies human judgment in ways that actually help.
Calibrating Your Skepticism
The METR study comes with an important caveat: researchers note that 50 hours of Cursor experience may be insufficient for developers to fully adapt their workflows. AI coding tools may genuinely pay off with more practice, different problem types, or less familiar codebases where the developer has no speed advantage to lose.
But the gap between marketing claims and independent measurement should make you skeptical of vendor benchmarks. When GitHub says Copilot makes developers "up to 55% faster," that's based on surveys, not stopwatches. When Microsoft says it "completes tasks 2x faster," the methodology deserves scrutiny.
For teams evaluating AI coding tools, the quality metrics matter more than the speed claims. Watch your PR defect rates. Track how often AI-generated code gets rewritten within weeks. Measure the time spent reviewing and fixing suggestions, not just the time "saved" accepting them.
The productivity paradox may resolve with better models, better tools, or better workflows. But right now, the evidence suggests experienced developers on familiar codebases should be skeptical of promises that don't match their measurements.