New Relic surveyed 200 US technology decision-makers this year — directors, VPs and C-suite at upper mid-market and enterprise companies, all of them already running generative and agentic AI in their engineering organisations. The research was run with Hanover Research and published in June.
Two findings from it, held next to each other:
94% rate AI-generated code as higher quality during review.
78% report more production incidents after deploying it.
Those are the same leaders. The same code. One judgement made while looking at a diff, the other made while looking at an incident channel.
They can't both be right, and it's not hard to work out which one is lying.
#Plausible Is Not the Same as Correct
Code review, as practised by most teams, is a surface-level inspection. That's not an insult — it's a design constraint. A reviewer has twenty minutes, no execution environment, no production data, and a diff stripped of most of its context. Given that, what can they actually assess?
Naming. Structure. Whether it follows house conventions. Whether the tests look like tests. Whether the comments explain anything. Whether it reads like code written by someone who knew what they were doing.
Every single one of those is a proxy for correctness, not correctness itself. They worked for decades because they correlated well with human competence. Someone who wrote messy, badly-named, uncommented code was usually also someone who'd got the logic wrong, and someone who wrote clean idiomatic code had usually thought it through. The surface signal carried real information about what was underneath.
Language models broke that correlation, and they broke it precisely and completely. They are extraordinarily good at every property a reviewer can see in twenty minutes — consistent naming, conventional structure, plausible tests, tidy comments — and only ordinarily good at the property nobody can see, which is whether it does the right thing on the inputs you didn't think about.
So 94% of leaders saying AI code reviews well is not evidence that AI code is good. It's evidence that AI code is reviewable-looking, which is a different property that we accidentally spent thirty years learning to treat as a synonym.
#The Rest of the Numbers
The gap isn't a rounding error, and the survey is fairly unsparing about its size.
82% experienced at least one production failure tied to AI-generated code in the past six months.
74% needed significant rework on at least a quarter of their AI-generated code over the past year.
86% report senior engineers spending more time fixing code.
Only 19% report no AI code challenges at all.
And the scale it's happening at: 67% of these leaders say AI now writes or substantially refactors between half and three-quarters of their organisation's weekly code output.
Then the finding that explains the mechanism. 62% admit their teams often ship AI-generated code without line-by-line verification — and 88% have formalised "vibe coding" in their production policies, meaning this isn't shadow practice anyone is hiding. It's the documented process.
Put it in order and the causal chain is embarrassingly simple. AI writes most of the code. Most of it isn't read line by line. It looks great in review because looking great is the thing AI is best at. It merges. Then production tells everybody the truth, roughly two weeks late.
#DORA Priced It
Google's DORA team published research in May on the ROI of AI-assisted software development, and their framing is the most useful thing anyone has produced on this so far, because it treats the downsides as line items rather than caveats.
They name two costs.
The verification tax: the expense of establishing whether AI-generated code is actually correct. This doesn't go away when you adopt AI — it moves. It leaves the writing step, where it used to live implicitly in the author's own understanding, and reappears in review, in QA, and in incident response. That's why 86% report senior engineers spending more time fixing code even as everyone ships more.
The instability tax: delivery instability rising as code volume grows faster than your pipeline's ability to absorb it. DORA's illustrative model for a 500-person engineering organisation includes a $344,000 negative downtime impact from change failure rate rising after adoption. That's inside a scenario where the overall return is still strongly positive — 39% first-year ROI, roughly an eight-month payback — which is the point. The tax isn't an argument against AI. It's a line on the invoice, and most organisations aren't reading it because they never itemised it.
DORA's other finding is worth carrying too: gains run 35–40% on greenfield work and around 10% or less on legacy code. If your engineers spend most of their time in a mature codebase, and most engineers do, the headline productivity numbers were never yours to begin with.
#What This Does to Your Metrics
Here's the part that's uncomfortable for a company that sells engineering metrics.
Several of the numbers our industry has trained leaders to watch have quietly stopped meaning what they meant.
Review approval rate is now close to noise. When approval largely reflects how plausible code looks, and AI is optimised for plausibility, a high approval rate tells you about presentation, not quality. It was always a weak signal. It's now a compromised one.
Deployment frequency is genuinely ambiguous. Deploying more often used to be strong evidence of a healthy pipeline, because getting to high frequency required automation, tests and small batches — you couldn't fake it. Now volume alone can push the number up while the underlying discipline erodes. Deployment frequency rising with change failure rate rising is not a team that improved. It's a team shipping faster than it can verify.
PR count and merge throughput are the worst offenders, because they respond most directly to AI and say the least about outcomes.
What still works is everything measured after the merge, because production is the one reviewer that cannot be fooled by good formatting.
Change failure rate. The single most honest number available to you right now. It doesn't care how the code looked.
Mean time to recovery. Arguably more important than it's ever been, because if more defects are reaching production, your recovery speed is doing more of the work of keeping you stable.
Rework and churn rate. What proportion of merged code gets rewritten or reverted within a few weeks? This is the closest thing to a direct measurement of "the review didn't catch it," and it maps onto that 74%-needing-significant-rework finding.
Revert rate and time to detect. How often do you back something out, and how long does the defect live before anyone notices? Time to detect in particular is the metric that gets worse silently, because nobody files a ticket for the two weeks before they found it.
The general rule: any metric a language model can move by writing more code is no longer measuring your team. Metrics that only move when something works or fails in production still are.
#What To Do
Instrument the back half of the pipeline first. Most teams have far better visibility into PRs and commits than into incidents, reverts and rework, because the front half is easier to collect. That imbalance was tolerable when the front half was informative. It isn't now.
Report change failure rate next to deployment frequency, always. Neither number is interpretable alone anymore, and quoting deployment frequency by itself has become actively misleading. Together they still tell you something true.
Stop treating "approved" as a quality event. Approval means someone thought it looked fine. That's worth something, but it's worth much less than it used to be, and any dashboard treating it as a quality gate is measuring theatre.
Decide, explicitly, where verification lives. The 62% shipping without line-by-line review aren't being reckless so much as honest — nobody can line-by-line 70% of a growing codebase. So the question isn't whether to verify, it's where. Stronger tests, contract and property-based checks, better observability, staged rollouts, faster reverts. Pick deliberately, because if you don't pick, the answer defaults to "in production, by your customers."
Push context to where the code is written. The cheapest defect is the one that never gets generated. If the agent writing the code knows your architectural boundaries, your size conventions and the standards your team actually enforces, you're spending less on verification because there's less to verify. Everything else on this list is downstream cleanup.
#The Caveat This Post Needs
The New Relic figures are self-reported perception data from 200 leaders. That's a real limitation and it cuts both ways. "94% say quality looks higher in review" is a statement about what leaders believe their reviewers concluded — which is exactly the kind of second-hand impression that surveys capture badly. The incident-side numbers are more trustworthy, because "did you have a production failure tied to AI code" is a thing people remember accurately and have little incentive to overstate.
The contradiction survives the caveat, though. Even discounting both halves heavily, a group of leaders reporting that code looks better in review and behaves worse in production is telling you something about the review signal, not about the code.
#The Short Version
Your code review is now measuring how good the code looks, and language models are exceptional at making code look good.
That doesn't make review worthless. It makes it a design and comprehension check — is this the right shape, will anyone understand it in a year, does it belong in this codebase at all — rather than a correctness gate. Those are valuable questions. They're just not the question everyone thinks they're asking when they click approve.
The correctness gate moved to production, whether you instrumented it or not. Change failure rate, MTTR, rework, revert rate: that's where your quality signal lives now.
Approval is an opinion. Production is a measurement.
#Related Reading
- Change Failure Rate: Balancing Speed and Quality - The metric that survived the transition
- Mean Time to Recovery: Building Resilient Engineering Teams - Why recovery speed matters more as defect rates rise
- Everyone Got Faster and Nothing Shipped Sooner - The flow side of the same problem
- Building a Strong Code Review Culture - What review is still genuinely good for
Coderbuds tracks the metrics that still mean something — change failure rate, recovery time, rework and deploy lag — and turns them into guidance your engineers and coding agents can actually act on. See what production says about your code.
