The ROI of AI Spending Is a Criterion Problem
On October 20, 2026, Gergely Orosz published an analysis in The Pragmatic Engineer titled “What’s the ROI of software engineers’ spending on AI?” He surveyed engineering leaders and senior engineers at 27 companies, collected aggregate data from CodeRabbit and Gitar, and reported what he found. The results are worth reading for anyone who plans workforce budgets, because they expose a problem that most organisations do not have the vocabulary to name.
CodeRabbit tracked 20,000 developers over a year and found a 128% increase in pull requests per developer. Gitar reported that “AI-native” teams ship 60 or more PRs per developer per week. The Pragmatic Engineer’s own survey of 27 companies showed monthly AI spending per developer ranging from under $150 at a publicly traded company with 1,500 engineers to $3,000 at a two-person .NET shop in Europe. A quant trading shop with roughly 2,000 developers reported total AI spending of $8 million per month in September, up from $500,000 in January, with the biggest spenders being traders rather than developers.
The part that should stop HR leaders in their tracks is this. Several companies in the survey reported that PR throughput did not change after AI adoption. A publicly traded SaaS company with about 600 engineers reported that fewer than 5% of developers hit their $1,200 monthly cap, and that “the velocity of PRs merged has not seemed to change much.” A Series D hospitality company with 200 engineers saw spending of $2,200 per developer per month but reported that “PR frequency has not much changed.” An AI SaaS company with 100 employees gave developers a $2,000 monthly allowance and the engineer there said: “We are perhaps marginally faster in a few places.”
Some companies reported large gains. A publicly traded company with 5,000 developers saw PR frequency double, from 4 to 8 per developer per week. A multinational with 10,000 employees saw a single team go from 1.5 to 4 PRs per developer per week, though the team lead attributed most of that to one outlier engineer. The two-person .NET shop reported a 5 to 10 fold increase.
The variation is the finding. Spending on AI tools and changes in output moved together in some companies and not in others, with no visible pattern connecting spend to results. The Pragmatic Engineer concluded that the ROI question is one “that few companies seem able to answer.”
What the measurement gap actually is
The inability to connect spending to output is a specific kind of problem, and industrial-organisational psychology has a name for it. In 1992, James Austin and Jeffrey Villanova published a review in the Journal of Applied Psychology called “The criterion problem” that distinguished between different types of performance measures and the errors each one produces. The paper is available through the American Psychological Association at https://psycnet.apa.org/record/1992-32024-001.
Austin and Villanova’s argument was that performance measurement has two stages. The first is identifying what to measure, which they called criterion conceptualisation. The second is building a workable measure of it, which they called criterion measurement. Most organisations collapse the two stages into one. They pick a measure that is easy to collect, treat it as the criterion, and move on. The problem is that the easy-to-collect measure captures only a slice of what the job produces, and the slice it captures is the one that was easiest to count rather than the one that mattered most.
PR count is a textbook case of what the criterion literature calls a single representational error. It measures one visible output and stands in for the full construct of developer productivity. The measure was already imperfect before AI. It is worse now, because AI changes the relationship between the activity being counted and the work it was supposed to represent.
Why AI breaks the old measure
The Pragmatic Engineer’s data contains the mechanism. At the publicly traded company with 5,000 employees that reported flat PR throughput, the team lead observed that PRs had grown to three times their previous size in lines of code per change request. Output counted by PR count looked flat. Output counted by lines of code looked very different. The choice of measure determines what you see.
At the publicly traded SaaS company with 600 engineers, the staff engineer reported that AI had accelerated specific activities substantially. Customer issue triage that previously took a full day was finished in a few minutes. Research that took a day or two of manual investigation was done within minutes. Proof-of-concept prototyping that took a week or two was finished in a few hours. PR throughput was flat because the team maintained its quality bar and rejected large or unnecessary changes. The time savings showed up in activities that PR count does not measure at all.
The Anthropic Economics team’s report on employment changes at Anthropic, published October 16, 2026, documents the same pattern at a different resolution. The authors found that code commits per engineer rose roughly fivefold over six months, but also that interactive Claude sessions spent about 80% of their wall-clock time idle, waiting for human input. The human bottleneck shifted from producing work to reviewing it. Reading all agent output would take more than half the work week for non-technical staff, based on measured output volume. The report is available at https://www.anthropic.com/research/employment-at-anthropic.
The implication for measurement is direct. When AI produces the code and the human reviews it, the human’s contribution moves upstream of the PR. It lives in the decision about what to ask the agent to build, in the review of what it produced, and in the rejection of work that was unnecessary or wrong. PR count captures the agent’s output. It does not capture the human’s judgment that preceded and followed it.
The criterion problem in practice
What the Pragmatic Engineer found across 27 companies maps onto the criterion literature with surprising precision. The companies that reported flat PR throughput despite high spend were not necessarily getting no return on their AI investment. They were getting a return that their chosen measure could not detect.
Customer issue triage that went from a full day to a few minutes is a productivity gain. Research that went from two days to minutes is a productivity gain. Proof-of-concept prototyping that went from two weeks to hours is a productivity gain. None of those gains show up in PR count. The staff engineer who reported them said that overall, “actual development speed had not increased much” when the team maintained its quality bar. That is a statement about a specific measure, not about the value of the spending.
The companies that reported large PR increases may have been capturing real gains, or they may have been capturing a change in how work is parcelled. The two-person .NET shop that went from 1 to 2 PRs per day to 10 or more was also clearing a ten-year-old backlog, which is work that produces PRs at high volume but represents a finite stock rather than a sustained rate. The quant trading shop measured ROI as desk profit and loss, which is a criterion that captures value regardless of how it was produced. That is the measurement approach the criterion literature would call a composite criterion, and it is the one most resistant to distortion from a single representational error.
What HR leaders should take from this
The Pragmatic Engineer’s data and the criterion literature point to the same conclusion. The question “what is the ROI of AI spending?” is unanswerable with a single metric, and the metric most companies have chosen, PR throughput, is one of the ones most likely to mislead.
Three things follow for HR leaders who need to evaluate AI tool spending in their own organisations.
Name the criterion before you measure it. Austin and Villanova’s distinction between conceptualisation and measurement matters here. Before asking whether AI tools produce a return, decide what the return is supposed to be. It might be faster delivery of customer fixes, shorter research cycles, or more prototypes reaching product decisions. Each of those is a different criterion, and each requires a different measure. PR count answers none of those questions.
Track activities the agent does not produce. The Pragmatic Engineer’s data shows that the largest time savings in some companies appeared in triage, research, and prototyping. These are activities that produce no PRs. If your ROI calculation counts only code output, you are measuring the machine’s work and ignoring the human’s. The Anthropic report found that 20% of surveyed employees described their role as shifting from doing work to managing Claude. That shift is real work, and it is invisible to throughput metrics.
Beware the backlog effect. A team clearing old work at high PR volume is generating output that looks like sustained productivity growth but is actually a finite stock being depleted. The two-person .NET shop in the survey was clearing a ten-year backlog. When the backlog runs out, the PR rate will fall, and if the criterion is PR count, the ROI will appear to collapse. The spending will not have changed. The measure will have caught up with reality.
The Pragmatic Engineer’s survey is a convenience sample of 27 companies, not a systematic study. The findings are self-reported. But the pattern it exposes, spending rising while the chosen output measure stays flat or varies unpredictably, is the pattern the criterion literature predicted decades before AI tools existed. The measure was always a proxy. AI made the proxy visible.
Sources: Orosz, G. (2026). What’s the ROI of software engineers’ spending on AI? The Pragmatic Engineer, October 20, 2026. https://newsletter.pragmaticengineer.com/p/whats-the-roi-of-software-engineers · Wilmers, N., Catipon, R., Massenkoff, M., and McCrory, P. (2026). How employment is changing at Anthropic. The Anthropic Institute. https://www.anthropic.com/research/employment-at-anthropic · Austin, J. T., and Villanova, J. S. (1992). The criterion problem: Four strategies for a changing landscape. Journal of Applied Psychology, 77(4), 418-432. https://psycnet.apa.org/record/1992-32024-001 · Hackman, J. R., and Oldham, G. R. (1976). Motivation through the design of work: Test of a theory. Organizational Behavior and Human Performance, 16(2), 250-279. https://doi.org/10.1016/0030-5073(76)90016-7