Measuring AI Coding’s Value Beyond The Hype

🔍 Read the full analysis: Measuring AI Coding’s Value Beyond The Hype on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Engineering teams are spending from about $150 to $3,000 per developer each month on AI coding tools, but many companies cannot yet say what that spending has delivered, according to a report by Gergely Orosz of The Pragmatic Engineer. Drawing on conversations at 27 workplaces and data from two code-review vendors, the October 20 report documents sharply different costs and productivity results as organizations prepare their 2027 budgets.

The report describes spending caps ranging from $150 per developer at a publicly traded company with 1,500 engineers to about $3,000 at Ix Technologies, a two-person European .NET agency. Other companies set caps between $1,000 and $4,000, with typical monthly spending around $1,000. These are examples from the workplaces surveyed, not a representative estimate of spending across the software industry.

Reported output also varies. CodeRabbit says developers on its platform created 113% more pull requests per week than a year earlier. Microsoft research covering tens of thousands of engineers during an early-2026 rollout found adopters merged about 24% more pull requests over four months than they otherwise would have. Microsoft’s researchers cautioned that a merged pull request is not the same as delivered value. At some surveyed companies, pull-request counts stayed flat; one said the requests became larger.

Orosz’s report identifies triage, code archaeology, prototypes and backlog cleanup on simple, consistent technology stacks as areas where teams described practical benefits. One staff engineer at a SaaS company said customer-issue triage that could take a day was reduced to minutes of agent work and up to an hour of human checking. Ix Technologies said it was using AI tools to work through a long-running backlog on a C# and Blazor stack. These are company accounts, not controlled measurements of the tools’ effect.

Why Budget Owners Need Better Measures

The immediate issue for engineering leaders is not simply whether AI tools produce more code, but whether that code helps teams deliver useful, reliable changes at a cost the business can justify. The wide spending range and uneven output figures make a single productivity claim difficult to apply across different teams and codebases. They also complicate budget decisions as some discounted subscriptions approach expiry and vendors adopt usage-based pricing.

The report recommends tracking outcomes rather than tool consumption, including cost per accepted change, rework, change-failure rates and whether teams close more user stories than arrive. Those measures connect spending to work that is accepted and maintained, rather than to token usage or raw pull-request volume. Security is another constraint: the source cites Veracode testing in which 44% of AI coding tasks produced a known vulnerability. That result is a test finding, not a claim that 44% of all AI-generated code in production is vulnerable.

Evidence Behind the Spending Debate

Orosz published the report on October 20 after speaking with engineering leaders and senior engineers at 27 workplaces. It also drew on aggregated data from CodeRabbit and Gitar. The article says its cited material comes from the freely readable portion of Orosz’s piece, so the findings should be read with that scope in mind. Vendor platform data describes users of those services and does not by itself establish results for companies that do not use them.

Other studies cited in the source illustrate why estimates differ. METR’s early-2025 study found experienced open-source developers took 19% longer with AI on the tasks studied. A February follow-up estimated an 18% speedup for returning developers and 4% for new developers, but both estimates had confidence intervals that included no change. METR described the follow-up data as unreliable, noting that 30% to 50% of developers withheld tasks they did not want to attempt without AI. Self-reported gains were higher, but METR said participants had previously overestimated AI’s effect.

Pricing is also changing. The source says GitHub moved Copilot plans to usage-based billing on June 1, charging tokens at each model’s API rate and removing the fallback to a cheaper model. It quotes GitHub saying agentic use brings higher computing and inference demands. Orosz also reports that API-plan spending at some companies peaked in June and July and had roughly halved since, which he attributes to Opus 5.5 and a shift toward open models.

“a question that few companies seem able to answer – at least for now”

— Gergely Orosz, The Pragmatic Engineer

What the Productivity Numbers Cannot Show

The available figures do not establish a single industry-wide productivity gain. The 27 workplaces were interviewed rather than presented as a representative survey, vendor data comes from users of specific platforms, and company-reported examples may not transfer to teams with different systems or work. Pull-request counts also do not show whether code is useful, secure, maintainable or costly to review.

It remains unclear how much of the reported output translates into shipped product improvements or reduced costs after accounting for review, rework, failures and security checks. The source’s claims about spending falling at some API-plan companies and the reasons for that decline are attributed to Orosz; they do not establish a universal trend. The article also does not provide a common, independently measured ROI calculation across the workplaces.

What Companies Should Track Next

The near-term test for engineering leaders is whether their own budgets can be tied to outcomes as subscription discounts end and usage-based charges expose model costs more directly. Teams can compare spending with accepted changes, rework, change failures and work completed relative to incoming demand, while tracking whether review and security capacity keep pace. Results should be compared across similar tasks and teams rather than treated as a universal return from adopting AI.

The source does not identify a shared industry standard or a scheduled follow-up study that will settle the question. For now, companies setting 2027 budgets will need to evaluate their own work and costs; further evidence will depend on more consistent measurement of delivered value, not just activity in coding tools.

Key Questions

How much are companies spending on AI coding tools?

The workplaces in Orosz’s report described spending limits ranging from about $150 to $3,000 per developer per month. The examples are not a representative industry-wide average.

Do the reported productivity figures prove AI coding delivers business value?

No. Pull-request counts and merged changes are measures of activity or output, but do not by themselves show customer benefit, code quality, maintenance costs or return on investment. Microsoft’s researchers explicitly cautioned that a merged pull request is not the same as the value it delivers.

Where did teams report the clearest benefits?

The report describes benefits in issue triage, understanding existing code, building prototypes and clearing backlogs, particularly on simple, consistent technology stacks. These examples are reported experiences rather than controlled proof that every team will see the same gains.

What should companies measure when evaluating AI coding costs?

The report points to cost per accepted change, rework, change-failure rates and stories closed compared with stories arriving. These measures connect tool spending with completed work and the costs of maintaining it.

Source: ThorstenMeyerAI.com