AI in software development and the trouble with measuring productivity

Since 2024 JetBrains has used its Developer Ecosystem Survey and interim polls to track which work developers are willing to hand to AI. In April to June 2025, 18 per cent were prepared to let go of writing application logic; in May to July 2026 it was 50 per cent. For writing tests the figure rose from 50 to 75 per cent, for refactoring from 38 to 60 per cent. Forty-three per cent would leave code review entirely to AI, against 33 per cent a year earlier, and 54 per cent prefer to keep debugging for themselves, against 76 per cent a year earlier. In its Developer Survey 2026, with more than thirty thousand respondents, Stack Overflow finds that 73 per cent of those using AI coding assistants and agents use them daily, and that a third of daily users work with AI for four hours or more a day. Use is concentrated in writing code on familiar ground (67 per cent) and debugging (61 per cent), much less in running production systems (20 per cent).

The volume of work passing through the system is growing with it. GitHub counted an average of 43.2 million merged pull requests a month in 2025, 23 per cent more than a year earlier, and between May and September 2025 its own coding agent opened more than a million pull requests. GitHub itself points out that the agent is used mainly in larger and older projects, and calls for controlled experiments to establish the effect.

Establishing that effect is proving difficult. In an experiment with experienced open-source developers, using data from February to June 2025, METR found that tasks took 19 per cent longer with AI. A follow-up from August 2025, with 57 developers and more than eight hundred tasks, no longer yields a reliable picture, according to METR. More and more developers declined to take part because they did not want to work without AI, and 30 to 50 per cent of participants held back tasks for the same reason. METR considers it likely that developers were getting more of a speed-up in early 2026 than in early 2025, but calls its own data very weak evidence of how large it is. Developers also distrust the existing yardstick: in JetBrains’ 2025 survey of 24,534 developers in 194 countries, 66 per cent do not believe, or doubt, that current metrics reflect their real contribution.

Inside organisations, the question ends up in the budget. Readers of The Pragmatic Engineer, more than nine hundred of them in a survey in early 2026, describe rising costs and usage limits, and one CTO says the CFO is hard to keep on board because the productivity gain is difficult to prove. Developers who focus mainly on quality report being swamped by reviewing ever more AI-written code. At Stack Overflow, 38 per cent of respondents say their organisation limits usage and 28 per cent that spending has been audited.

How solid is the evidence? Five independent organisations see the same direction in use and delegation within a single year, and JetBrains has a series going back to 2024. What it delivers in speed and quality has not yet been established, and the large surveys rely on self-selected respondents, so the picture is not yet settled.