The Truth About AI Coding Tools: Gains, Risks, and Real Impact

A recent study by METR (Model Evaluation & Testing Research) suggests that the productivity boost promised by AI coding tools isn't universal. Researchers conducted controlled experiments with developers working on real-world tasks using tools like Copilot.

The findings were nuanced: AI tools helped novice developers complete simple tasks faster. However, for experienced engineers working with large, legacy codebases, productivity actually declined. Many found themselves spending more time reviewing and correcting AI-suggested code than writing it themselves.

These mixed results mirror broader trends observed in the recent Dev Barometer Q3 2025, which found that while many developers are indeed benefiting from AI—saving an average of 7.3 hours per week, using AI tools in nearly a quarter of their code, and investing time into AI/ML upskilling—companies often lag in strategy maturity, infrastructure, and resource allocation. The study underscores that the productivity boost of AI doesn’t just depend on the tools themselves, but on the surrounding context: how organizations support learning, embrace remote work, and embed AI into workflows.

The report warns companies to approach AI adoption with context-awareness. Blanket deployments may not yield the expected productivity dividends, especially in complex environments like enterprise software or large open-source projects.

The study recommends using AI selectively: for boilerplate code, documentation generation, or prototyping, while relying on traditional methods for complex logic.