AI Tools May Slow Down Experienced Developers, Claims METR Study

METR's study finds AI coding tools may hinder productivity for experienced developers, increasing task completion time by 19%.

Key Points

  • • AI tools increase completion time for experienced developers by 19%.
  • • Participants expected a 24% reduction in task times but were mistaken.
  • • Developers accepted less than 44% of AI suggestions and spent significant time reviewing outputs.
  • • AI tools' effectiveness varies based on user experience and task complexity.

A recent study conducted by the non-profit AI research group METR reveals that AI coding tools, such as Cursor and GitHub Copilot, may actually hinder the productivity of experienced developers rather than enhance it. In a randomized controlled trial that included 16 seasoned open-source developers, the study evaluated 246 coding tasks, with half allowing the use of AI tools. Surprisingly, the researchers found that tasks completed with AI tools took 19% longer than those completed without.

Initially, developers expected a 24% reduction in task completion times when using these AI tools. However, their productivity assumptions proved overly optimistic. Key factors contributing to this slowdown included the time spent crafting prompts for the AI and waiting for responses. Developers accepted fewer than 44% of AI-generated suggestions and expended about 9% of their time verifying and modifying AI outputs. Moreover, while a significant portion of developers (94%) had experience with web-based language models, only 56% had familiarity with Cursor, potentially affecting their efficiency.

Despite these findings, the METR study emphasizes caution against generalizing the results. Other research indicates that AI tools can boost productivity in different circumstances, highlighting that effectiveness can vary based on task complexity and the user's familiarity with the tools. As AI technology rapidly evolves, the study suggests improvements may soon alleviate current productivity constraints.