New Study Reveals AI Tools May Hinder Developer Productivity
Metr's study finds that AI coding tools may slow down experienced developers by 19%, countering expectations of improved productivity.
Key Points
- • AI tools slowed task completion by 19% for experienced developers.
- • Researchers initially expected AI usage to boost speed by 24%.
- • Developers accepted less than 44% of AI's code suggestions.
- • Participants averaged five years of coding experience but only moderate AI familiarity.
A study by the nonprofit research organization Metr highlights troubling findings regarding the impact of AI tools on the productivity of experienced software developers. Contrary to prior expectations that AI assistance would enhance speed, the research indicates a concerning trend whereby utilizing AI actually slows down coding tasks.
The comprehensive study involved 16 developers who undertook 246 tasks, discovering that those who used AI tools experienced a 19% increase in task completion time when compared to a control group not utilizing AI assistance. Researchers initially anticipated that AI-assisted developers would complete tasks 24% more quickly, demonstrating a significant gap in expected versus actual performance.
Several factors contributed to this slowdown: developers accepted fewer than 44% of AI-generated code suggestions. Furthermore, they spent approximately 9% of their work time reviewing and correcting AI outputs, which added significant delays to the coding process. Even though AI tools reportedly offered some time savings during the coding and research phases, this was ultimately negated by the time developers spent prompting the AI, waiting for responses, and verifying the accuracy of generated code.
Interestingly, study participants tended to overestimate the effectiveness of AI tools, indicating a disconnect between user expectations and actual utility. All participants had substantial coding experience, averaging five years, yet only moderate familiarity with AI technologies. This raises questions about how less experienced programmers might fare using similar tools. Moreover, the complexity of the code repositories used in the study could skew results, as such environments typically require precise and high-quality code contributions.
As AI tools continue to evolve, these findings suggest a critical need for developers and organizations to re-evaluate reliance on AI for productivity in software development tasks. The insights from Metr's research emphasize that while AI can be beneficial, it may also introduce challenges that dilute its intended productivity enhancements.