Study Reveals Decline in Developer Productivity with AI Coding Tools
Recent study finds AI coding tools decrease productivity for experienced developers.
Key Points
- • AI tools decreased productivity of experienced developers by 19%.
- • Developers overestimated the expected boost from AI tools by 20%.
- • AI-assisted developers spent more time reviewing outputs and being idle.
- • Only 44% of AI-generated code was accepted by developers.
A new study conducted by Model Evaluation & Threat Research (METR) has brought to light concerning findings regarding the use of AI coding tools among experienced software developers. Contrary to popular belief that these tools enhance productivity, the research indicates a notable decrement, with a documented 19% increase in task completion time among developers who utilized AI assistance compared to their non-AI-using counterparts.
The study involved 16 experienced developers who were divided into two groups: one team had access to AI tools, primarily using Cursor with Claude 3.5/3.7 Sonnet, while the other group worked without AI assistance. Surprisingly, developers who employed AI tools reported a misguided expectation of a 20% productivity boost, revealing a significant overestimation of the tools' effectiveness. Despite their confidence, those using AI spent over 20% more time reviewing AI outputs, waiting for the AI, or remaining idle, contrasting with the more proactive coding engagement of the non-AI group.
METR researcher Nate Rush noted his astonishment at the negative productivity results, which he believes may reflect the specific context of their study. He cautioned that these findings may not universally apply as AI tools are continuously evolving and might yield different results in various circumstances. Additionally, Steve Newman, a cofounder of Google Docs, acknowledged the study's credibility and underscored the limitations of current AI tools in effectively supporting developer productivity. The research also highlighted that only 44% of AI-generated code was accepted by developers, who spent an average of 9% of their time refining outputs from the AI.
As developers react to these findings, there is a growing sentiment for a more cautious and contextualized use of AI tools in coding practices. Further research will be essential to track the evolving dynamics as advancements in AI technology continue.
With new capabilities emerging, how developers choose to integrate AI into their workflows remains a crucial element to monitor moving forward, particularly given the mixed results regarding productivity enhancements.