SAN FRANCISCO: A recent study by the AI research nonprofit METR reveals that using advanced AI tools can slow down experienced software developers rather than speeding up their work, particularly when they are working with familiar codebases.Â
The study, conducted earlier this year, involved developers using Cursor, a popular AI coding assistant, to complete tasks in open-source projects they were already familiar with.
Before the study, developers believed AI tools would decrease task completion time by 24%. However, the findings showed the opposite – AI tools actually increased task time by 19%. Despite this, developers still estimated a 20% reduction in time, indicating a mismatch between expectations and reality.
The study’s lead authors, Joel Becker and Nate Rush, were surprised by the results, as Rush had expected a significant speed improvement. This challenges the commonly held belief that AI tools enhance productivity for experienced developers, a notion that has driven considerable investment in AI solutions for software development.
While AI has been touted as a potential solution for automating entry-level coding jobs, the METR study suggests that it does not provide the same productivity gains in all scenarios. In particular, developers who are familiar with large, established open-source codebases experienced delays due to the time spent correcting AI suggestions, which were often directionally correct but not fully accurate.
The authors acknowledged that the slowdown may not apply to junior engineers or those unfamiliar with the codebases they are working on. Despite the slowdown, most participants in the study and the authors themselves continue to use AI tools, finding them helpful for making the development process more efficient and less effortful.
