METR, a non-profit research organization that studies the capabilities of AI, conducted an in-depth study of the performance of experienced software developers using the popular AI programming assistant Cursor. The results contradicted the popular belief that AI tools speed up task completion times, but this is not true for all IT professionals.
METR, a non-profit research organization that studies the capabilities of AI, conducted an in-depth study of the performance of experienced software developers using the popular AI programming assistant Cursor. The results contradicted the popular belief that AI tools speed up task completion times, but this is not true for all IT professionals.
Before the study, developers involved in open source projects believed that using AI would speed up their work by 24%. Even after completing tasks with AI, developers believed that they had reduced their time by 20%. But the study found that using AI did the opposite, increasing task completion time by 19%, Reuters reports .
Even the second studies themselves were shocked by the results, expecting "twice the acceleration."
The findings refute the belief that AI always makes expensive engineers much more productive—a factor that attracts significant investment in companies selling AI products to aid in software development.
Previous research on productivity has found significant benefits: one study found that using AI made coders 56% faster, while another found that developers were able to complete 26% more tasks in a given amount of time.
But a new METR study shows that this doesn’t work for all programmers. In particular, experienced developers who are familiar with the quirks and requirements of large, established open source databases are experiencing slowdowns.
The slowdown was due to the fact that developers needed to spend time reviewing and correcting what the artificial intelligence models suggested.
"When we looked at the videos, we found that the AI was making some suggestions about its work, and those suggestions were often correct, but not quite what was needed," said one of the study's authors, Joel Becker.
However, the authors warn that the slowdown will occur in the work of junior engineers or engineers working with code bases that are unfamiliar to them.
However, most of the study participants, as well as the study authors, continue to use Cursor today. The authors believe this is because AI makes the development process easier and therefore more enjoyable, similar to editing an essay instead of staring at a blank page.
"Developers have other goals than just getting things done as quickly as possible," Becker said. "So they take this less time-consuming route."
https://t.me/joinupstaff