GenAI tools don't increase coding efficiency
While AI is often touted as being the solution for all kinds of tasks, when it comes to developing software it seems that it may not always be improving things.
A new study from Uplevel suggests that today's GenAI-based developer tools don't tend to increase coding efficiency and can actually increase bug rates.
The research was conducted by Uplevel Data Labs, Uplevel's data science arm. It examined a sample of 800 software developers on large engineering teams whose organizations had adopted Microsoft's GitHub Copilot, a GenAI-based coding assistant and developer tool.
It finds developers with Copilot did not see an increase in coding speed. In terms of pull request (PR) cycle time (the time to merge code into a repository) and PR throughput (number of pull requests merged), Copilot neither helped nor hurt developers.
Following Copilot implementation, developers saw a 41 percent increase in bugs within pull requests -- suggesting the tool may impact code quality.
The study also suggests that use of Copilot only has a limited effect on the time developers spend on out of hours work. It decreased by 17 percent for those with Copilot access but, notably, by 28 percent for a control group of those without.
"Engineering teams today seek to allocate their time to the highest value work, complete that work as effectively as possible, and do so without burning out," says Joe Levy, CEO of Uplevel. "They look at data to drive decision-making, and right now, the data doesn’t show appreciable gains in these specific areas through generative AI. But innovation moves quickly, and we’re not suggesting that developers ignore GenAI-based tools like Copilot, Gemini or CodeWhisperer. These tools are all new, there is a learning curve, and most teams have yet to land on the most effective use cases that improve productivity. We will continue to watch these insights as the adoption of GenAI continues to grow, and we recommend anyone investing in GenAI tools do the same."
The full report is available from the Uplevel site.
Image credit: dampoint/depositphotos.com<