Data team productivity threatens project success

Although 81 percent of respondents to a new survey say that their data team's overall productivity has improved in the last 12 months, 95 percent of teams are still at or over their capacity.

The study of over 500 US-based data scientists, data engineers, data analysts, enterprise architects and chief data officers by Ascend.io finds automation is emerging as the most promising path to increase data team capacity and productivity.

A majority (85 percent) are planning to implement automation technologies in the next year even though only 3.5 percent of the same respondents report they currently have automation technologies in place.

"Data team productivity remains the single biggest threat to the success of data projects and workloads," says Sean Knapp, CEO and founder of Ascend.io. "In fact, data team capacity has only marginally improved year over year, yet the demands on these teams continue to grow exponentially -- far beyond what teams can feasibly keep up with."

Nearly all data teams (93 percent) anticipate the number of data pipelines in their organization will increase between now and the end of the year -- with 57 percent projecting an increase of 50 percent or more. Amid the rising number of data pipelines across their organization, nearly three out of four respondents (72 percent) say that the need for data products is growing faster than their team size. This is especially true among data engineer respondents, where 82 percent state that the need for data products is increasing at a faster rate than their team size.

Interestingly though respondents are split when it comes to where the backlog lies. Data scientists are 3.3 times more likely to say data science is the most bottle-necked, data engineers are twice as likely to indicate data engineering, data analysts are 1.9 times more likely to say data analysis, and enterprise architects are 1.5 times more likely to indicate data architecture.

In terms of dealing with the problem there's a strong interest in low-code tools that provide greater flexibility, with 81 percent saying they would be more inclined to use a no-code or low-code tool if it offered the ability to use their preferred programming languages, up from 73 percent in 2021. Respondents also cite a strong interest in data mesh frameworks, with 76 percent planning to implement a data mesh in the next 12 to 24 months.

You can find out more about Ascend.io's 2022 DataAware Pulse Survey on the company's blog.

Image credit: HeyDmitriy/depositphotos.com