Data teams struggle to keep pace with demand

According to new research 96 percent of data teams are operating at or over capacity, thanks to a surge in demand for data pipelines.

The study by data engineering company Ascend.io shows 93 percent of respondents anticipate the number of data pipelines in their organization increasing between now and the end of the year, with 56 percent predicting the number to increase by more than 50 percent.

This surge appears to be affecting some roles more than others, with data scientists 2.4 times more likely than other roles to anticipate significant growth in pipeline demands. Encouragingly, the majority (79 percent) of respondents say that their infrastructure and systems are able to scale to meet their increased data volume processing needs, which highlights that the new problem with scale is with team capacity and less on technology capacity.

Their organization's need for data products is growing faster than their team size according 74 percent of respondents. This is more true for data engineers, 81 percent of whom say the need for data products is increasing at a faster rate than their team size.

"Data pipelines are fueling nearly every data-driven initiative across the business," says Sean Knapp, CEO and founder of Ascend.io. "However, as innovations at the infrastructure layer continue to enable processing of greater volumes and velocities of data, businesses face a new scaling challenge: how to enable their teams to achieve more, and faster. Our research shows that team sizes are not scaling at a fast enough rate to keep up with the needs of the business. Combined with our data that highlights almost every data professional today is already at capacity, this leaves little room for strategic work and innovation."

Perhaps not surprisingly teams are turning to new products or tools (53 percent) or implementing automation technology (53 percent). Additional means to increase bandwidth include hiring more staff (47 percent) and re-platforming and retiring legacy technologies (30 percent).

However, concerns remain around the usability of low- and no-code technology. In a 2020 study, 80 percent of respondents reported they were already using or considering low-and no-code solutions to increase their team’s bandwidth. Interestingly though, the 2021 research indicates that only four percent of data professionals prefer a no-code user interface.

You can read more on the Ascend.io blog and there's an infographic overview of the findings below.

Image Credit: aslysun / Shutterstock