Data engineering teams struggle to keep up with demand

New research from Ascend.io finds that since the onset of the COVID-19 pandemic 78 percent of data professionals have been asked to take on responsibilities outside of their core job function, with 97 percent now signaling their teams are at or over capacity.

The study of over 300 data scientists, data engineers and enterprise architects in the US also reveals that to increase capacity 89 percent of data professionals are turning to automation, low-code, or no-code technologies, with 73 percent citing automation as an opportunity for career advancement.

"Organizations are quickly discovering that data engineers are essential to unlocking the value of data and to removing bottlenecks across the entire data team," says Sean Knapp, CEO and founder of Ascend.io. "LinkedIn's 2020 Emerging Jobs Report found that data engineering has surged onto the scene, quickly becoming one of the top-ten jobs experiencing tremendous growth. At present, there are simply not enough data engineers to meet the demand. To enable more data professionals to tackle the growing backlog of data engineering tasks, tools such as automation and low- and no-code technology can provide tremendous leverage and scalability for existing data engineers, while at the same time enabling a new era of citizen data engineers."

Several things hamper data staff in their ability to meet the data needs of the business. Data scientists are mostly impacted by having to ask others for access to data or systems (48 percent), whereas data engineers are mostly held back by maintenance of existing and legacy systems (54 percent).

More data professionals are seeing the value of low- and no-cost tools, with 80 percent of respondents already using or considering them to support their team. 25 percent of data engineers are using low- and no-code solutions today and 53 percent are currently considering such technology, compared to 48 percent of enterprise architects and 42 percent of data scientists.

You can find out more on the Ascend blog and there's an infographic summary of the findings below.

Image credit: Wavebreakmedia/depositphotos.com