Enterprises struggle to make data available for insights

The value of big data, we're told, is that it allows organizations to derive useful insights which can drive digital transformation projects.

However, a new study from cloud warehouse data transformation company Matillion and IDG Research shows that making data available for insights is a barrier for 90 percent of enterprises.

The study, which polled over 200 IT, data science, and data engineering professionals at North American organizations with at least 1,000 employees, finds that data volumes are growing at a rapid rate, on average 63 percent a month.

But 12 percent of respondents report that their data volumes are growing at 100 percent or more per month. In addition more than 20 percent of those surveyed report drawing from 1,000 or more data sources.

Things are further complicated by the fact that responsibility for exploiting data is divided between IT groups (55 percent) and business units (45 percent). Only 23 percent of respondents have completely centralized their BI and analytics teams, but virtually all plan to tap the cloud for data management, migrate or continue to migrate data to the cloud over the next 24 months.

The study shows that 90 percent have already placed some data in cloud data warehouses (CDWs). 37 percent of organizational data is in CDWs, 35 percent is in on-premise data warehouses, and 25 percent is in offsite, non-CDWs. Three CDW suppliers dominate, Amazon Redshift (used by 54 percent of  respondents), Google BigQuery (50 percent), and Snowflake (26 percent).

Yet despite this shift to the cloud, more than 90 percent say it's challenging to make data available in a format usable for analytics. Respondents cite several obstacles slowing their data analytics projects, including a lack of necessary data granularity, manual coding of data pipelines and difficulty connecting with multiple data sources. In addition 37 percent struggle with data transformation.

"Without the right data management strategy, growing data volumes go from a competitive advantage to a company-wide struggle to make it useful," says Matthew Scullion, CEO of Matillion. "Our research underscores the demand and the urgency to use the flexibility and scalability of the cloud -- not just to store data, but also to accelerate insights through data transformation."

You can get the full results of the research here and there's a summary of the findings in the infographic below.

Photo Credit: Billion Photos/Shutterstock