Business intelligence analysts spend too much time cleaning up data

We hear a lot about the potential benefits of big data, but a new study reveals that those benefits are won at a cost of considerable time spent in cleaning up and preparing raw information.

The study by data integration company Xplenty surveyed over 200 business intelligence professionals and finds that a third of them spend 50-90 percent of their time just cleaning raw data.

Looking at the 'extract, transform and load' (ETL) process, including preferences for on-premise or cloud-based solutions, perceived challenges, and the amount of time spent on ETL, the results show that 97 percent of those surveyed say that ETL is critical for their business intelligence efforts.

More than half (51 percent) of those polled say that they currently use on-premise ETL solutions. However, 51 percent of these say that they are 'strongly considering' moving all ETL processes to the cloud.

"While many organizations still rely heavily on existing on-premise IT for ETL, the desire to shift to a more cloud-based model has never been stronger," says Yaniv Mor, CEO & Co-Founder of Xplenty. "Cloud ETL offers a host of benefits over on-premise, from increased agility in resource deployment to reduced costs. As such, the cloud is an increasingly attractive option from both a performance and operational perspective".

When asked what the biggest challenges were in making data ready for analysis, 55 percent say integrating data from different platforms, followed by transforming, cleansing and formatting incoming data (39 percent), integrating relational and non-relational data (32 percent), and the sheer volume of data that needs to be managed (21 percent) at any given time.

"BI professionals should be spending the majority of their time evaluating data and deciphering patterns gleaned through the analytics process -- not readying data for analytics," adds Mor. "The more time they spend making raw data analytics usable, the less time they have to generate real value from it. We have to accelerate Big Data's 'time-to-insight,' boosting efficiency and bringing more immediate answers to an organization so that they can more quickly take advantage of them".

More about how analysts are spending time as 'data janitors' is available on Xplenty's blog.

Photo Credit: Amy Walters /Shutterstock