Data quality incidents take two days or more to resolve

The latest State of Data Quality survey from Bigeye finds that more than half of respondents have experienced five or more data issues over the last three months.

Another 40 percent say they have experienced moderate to severe data incidents within the last six months, and that it took a major effort to avoid damage to the business. These incidents range from severe enough to impact the company's bottom line, to reducing engineer productivity.

70 percent of respondents report at least two data incidents that diminished the productivity of their teams. What's more incidents are taking an average of 48 hours to troubleshoot but can often have repercussions lasting weeks or even months. This means organizations with more than five data incidents a month are essentially lurching from incident to incident, with little ability to trust data or invest in larger data infrastructure projects.

"Coming from a data team before starting Bigeye, I knew anecdotally how much of a burden data quality and pipeline reliability issues were. These survey results confirmed my experience: data quality issues are the biggest blockers preventing data teams from being successful," says Kyle Kirwan, Bigeye's CEO and co-founder. "We’ve heard that around 250-500 hours are lost every quarter, just dealing with data pipeline issues."

The study finds that 86 percent of respondents report that some combination of data engineers, software engineers, and data analysts are in charge of data at their organization. 59.9 percent say that data engineers are the first line of defense in dealing with data incidents.

While the research shows engineers are looking for proactive, automated solutions to address data quality issues, in most companies, data quality initiatives are generally prioritized only after a data quality incident that negatively impacts the bottom line. This means that solutions are too often reactive fixes. 89 percent say that data monitoring is either somewhat or very critical or to their operations.

The full report is available from the Bigeye site.

Image credit: lightsource/depositphotos.com