Dirty data and why it’s a problem for business [Q&A]

Organizations are sitting on troves of information yet struggle to leverage this data for quick decision-making. The challenge isn't just about having data, but working with it in its natural state -- which often includes ‘dirty data' not cleaned of typos or errors.

We spoke to CEO of analytics company WisdomAI, Soham Mazumdar, to find out more about this challenge and how businesses can deal with it.

BN: What data challenges do enterprises face today when trying to extract meaningful business insights?

SM: Companies invest heavily in infrastructure but face bottlenecks with rigid dashboards, poor data hygiene, and siloed information. Most enterprises need specialized teams to run reports, creating significant delays when business leaders need answers quickly. The interface where people consume data remains outdated despite advancements in cloud data engines and data science.

BN: How has the traditional approach to business intelligence created bottlenecks for decision-makers?

SM: Traditional BI stacks slow decision-making because every question has to fight its way through disconnected data silos and a relay team of specialists. When a chief revenue officer needs to know how to close the quarter, the answer typically passes through half a dozen hands -- analysts wrangling CRM extracts, data engineers stitching files together, and dashboard builders refreshing reports -- turning a simple query into a multi-day project. Our platform breaks down those silos and puts the full depth of data one keystroke away, so the CRO can drill from headline metrics all the way to row-level detail in seconds. No waiting in the analyst queue, no predefined dashboards that can’t keep up with new questions -- just true self-service insights delivered at the speed the business moves.

BN: What makes working with ‘dirty’ or unstructured data particularly challenging for organizations?

SM: Working with data where and how it is -- that's essentially the holy grail for enterprise business intelligence. Traditional systems aren't built to handle unstructured data or ‘dirty’ data with typos and errors. When information exists across varied sources – databases, documents, telemetry data -- organizations struggle to integrate this information cohesively. Without capabilities to handle these diverse data types, valuable context remains isolated in separate systems.

BN: How does WisdomAI's approach to data analytics differ from conventional business intelligence tools?

SM: We've built an agentic data insights platform that works with data where it is -- structured, unstructured, and even ‘dirty’ data. Rather than asking analytics teams to run reports, business managers can directly ask questions and drill into details. Our platform can be trained on any data warehousing system by analyzing query logs. We're compatible with major cloud data services like Snowflake, Microsoft Fabric, Google's BigQuery, Amazon's Redshift, Databricks, and Postgres and also just document formats like excel, PDF, powerpoint etc. Unlike conventional tools designed primarily for analysts, our conversational interface empowers business users to get answers directly, while our multi-agent architecture enables complex queries across diverse data systems.

BN: Could you walk us through a real-world example of how this technology transforms decision-making processes?

SM: Let’s say a chief revenue officer asks, "How am I going to close my quarter?" Our platform immediately offers a list of pending deals to focus on, along with information on what's delaying each one -- such as specific questions customers are waiting to have answered. This happens with five keystrokes instead of five specialists and days of delay.

We've seen transformative results with multiple customers. For F500 oil and gas company, ConocoPhillips, drilling engineers and operators now use our platform to query complex well data directly in natural language. Before WisdomAI, these engineers needed technical help for even basic operational questions about well status or job performance. Now they can instantly access this information while simultaneously comparing against best practices in their drilling manuals -- all through the same conversational interface. They evaluated numerous AI vendors in a six-month process, and our solution delivered a 50 percent accuracy improvement over the closest competitor.

At a hyper growth Cyber Security company Descope, WisdomAI is used as a virtual data analyst for Sales and Finance. We reduced report creation time from two to three days to just two or three hours -- a 90 percent decrease. This transformed their weekly sales meetings from data-gathering exercises to strategy sessions focused on actionable insights. As their CRO notes, "Wisdom AI brings data to my fingertips. It really democratizes the data, bringing me the power to go answer questions and move on with my day, rather than define your question, wait for somebody to build that answer, and then get it in five days." This ability to make data-driven decisions with unprecedented speed has been particularly crucial for a fast-growing company in the competitive identity management market.

BN: What role does generative AI play in your approach to data analytics, and how do you address concerns about hallucinations?

SM: Our AI-Ready Context Model trains on the organization's data to create a universal context understanding that answers questions with high semantic accuracy while maintaining data privacy and governance. Furthermore, we use generative AI to formulate well-scoped queries that allow us to extract data from the different systems, as opposed to feeding raw data into the LLMs. This is crucial for addressing hallucination and safety concerns with LLMs.

BN: What does the future of enterprise data analytics look like as AI continues to evolve?

SM: The future of analytics is moving from specialist-driven reports to self-service intelligence accessible to everyone. BI tools have been around for 20+ years, but adoption hasn't even reached 20 percent of company employees. Meanwhile, in just twelve months, 60 percent of workplace users adopted ChatGPT, many using it for data analysis. This dramatic difference shows the potential for conversational interfaces to increase adoption. We're seeing a fundamental shift where all employees can directly interrogate data without technical skills. The winning approach will combine the computational power of AI with natural human interaction, allowing insights to find users proactively rather than requiring them to hunt through dashboards.

Image credit: Carlos Muza/Unsplash