Have a business challenge? It's really a big data problem in disguise

Throughout history, businesses have relied on certain measures to project their success -- sales growth, customer retention, churn rate, and so on. However, those are all lagging indicators for modern businesses that want to stay ahead of the competition; all these potential business challenges must fall under a single new metric: time to insight. Similarly, think about what happened with high-frequency trading, where suddenly, years of research and algorithms didn’t matter if your competition could act on information orders of magnitude faster than you -- it was like having a crystal ball to the future!

The advent of big data has companies flooded by a tsunami of information. The problem with this is twofold: big data is difficult to wrangle in order to provide insights in the first place, and once the right architecture is deployed, it can then be hard to choose the proper parameters to provide those insights.

Enterprises that know what their business is doing at every step in real time will win in today’s demanding climate, where customers expect companies to know what they want and when they want it. The key to getting those real-time insights -- and ensuring they are immediately actionable -- is providing native access to data.

Companies have more data now than at any other point in history. In fact, 90 percent of the total data in the world has been created in the last two years. And unstructured data accounts for about 80 percent of enterprise data. The best way to ensure that data is immediately actionable is to use a business intelligence (BI) tool that can access it natively. This makes it easier to incorporate the data into analysis, allowing companies to receive more input on factors like customer behavior across different channels.

To allow native access to data, or a "data-native approach," companies need to update their legacy data storage so it has the agility to handle the onslaught of big data. In past generations, it was enough to store data in a data warehouse and then move it, duplicate it, and perform an extract, transform, load (ETL) process so it could be actionable in another part of the architecture. With big data, this process is complex and inefficient, clogging up the system with unnecessary data duplication and grinding time to insight to a halt. With a data-native approach, data analysis occurs right on top of where the data is stored, enabling companies to know exactly what their customers are doing -- and want -- in real time.

One company took a data-native approach to its marketing, pinpointing its offline billboard advertising locations and comparing them alongside sales in stores near the location. By analyzing data relevant to both these two metrics, the company was able to see the good news and the bad news. On the plus side, its ads were working-- products in stores near billboards were selling quickly. But, unfortunately, those products were also selling out, meaning that their billboard would be up for more than a week after customers were no longer able to walk into a store and purchase the product. By adjusting marketing spend and supply to those stores, the company was able to use its data to increase their sales while not wasting any extra advertising dollars. This outcome could only be accomplished through real-time metrics on the full granularity of their data -- not just summaries or averages -- and the only way to achieve true real-time metrics is through a data-native approach. Previously, the company’s actions were misguided by what is classically known as Simpson’s paradox, in which a trend appears in different groups of data but disappears or reverses when these groups are combined. This company had always been measuring performance at a regional level where supply to advertising spend averaged out, but the problem was that the averages were hiding opportunities to optimize at the individual store and billboard level.

In addition to the huge advantage of gaining real-time visibility into a company’s information, taking a data-native approach also results in cost savings when it comes to handling data. Traditionally, companies have struggled to manage unstructured data through a laborious ETL process. And these different pieces of data have been siloed, meaning that gaining insight from connections made through different data sets was limited by the time-intensive, physical realities of moving data. Handling raw data natively also ensures its fidelity, as there is inherent information in raw data that is lost when it is processed or becomes more complex.

Think of raw data like Google Books. Before, if a person wanted to research all books that contained a certain phrase or topic, they’d have to go to a library, go to the card catalog, and rely on it to give them hints to where their data might live. Then they’d have to manually find that information, and even then it might be incomplete. After all, it’s limited by the number of books in one library. Now, people can go to Google Books, look up any phrase and have instant access to all sources of that information within seconds. Similarly, when working in a data-native system, all these pieces of information are immediately apparent, since they are located in one place and can be analyzed in detail right then and there. But outside of that system, each data point would be separate from all the other ones, and a data scientist would have to manually move each data point to the same location in order to begin analysis. The task of pairing up these data points for analysis would add additional time to insight.

This data-native method is not a futuristic concept that needs proving out. It is a reality for gaining rapid business intelligence in many fields today, including financial services and cybersecurity, as well as Internet of Things companies. These are all very challenging sectors, with new innovations and threats emerging every single day. For any enterprise to rise to the top in their industry, they have to be agile, equipped to reduce risk, be able to erase their bottom line issues, and innovate at a rapid pace.

Companies must start examining their business challenges through the lens of big data, using a data-native approach. Doing so will not only help them win the time-to-insight race among their competition, but will help them increase their market share.

Photo: Ronald Sumners/Shutterstock

Shant Hovsepian is the Co-Founder and CTO of Arcadia Data. Prior to this, he worked as Member of Technical Staff at Aster Data Systems. He served as Software Engineering Intern at Google.