What can machine learning do at scale?

In my series, I’ve looked at the different ways in which data can be deployed to help people make decisions. Over time, more of the decision-making process has shifted from people manually collating data from different sources in their heads to using data sets that can be automatically joined together. This networked approach to data makes it easier for people of all skill levels to work with data.

This has evolved to make more use of automation over time. By making it easier for individuals to link up data sets and form connections between these assets, businesses have been able to spread analytics to more users within their organizations. This is now being taken further with machine learning.

Spotting patterns in data can be easier for machines that can work with huge sets of data from multiple systems. For businesses that have data spread across many different ERP systems, this can make it easier to reconcile interactions around the world with customers as well as managing the supply chain that links products, services and providers together. For large enterprises that have grown through acquisition or mergers with other businesses, this internal infrastructure can have volumes of data that could provide great insights if they can be joined in the right ways.

Building out how to use insight across the business

For companies with multiple applications in one area, getting consistent data can be a massive challenge. Even when the same application is used, the requirements for national or regional variations can often mean lots of customization has taken place. Linking up each record so that the complete picture is visible is therefore not easy.

Taking manual work out of this process can help speed up getting access to results. Using machine learning and automation, getting that single overview of all customer activity from multiple systems is possible. However, this should only be the start for how that data can be used.

For individuals, the value from this automation is how quickly they can bring in their own data and have it integrated with this centrally managed data set. Can they join up their own information on customers -- complete with their own customer identifiers or information sets -- with other data sets automatically, and then get recommendations on how to present this data back out for their own use? How about for sharing with other groups that have their own questions to answer?

The role for machine learning and automation here is that each of these tasks should not require a data scientist to come in and prepare the data for others to use. Instead, automating the internal preparation process should make it easier for all users to get the answers that they are looking for.

Similarly, sharing this data outwards is a challenge. Simply delivering a report or spreadsheet of your findings can help you, but it won’t necessarily help others to use the results. At best, you end up having to share the same sets of data for people to go through, which results in duplicate work and additional maintenance burden; at worst, discrepancies in the data lead to wasted time or mistakes in judgement. Instead, networking people with these data sets can help them find out what insights they actually want to discover.

Getting insights out to more people with Machine Learning

For machine learning implementations, getting this link between business goals and the decisions that people make every day relies on data. Machine learning can be used to automate the process for analyzing new data against current information, and then recommending potential decisions to those that are involved. However, this has to be seen in context.

Traditionally, trained data scientists are required to create the right frameworks for machine learning. Machine learning implementations can be based on supervised development and training of algorithms and sophisticated analytics models to reflect an activity. Once these initial projects are set up, the data science team has to run more real-world data through their models and then score them to judge how accurate they are. Following this work, the team will work to find the best fit for all the data that is involved and retrain the models to be more accurate. This process is manual and depends on data scientists with an advanced set of skills.

Today, companies are looking to automate more of the steps involved in sorting the data, creating these models and scoring them for accuracy. There are several reasons behind this: first is that there is a dearth of skilled data scientists available for companies to recruit. More than 2.7 million data science and analytics roles will exist by 2020 according to research by IBM, and demand for these skills will grow by 39 percent. This also means that there is a premium for roles with these skills, putting average salaries well over $80,000 and roles with specialist skills over $150,000.

While these particular skills will continue to be in demand, there are also opportunities to automate a good portion of this process decreasing the need for human intervention across the whole process. At the very least this means that data scientists can accomplish more with similar or less effort; but, more likely, it can remove the need for specialist skills and decrease the sophistication of skills required. As a result, automation can extend these capabilities around machine learning to more Line of Business teams.

All of a sudden, instead of having a handful of data scientists working on complex models for months, we now have many people throughout the organization able to bring data together, spot patterns and opportunities, and make predictions about the business on their own and in less time.

Does this replace the data scientist? Absolutely not. Using machine learning to automate the analytics process invariably means reducing its sophistication and limiting what can be achieved. But that doesn’t diminish the value of automation. On the contrary, the value is in the greatly increased scale. By helping set guard rails on what can be done with data, data scientists can help those line of business teams to help themselves. Now, many more people are producing insights across the organization. At the same time, those data scientists can make use of their full set of capabilities to look in more detail at harder problems or more varied issues where the potential to automate workflows has not yet been considered.

There is still enormous benefit in taking some of these capabilities around machine learning, making them vastly more consumable through automation, and then scaling out the results to the broader organization to make more use of advanced analytics. Just as data scientists are initially needed to get machine learning working properly, automation relies on data science principles and efforts at the beginning to set the right course for future users to benefit.

By looking at the whole process and other teams’ sets of data within a whole network, machine learning can be used to scale up analytics across the whole business. This will include making recommendations that affect wider workflows and activities in the best way possible rather than looking solely at the individual or the team.

Networking people and data

This approach to analytics should provide better levels of insight to everyone across the business. However, it relies on everyone being able to get those insights delivered to them with accuracy and timeliness. Networking people together around sets of data should make it easier to collaborate, rather than relying on those individuals seeing reports or shared documents.

It’s possible to go through business processes in more detail and look at the role that data can play. This can then encourage more use of data within the business as part of a feedback loop. The mainstream adoption of machine learning and analytics can make the results available faster, but ultimately it comes down to helping the people involved to work more effectively. Scaling up this increase in effectiveness comes down to making analytics easier to consume across the business.

Pedro Arellano, vice president, Product Strategy at Birst.

Published under license from ITProPortal.com, a Future plc Publication. All rights reserved.

Photo credit: Lightspring / Shutterstock