Why we need to make data integration obsolete [Q&A]

Nowadays there are apps for almost everything. As users they clutter up our phones and tablets, but they present problems for businesses too, in particular the storing and analysis of the data they create.

Data is often tied to a specific app and each app needs its own copy, so enterprises can end up with lots of copies of the same information leading to issues with control, compliance and more.

Karanjot Jaswal, co-founder and chief technology officer of Cinchy believes that rather than focusing on these data silos and fragmented data we should be making data integration obsolete. We talked to him to find out more.

BN: What's the basic problem with how data and apps work together?

KJ: The basic problem with how data and apps work is that they don't really work together at all. Applications trap data into database silos, making true collaboration on organizational data next to impossible. Apps also erode the access controls that are placed on data, making data governance and data compliance a distinct (often impossible) challenge. Finally, apps also force us to make copies of data via point-to-point integration, which is a tax on innovation that consumes an increasingly large share of IT project budgets. Data is the crown jewel, the true gift; the application is at best a tool to create, collate and deliver the data. But you wouldn't know that from most operating practices.

Even the most forward-focused organizations struggle to develop a truly 'data-centric' culture because we first have to upend what we currently have in place: an 'app-centric' culture. It's certainly true that there's been rampant innovation within the data universe -- and again, that's a good thing -- but somewhere inside that mix, the true value of data has been disregarded.

The fact that we have more apps than ever to help us solve increasingly granular business challenges is a good thing, but that doesn’t necessarily mean that current application designs are secure or scalable. The endpoint should be that the data is serving the needs of the organization -- it should be instantly accessible to the right people at the right time for the right reasons, enable cross-disciplinary collaboration, and drive the development of new solutions to meet business needs. We need to make data central to the organization -- not applications. For now, we're a long way away from that happening.

BN: Why isn't data integration the answer?

KJ: Conceptually, there's nothing wrong with integration -- it's fundamental and critical. We need to integrate data into all business functions. The problem is that while we’ve seen constant progress in data creation, data analysis, data storage, we've seen very little progress in data integration.

Today, there's an app for everything -- which, to be sure, is usually a good thing -- and there's usually a database for every app. Over time, those databases mutate into hundreds of data silos, immutable and often impenetrable. Each of these goes through hundreds of integrations, exponentially increasing fragmentation, complexity, and costs with each new iteration. This is why functions related to integration alone -- think ETLs, APIs, microservices and so forth -- can consume half of the entire IT budget. And by continually duplicating data, the business becomes more sluggish than agile. It gets worse: Each new act of duplication exposes the data -- and the enterprise overall -- to regulatory risks, security vulnerabilities and privacy liabilities.

Drill down into one basic function. In today's IT ecosystems, even sensitive data is routinely copied, frequently with third party apps that are hosted in jurisdictions with questionable regulatory conditions. Each new copy, however necessary, potentially violates security measures, threatens security protocols, and adds to versioning complexity.

And at the 500-foot level, the simple act of copying undermines the value inherent in data. We don't copy currency, or intellectual property or other precious assets -- shouldn't data be at that level? Yet the way data integration has been done for the last 40 years requires endless data copying.

BN: So how can we share data between apps in real time?

KJ: It starts with separating the data from the apps. We advocate the emerging technology category known as dataware, which enables the decoupling of operational data from applications specifically to enhance collaboration without copying -- in sum, it introduces a network-based data management architecture, a bit like a digital brain, to enable people and apps to 'plug in' and collaborate on real-time operational data. There are many other benefits too: For example, it eliminates the need to perform point-to-point data integration when developing business solutions such as new web apps, 360 views, analytics, and workflow automations. In fact, dataware can power unlimited solutions with the same physical data and support schematic evolution without negative consequences, a capability known as 'schema plasticity'.

But this technology isn't working in a vacuum -- it's supported by emerging methodologies and standards like Zero-Copy Integration which defines a highly-efficient framework for innovation that is based on the decoupling of data from apps in order to eliminate data silos and data integration; the establishment of a data mesh to decentralize data governance; the use of active metadata to reduce complex code, and the auto- enforcement of data access controls that are set once at the data level, rather than app by app.

BN: Isn't there a risk of greater latency if data is separated from apps?

KJ: Within the context of a single app the data is already separated from the code, and what we're talking about here is continuing with that separation, but without separating the data from each other. An 'integration-free' approach results in less compute, less storage, and less latency.

BN: Can eliminating integration help solve security, compliance or governance headaches?

KJ: The elimination of data integration that dataware makes possible (and the Zero-Copy Integration framework requires) has a seismic impact on the ability of organizations to control the data they manage. The radical simplification of data architecture that dataware supports is helping organizations transform, on a project by project basis, from traditional IT ecosystems defined by chaos and copies to a future state that’s based on control and collaboration.

When you think about it, the approach of reducing and ultimately eliminating copies in order to protect value is actually similar to how societies already govern things like currency, identities, and intellectual property -- and it works for data, too. Just intuitively, it's far easier to protect, track, and control one instance of something as opposed to thousands of replicas. Furthermore, once data is established as a unique, canonical entity, it can be transformed into something known as 'autonomous data' which is self-protecting, self-correcting, self-versioning, and self-tracking.

With this, the implications for radical improvements to compliance and auditability become apparent.

Image Credit: aslysun / Shutterstock