Dark data and why you should worry about it

How much of your company's data do you actually use? According to search technology specialist Lucidworks, businesses typically only analyze around 10 percent of the data they collect.

The rest becomes what the company calls 'dark data' -- information that lurks unused. Much of this data is unstructured and doesn't fit into any convenient database format. This means that companies don't have the tools to make sense of it or simply find they have too much to handle.

Lucidworks reckons that 7.5 septillion (7,700,000,000,000,000,000,000) gigabytes of data are generated worldwide each day and a large proportion of that will go dark. This data may contain sensitive information or details that could be used to commit crime, so it needs to be looked after. But it also has the potential to uncover details of undetected fraud, or boost business by revealing patterns in consumer habits, or finding untapped sources of revenue.

There's an interactive graphic on the issue of dark data on the Lucidworks site. You can see a short extract from it below to give you a flavor.