Shedding Light On The Topic Of Dark Data

Let’s say you’re finding an item in the storeroom. But you can’t find it because of all the other things that are taking up space and blocking your way. Data scientists face a similar issue when finding the data they need amidst the unused data taking up space in the data storage. This unused data is known as dark data. Dark data will be introduced in this article.

Dark data or dusty data
Dark data is like the boxes of files collecting dust and spider webs in the storeroom. Image designed using Canva.

What is dark data?

Dark data, also known as dusty data, is the unstructured data that an organisation continuously collects and stores but not organise and use.

“Collected and stored…

NOT organised, NOT used…”

How I would write about dark data in a poetic way.

There are three dimensions of dark data:

Organisations usually collect unstructured data “just in case” they need them.

But they don’t know how to use or analyse unstructured data, so they end up leaving the data unused and forgotten.

Just like the items that collect dust in the dark store room at home.

Differences between the 3 dimensions of dark data
A small comparison of the three dimensions. Image designed using Canva.

Importance of dealing with dark data

There are several reasons to not leave dark data ignored.

Firstly, dark data affects data quality in terms of accessibility.

Accessing the necessary data is like finding a needle in a haystack consisting mostly of dark data.

IBM estimated that 90% of all data generated is dark data.

So, obviously, this haystack of unused data would be taking up too much storage space, which brings us to the next point.

Storage can be costly, so not making good use of all the data stored including the dark data would be like throwing money out the window.

Another reason to put dark data to good use is cybersecurity.

Most organisations may think that it is useless, so they don’t bother protecting it with cybersecurity tools.

However, hackers might disagree about its uselessness.

Yes, it can actually be valuable if organisations know how to use it, which we’ll get to later.

If hackers can steal the dark data, it might place the organisations at risk.

So, if the organisations leverage it, they’ll have the incentive to protect it from data theft.

reasons to deal with dark data
There are 4 main reasons to deal with dark data: data quality, costly storage, cybersecurity and hidden insights. Image designed using Canva.

Since 77% of managers and executives in the world believe that finding and capturing dark data should be a top priority, let’s look at the ways organisations can do so.

Suggestions for using dark data

According to TechRepublic, the very first step organisations have to take is to find out what they have.

It just makes sense to know what their dark data looks like before they can do anything with it, right?

Once they know what they have, they need to determine if it provides value to their company or not.

If it doesn’t, they can clear the data clutter.

If it does, they can keep the dark data and tap into it.

For the second option, they can add external data to complement the available dark data to support decision-making.

While looking at both dark and external data, they have to check the data’s quality.

Common use cases

At this point, it’s becoming clear that using dark data can uncover important information that help organisations make strategic decisions.

According to Precisely, here are some common use cases:

Customer support logs

A business typically keeps records of customer support interactions in a CRM (customer relationship management) system including:

  • when a customer contacted the business
  • products or services used
  • which communication channel was used
  • the pipeline stage

Instead of abandoning such data or using it only when there’s an issue, the data can be leveraged to understand or predict when customers are most likely to make contact, their preferred channels, their preferred products or services, etc.

Non-textual data

Textual data may be easier to process but non-textual data also has the potential to provide insights.

For example, a research publication described how administrative textual and non-textual data from public archives were used to predict the performance of Danish startups for innovation support decisions.

The performance indicators were survival, high employment growths, a return on assets of above 20%, new patent apps and involvement in an innovation subsidy program.

What the research may have considered as non-textual data include the characteristics of the founders and startups that may not be captured in the textual data they obtained.

Legacy system log

There are situations whereby organisations keep older types of systems like a software that works on an older version of the operating system (i.e. Windows).

It would be hard to imagine using modern analytics tools to understand these old systems but it’s actually possible.

These system logs just need to be moved into an analytics platform like Hadoop for legacy system data to be useful.

Common dark data use cases
Examples of dark data opportunities are customer support logs, non-textual data and legacy system logs. Image designed using Canva.

Lessons

Here’s a little summary of what we learnt from this article:

Dark data is unstructured data collected and stored but not organised and used.

Its 3 dimensions are traditional unstructured data, non-traditional unstructured data and deep web data.

Dark data is normally treated like the dusty, unused items in the storeroom at home because organisations don’t know what to do with traditional unstructured data, non-traditional unstructured data or deep web data.

However, leaving it abandoned can result in issues for data quality, costly storage, cybersecurity risks and missing out on hidden insights.

Dark data can be a gold mine if used well.

So, the first step for organisations to take is to get to know the dark data, and then decide if it is worth keeping or not.