Big Data has been around long enough now that it’s clear that analyzing big data can deliver some sweet benefits. What’s less discussed are the challenges companies face in finding those benefits.

A Key Challenge:  Irrelevant Data. My friend Shahin Khan recently tweeted about one of the key challenges with big data.

@ShahinKhan The ratio of relevant data to irrelevant data will asymptotically approach zero. #BigData

Shahin is Managing Partner of Orion Marketing, was VP/GM of the HPC and Visualization products at Sun Microsystems and we met selling high performance computers to scientists, engineers, and mathematicians (my specialty at the time was Finite Element Analysis). So we both lived in a world where “asymptotically approach zero” has important meaning.

If that phrase isn’t terribly meaningful to you, Shahin’s observation means:

• There is far, far more irrelevant data than there is relevant data.
• In fact, there is SO MUCH irrelevant data that relevant data tends to become overwhelmed by the irrelevant – meaning we all have to take tremendous care in what we claim big data will tell us and in anticipating the very hard work required to get value from big data.
• It does not mean there isn’t relevant data nor that there isn’t a lot of it. It also does not mean that there isn’t tremendous value in the relevant data.

Another Key Challenge:  Irrelevant Results. I’m seeing such poor application of big data analysis that I Tweeted a suggestion for an adjunct to Shahin’s tweet:

@DRTVGuru The ratio of #BigData results that give insight to #BigData results that are meaningless or harmful will asymptotically approach zero.

In other words, businesses are becoming overwhelmed with harmful “analysis” from big data and harmful “findings” from big data. Identifying the meaningful ones sometimes feels like searching for a needle in a haystack.

Here are some rules of the road I use with big data.

There is tremendous potential profit that can be driving with what we learn through big data if we honor what it is:  Another source of analysis to complement traditional marketing research and business analysis.

So here’s a set of rules of the road:

• Executives have to be wary – internal politics often lead data analysis into meaninglessness AND the massive potential profits for big data vendors (both data suppliers and data analyzers) lead them to sell big data services that don’t help you.
• Too often big data analysts end up ensuring they find the answer the company or managers want — NOT the answer that will make the company big profits.
• Big data is NOT inherently more important than traditional quantitative marketing research – nor is it inherently more accurate or more reliable. It all depends on what you need to learn.
• Big data is NOT inherently more accurate or reliable than traditional qualitative marketing research. It all depends on what you need to learn.
• No matter how much big data work or marketing research you do, the successful businesses are those who read the analysis then make smart choices informed both by data and by instinct.

So go forth and collect big data (I’ve added some additional tweets below from Shahin about big data that you might find interesting). But take care. Big data isn’t magic. And big data doesn’t replace what you’ve done before. Even more, neither big data nor research supplant the need for making informed instinctive judgements in order to succeed.

And you know what? That’s why I love business. It’s thick, complicated, fun…and you have to think at a lot of levels to have the best success.

More from Shahin on big data…

@ShahinKhan Whatever caused you to collect data will cause you to collect a lot more data. Plan for scale. #BigData

@ShahinKhan Given enough data, you can simultaneously prove opposites. #BeautifulMind #BigData #Multiverse

@ShahinKhan The ratio of relevant data to irrelevant data will asymptotically approach zero. #BigData

@ShahinKhan Data is cheaper to keep than to delete. Multiple copies, in fact. #BigData

Copyright 2015 – Doug Garnett – All Rights Reserved