A few years back I read a post on RetailWire that pondered whether retailers will need traditional research once mobile tracking is “in place”. The question reveals a very common flaw in how people think about research. And that flaw continues today amid all the AI and machine-learning hype.

Developing conclusions from mobile data is similar to scientists reading the fossil record. When I was a kid, scientists had been observing the fossil record for hundreds of years. So, they really thought they knew what the truth was. Dinosaurs were reptiles, they had reptilian skin, they were cold-blooded, lived isolated lives, and modern day lizards are their direct descendants.

Fast forward to 2017. My kids learned that some (many) dinosaurs had feathers, that some (many) were herd animals, they were pretty fast moving, that some lived in family based units, some hunted together, and that birds essentially evolved from dinosaurs.

The original scientists weren’t bad at their jobs. In fact, they were brilliant. The problem was all they had was observed data. They created solid, grand theories from the observed facts they knew.

With Observed Data You Never Know What You Don’t Know. Paleontologists erred in their theories because there were thousands and millions of fossil truths they couldn’t see – they hadn’t yet been discovered or analyzed. They also projected modern day perceptions onto million year old facts to try to have them make sense – a classic error in research.

The specific errors in observational research are different with modern consumer observations. But the fundamental truth – that you don’t know what you don’t know – is always true and should lead to far more caution than modern observational data is given.

Ethnographic observers are in a similar bind as are direct marketers who rely purely on response. No matter how hard we work, observation misses more knowledge about human consumers than it captures. And without that knowledge we easily mislead ourselves into error.

Mobile data puts us in a similar spot. So does purchase history, shopper behavior, browsing behavior online, phone call data, and almost every other type of data the modern AI theories are based on.

The big miss:  BEHAVIORAL DATA NEVER TELLS US “WHY”. Why can’t Amazon recommend books that I find interesting despite having a 20 year history of what I’ve bought? And remember, Amazon reflects the state of the art for this kind of processing today.

The reason is clear:  they only know WHAT I bought – they don’t know WHY I bought it. They’d have to have the “WHY” to even begin to suggest useful recommendations.

We need to respect better that the key to consumer behavior is understanding motivation – why someone makes that choice, takes that action, buys that product.

No amount of clever statistics, comparative analysis, or merging of data sets can get past the missing “why”. They may help, but the limitation of observed data always remains.

That often makes behavioral data a type of marketer Rorshach test. Marketers and companies often project motivation assumptions onto data sets. Sometimes those projections reflect marketing department group think. At other times, the projections are the things that help individual careers. Or, they project the results of the latest session with a highly-paid consultant; or the answers they think bosses or shareholders want to hear.

What I’ve observed from big data efforts is that they rarely find actionable consumer truths. Most often teams will give me “major findings” that turn out to be what we already knew or were simple to deduce from other information.

Wise companies will continue to rely heavily on research that helps us see motivation because motivation is the key to changing profit in big ways. In modern research, its not just mobile that lacks insight into motivation. True “ethnographic research” is purely observational and is quite weak at discovering things that drive sales. (Perhaps that’s why so many firms claim to do ethnographic research but really do in-home one-on-one interviews).

To get to motivation, you have to include qualitative research of some form along with wise use of original quantitative research. It has to be executed by professionals and should be compared with findings from data.

All research has to be interpreted with care to avoid similar theoretical jumps to the errors noted above. But somehow, I find the challenges in qualitative data much more evident where the errors in things like mobile data are dramatically more insidious.

Data IS important. Companies need to dig for valuable insight in their data. But let’s stop getting carried away. Data can only be valuable when combined with traditional marketing research as well as wisdom and insightful analysis.

Only then will we be following Deming’s admonition to remember that “Information is not knowledge. Let’s not confuse the two.”

Copyright 2017 – Doug Garnett – All Rights Reserved