Quite often businesses succeed in spite of specific choices – not because of those choices. Yet most never stop to consider which it is – choosing to believe they must have been smart rather than admit what they don’t know.

Take the idea that media buyers for DRTV ad campaigns should be chosen based on direct, head-to-head competition between media vendors.

Constructing a valid media vendor test that accurately judges each firm’s abilities is far, far harder than it seems. So here’s The Myth of the Media Shootout (link here), an article I wrote for the October edition of Response Magazine. It looks at a popular competitive testing myth in direct response television.

The myth suggests simple head-to-head tests will choose the best media vendor. But the truth is that an enormous test budget (hundreds of thousands to each media vendor) over an extended period of time (six months would be about right) would be required to make the test valid. And even then, the margin of error is roughly +/-15%.

Since that’s too expensive and takes too long, no one spends that money. Instead, they use the same approach but cut the budget. In practice DRTV advertisers will dedicate a whole $25K to $50K over 2 to 4 weeks and tell everyone it’s valid. Since this invalid test gives the appearance of smart management methodology, everyone will also congratulate themselves on being smart.

Hence they fall back on one of the most popular moves to avoid costly testing: “We can’t afford the $60,000-$100,000 it takes to accurately research that topic. So we’ll put out an online survey using Survey Monkey and tell people we learned just as much as if we spent the $60K to $100K.”

“But we succeeded.” There are people out there who have used this type of testing and will tell me how well it’s worked. So let’s return to my opening idea… “Did you succeed BECAUSE OF that testing or DESPITE that testing?” I have yet to see a case where the success came BECAUSE OF flawed testing methodology.

This reality is not just a DRTV issue. This problem runs rampant in advertising as well as business. Companies regularly create selection processes where the criteria and methodology used have no validity.

Many, many choices in business cannot be made analytically without spending far more money on the test than it’s possible to save by making the “best choice”.

So read & enjoy. When you can, spend the money to make smarter choices. But when you can’t spend the money, don’t lie to yourself.

And no matter what you do, ask the question: Did we succeed because we made smart choices…or despite making poor choices? It’s a challenging question in business but one which leads to tremendous success.

Copyright 2013 – Doug Garnett – All Rights Reserved