I wasn’t entirely shocked to see that the Old Spice social media video campaign had won an Effie – a lot of people in the ad business seem to have decided this campaign was the grand epiphany of social media effectiveness. Except, I’d done some reading about the effectiveness of the campaign and found its results entirely unclear.

So let’s consider the “Effie” award from the American Marketing Association. Their website tells us:

“The Effie Awards were founded in 1968 by the American Marketing Association, New York Chapter, as an awards program to recognize the most effective advertising efforts in the United States each year.”

Great description. They must be better evaluated than the curated art shows that most other awards have become (like Cannes, Clio’s, and most others).

What led to awarding the 2011 Old Spice Campaign? Here’s a link to the Effie PDF about the campaign (click here). The PDF regales us with statistics claimed to reflect effectiveness. Mostly they are the usual big online numbers. And I notice:

1. They claim the total cost of the campaign was less than $500,000. Guess they didn’t think it was important to mention the $3M to $10M spent creating the first spot and airing it in spring of 2011 starting with the Superbowl. (TV is highly effective at driving social media action.)

2. Old Spice sales for the product line had already gathered tremendous momentum in 2010. Their brief claims sales were up over 60% in 2010 vs. prior year. Hmmm. That’s a nice starting point for more growth.

3. They report all the expectedly big social media numbers. But I’ve written elsewhere my skepticism about accepting big numbers just ’cause they’re big. (Link here.)

4. Then there’s complete chaos. They claim Nielsen says UNIT sales were up 125%. But they offer a graph with units on the side but which labels the 125% increase as “YouTube Responses”. Huh? Which was it? What’s really going on here? That’s far too sloppy to evaluate “effectiveness” and I’d drop one of my students most of a letter grade for that kind of mistake.

5. The last line caps it all off. Question: “Anything Else Going On That Might Have Affected Results?” Answer: “No other factors”. No mention of the TV campaign, the coupon campaigns, co-op ads or specials at the store? Riiiiiight.

Except, there is this article in AdAge. (Link here.) This article concludes that the Old Spice Q2 sales increase was attributable to the coupon campaign – not social media.

While the social media campaign was on, P&G also mounted a significant coupon campaign for the product (guess they forgot about that when submitting for an Effie). In the end, the Old Spice increase was roughly the same as increases of competitors who had mounted the same type of coupon campaign. And competitors (like Axe) who hadn’t used coupons hadn’t grown.

In other words: Coupon appear to be responsible for the product line growth – not social media efforts. And this means any claim of superior effectiveness needs a huge asterisk right next to it.

What Can We Conclude About the Old Spice Campaign? Clearly:

1. The brief is wrong when it suggests there were “No Other Factors”. Old Spice is part of a highly robust market with co-op ads, coupons, and a lot of retail action. Nothing – NOTHING – operates in a vacuum.

2. The brief is meaningless without a picture of the competitive environment and relative growth during this period.

While I don’t know what process led to the award, it seems realistic to suggest that the award should have caveats – if its not pulled entirely. At least, this is what an award show that cared deeply about effectiveness would do.

What Can We Conclude About the Effies? Not much. This is a single award and campaign. However, they didn’t do serious fact checking (it’s not like AdAge is a tiny and unimportant advertising publication). And they published a write up that is factually incorrect.

So do Effies have any connection with effectiveness? They might not. The Old Spice campaign was very clever, highly unusual, a new application of social, etc…. But none of that matters when you call your award an “Effie”.

The Sad Award Truth. Effies aren’t alone – they just claim to be better. Truth is, there is NO way to create an award that credits effective advertising.

Awards start to fail at one of advertising’s first critical steps: targeting. Effective advertising targets an end consumer. Judges are NOT those consumers and they never view award submissions within the reality in which consumers consider the ads. (Only industry associations for vertical markets can have any hope of judging target market impact.)

Once this failure starts, there is no way for awards to be anything more than they’ve always been: the more carefully crafted ones are curated art shows and the less well crafted are mere popularity contests. To create an award that is anything else would require judges to dedicate hundreds of hours of their time and be supported by a staff dedicated to developing independent judgement.

Still, the Effies could have done better. This campaign might still get an Effie if they’d had a complete submission that showed more effectiveness insight. And perhaps there was data submitted that wasn’t made public. But the public release showed egregious gaps. There’s no way those should be rewarded.

Copyright 2012 – Doug Garnett – All Rights Reserved