The Dangers of Direct Response Metrics for Online Retailers

I just started developing online retail for Studio Moderna at the end of 2004.

It was an amazing time. A much simpler world when it was easy being a marketing hero just using traditional direct response metrics.

We lived and died by conversion rates, cost per order and direct campaign profitability. Sure, we used other metrics as well, but these were the Holy Grail.

But then the world started changing.

What we’ve been taught about direct response measurement online “suddenly” became outdated, and is today even dangerous if used as our primary compass.

1. Reliable Direct Response Measurement is a Wet Dream

Direct marketers like to see the world as a simple funnel. And we all love the safety we get from making decisions based on the funnel.

Is a campaign generating a low conversion rate or too high cost per order?

No problem. Kill it. That’s the safe thing to do.

Why keep investing in something that evidently isn’t generating direct sales?

Unfortunately, life is not that simple, and the “funnel model” is practically dead.

The Scary New World and Google’s Zero Moment of Truth Model

As shown by Google’s ZMOT Study, the consumer path to purchase has become increasingly complex:

  • In 2011 consumers used an average of 10.4 info sources before buying. And that’s twice as much as in 2010.
  • Purchase research for most product categories peaks at 2 – 3 months before making the final decision.
  • Pre-shopping research (zero moment of truth) now has more impact than stimulus and the in-store purchase decision (first moment of truth).

And there’s more: customers buying online do 9.2 searches on average before buying, and those buying offline do 7.6 searches (source).

It’s Not Really a Funnel …

The decision making process isn’t even close to being a funnel.

It’s actually closer to this, just incredibly more complex in real life:

We can’t really easily measure the direct impact of offline media, so let’s skip that for now.

But even just looking at the online channels we have a problem …

The Danger of Relying Only on Google Analytics Conversion Rates

Tools like Google Analytics will reliably report the “last click” source that brought visitors to your website just before their “first moment of truth” purchase decision.

The problem is, they only tell you which sources brought you customers that were at the very end of their purchase cycle.

They don’t and can’t reliably tell you what stimulated the customer to start researching the purchase and how he or she arrived to that final moment of purchase.

The result? Making decisions just based on “last click” metrics means you’re only investing in grabbing late-stage customers, but are actually ignoring early-stage potential customers who haven’t yet decided to buy from you.

And that means ignoring your greatest revenue growth potential …

 2. The False Security of Online Multi-Channel Attribution Models

But wait, hasn’t this problem already been solved with online multi-channel attribution models?

Nope.

Multi-channel attribution models try to give us this:

Just one problem: the current technical limitations (not to mention the upcoming privacy troubles) make reliable multi-channel attribution impossible.

They certainly tell us part of the story, and that’s a hell of a lot better than nothing. But still, only part of the story.

Can’t Measure Cross-Browser and Cross-Computer Behavior

You start researching a purchase at work, and then buy in the evening at home. Or even research in one browser, and buy through another.

The end result – Google Analytics, or any other online analytics tool, can’t measure this behavior (unless you’ve somehow connected “both” users, perhaps via login).

Imagine this scenario:

  1. John clicks on your ad at work, generating a cost for you. He looks at the product, does some more research, and decides to by later.
  2. In the evening he enters your URL directly and buys on the spot, but with no ad cost to you.

He wouldn’t have bought if he hadn’t clicked on the ad at work. And yet that ad gets 0 attribution for the sale, and the direct URL entry gets the full attribution.

Looking just at direct results you might make the decision that the ad campaign isn’t working. Of course it’s not working, if it’s not generating direct sales!

But is it really not?

Can’t Measure Cross-Device Behavior

Now add different devices into the mix.

Same story!

Can’t Measure Third-Party Services

And finally, just to make things even more interesting, don’t forget that third-party services people use to research your product also have an impact.

Reading your product review on Yelp, Amazon, Zagat or other services. Seeing your posts on Facebook, but not clicking. And more.

You can’t measure the impact of any of this using traditional online analytics tools … and yet, you can be certain there is an impact.

Is It Google Analytics’ Fault?

Absolutely not. And neither is it Omniture’s, KISSmetrics’ etc.

They’re all amazing tools. It’s just how the internet works that’s the problem.

Don’t Worry. It’s OK!

We have more data than ever before. But, we don’t know everything.

Our challenge aren’t the tools. We can’t make data perfect.

Our challenge is changing our mental model.

  1. Don’t rely only on direct response metrics.
  2. Don’t judge campaigns and traffic sources purely on their direct response results.
  3. Don’t focus just on late purchase stage customers and traffic sources, unless you want guaranteed low growth.
  4. Develop new online analytics models and frameworks to measure the entire purchase cycle.
  5. Evangelize a different approach to online measurement within your organization.

And above all, admit to yourself that you just can’t measure everything. Plus, you can get started with the follow-up post :)

To drive the point home, here’s an amazing quote from Jim Novo, from a conversation we had on Avinash’s blog (BTW – this post is a must-read for understanding the challenges of multi-channel attribution):

There are simply limits on what can be “proven” given various constraints, and that’s where experience and a certain amount of gut feel based on knowledge of customer kick in.

If you can’t measure it properly, just say so. So much damage has been done in this area by creating false confidence, especially around the value of sequential attribution models where people sit around and assign gut values to the steps.

Acting on faulty models is worse than having no information at all.

Update [2012-09-19]: Jim Novo posted an amazing feedback to my post here. My response in the comments section. And I do agree with Jim’s points 100%.

Comments

  1. track viewthru conversions, give it a 30-40 days cookie window. The numbers usually are very interesting, close or even abouve last-click conversions. a viewthru conversion is a metric that tells you: number of converted visitors / number of visitors that *saw* your ad on Google Display Network. you can de-duplicate the number relative to search ad-clicks. Bringing this beyond Google assets is alse feasible, using your own adserver and its conversion metrics.

    • Rok Hrastnik says:

      Andrej, viewthru conversions are certainly an additional relevant signal, although their reliability is always hugely problematic.

      1. The larger the share of audience reached, the lower the relevance.

      2. Very misleading if used in conjunction with remarketing campaigns (what caused the return visit: ad impression or natural behavior?).

      So, in my opinion and experience, only a somewhat valuable signal. But always better than nothing:)

  2. Steve Rogai says:

    Yeah well the funnel system does work still, but I agree that blinding your eye to logic-based tools may benefit your brand or product placement through better pin-point accuracy. I see what you mean.

    • Rok Hrastnik says:

      Totally agree. It still works, to a certain extent. But not as the one and only measure of success, and therefore of allocating budgets and human resources.

  3. Jim Novo says:

    Rok, thanks for quoting me and I think we probably agree on the topic of multi-channel attribution, but not “Marketing Funnel is Dead” and “Direct Response Metrics are a Wet Dream”. It’s surely true people abuse these models and data by using them for purposes they were not intended for, I think we agree on that. I wouldn’t throw them out the window though, because used for the right reasons, they can be quite valuable. I expanded on this topic in a post, your comments welcome:

    http://blog.jimnovo.com/2012/09/18/funnel-measurement-dead/

  4. Rok Hrastnik says:

    Jim, first of all, thank you for taking the time for such an in-depth response & post. I’ve already enjoyed our conversation on Avinash’s post, and this new discussion even more.

    And, in reality, I have nothing to add to your blog post. I agree with all the points you’ve made 100%.

    I never intented for my posts to state that “Direct Response Measurement overall is a Wet Dream”.

    In The Dangers of Direct Response Metrics for Online Retailers my intent was to demonstrate that it’s dangerous to rely only on the simple direct conversion funnel (click > convert), which is still mostly used by (direct) marketers to evaluate media spend / campaign efficiency.

    My key point was that the purchase decision process in most cases isn’t instant, and the tools we have available today do not make it easily possible to measure the impact of all the touch points leading to the purchase, and we therefore should not rely only on direct response when optimizing our marketing investments.

    I’m a direct marketer by origin. I love direct response metrics. However, as stated in my post, they are not enough. We can no longer rely ONLY on direct conversion rates, cost per order etc.

    Unfortunatelly, most still do that today. It’s not just the marketers. I’ve met many who try to do things differently, but are then blocked by “black & white data” oriented CEOs and CFOs.

    In Moving Beyond Direct Conversions (1): Adapt to the Purchase Cycle I tried to expand on the idea, and proposed a) adapting goal KPIs to campaign goals (directly related with the purchase cycle stage they are primarily targeting) and b) expanding our basic direct response metrics “pool” with not only assisted conversions, but also engagement and other softer metrics.

    But, the bottom line is, I completely agree with all of your points.

    I might have been somewhat too aggressive in trying to bring my point home. A large part of that is due to the mistakes I’ve made myself in the past.

    Almost a decade ago, when I was more or less starting in online retail and building my team, my focus was almost exclusively on direct response metrics. I was all about direct CRs, direct CPOs etc. All I cared about was direct campaign / media investment profitability (not counting lead generation, though). And I trained my team the same way.

    My first real eye-opener was an in-depth analysis we did on the impact of our email program, finding that it has about a 3x stronger indirect impact VS direct conversion impact.

    Later, with more experience, I started implementing more complex analytical frameworks, but a lot of the damage has already been done. I had already very successfully “converted” a lot of the people to the “wrong path” (yeah, I know, sounds a little too poetic:) — to such an extent, that it than proved difficult to correct my own mistakes.

    So, much of my vigor comes from my own frustration over the mistakes I’ve made in my early days as a marketer/online retailer.

    The second reason is that I’ve seen far too many marketers, CEOs and CFOs focus only on measuring and optimizing for the direct response, consequently blocking their customer growth.

    Jim, thank you for the discussion. Enjoyed it as always!

  5. Rok Hrastnik says:

    Thanks for the “nod”.
    But just to clarify, I never said it’s becoming “worthless”:)
    I actually agree 100% with Jim’s points.
    More in the comments section of mine and Jim’s blog:)
    All the best,
    Rok

Trackbacks

  1. [...] some very thoughtful points about measuring the consumer paths to purchases on ecommerce sites in The Dangers of Direct Response Metrics for Online Retailers. It includes some great insights from Google’s Zero Moment of Truth Model and [...]

  2. [...] purchases on ecommerce sites in The Dangers of Direct Response Metrics for Online Retailers. It includes some great insights from Google’s zero [...]

  3. [...] example, here is a post worth reading on this topic by Rok Hrastnik.  There are some very good points in this post on why using a funnel to attribute media value is [...]